From owner-svn-src-stable@freebsd.org Sun Sep 27 01:06:46 2015 Return-Path: Delivered-To: svn-src-stable@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 6E7ACA0A2EA; Sun, 27 Sep 2015 01:06:46 +0000 (UTC) (envelope-from kib@FreeBSD.org) Received: from repo.freebsd.org (repo.freebsd.org [IPv6:2001:1900:2254:2068::e6a:0]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 4588D2D9; Sun, 27 Sep 2015 01:06:46 +0000 (UTC) (envelope-from kib@FreeBSD.org) Received: from repo.freebsd.org ([127.0.1.70]) by repo.freebsd.org (8.15.2/8.15.2) with ESMTP id t8R16kDC087500; Sun, 27 Sep 2015 01:06:46 GMT (envelope-from kib@FreeBSD.org) Received: (from kib@localhost) by repo.freebsd.org (8.15.2/8.15.2/Submit) id t8R16kIY087499; Sun, 27 Sep 2015 01:06:46 GMT (envelope-from kib@FreeBSD.org) Message-Id: <201509270106.t8R16kIY087499@repo.freebsd.org> X-Authentication-Warning: repo.freebsd.org: kib set sender to kib@FreeBSD.org using -f From: Konstantin Belousov Date: Sun, 27 Sep 2015 01:06:46 +0000 (UTC) To: src-committers@freebsd.org, svn-src-all@freebsd.org, svn-src-stable@freebsd.org, svn-src-stable-10@freebsd.org Subject: svn commit: r288282 - stable/10/sys/fs/fifofs X-SVN-Group: stable-10 MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit X-BeenThere: svn-src-stable@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: SVN commit messages for all the -stable branches of the src tree List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 27 Sep 2015 01:06:46 -0000 Author: kib Date: Sun Sep 27 01:06:45 2015 New Revision: 288282 URL: https://svnweb.freebsd.org/changeset/base/288282 Log: MFC r288044: Ensure that when a blockable open of fifo returns success, a valid file descriptor opened for complimentary access exists as well. Modified: stable/10/sys/fs/fifofs/fifo_vnops.c Directory Properties: stable/10/ (props changed) Modified: stable/10/sys/fs/fifofs/fifo_vnops.c ============================================================================== --- stable/10/sys/fs/fifofs/fifo_vnops.c Sat Sep 26 22:57:10 2015 (r288281) +++ stable/10/sys/fs/fifofs/fifo_vnops.c Sun Sep 27 01:06:45 2015 (r288282) @@ -64,6 +64,8 @@ struct fifoinfo { struct pipe *fi_pipe; long fi_readers; long fi_writers; + u_int fi_rgen; + u_int fi_wgen; }; static vop_print_t fifo_print; @@ -137,6 +139,7 @@ fifo_open(ap) struct thread *td; struct fifoinfo *fip; struct pipe *fpipe; + u_int gen; int error, stops_deferred; vp = ap->a_vp; @@ -164,6 +167,7 @@ fifo_open(ap) PIPE_LOCK(fpipe); if (ap->a_mode & FREAD) { fip->fi_readers++; + fip->fi_rgen++; if (fip->fi_readers == 1) { fpipe->pipe_state &= ~PIPE_EOF; if (fip->fi_writers > 0) @@ -179,6 +183,7 @@ fifo_open(ap) return (ENXIO); } fip->fi_writers++; + fip->fi_wgen++; if (fip->fi_writers == 1) { fpipe->pipe_state &= ~PIPE_EOF; if (fip->fi_readers > 0) @@ -187,6 +192,7 @@ fifo_open(ap) } if ((ap->a_mode & O_NONBLOCK) == 0) { if ((ap->a_mode & FREAD) && fip->fi_writers == 0) { + gen = fip->fi_wgen; VOP_UNLOCK(vp, 0); stops_deferred = sigallowstop(); error = msleep(&fip->fi_readers, PIPE_MTX(fpipe), @@ -194,7 +200,7 @@ fifo_open(ap) if (stops_deferred) sigdeferstop(); vn_lock(vp, LK_EXCLUSIVE | LK_RETRY); - if (error) { + if (error != 0 && gen == fip->fi_wgen) { fip->fi_readers--; if (fip->fi_readers == 0) { PIPE_LOCK(fpipe); @@ -214,6 +220,7 @@ fifo_open(ap) */ } if ((ap->a_mode & FWRITE) && fip->fi_readers == 0) { + gen = fip->fi_rgen; VOP_UNLOCK(vp, 0); stops_deferred = sigallowstop(); error = msleep(&fip->fi_writers, PIPE_MTX(fpipe), @@ -221,7 +228,7 @@ fifo_open(ap) if (stops_deferred) sigdeferstop(); vn_lock(vp, LK_EXCLUSIVE | LK_RETRY); - if (error) { + if (error != 0 && gen == fip->fi_rgen) { fip->fi_writers--; if (fip->fi_writers == 0) { PIPE_LOCK(fpipe); From owner-svn-src-stable@freebsd.org Sun Sep 27 01:15:18 2015 Return-Path: Delivered-To: svn-src-stable@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 24DD2A0A8E7; Sun, 27 Sep 2015 01:15:18 +0000 (UTC) (envelope-from alc@FreeBSD.org) Received: from repo.freebsd.org (repo.freebsd.org [IPv6:2001:1900:2254:2068::e6a:0]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 161729C6; Sun, 27 Sep 2015 01:15:18 +0000 (UTC) (envelope-from alc@FreeBSD.org) Received: from repo.freebsd.org ([127.0.1.70]) by repo.freebsd.org (8.15.2/8.15.2) with ESMTP id t8R1FH3I091603; Sun, 27 Sep 2015 01:15:17 GMT (envelope-from alc@FreeBSD.org) Received: (from alc@localhost) by repo.freebsd.org (8.15.2/8.15.2/Submit) id t8R1FHu7091602; Sun, 27 Sep 2015 01:15:17 GMT (envelope-from alc@FreeBSD.org) Message-Id: <201509270115.t8R1FHu7091602@repo.freebsd.org> X-Authentication-Warning: repo.freebsd.org: alc set sender to alc@FreeBSD.org using -f From: Alan Cox Date: Sun, 27 Sep 2015 01:15:17 +0000 (UTC) To: src-committers@freebsd.org, svn-src-all@freebsd.org, svn-src-stable@freebsd.org, svn-src-stable-10@freebsd.org Subject: svn commit: r288283 - stable/10/sys/vm X-SVN-Group: stable-10 MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit X-BeenThere: svn-src-stable@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: SVN commit messages for all the -stable branches of the src tree List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 27 Sep 2015 01:15:18 -0000 Author: alc Date: Sun Sep 27 01:15:17 2015 New Revision: 288283 URL: https://svnweb.freebsd.org/changeset/base/288283 Log: MFC r287944 Eliminate (many) unnecessary calls to pmap_remove_all(). Modified: stable/10/sys/vm/vm_page.c Directory Properties: stable/10/ (props changed) Modified: stable/10/sys/vm/vm_page.c ============================================================================== --- stable/10/sys/vm/vm_page.c Sun Sep 27 01:06:45 2015 (r288282) +++ stable/10/sys/vm/vm_page.c Sun Sep 27 01:15:17 2015 (r288283) @@ -3009,7 +3009,8 @@ vm_page_set_invalid(vm_page_t m, int bas bits = VM_PAGE_BITS_ALL; else bits = vm_page_bits(base, size); - if (m->valid == VM_PAGE_BITS_ALL && bits != 0) + if (object->ref_count != 0 && m->valid == VM_PAGE_BITS_ALL && + bits != 0) pmap_remove_all(m); KASSERT((bits == 0 && m->valid == VM_PAGE_BITS_ALL) || !pmap_page_is_mapped(m), From owner-svn-src-stable@freebsd.org Sun Sep 27 01:19:43 2015 Return-Path: Delivered-To: svn-src-stable@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 8C8DFA0AAC4; Sun, 27 Sep 2015 01:19:43 +0000 (UTC) (envelope-from mav@FreeBSD.org) Received: from repo.freebsd.org (repo.freebsd.org [IPv6:2001:1900:2254:2068::e6a:0]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 7D645B6A; Sun, 27 Sep 2015 01:19:43 +0000 (UTC) (envelope-from mav@FreeBSD.org) Received: from repo.freebsd.org ([127.0.1.70]) by repo.freebsd.org (8.15.2/8.15.2) with ESMTP id t8R1JhTG091855; Sun, 27 Sep 2015 01:19:43 GMT (envelope-from mav@FreeBSD.org) Received: (from mav@localhost) by repo.freebsd.org (8.15.2/8.15.2/Submit) id t8R1Jhwv091854; Sun, 27 Sep 2015 01:19:43 GMT (envelope-from mav@FreeBSD.org) Message-Id: <201509270119.t8R1Jhwv091854@repo.freebsd.org> X-Authentication-Warning: repo.freebsd.org: mav set sender to mav@FreeBSD.org using -f From: Alexander Motin Date: Sun, 27 Sep 2015 01:19:43 +0000 (UTC) To: src-committers@freebsd.org, svn-src-all@freebsd.org, svn-src-stable@freebsd.org, svn-src-stable-10@freebsd.org Subject: svn commit: r288284 - stable/10/sys/dev/uart X-SVN-Group: stable-10 MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit X-BeenThere: svn-src-stable@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: SVN commit messages for all the -stable branches of the src tree List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 27 Sep 2015 01:19:43 -0000 Author: mav Date: Sun Sep 27 01:19:42 2015 New Revision: 288284 URL: https://svnweb.freebsd.org/changeset/base/288284 Log: MFC r287747: Add ID for Intel Panther Point KT Controller Found on ASUS P8Q77-M motherboard. Submitted by: Dmitry Luhtionov Modified: stable/10/sys/dev/uart/uart_bus_pci.c Directory Properties: stable/10/ (props changed) Modified: stable/10/sys/dev/uart/uart_bus_pci.c ============================================================================== --- stable/10/sys/dev/uart/uart_bus_pci.c Sun Sep 27 01:15:17 2015 (r288283) +++ stable/10/sys/dev/uart/uart_bus_pci.c Sun Sep 27 01:19:42 2015 (r288284) @@ -120,6 +120,7 @@ static const struct pci_id pci_ns8250_id 8 * DEFAULT_RCLK }, { 0x8086, 0x1c3d, 0xffff, 0, "Intel AMT - KT Controller", 0x10 }, { 0x8086, 0x1d3d, 0xffff, 0, "Intel C600/X79 Series Chipset KT Controller", 0x10 }, +{ 0x8086, 0x1e3d, 0xffff, 0, "Intel Panther Point KT Controller", 0x10 }, { 0x8086, 0x2a07, 0xffff, 0, "Intel AMT - PM965/GM965 KT Controller", 0x10 }, { 0x8086, 0x2a47, 0xffff, 0, "Mobile 4 Series Chipset KT Controller", 0x10 }, { 0x8086, 0x2e17, 0xffff, 0, "4 Series Chipset Serial KT Controller", 0x10 }, From owner-svn-src-stable@freebsd.org Sun Sep 27 01:20:56 2015 Return-Path: Delivered-To: svn-src-stable@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 1A526A0AB7E; Sun, 27 Sep 2015 01:20:56 +0000 (UTC) (envelope-from mav@FreeBSD.org) Received: from repo.freebsd.org (repo.freebsd.org [IPv6:2001:1900:2254:2068::e6a:0]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 0B55ED40; Sun, 27 Sep 2015 01:20:56 +0000 (UTC) (envelope-from mav@FreeBSD.org) Received: from repo.freebsd.org ([127.0.1.70]) by repo.freebsd.org (8.15.2/8.15.2) with ESMTP id t8R1Ktq7095071; Sun, 27 Sep 2015 01:20:55 GMT (envelope-from mav@FreeBSD.org) Received: (from mav@localhost) by repo.freebsd.org (8.15.2/8.15.2/Submit) id t8R1Kt63095070; Sun, 27 Sep 2015 01:20:55 GMT (envelope-from mav@FreeBSD.org) Message-Id: <201509270120.t8R1Kt63095070@repo.freebsd.org> X-Authentication-Warning: repo.freebsd.org: mav set sender to mav@FreeBSD.org using -f From: Alexander Motin Date: Sun, 27 Sep 2015 01:20:55 +0000 (UTC) To: src-committers@freebsd.org, svn-src-all@freebsd.org, svn-src-stable@freebsd.org, svn-src-stable-9@freebsd.org Subject: svn commit: r288285 - stable/9/sys/dev/uart X-SVN-Group: stable-9 MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit X-BeenThere: svn-src-stable@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: SVN commit messages for all the -stable branches of the src tree List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 27 Sep 2015 01:20:56 -0000 Author: mav Date: Sun Sep 27 01:20:55 2015 New Revision: 288285 URL: https://svnweb.freebsd.org/changeset/base/288285 Log: MFC r287747: Add ID for Intel Panther Point KT Controller Found on ASUS P8Q77-M motherboard. Submitted by: Dmitry Luhtionov Modified: stable/9/sys/dev/uart/uart_bus_pci.c Directory Properties: stable/9/ (props changed) stable/9/sys/ (props changed) stable/9/sys/dev/ (props changed) Modified: stable/9/sys/dev/uart/uart_bus_pci.c ============================================================================== --- stable/9/sys/dev/uart/uart_bus_pci.c Sun Sep 27 01:19:42 2015 (r288284) +++ stable/9/sys/dev/uart/uart_bus_pci.c Sun Sep 27 01:20:55 2015 (r288285) @@ -120,6 +120,7 @@ static const struct pci_id pci_ns8250_id 8 * DEFAULT_RCLK }, { 0x8086, 0x1c3d, 0xffff, 0, "Intel AMT - KT Controller", 0x10 }, { 0x8086, 0x1d3d, 0xffff, 0, "Intel C600/X79 Series Chipset KT Controller", 0x10 }, +{ 0x8086, 0x1e3d, 0xffff, 0, "Intel Panther Point KT Controller", 0x10 }, { 0x8086, 0x2a07, 0xffff, 0, "Intel AMT - PM965/GM965 KT Controller", 0x10 }, { 0x8086, 0x2a47, 0xffff, 0, "Mobile 4 Series Chipset KT Controller", 0x10 }, { 0x8086, 0x2e17, 0xffff, 0, "4 Series Chipset Serial KT Controller", 0x10 }, From owner-svn-src-stable@freebsd.org Sun Sep 27 01:26:42 2015 Return-Path: Delivered-To: svn-src-stable@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 8043A9CF025; Sun, 27 Sep 2015 01:26:42 +0000 (UTC) (envelope-from alc@FreeBSD.org) Received: from repo.freebsd.org (repo.freebsd.org [IPv6:2001:1900:2254:2068::e6a:0]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 71814155; Sun, 27 Sep 2015 01:26:42 +0000 (UTC) (envelope-from alc@FreeBSD.org) Received: from repo.freebsd.org ([127.0.1.70]) by repo.freebsd.org (8.15.2/8.15.2) with ESMTP id t8R1QgvK095879; Sun, 27 Sep 2015 01:26:42 GMT (envelope-from alc@FreeBSD.org) Received: (from alc@localhost) by repo.freebsd.org (8.15.2/8.15.2/Submit) id t8R1QgK8095878; Sun, 27 Sep 2015 01:26:42 GMT (envelope-from alc@FreeBSD.org) Message-Id: <201509270126.t8R1QgK8095878@repo.freebsd.org> X-Authentication-Warning: repo.freebsd.org: alc set sender to alc@FreeBSD.org using -f From: Alan Cox Date: Sun, 27 Sep 2015 01:26:42 +0000 (UTC) To: src-committers@freebsd.org, svn-src-all@freebsd.org, svn-src-stable@freebsd.org, svn-src-stable-10@freebsd.org Subject: svn commit: r288286 - stable/10/sys/vm X-SVN-Group: stable-10 MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit X-BeenThere: svn-src-stable@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: SVN commit messages for all the -stable branches of the src tree List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 27 Sep 2015 01:26:42 -0000 Author: alc Date: Sun Sep 27 01:26:41 2015 New Revision: 288286 URL: https://svnweb.freebsd.org/changeset/base/288286 Log: MFC r284654 Avoid pmap_is_modified() on pages that can't be mapped. Modified: stable/10/sys/vm/vm_pageout.c Directory Properties: stable/10/ (props changed) Modified: stable/10/sys/vm/vm_pageout.c ============================================================================== --- stable/10/sys/vm/vm_pageout.c Sun Sep 27 01:20:55 2015 (r288285) +++ stable/10/sys/vm/vm_pageout.c Sun Sep 27 01:26:41 2015 (r288286) @@ -1115,9 +1115,11 @@ vm_pageout_scan(struct vm_domain *vmd, i * then the page may still be modified until the last of those * mappings are removed. */ - vm_page_test_dirty(m); - if (m->dirty == 0 && object->ref_count != 0) - pmap_remove_all(m); + if (object->ref_count != 0) { + vm_page_test_dirty(m); + if (m->dirty == 0) + pmap_remove_all(m); + } if (m->valid == 0) { /* From owner-svn-src-stable@freebsd.org Sun Sep 27 01:33:47 2015 Return-Path: Delivered-To: svn-src-stable@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id B6BAD9CF5F8; Sun, 27 Sep 2015 01:33:47 +0000 (UTC) (envelope-from kib@FreeBSD.org) Received: from repo.freebsd.org (repo.freebsd.org [IPv6:2001:1900:2254:2068::e6a:0]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id A5832809; Sun, 27 Sep 2015 01:33:47 +0000 (UTC) (envelope-from kib@FreeBSD.org) Received: from repo.freebsd.org ([127.0.1.70]) by repo.freebsd.org (8.15.2/8.15.2) with ESMTP id t8R1XlLe099862; Sun, 27 Sep 2015 01:33:47 GMT (envelope-from kib@FreeBSD.org) Received: (from kib@localhost) by repo.freebsd.org (8.15.2/8.15.2/Submit) id t8R1Xix6099851; Sun, 27 Sep 2015 01:33:44 GMT (envelope-from kib@FreeBSD.org) Message-Id: <201509270133.t8R1Xix6099851@repo.freebsd.org> X-Authentication-Warning: repo.freebsd.org: kib set sender to kib@FreeBSD.org using -f From: Konstantin Belousov Date: Sun, 27 Sep 2015 01:33:44 +0000 (UTC) To: src-committers@freebsd.org, svn-src-all@freebsd.org, svn-src-stable@freebsd.org, svn-src-stable-10@freebsd.org Subject: svn commit: r288287 - in stable/10/sys: amd64/amd64 arm/arm i386/i386 ia64/ia64 kern mips/mips powerpc/powerpc sparc64/sparc64 sys X-SVN-Group: stable-10 MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit X-BeenThere: svn-src-stable@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: SVN commit messages for all the -stable branches of the src tree List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 27 Sep 2015 01:33:47 -0000 Author: kib Date: Sun Sep 27 01:33:43 2015 New Revision: 288287 URL: https://svnweb.freebsd.org/changeset/base/288287 Log: MFC r288000: Add support for weak symbols to the kernel linkers. Modified: stable/10/sys/amd64/amd64/elf_machdep.c stable/10/sys/arm/arm/elf_machdep.c stable/10/sys/i386/i386/elf_machdep.c stable/10/sys/ia64/ia64/elf_machdep.c stable/10/sys/kern/link_elf.c stable/10/sys/kern/link_elf_obj.c stable/10/sys/mips/mips/elf_machdep.c stable/10/sys/powerpc/powerpc/elf32_machdep.c stable/10/sys/powerpc/powerpc/elf64_machdep.c stable/10/sys/sparc64/sparc64/elf_machdep.c stable/10/sys/sys/linker.h Directory Properties: stable/10/ (props changed) Modified: stable/10/sys/amd64/amd64/elf_machdep.c ============================================================================== --- stable/10/sys/amd64/amd64/elf_machdep.c Sun Sep 27 01:26:41 2015 (r288286) +++ stable/10/sys/amd64/amd64/elf_machdep.c Sun Sep 27 01:33:43 2015 (r288287) @@ -167,6 +167,7 @@ elf_reloc_internal(linker_file_t lf, Elf Elf_Size rtype, symidx; const Elf_Rel *rel; const Elf_Rela *rela; + int error; switch (type) { case ELF_RELOC_REL: @@ -202,29 +203,29 @@ elf_reloc_internal(linker_file_t lf, Elf break; case R_X86_64_64: /* S + A */ - addr = lookup(lf, symidx, 1); + error = lookup(lf, symidx, 1, &addr); val = addr + addend; - if (addr == 0) + if (error != 0) return -1; if (*where != val) *where = val; break; case R_X86_64_PC32: /* S + A - P */ - addr = lookup(lf, symidx, 1); + error = lookup(lf, symidx, 1, &addr); where32 = (Elf32_Addr *)where; val32 = (Elf32_Addr)(addr + addend - (Elf_Addr)where); - if (addr == 0) + if (error != 0) return -1; if (*where32 != val32) *where32 = val32; break; case R_X86_64_32S: /* S + A sign extend */ - addr = lookup(lf, symidx, 1); + error = lookup(lf, symidx, 1, &addr); val32 = (Elf32_Addr)(addr + addend); where32 = (Elf32_Addr *)where; - if (addr == 0) + if (error != 0) return -1; if (*where32 != val32) *where32 = val32; @@ -241,8 +242,8 @@ elf_reloc_internal(linker_file_t lf, Elf case R_X86_64_GLOB_DAT: /* S */ case R_X86_64_JMP_SLOT: /* XXX need addend + offset */ - addr = lookup(lf, symidx, 1); - if (addr == 0) + error = lookup(lf, symidx, 1, &addr); + if (error != 0) return -1; if (*where != addr) *where = addr; Modified: stable/10/sys/arm/arm/elf_machdep.c ============================================================================== --- stable/10/sys/arm/arm/elf_machdep.c Sun Sep 27 01:26:41 2015 (r288286) +++ stable/10/sys/arm/arm/elf_machdep.c Sun Sep 27 01:33:43 2015 (r288287) @@ -132,6 +132,7 @@ elf_reloc_internal(linker_file_t lf, Elf Elf_Word rtype, symidx; const Elf_Rel *rel; const Elf_Rela *rela; + int error; switch (type) { case ELF_RELOC_REL: @@ -167,8 +168,8 @@ elf_reloc_internal(linker_file_t lf, Elf break; case R_ARM_ABS32: - addr = lookup(lf, symidx, 1); - if (addr == 0) + error = lookup(lf, symidx, 1, &addr); + if (error != 0) return -1; *where += addr; break; @@ -183,8 +184,8 @@ elf_reloc_internal(linker_file_t lf, Elf break; case R_ARM_JUMP_SLOT: - addr = lookup(lf, symidx, 1); - if (addr) { + error = lookup(lf, symidx, 1, &addr); + if (error == 0) { *where = addr; return (0); } Modified: stable/10/sys/i386/i386/elf_machdep.c ============================================================================== --- stable/10/sys/i386/i386/elf_machdep.c Sun Sep 27 01:26:41 2015 (r288286) +++ stable/10/sys/i386/i386/elf_machdep.c Sun Sep 27 01:33:43 2015 (r288287) @@ -177,6 +177,7 @@ elf_reloc_internal(linker_file_t lf, Elf Elf_Word rtype, symidx; const Elf_Rel *rel; const Elf_Rela *rela; + int error; switch (type) { case ELF_RELOC_REL: @@ -212,8 +213,8 @@ elf_reloc_internal(linker_file_t lf, Elf break; case R_386_32: /* S + A */ - addr = lookup(lf, symidx, 1); - if (addr == 0) + error = lookup(lf, symidx, 1, &addr); + if (error != 0) return -1; addr += addend; if (*where != addr) @@ -221,8 +222,8 @@ elf_reloc_internal(linker_file_t lf, Elf break; case R_386_PC32: /* S + A - P */ - addr = lookup(lf, symidx, 1); - if (addr == 0) + error = lookup(lf, symidx, 1, &addr); + if (error != 0) return -1; addr += addend - (Elf_Addr)where; if (*where != addr) @@ -239,8 +240,8 @@ elf_reloc_internal(linker_file_t lf, Elf break; case R_386_GLOB_DAT: /* S */ - addr = lookup(lf, symidx, 1); - if (addr == 0) + error = lookup(lf, symidx, 1, &addr); + if (error != 0) return -1; if (*where != addr) *where = addr; Modified: stable/10/sys/ia64/ia64/elf_machdep.c ============================================================================== --- stable/10/sys/ia64/ia64/elf_machdep.c Sun Sep 27 01:26:41 2015 (r288286) +++ stable/10/sys/ia64/ia64/elf_machdep.c Sun Sep 27 01:33:43 2015 (r288287) @@ -127,17 +127,18 @@ elf64_dump_thread(struct thread *td, voi } -static Elf_Addr -lookup_fdesc(linker_file_t lf, Elf_Size symidx, elf_lookup_fn lookup) +static int +lookup_fdesc(linker_file_t lf, Elf_Size symidx, elf_lookup_fn lookup, + Elf_Addr *addr1) { linker_file_t top; Elf_Addr addr; const char *symname; - int i; + int i, error; static int eot = 0; - addr = lookup(lf, symidx, 0); - if (addr == 0) { + error = lookup(lf, symidx, 0, &addr); + if (error != 0) { top = lf; symname = elf_get_symname(top, symidx); for (i = 0; i < top->ndeps; i++) { @@ -148,30 +149,33 @@ lookup_fdesc(linker_file_t lf, Elf_Size break; } if (addr == 0) - return (0); + return (EINVAL); } if (eot) - return (0); + return (EINVAL); /* * Lookup and/or construct OPD */ for (i = 0; i < 8192; i += 2) { - if (fptr_storage[i] == addr) - return (Elf_Addr)(fptr_storage + i); + if (fptr_storage[i] == addr) { + *addr1 = (Elf_Addr)(fptr_storage + i); + return (0); + } if (fptr_storage[i] == 0) { fptr_storage[i] = addr; fptr_storage[i+1] = link_elf_get_gp(lf); - return (Elf_Addr)(fptr_storage + i); + *addr1 = (Elf_Addr)(fptr_storage + i); + return (0); } } printf("%s: fptr table full\n", __func__); eot = 1; - return (0); + return (EINVAL); } /* Process one elf relocation with addend. */ @@ -184,6 +188,7 @@ elf_reloc_internal(linker_file_t lf, Elf Elf_Size rtype, symidx; const Elf_Rel *rel; const Elf_Rela *rela; + int error; switch (type) { case ELF_RELOC_REL: @@ -223,8 +228,8 @@ elf_reloc_internal(linker_file_t lf, Elf case R_IA_64_NONE: break; case R_IA_64_DIR64LSB: /* word64 LSB S + A */ - addr = lookup(lf, symidx, 1); - if (addr == 0) + error = lookup(lf, symidx, 1, &addr); + if (error != 0) return (-1); *where = addr + addend; break; @@ -233,16 +238,16 @@ elf_reloc_internal(linker_file_t lf, Elf printf("%s: addend ignored for OPD relocation\n", __func__); } - addr = lookup_fdesc(lf, symidx, lookup); - if (addr == 0) + error = lookup_fdesc(lf, symidx, lookup, &addr); + if (error != 0) return (-1); *where = addr; break; case R_IA_64_REL64LSB: /* word64 LSB BD + A */ break; case R_IA_64_IPLTLSB: - addr = lookup_fdesc(lf, symidx, lookup); - if (addr == 0) + error = lookup_fdesc(lf, symidx, lookup, &addr); + if (error != 0) return (-1); where[0] = *((Elf_Addr*)addr) + addend; where[1] = *((Elf_Addr*)addr + 1); Modified: stable/10/sys/kern/link_elf.c ============================================================================== --- stable/10/sys/kern/link_elf.c Sun Sep 27 01:26:41 2015 (r288286) +++ stable/10/sys/kern/link_elf.c Sun Sep 27 01:33:43 2015 (r288287) @@ -158,7 +158,7 @@ static int link_elf_each_function_nameva static void link_elf_reloc_local(linker_file_t); static long link_elf_symtab_get(linker_file_t, const Elf_Sym **); static long link_elf_strtab_get(linker_file_t, caddr_t *); -static Elf_Addr elf_lookup(linker_file_t, Elf_Size, int); +static int elf_lookup(linker_file_t, Elf_Size, int, Elf_Addr *); static kobj_method_t link_elf_methods[] = { KOBJMETHOD(linker_lookup_symbol, link_elf_lookup_symbol), @@ -1498,8 +1498,8 @@ elf_get_symname(linker_file_t lf, Elf_Si * This is not only more efficient, it's also more correct. It's not always * the case that the symbol can be found through the hash table. */ -static Elf_Addr -elf_lookup(linker_file_t lf, Elf_Size symidx, int deps) +static int +elf_lookup(linker_file_t lf, Elf_Size symidx, int deps, Elf_Addr *res) { elf_file_t ef = (elf_file_t)lf; const Elf_Sym *sym; @@ -1507,8 +1507,10 @@ elf_lookup(linker_file_t lf, Elf_Size sy Elf_Addr addr, start, base; /* Don't even try to lookup the symbol if the index is bogus. */ - if (symidx >= ef->nchains) - return (0); + if (symidx >= ef->nchains) { + *res = 0; + return (EINVAL); + } sym = ef->symtab + symidx; @@ -1518,9 +1520,12 @@ elf_lookup(linker_file_t lf, Elf_Size sy */ if (ELF_ST_BIND(sym->st_info) == STB_LOCAL) { /* Force lookup failure when we have an insanity. */ - if (sym->st_shndx == SHN_UNDEF || sym->st_value == 0) - return (0); - return ((Elf_Addr)ef->address + sym->st_value); + if (sym->st_shndx == SHN_UNDEF || sym->st_value == 0) { + *res = 0; + return (EINVAL); + } + *res = ((Elf_Addr)ef->address + sym->st_value); + return (0); } /* @@ -1533,8 +1538,10 @@ elf_lookup(linker_file_t lf, Elf_Size sy symbol = ef->strtab + sym->st_name; /* Force a lookup failure if the symbol name is bogus. */ - if (*symbol == 0) - return (0); + if (*symbol == 0) { + *res = 0; + return (EINVAL); + } addr = ((Elf_Addr)linker_file_lookup_symbol(lf, symbol, deps)); @@ -1544,7 +1551,8 @@ elf_lookup(linker_file_t lf, Elf_Size sy else if (elf_set_find(&set_vnet_list, addr, &start, &base)) addr = addr - start + base; #endif - return addr; + *res = addr; + return (0); } static void Modified: stable/10/sys/kern/link_elf_obj.c ============================================================================== --- stable/10/sys/kern/link_elf_obj.c Sun Sep 27 01:26:41 2015 (r288286) +++ stable/10/sys/kern/link_elf_obj.c Sun Sep 27 01:33:43 2015 (r288287) @@ -144,7 +144,8 @@ static void link_elf_reloc_local(linker_ static long link_elf_symtab_get(linker_file_t, const Elf_Sym **); static long link_elf_strtab_get(linker_file_t, caddr_t *); -static Elf_Addr elf_obj_lookup(linker_file_t lf, Elf_Size symidx, int deps); +static int elf_obj_lookup(linker_file_t lf, Elf_Size symidx, int deps, + Elf_Addr *); static kobj_method_t link_elf_methods[] = { KOBJMETHOD(linker_lookup_symbol, link_elf_lookup_symbol), @@ -1224,38 +1225,46 @@ elf_obj_cleanup_globals_cache(elf_file_t * This is not only more efficient, it's also more correct. It's not always * the case that the symbol can be found through the hash table. */ -static Elf_Addr -elf_obj_lookup(linker_file_t lf, Elf_Size symidx, int deps) +static int +elf_obj_lookup(linker_file_t lf, Elf_Size symidx, int deps, Elf_Addr *res) { elf_file_t ef = (elf_file_t)lf; Elf_Sym *sym; const char *symbol; - Elf_Addr ret; + Elf_Addr res1; /* Don't even try to lookup the symbol if the index is bogus. */ - if (symidx >= ef->ddbsymcnt) - return (0); + if (symidx >= ef->ddbsymcnt) { + *res = 0; + return (EINVAL); + } sym = ef->ddbsymtab + symidx; /* Quick answer if there is a definition included. */ - if (sym->st_shndx != SHN_UNDEF) - return (sym->st_value); + if (sym->st_shndx != SHN_UNDEF) { + *res = sym->st_value; + return (0); + } /* If we get here, then it is undefined and needs a lookup. */ switch (ELF_ST_BIND(sym->st_info)) { case STB_LOCAL: /* Local, but undefined? huh? */ - return (0); + *res = 0; + return (EINVAL); case STB_GLOBAL: + case STB_WEAK: /* Relative to Data or Function name */ symbol = ef->ddbstrtab + sym->st_name; /* Force a lookup failure if the symbol name is bogus. */ - if (*symbol == 0) - return (0); - ret = ((Elf_Addr)linker_file_lookup_symbol(lf, symbol, deps)); + if (*symbol == 0) { + *res = 0; + return (EINVAL); + } + res1 = (Elf_Addr)linker_file_lookup_symbol(lf, symbol, deps); /* * Cache global lookups during module relocation. The failure @@ -1267,18 +1276,20 @@ elf_obj_lookup(linker_file_t lf, Elf_Siz * restored to SHN_UNDEF in elf_obj_cleanup_globals_cache(), * above. */ - if (ret != 0) { + if (res1 != 0) { sym->st_shndx = SHN_FBSD_CACHED; - sym->st_value = ret; + sym->st_value = res1; + *res = res1; + return (0); + } else if (ELF_ST_BIND(sym->st_info) == STB_WEAK) { + sym->st_value = 0; + *res = 0; + return (0); } - return (ret); - - case STB_WEAK: - printf("link_elf_obj: Weak symbols not supported\n"); - return (0); + return (EINVAL); default: - return (0); + return (EINVAL); } } Modified: stable/10/sys/mips/mips/elf_machdep.c ============================================================================== --- stable/10/sys/mips/mips/elf_machdep.c Sun Sep 27 01:26:41 2015 (r288286) +++ stable/10/sys/mips/mips/elf_machdep.c Sun Sep 27 01:33:43 2015 (r288287) @@ -174,6 +174,7 @@ elf_reloc_internal(linker_file_t lf, Elf Elf_Word rtype = (Elf_Word)0, symidx; const Elf_Rel *rel = NULL; const Elf_Rela *rela = NULL; + int error; /* * Stash R_MIPS_HI16 info so we can use it when processing R_MIPS_LO16 @@ -213,8 +214,8 @@ elf_reloc_internal(linker_file_t lf, Elf break; case R_MIPS_32: /* S + A */ - addr = lookup(lf, symidx, 1); - if (addr == 0) + error = lookup(lf, symidx, 1, &addr); + if (error != 0) return (-1); addr += addend; if (*where != addr) @@ -222,8 +223,8 @@ elf_reloc_internal(linker_file_t lf, Elf break; case R_MIPS_26: /* ((A << 2) | (P & 0xf0000000) + S) >> 2 */ - addr = lookup(lf, symidx, 1); - if (addr == 0) + error = lookup(lf, symidx, 1, &addr); + if (error != 0) return (-1); addend &= 0x03ffffff; @@ -241,8 +242,8 @@ elf_reloc_internal(linker_file_t lf, Elf break; case R_MIPS_64: /* S + A */ - addr = lookup(lf, symidx, 1); - if (addr == 0) + error = lookup(lf, symidx, 1, &addr); + if (error != 0) return (-1); addr += addend; if (*(Elf64_Addr*)where != addr) @@ -251,8 +252,8 @@ elf_reloc_internal(linker_file_t lf, Elf case R_MIPS_HI16: /* ((AHL + S) - ((short)(AHL + S)) >> 16 */ if (rela != NULL) { - addr = lookup(lf, symidx, 1); - if (addr == 0) + error = lookup(lf, symidx, 1, &addr); + if (error != 0) return (-1); addr += addend; *where &= 0xffff0000; @@ -266,8 +267,8 @@ elf_reloc_internal(linker_file_t lf, Elf case R_MIPS_LO16: /* AHL + S */ if (rela != NULL) { - addr = lookup(lf, symidx, 1); - if (addr == 0) + error = lookup(lf, symidx, 1, &addr); + if (error != 0) return (-1); addr += addend; *where &= 0xffff0000; @@ -275,8 +276,8 @@ elf_reloc_internal(linker_file_t lf, Elf } else { ahl += (int16_t)addend; - addr = lookup(lf, symidx, 1); - if (addr == 0) + error = lookup(lf, symidx, 1, &addr); + if (error != 0) return (-1); addend &= 0xffff0000; @@ -292,8 +293,8 @@ elf_reloc_internal(linker_file_t lf, Elf break; case R_MIPS_HIGHER: /* %higher(A+S) */ - addr = lookup(lf, symidx, 1); - if (addr == 0) + error = lookup(lf, symidx, 1, &addr); + if (error != 0) return (-1); addr += addend; *where &= 0xffff0000; @@ -301,8 +302,8 @@ elf_reloc_internal(linker_file_t lf, Elf break; case R_MIPS_HIGHEST: /* %highest(A+S) */ - addr = lookup(lf, symidx, 1); - if (addr == 0) + error = lookup(lf, symidx, 1, &addr); + if (error != 0) return (-1); addr += addend; *where &= 0xffff0000; Modified: stable/10/sys/powerpc/powerpc/elf32_machdep.c ============================================================================== --- stable/10/sys/powerpc/powerpc/elf32_machdep.c Sun Sep 27 01:26:41 2015 (r288286) +++ stable/10/sys/powerpc/powerpc/elf32_machdep.c Sun Sep 27 01:33:43 2015 (r288287) @@ -164,6 +164,7 @@ elf_reloc_internal(linker_file_t lf, Elf Elf_Addr addend; Elf_Word rtype, symidx; const Elf_Rela *rela; + int error; switch (type) { case ELF_RELOC_REL: @@ -187,16 +188,16 @@ elf_reloc_internal(linker_file_t lf, Elf break; case R_PPC_ADDR32: /* word32 S + A */ - addr = lookup(lf, symidx, 1); - if (addr == 0) - return -1; + error = lookup(lf, symidx, 1, &addr); + if (error != 0) + return -1; addr += addend; *where = addr; break; case R_PPC_ADDR16_LO: /* #lo(S) */ - addr = lookup(lf, symidx, 1); - if (addr == 0) + error = lookup(lf, symidx, 1, &addr); + if (error != 0) return -1; /* * addend values are sometimes relative to sections @@ -211,8 +212,8 @@ elf_reloc_internal(linker_file_t lf, Elf break; case R_PPC_ADDR16_HA: /* #ha(S) */ - addr = lookup(lf, symidx, 1); - if (addr == 0) + error = lookup(lf, symidx, 1, &addr); + if (error != 0) return -1; /* * addend values are sometimes relative to sections Modified: stable/10/sys/powerpc/powerpc/elf64_machdep.c ============================================================================== --- stable/10/sys/powerpc/powerpc/elf64_machdep.c Sun Sep 27 01:26:41 2015 (r288286) +++ stable/10/sys/powerpc/powerpc/elf64_machdep.c Sun Sep 27 01:33:43 2015 (r288287) @@ -135,6 +135,7 @@ elf_reloc_internal(linker_file_t lf, Elf Elf_Addr addend; Elf_Word rtype, symidx; const Elf_Rela *rela; + int error; switch (type) { case ELF_RELOC_REL: @@ -157,9 +158,9 @@ elf_reloc_internal(linker_file_t lf, Elf break; case R_PPC64_ADDR64: /* doubleword64 S + A */ - addr = lookup(lf, symidx, 1); - if (addr == 0) - return -1; + error = lookup(lf, symidx, 1, &addr); + if (error != 0) + return -1; addr += addend; *where = addr; break; @@ -169,7 +170,7 @@ elf_reloc_internal(linker_file_t lf, Elf break; case R_PPC_JMP_SLOT: /* function descriptor copy */ - addr = lookup(lf, symidx, 1); + lookup(lf, symidx, 1, &addr); memcpy(where, (Elf_Addr *)addr, 3*sizeof(Elf_Addr)); __asm __volatile("dcbst 0,%0; sync" :: "r"(where) : "memory"); break; Modified: stable/10/sys/sparc64/sparc64/elf_machdep.c ============================================================================== --- stable/10/sys/sparc64/sparc64/elf_machdep.c Sun Sep 27 01:26:41 2015 (r288286) +++ stable/10/sys/sparc64/sparc64/elf_machdep.c Sun Sep 27 01:33:43 2015 (r288287) @@ -343,6 +343,7 @@ elf_reloc(linker_file_t lf, Elf_Addr rel Elf_Addr value; Elf_Addr mask; Elf_Addr addr; + int error; if (type != ELF_RELOC_RELA) return (-1); @@ -371,8 +372,8 @@ elf_reloc(linker_file_t lf, Elf_Addr rel value = rela->r_addend; if (RELOC_RESOLVE_SYMBOL(rtype)) { - addr = lookup(lf, symidx, 1); - if (addr == 0) + error = lookup(lf, symidx, 1, &addr); + if (error != 0) return (-1); value += addr; if (RELOC_BARE_SYMBOL(rtype)) Modified: stable/10/sys/sys/linker.h ============================================================================== --- stable/10/sys/sys/linker.h Sun Sep 27 01:26:41 2015 (r288286) +++ stable/10/sys/sys/linker.h Sun Sep 27 01:33:43 2015 (r288287) @@ -259,7 +259,7 @@ extern int kld_debug; #endif -typedef Elf_Addr elf_lookup_fn(linker_file_t, Elf_Size, int); +typedef int elf_lookup_fn(linker_file_t, Elf_Size, int, Elf_Addr *); /* Support functions */ int elf_reloc(linker_file_t _lf, Elf_Addr base, const void *_rel, int _type, elf_lookup_fn _lu); From owner-svn-src-stable@freebsd.org Sun Sep 27 01:35:33 2015 Return-Path: Delivered-To: svn-src-stable@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 9FE619CF7E1; Sun, 27 Sep 2015 01:35:33 +0000 (UTC) (envelope-from alc@FreeBSD.org) Received: from repo.freebsd.org (repo.freebsd.org [IPv6:2001:1900:2254:2068::e6a:0]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 90F3396C; Sun, 27 Sep 2015 01:35:33 +0000 (UTC) (envelope-from alc@FreeBSD.org) Received: from repo.freebsd.org ([127.0.1.70]) by repo.freebsd.org (8.15.2/8.15.2) with ESMTP id t8R1ZXfp000102; Sun, 27 Sep 2015 01:35:33 GMT (envelope-from alc@FreeBSD.org) Received: (from alc@localhost) by repo.freebsd.org (8.15.2/8.15.2/Submit) id t8R1ZXEd000101; Sun, 27 Sep 2015 01:35:33 GMT (envelope-from alc@FreeBSD.org) Message-Id: <201509270135.t8R1ZXEd000101@repo.freebsd.org> X-Authentication-Warning: repo.freebsd.org: alc set sender to alc@FreeBSD.org using -f From: Alan Cox Date: Sun, 27 Sep 2015 01:35:33 +0000 (UTC) To: src-committers@freebsd.org, svn-src-all@freebsd.org, svn-src-stable@freebsd.org, svn-src-stable-10@freebsd.org Subject: svn commit: r288288 - stable/10/sys/vm X-SVN-Group: stable-10 MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit X-BeenThere: svn-src-stable@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: SVN commit messages for all the -stable branches of the src tree List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 27 Sep 2015 01:35:33 -0000 Author: alc Date: Sun Sep 27 01:35:32 2015 New Revision: 288288 URL: https://svnweb.freebsd.org/changeset/base/288288 Log: MFC r287121 Testing whether a page is dirty does not require the page lock. Modified: stable/10/sys/vm/vm_pageout.c Directory Properties: stable/10/ (props changed) Modified: stable/10/sys/vm/vm_pageout.c ============================================================================== --- stable/10/sys/vm/vm_pageout.c Sun Sep 27 01:33:43 2015 (r288287) +++ stable/10/sys/vm/vm_pageout.c Sun Sep 27 01:35:32 2015 (r288288) @@ -408,10 +408,13 @@ more: ib = 0; break; } - vm_page_lock(p); vm_page_test_dirty(p); - if (p->dirty == 0 || - p->queue != PQ_INACTIVE || + if (p->dirty == 0) { + ib = 0; + break; + } + vm_page_lock(p); + if (p->queue != PQ_INACTIVE || p->hold_count != 0) { /* may be undergoing I/O */ vm_page_unlock(p); ib = 0; @@ -435,10 +438,11 @@ more: if ((p = vm_page_next(ps)) == NULL || vm_page_busied(p)) break; - vm_page_lock(p); vm_page_test_dirty(p); - if (p->dirty == 0 || - p->queue != PQ_INACTIVE || + if (p->dirty == 0) + break; + vm_page_lock(p); + if (p->queue != PQ_INACTIVE || p->hold_count != 0) { /* may be undergoing I/O */ vm_page_unlock(p); break; From owner-svn-src-stable@freebsd.org Sun Sep 27 01:37:31 2015 Return-Path: Delivered-To: svn-src-stable@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 4D9F89CF993; Sun, 27 Sep 2015 01:37:31 +0000 (UTC) (envelope-from kib@FreeBSD.org) Received: from repo.freebsd.org (repo.freebsd.org [IPv6:2001:1900:2254:2068::e6a:0]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 325B9B01; Sun, 27 Sep 2015 01:37:31 +0000 (UTC) (envelope-from kib@FreeBSD.org) Received: from repo.freebsd.org ([127.0.1.70]) by repo.freebsd.org (8.15.2/8.15.2) with ESMTP id t8R1bVvh000245; Sun, 27 Sep 2015 01:37:31 GMT (envelope-from kib@FreeBSD.org) Received: (from kib@localhost) by repo.freebsd.org (8.15.2/8.15.2/Submit) id t8R1bUYD000243; Sun, 27 Sep 2015 01:37:30 GMT (envelope-from kib@FreeBSD.org) Message-Id: <201509270137.t8R1bUYD000243@repo.freebsd.org> X-Authentication-Warning: repo.freebsd.org: kib set sender to kib@FreeBSD.org using -f From: Konstantin Belousov Date: Sun, 27 Sep 2015 01:37:30 +0000 (UTC) To: src-committers@freebsd.org, svn-src-all@freebsd.org, svn-src-stable@freebsd.org, svn-src-stable-10@freebsd.org Subject: svn commit: r288289 - stable/10/sys/powerpc/powerpc X-SVN-Group: stable-10 MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit X-BeenThere: svn-src-stable@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: SVN commit messages for all the -stable branches of the src tree List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 27 Sep 2015 01:37:31 -0000 Author: kib Date: Sun Sep 27 01:37:30 2015 New Revision: 288289 URL: https://svnweb.freebsd.org/changeset/base/288289 Log: MFC r288001: Use tabs for indend. Modified: stable/10/sys/powerpc/powerpc/elf32_machdep.c stable/10/sys/powerpc/powerpc/elf64_machdep.c Directory Properties: stable/10/ (props changed) Modified: stable/10/sys/powerpc/powerpc/elf32_machdep.c ============================================================================== --- stable/10/sys/powerpc/powerpc/elf32_machdep.c Sun Sep 27 01:35:32 2015 (r288288) +++ stable/10/sys/powerpc/powerpc/elf32_machdep.c Sun Sep 27 01:37:30 2015 (r288289) @@ -184,8 +184,8 @@ elf_reloc_internal(linker_file_t lf, Elf switch (rtype) { - case R_PPC_NONE: - break; + case R_PPC_NONE: + break; case R_PPC_ADDR32: /* word32 S + A */ error = lookup(lf, symidx, 1, &addr); @@ -193,9 +193,9 @@ elf_reloc_internal(linker_file_t lf, Elf return -1; addr += addend; *where = addr; - break; + break; - case R_PPC_ADDR16_LO: /* #lo(S) */ + case R_PPC_ADDR16_LO: /* #lo(S) */ error = lookup(lf, symidx, 1, &addr); if (error != 0) return -1; @@ -224,17 +224,17 @@ elf_reloc_internal(linker_file_t lf, Elf addr = relocbase + addend; else addr += addend; - *hwhere = ((addr >> 16) + ((addr & 0x8000) ? 1 : 0)) + *hwhere = ((addr >> 16) + ((addr & 0x8000) ? 1 : 0)) & 0xffff; break; case R_PPC_RELATIVE: /* word32 B + A */ - *where = elf_relocaddr(lf, relocbase + addend); - break; + *where = elf_relocaddr(lf, relocbase + addend); + break; default: - printf("kldload: unexpected relocation type %d\n", - (int) rtype); + printf("kldload: unexpected relocation type %d\n", + (int) rtype); return -1; } return(0); Modified: stable/10/sys/powerpc/powerpc/elf64_machdep.c ============================================================================== --- stable/10/sys/powerpc/powerpc/elf64_machdep.c Sun Sep 27 01:35:32 2015 (r288288) +++ stable/10/sys/powerpc/powerpc/elf64_machdep.c Sun Sep 27 01:37:30 2015 (r288289) @@ -154,20 +154,20 @@ elf_reloc_internal(linker_file_t lf, Elf switch (rtype) { - case R_PPC_NONE: - break; + case R_PPC_NONE: + break; case R_PPC64_ADDR64: /* doubleword64 S + A */ error = lookup(lf, symidx, 1, &addr); if (error != 0) return -1; addr += addend; - *where = addr; - break; + *where = addr; + break; case R_PPC_RELATIVE: /* doubleword64 B + A */ - *where = elf_relocaddr(lf, relocbase + addend); - break; + *where = elf_relocaddr(lf, relocbase + addend); + break; case R_PPC_JMP_SLOT: /* function descriptor copy */ lookup(lf, symidx, 1, &addr); @@ -176,8 +176,8 @@ elf_reloc_internal(linker_file_t lf, Elf break; default: - printf("kldload: unexpected relocation type %d\n", - (int) rtype); + printf("kldload: unexpected relocation type %d\n", + (int) rtype); return -1; } return(0); From owner-svn-src-stable@freebsd.org Sun Sep 27 04:17:09 2015 Return-Path: Delivered-To: svn-src-stable@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 716319B8A43; Sun, 27 Sep 2015 04:17:09 +0000 (UTC) (envelope-from alc@FreeBSD.org) Received: from repo.freebsd.org (repo.freebsd.org [IPv6:2001:1900:2254:2068::e6a:0]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 61FD28B6; Sun, 27 Sep 2015 04:17:09 +0000 (UTC) (envelope-from alc@FreeBSD.org) Received: from repo.freebsd.org ([127.0.1.70]) by repo.freebsd.org (8.15.2/8.15.2) with ESMTP id t8R4H9mt066679; Sun, 27 Sep 2015 04:17:09 GMT (envelope-from alc@FreeBSD.org) Received: (from alc@localhost) by repo.freebsd.org (8.15.2/8.15.2/Submit) id t8R4H9jX066678; Sun, 27 Sep 2015 04:17:09 GMT (envelope-from alc@FreeBSD.org) Message-Id: <201509270417.t8R4H9jX066678@repo.freebsd.org> X-Authentication-Warning: repo.freebsd.org: alc set sender to alc@FreeBSD.org using -f From: Alan Cox Date: Sun, 27 Sep 2015 04:17:09 +0000 (UTC) To: src-committers@freebsd.org, svn-src-all@freebsd.org, svn-src-stable@freebsd.org, svn-src-stable-10@freebsd.org Subject: svn commit: r288292 - stable/10/lib/libc/sys X-SVN-Group: stable-10 MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit X-BeenThere: svn-src-stable@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: SVN commit messages for all the -stable branches of the src tree List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 27 Sep 2015 04:17:09 -0000 Author: alc Date: Sun Sep 27 04:17:08 2015 New Revision: 288292 URL: https://svnweb.freebsd.org/changeset/base/288292 Log: MFC r285428 Correct the description of MADV_DONTNEED. Modified: stable/10/lib/libc/sys/madvise.2 Directory Properties: stable/10/ (props changed) Modified: stable/10/lib/libc/sys/madvise.2 ============================================================================== --- stable/10/lib/libc/sys/madvise.2 Sun Sep 27 04:03:11 2015 (r288291) +++ stable/10/lib/libc/sys/madvise.2 Sun Sep 27 04:17:08 2015 (r288292) @@ -28,7 +28,7 @@ .\" @(#)madvise.2 8.1 (Berkeley) 6/9/93 .\" $FreeBSD$ .\" -.Dd January 30, 2014 +.Dd July 12, 2015 .Dt MADVISE 2 .Os .Sh NAME @@ -79,9 +79,9 @@ pages in from backing store, but quickly into the calling process. .It Dv MADV_DONTNEED Allows the VM system to decrease the in-memory priority -of pages in the specified range. -Additionally future references to -this address range will incur a page fault. +of pages in the specified address range. +Consequently, future references to this address range are more likely +to incur a page fault. .It Dv MADV_FREE Gives the VM system the freedom to free pages, and tells the system that information in the specified page range From owner-svn-src-stable@freebsd.org Sun Sep 27 04:25:08 2015 Return-Path: Delivered-To: svn-src-stable@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 30646A0A01A; Sun, 27 Sep 2015 04:25:08 +0000 (UTC) (envelope-from alc@FreeBSD.org) Received: from repo.freebsd.org (repo.freebsd.org [IPv6:2001:1900:2254:2068::e6a:0]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 14466C95; Sun, 27 Sep 2015 04:25:08 +0000 (UTC) (envelope-from alc@FreeBSD.org) Received: from repo.freebsd.org ([127.0.1.70]) by repo.freebsd.org (8.15.2/8.15.2) with ESMTP id t8R4P7cr070734; Sun, 27 Sep 2015 04:25:07 GMT (envelope-from alc@FreeBSD.org) Received: (from alc@localhost) by repo.freebsd.org (8.15.2/8.15.2/Submit) id t8R4P7tY070733; Sun, 27 Sep 2015 04:25:07 GMT (envelope-from alc@FreeBSD.org) Message-Id: <201509270425.t8R4P7tY070733@repo.freebsd.org> X-Authentication-Warning: repo.freebsd.org: alc set sender to alc@FreeBSD.org using -f From: Alan Cox Date: Sun, 27 Sep 2015 04:25:07 +0000 (UTC) To: src-committers@freebsd.org, svn-src-all@freebsd.org, svn-src-stable@freebsd.org, svn-src-stable-10@freebsd.org Subject: svn commit: r288293 - stable/10/share/man/man9 X-SVN-Group: stable-10 MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit X-BeenThere: svn-src-stable@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: SVN commit messages for all the -stable branches of the src tree List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 27 Sep 2015 04:25:08 -0000 Author: alc Date: Sun Sep 27 04:25:07 2015 New Revision: 288293 URL: https://svnweb.freebsd.org/changeset/base/288293 Log: MFC r286513, r286784 Revise the text about the atomicity of the defined operations across multiple processors. In particular, clearly state that the operations are always atomic when they are applied to the default memory type that is used by the kernel (and applications). Stop describing an acquire operation as a read barrier and a release operation as a write barrier. That description has never been correct, and it has caused confusion. Also, explicitly say that a thread doesn't see its own accesses being reordered. The reordering of a thread's accesses is only (potentially) visible to another thread. Modified: stable/10/share/man/man9/atomic.9 Directory Properties: stable/10/ (props changed) Modified: stable/10/share/man/man9/atomic.9 ============================================================================== --- stable/10/share/man/man9/atomic.9 Sun Sep 27 04:17:08 2015 (r288292) +++ stable/10/share/man/man9/atomic.9 Sun Sep 27 04:25:07 2015 (r288293) @@ -23,7 +23,7 @@ .\" .\" $FreeBSD$ .\" -.Dd June 20, 2015 +.Dd August 14, 2015 .Dt ATOMIC 9 .Os .Sh NAME @@ -67,8 +67,8 @@ .Ft int .Fn atomic_testandset_ "volatile *p" "u_int v" .Sh DESCRIPTION -Each of the atomic operations is guaranteed to be atomic in the presence of -interrupts. +Each of the atomic operations is guaranteed to be atomic across multiple +threads and in the presence of interrupts. They can be used to implement reference counts or as building blocks for more advanced synchronization primitives such as mutexes. .Ss Types @@ -108,71 +108,94 @@ unsigned 16-bit integer .El .Pp These must not be used in MI code because the instructions to implement them -efficiently may not be available. -.Ss Memory Barriers -Memory barriers are used to guarantee the order of data accesses in -two ways. -First, they specify hints to the compiler to not re-order or optimize the -operations. -Second, on architectures that do not guarantee ordered data accesses, -special instructions or special variants of instructions are used to indicate -to the processor that data accesses need to occur in a certain order. -As a result, most of the atomic operations have three variants in order to -include optional memory barriers. -The first form just performs the operation without any explicit barriers. -The second form uses a read memory barrier, and the third variant uses a write -memory barrier. -.Pp -The second variant of each operation includes an +efficiently might not be available. +.Ss Acquire and Release Operations +By default, a thread's accesses to different memory locations might not be +performed in +.Em program order , +that is, the order in which the accesses appear in the source code. +To optimize the program's execution, both the compiler and processor might +reorder the thread's accesses. +However, both ensure that their reordering of the accesses is not visible to +the thread. +Otherwise, the traditional memory model that is expected by single-threaded +programs would be violated. +Nonetheless, other threads in a multithreaded program, such as the +.Fx +kernel, might observe the reordering. +Moreover, in some cases, such as the implementation of synchronization between +threads, arbitrary reordering might result in the incorrect execution of the +program. +To constrain the reordering that both the compiler and processor might perform +on a thread's accesses, the thread should use atomic operations with .Em acquire -memory barrier. -This barrier ensures that the effects of this operation are completed before the -effects of any later data accesses. -As a result, the operation is said to have acquire semantics as it acquires a -pseudo-lock requiring further operations to wait until it has completed. -To denote this, the suffix +and +.Em release +semantics. +.Pp +Most of the atomic operations on memory have three variants. +The first variant performs the operation without imposing any ordering +constraints on memory accesses to other locations. +The second variant has acquire semantics, and the third variant has release +semantics. +In effect, operations with acquire and release semantics establish one-way +barriers to reordering. +.Pp +When an atomic operation has acquire semantics, the effects of the operation +must have completed before any subsequent load or store (by program order) is +performed. +Conversely, acquire semantics do not require that prior loads or stores have +completed before the atomic operation is performed. +To denote acquire semantics, the suffix .Dq Li _acq is inserted into the function name immediately prior to the .Dq Li _ Ns Aq Fa type suffix. -For example, to subtract two integers ensuring that any later writes will -happen after the subtraction is performed, use +For example, to subtract two integers ensuring that subsequent loads and +stores happen after the subtraction is performed, use .Fn atomic_subtract_acq_int . .Pp -The third variant of each operation includes a -.Em release -memory barrier. -This ensures that all effects of all previous data accesses are completed -before this operation takes place. -As a result, the operation is said to have release semantics as it releases -any pending data accesses to be completed before its operation is performed. -To denote this, the suffix +When an atomic operation has release semantics, the effects of all prior +loads or stores (by program order) must have completed before the operation +is performed. +Conversely, release semantics do not require that the effects of the +atomic operation must have completed before any subsequent load or store is +performed. +To denote release semantics, the suffix .Dq Li _rel is inserted into the function name immediately prior to the .Dq Li _ Ns Aq Fa type suffix. -For example, to add two long integers ensuring that all previous -writes will happen first, use +For example, to add two long integers ensuring that all prior loads and +stores happen before the addition, use .Fn atomic_add_rel_long . .Pp -A practical example of using memory barriers is to ensure that data accesses -that are protected by a lock are all performed while the lock is held. -To achieve this, one would use a read barrier when acquiring the lock to -guarantee that the lock is held before any protected operations are performed. -Finally, one would use a write barrier when releasing the lock to ensure that -all of the protected operations are completed before the lock is released. +The one-way barriers provided by acquire and release operations allow the +implementations of common synchronization primitives to express their +ordering requirements without also imposing unnecessary ordering. +For example, for a critical section guarded by a mutex, an acquire operation +when the mutex is locked and a release operation when the mutex is unlocked +will prevent any loads or stores from moving outside of the critical +section. +However, they will not prevent the compiler or processor from moving loads +or stores into the critical section, which does not violate the semantics of +a mutex. .Ss Multiple Processors -The current set of atomic operations do not necessarily guarantee atomicity -across multiple processors. -To guarantee atomicity across processors, not only does the individual -operation need to be atomic on the processor performing the operation, but -the result of the operation needs to be pushed out to stable storage and the -caches of all other processors on the system need to invalidate any cache -lines that include the affected memory region. -On the +In multiprocessor systems, the atomicity of the atomic operations on memory +depends on support for cache coherence in the underlying architecture. +In general, cache coherence on the default memory type, +.Dv VM_MEMATTR_DEFAULT , +is guaranteed by all architectures that are supported by +.Fx . +For example, cache coherence is guaranteed on write-back memory by the +.Tn amd64 +and .Tn i386 -architecture, the cache coherency model requires that the hardware perform -this task, thus the atomic operations are atomic across multiple processors. +architectures. +However, on some architectures, cache coherence might not be enabled on all +memory types. +To determine if cache coherence is enabled for a non-default memory type, +consult the architecture's documentation. On the .Tn ia64 architecture, coherency is only guaranteed for pages that are configured to From owner-svn-src-stable@freebsd.org Sun Sep 27 04:36:10 2015 Return-Path: Delivered-To: svn-src-stable@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id F09C2A0A5C4; Sun, 27 Sep 2015 04:36:10 +0000 (UTC) (envelope-from alc@FreeBSD.org) Received: from repo.freebsd.org (repo.freebsd.org [IPv6:2001:1900:2254:2068::e6a:0]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id E1056235; Sun, 27 Sep 2015 04:36:10 +0000 (UTC) (envelope-from alc@FreeBSD.org) Received: from repo.freebsd.org ([127.0.1.70]) by repo.freebsd.org (8.15.2/8.15.2) with ESMTP id t8R4aAM5074957; Sun, 27 Sep 2015 04:36:10 GMT (envelope-from alc@FreeBSD.org) Received: (from alc@localhost) by repo.freebsd.org (8.15.2/8.15.2/Submit) id t8R4aAG5074955; Sun, 27 Sep 2015 04:36:10 GMT (envelope-from alc@FreeBSD.org) Message-Id: <201509270436.t8R4aAG5074955@repo.freebsd.org> X-Authentication-Warning: repo.freebsd.org: alc set sender to alc@FreeBSD.org using -f From: Alan Cox Date: Sun, 27 Sep 2015 04:36:10 +0000 (UTC) To: src-committers@freebsd.org, svn-src-all@freebsd.org, svn-src-stable@freebsd.org, svn-src-stable-10@freebsd.org Subject: svn commit: r288294 - stable/10/sys/vm X-SVN-Group: stable-10 MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit X-BeenThere: svn-src-stable@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: SVN commit messages for all the -stable branches of the src tree List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 27 Sep 2015 04:36:11 -0000 Author: alc Date: Sun Sep 27 04:36:09 2015 New Revision: 288294 URL: https://svnweb.freebsd.org/changeset/base/288294 Log: MFC r285282 The intention of r254304 was to scan the active queue continuously. However, I've observed the active queue scan stopping when there are frequent free page shortages and the inactive queue is steadily refilled by other mechanisms, such as the sequential access heuristic in vm_fault() or madvise(2). To remedy this problem, record the time of the last active queue scan, and always scan a number of pages proportional to the time since the last scan, regardless of whether that last scan was a timeout-triggered ("pass == 0") or free-page-shortage-triggered ("pass > 0") scan. Also, on a timeout-triggered scan, allow a full scan of the active queue when the system is short of inactive pages. Modified: stable/10/sys/vm/vm_page.h stable/10/sys/vm/vm_pageout.c Directory Properties: stable/10/ (props changed) Modified: stable/10/sys/vm/vm_page.h ============================================================================== --- stable/10/sys/vm/vm_page.h Sun Sep 27 04:25:07 2015 (r288293) +++ stable/10/sys/vm/vm_page.h Sun Sep 27 04:36:09 2015 (r288294) @@ -227,6 +227,7 @@ struct vm_domain { long vmd_segs; /* bitmask of the segments */ boolean_t vmd_oom; int vmd_pass; /* local pagedaemon pass */ + int vmd_last_active_scan; struct vm_page vmd_marker; /* marker for pagedaemon private use */ }; Modified: stable/10/sys/vm/vm_pageout.c ============================================================================== --- stable/10/sys/vm/vm_pageout.c Sun Sep 27 04:25:07 2015 (r288293) +++ stable/10/sys/vm/vm_pageout.c Sun Sep 27 04:36:09 2015 (r288294) @@ -926,9 +926,10 @@ vm_pageout_scan(struct vm_domain *vmd, i vm_page_t m, next; struct vm_pagequeue *pq; vm_object_t object; + long min_scan; int act_delta, addl_page_shortage, deficit, maxscan, page_shortage; int vnodes_skipped = 0; - int maxlaunder; + int maxlaunder, scan_tick, scanned; int lockmode; boolean_t queues_locked; @@ -1359,34 +1360,37 @@ relock_queues: * If we're just idle polling attempt to visit every * active page within 'update_period' seconds. */ - if (pass == 0 && vm_pageout_update_period != 0) { - maxscan /= vm_pageout_update_period; - page_shortage = maxscan; - } + scan_tick = ticks; + if (vm_pageout_update_period != 0) { + min_scan = pq->pq_cnt; + min_scan *= scan_tick - vmd->vmd_last_active_scan; + min_scan /= hz * vm_pageout_update_period; + } else + min_scan = 0; + if (min_scan > 0 || (page_shortage > 0 && maxscan > 0)) + vmd->vmd_last_active_scan = scan_tick; /* - * Scan the active queue for things we can deactivate. We nominally - * track the per-page activity counter and use it to locate - * deactivation candidates. - */ - m = TAILQ_FIRST(&pq->pq_pl); - while (m != NULL && maxscan-- > 0 && page_shortage > 0) { + * Scan the active queue for pages that can be deactivated. Update + * the per-page activity counter and use it to identify deactivation + * candidates. + */ + for (m = TAILQ_FIRST(&pq->pq_pl), scanned = 0; m != NULL && (scanned < + min_scan || (page_shortage > 0 && scanned < maxscan)); m = next, + scanned++) { KASSERT(m->queue == PQ_ACTIVE, ("vm_pageout_scan: page %p isn't active", m)); next = TAILQ_NEXT(m, plinks.q); - if ((m->flags & PG_MARKER) != 0) { - m = next; + if ((m->flags & PG_MARKER) != 0) continue; - } KASSERT((m->flags & PG_FICTITIOUS) == 0, ("Fictitious page %p cannot be in active queue", m)); KASSERT((m->oflags & VPO_UNMANAGED) == 0, ("Unmanaged page %p cannot be in active queue", m)); if (!vm_pageout_page_lock(m, &next)) { vm_page_unlock(m); - m = next; continue; } @@ -1439,7 +1443,6 @@ relock_queues: } else vm_page_requeue_locked(m); vm_page_unlock(m); - m = next; } vm_pagequeue_unlock(pq); #if !defined(NO_SWAPPING) @@ -1627,6 +1630,7 @@ vm_pageout_worker(void *arg) */ KASSERT(domain->vmd_segs != 0, ("domain without segments")); + domain->vmd_last_active_scan = ticks; vm_pageout_init_marker(&domain->vmd_marker, PQ_INACTIVE); /* From owner-svn-src-stable@freebsd.org Sun Sep 27 04:47:09 2015 Return-Path: Delivered-To: svn-src-stable@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id CDA3EA0ABAE; Sun, 27 Sep 2015 04:47:09 +0000 (UTC) (envelope-from alc@FreeBSD.org) Received: from repo.freebsd.org (repo.freebsd.org [IPv6:2001:1900:2254:2068::e6a:0]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id A3C42A52; Sun, 27 Sep 2015 04:47:09 +0000 (UTC) (envelope-from alc@FreeBSD.org) Received: from repo.freebsd.org ([127.0.1.70]) by repo.freebsd.org (8.15.2/8.15.2) with ESMTP id t8R4l9nm079133; Sun, 27 Sep 2015 04:47:09 GMT (envelope-from alc@FreeBSD.org) Received: (from alc@localhost) by repo.freebsd.org (8.15.2/8.15.2/Submit) id t8R4l9lg079132; Sun, 27 Sep 2015 04:47:09 GMT (envelope-from alc@FreeBSD.org) Message-Id: <201509270447.t8R4l9lg079132@repo.freebsd.org> X-Authentication-Warning: repo.freebsd.org: alc set sender to alc@FreeBSD.org using -f From: Alan Cox Date: Sun, 27 Sep 2015 04:47:09 +0000 (UTC) To: src-committers@freebsd.org, svn-src-all@freebsd.org, svn-src-stable@freebsd.org, svn-src-stable-10@freebsd.org Subject: svn commit: r288296 - stable/10/sys/vm X-SVN-Group: stable-10 MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit X-BeenThere: svn-src-stable@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: SVN commit messages for all the -stable branches of the src tree List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 27 Sep 2015 04:47:09 -0000 Author: alc Date: Sun Sep 27 04:47:08 2015 New Revision: 288296 URL: https://svnweb.freebsd.org/changeset/base/288296 Log: MFC r288025 Correct a non-fatal error in vm_pageout_worker(). vm_pageout_worker() should not assume that vm_pages_needed will remain set while it sleeps. Other threads can clear vm_pages_needed by performing a sufficient number of vm_page_free() calls, e.g., process termination. The effect of this error was that vm_pageout_worker() would free and/or launder pages when, in fact, there was no shortage of free pages. Rewrite a nearby comment to describe all of the possible cases and not just the most common case. The problem being that the comment made the most common case seem like the only case. Modified: stable/10/sys/vm/vm_pageout.c Directory Properties: stable/10/ (props changed) Modified: stable/10/sys/vm/vm_pageout.c ============================================================================== --- stable/10/sys/vm/vm_pageout.c Sun Sep 27 04:40:54 2015 (r288295) +++ stable/10/sys/vm/vm_pageout.c Sun Sep 27 04:47:08 2015 (r288296) @@ -1651,9 +1651,15 @@ vm_pageout_worker(void *arg) } if (vm_pages_needed) { /* - * Still not done, take a second pass without waiting - * (unlimited dirty cleaning), otherwise sleep a bit - * and try again. + * We're still not done. Either vm_pages_needed was + * set by another thread during the previous scan + * (typically, this happens during a level 0 scan) or + * vm_pages_needed was already set and the scan failed + * to free enough pages. If we haven't yet performed + * a level >= 2 scan (unlimited dirty cleaning), then + * upgrade the level and scan again now. Otherwise, + * sleep a bit and try again later. While sleeping, + * vm_pages_needed can be cleared. */ if (domain->vmd_pass > 1) msleep(&vm_pages_needed, @@ -1664,15 +1670,14 @@ vm_pageout_worker(void *arg) * Good enough, sleep until required to refresh * stats. */ - domain->vmd_pass = 0; msleep(&vm_pages_needed, &vm_page_queue_free_mtx, PVM, "psleep", hz); - } if (vm_pages_needed) { cnt.v_pdwakeups++; domain->vmd_pass++; - } + } else + domain->vmd_pass = 0; mtx_unlock(&vm_page_queue_free_mtx); vm_pageout_scan(domain, domain->vmd_pass); } From owner-svn-src-stable@freebsd.org Sun Sep 27 05:26:23 2015 Return-Path: Delivered-To: svn-src-stable@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 7E923A0A876; Sun, 27 Sep 2015 05:26:23 +0000 (UTC) (envelope-from alc@FreeBSD.org) Received: from repo.freebsd.org (repo.freebsd.org [IPv6:2001:1900:2254:2068::e6a:0]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 6EEC0FF0; Sun, 27 Sep 2015 05:26:23 +0000 (UTC) (envelope-from alc@FreeBSD.org) Received: from repo.freebsd.org ([127.0.1.70]) by repo.freebsd.org (8.15.2/8.15.2) with ESMTP id t8R5QNcC095378; Sun, 27 Sep 2015 05:26:23 GMT (envelope-from alc@FreeBSD.org) Received: (from alc@localhost) by repo.freebsd.org (8.15.2/8.15.2/Submit) id t8R5QNAu095377; Sun, 27 Sep 2015 05:26:23 GMT (envelope-from alc@FreeBSD.org) Message-Id: <201509270526.t8R5QNAu095377@repo.freebsd.org> X-Authentication-Warning: repo.freebsd.org: alc set sender to alc@FreeBSD.org using -f From: Alan Cox Date: Sun, 27 Sep 2015 05:26:23 +0000 (UTC) To: src-committers@freebsd.org, svn-src-all@freebsd.org, svn-src-stable@freebsd.org, svn-src-stable-10@freebsd.org Subject: svn commit: r288300 - stable/10/sys/vm X-SVN-Group: stable-10 MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit X-BeenThere: svn-src-stable@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: SVN commit messages for all the -stable branches of the src tree List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 27 Sep 2015 05:26:23 -0000 Author: alc Date: Sun Sep 27 05:26:22 2015 New Revision: 288300 URL: https://svnweb.freebsd.org/changeset/base/288300 Log: MFC r283795 Document vm_page_alloc_contig()'s support for the VM_ALLOC_NODUMP option. Modified: stable/10/sys/vm/vm_page.c Directory Properties: stable/10/ (props changed) Modified: stable/10/sys/vm/vm_page.c ============================================================================== --- stable/10/sys/vm/vm_page.c Sun Sep 27 05:16:06 2015 (r288299) +++ stable/10/sys/vm/vm_page.c Sun Sep 27 05:26:22 2015 (r288300) @@ -1710,6 +1710,7 @@ vm_page_alloc_contig_vdrop(struct spglis * * optional allocation flags: * VM_ALLOC_NOBUSY do not exclusive busy the page + * VM_ALLOC_NODUMP do not include the page in a kernel core dump * VM_ALLOC_NOOBJ page is not associated with an object and * should not be exclusive busy * VM_ALLOC_SBUSY shared busy the allocated page From owner-svn-src-stable@freebsd.org Sun Sep 27 05:45:17 2015 Return-Path: Delivered-To: svn-src-stable@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 3E7FA9D0366; Sun, 27 Sep 2015 05:45:17 +0000 (UTC) (envelope-from alc@FreeBSD.org) Received: from repo.freebsd.org (repo.freebsd.org [IPv6:2001:1900:2254:2068::e6a:0]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 2EEEEBA2; Sun, 27 Sep 2015 05:45:17 +0000 (UTC) (envelope-from alc@FreeBSD.org) Received: from repo.freebsd.org ([127.0.1.70]) by repo.freebsd.org (8.15.2/8.15.2) with ESMTP id t8R5jH8k003633; Sun, 27 Sep 2015 05:45:17 GMT (envelope-from alc@FreeBSD.org) Received: (from alc@localhost) by repo.freebsd.org (8.15.2/8.15.2/Submit) id t8R5jHpv003632; Sun, 27 Sep 2015 05:45:17 GMT (envelope-from alc@FreeBSD.org) Message-Id: <201509270545.t8R5jHpv003632@repo.freebsd.org> X-Authentication-Warning: repo.freebsd.org: alc set sender to alc@FreeBSD.org using -f From: Alan Cox Date: Sun, 27 Sep 2015 05:45:17 +0000 (UTC) To: src-committers@freebsd.org, svn-src-all@freebsd.org, svn-src-stable@freebsd.org, svn-src-stable-10@freebsd.org Subject: svn commit: r288302 - stable/10/sys/vm X-SVN-Group: stable-10 MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit X-BeenThere: svn-src-stable@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: SVN commit messages for all the -stable branches of the src tree List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 27 Sep 2015 05:45:17 -0000 Author: alc Date: Sun Sep 27 05:45:16 2015 New Revision: 288302 URL: https://svnweb.freebsd.org/changeset/base/288302 Log: MFC r266588 There is no reason to perform the pmap_remove() on the kernel pmap while the kmem object lock is held. Do the pmap_remove() before acquiring the kmem object lock. Modified: stable/10/sys/vm/vm_kern.c Directory Properties: stable/10/ (props changed) Modified: stable/10/sys/vm/vm_kern.c ============================================================================== --- stable/10/sys/vm/vm_kern.c Sun Sep 27 05:29:34 2015 (r288301) +++ stable/10/sys/vm/vm_kern.c Sun Sep 27 05:45:16 2015 (r288302) @@ -396,9 +396,9 @@ kmem_unback(vm_object_t object, vm_offse KASSERT(object == kmem_object || object == kernel_object, ("kmem_unback: only supports kernel objects.")); + pmap_remove(kernel_pmap, addr, addr + size); offset = addr - VM_MIN_KERNEL_ADDRESS; VM_OBJECT_WLOCK(object); - pmap_remove(kernel_pmap, addr, addr + size); for (i = 0; i < size; i += PAGE_SIZE) { m = vm_page_lookup(object, OFF_TO_IDX(offset + i)); vm_page_unwire(m, 0); From owner-svn-src-stable@freebsd.org Mon Sep 28 12:30:24 2015 Return-Path: Delivered-To: svn-src-stable@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id E223DA0BC7C; Mon, 28 Sep 2015 12:30:24 +0000 (UTC) (envelope-from mav@FreeBSD.org) Received: from repo.freebsd.org (repo.freebsd.org [IPv6:2001:1900:2254:2068::e6a:0]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id C66481760; Mon, 28 Sep 2015 12:30:24 +0000 (UTC) (envelope-from mav@FreeBSD.org) Received: from repo.freebsd.org ([127.0.1.70]) by repo.freebsd.org (8.15.2/8.15.2) with ESMTP id t8SCUOaL068187; Mon, 28 Sep 2015 12:30:24 GMT (envelope-from mav@FreeBSD.org) Received: (from mav@localhost) by repo.freebsd.org (8.15.2/8.15.2/Submit) id t8SCUMNA068177; Mon, 28 Sep 2015 12:30:22 GMT (envelope-from mav@FreeBSD.org) Message-Id: <201509281230.t8SCUMNA068177@repo.freebsd.org> X-Authentication-Warning: repo.freebsd.org: mav set sender to mav@FreeBSD.org using -f From: Alexander Motin Date: Mon, 28 Sep 2015 12:30:22 +0000 (UTC) To: src-committers@freebsd.org, svn-src-all@freebsd.org, svn-src-stable@freebsd.org, svn-src-stable-10@freebsd.org Subject: svn commit: r288338 - in stable/10/sys/cam: ata scsi X-SVN-Group: stable-10 MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit X-BeenThere: svn-src-stable@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: SVN commit messages for all the -stable branches of the src tree List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 28 Sep 2015 12:30:25 -0000 Author: mav Date: Mon Sep 28 12:30:22 2015 New Revision: 288338 URL: https://svnweb.freebsd.org/changeset/base/288338 Log: MFC r287289: Attach pass driver to LUNs is OFFLINE state. Previously such LUNs were silently ignored. But while they indeed unable to process most of SCSI commands, some, like RTPG, they still can. Modified: stable/10/sys/cam/ata/ata_xpt.c stable/10/sys/cam/scsi/scsi_cd.c stable/10/sys/cam/scsi/scsi_ch.c stable/10/sys/cam/scsi/scsi_da.c stable/10/sys/cam/scsi/scsi_pt.c stable/10/sys/cam/scsi/scsi_sa.c stable/10/sys/cam/scsi/scsi_xpt.c Directory Properties: stable/10/ (props changed) Modified: stable/10/sys/cam/ata/ata_xpt.c ============================================================================== --- stable/10/sys/cam/ata/ata_xpt.c Mon Sep 28 12:23:10 2015 (r288337) +++ stable/10/sys/cam/ata/ata_xpt.c Mon Sep 28 12:30:22 2015 (r288338) @@ -1090,7 +1090,8 @@ notsata: periph_qual = SID_QUAL(inq_buf); - if (periph_qual != SID_QUAL_LU_CONNECTED) + if (periph_qual != SID_QUAL_LU_CONNECTED && + periph_qual != SID_QUAL_LU_OFFLINE) break; /* Modified: stable/10/sys/cam/scsi/scsi_cd.c ============================================================================== --- stable/10/sys/cam/scsi/scsi_cd.c Mon Sep 28 12:23:10 2015 (r288337) +++ stable/10/sys/cam/scsi/scsi_cd.c Mon Sep 28 12:30:22 2015 (r288338) @@ -392,7 +392,8 @@ cdasync(void *callback_arg, u_int32_t co if (cgd->protocol != PROTO_SCSI) break; - + if (SID_QUAL(&cgd->inq_data) != SID_QUAL_LU_CONNECTED) + break; if (SID_TYPE(&cgd->inq_data) != T_CDROM && SID_TYPE(&cgd->inq_data) != T_WORM) break; Modified: stable/10/sys/cam/scsi/scsi_ch.c ============================================================================== --- stable/10/sys/cam/scsi/scsi_ch.c Mon Sep 28 12:23:10 2015 (r288337) +++ stable/10/sys/cam/scsi/scsi_ch.c Mon Sep 28 12:30:22 2015 (r288338) @@ -337,7 +337,8 @@ chasync(void *callback_arg, u_int32_t co if (cgd->protocol != PROTO_SCSI) break; - + if (SID_QUAL(&cgd->inq_data) != SID_QUAL_LU_CONNECTED) + break; if (SID_TYPE(&cgd->inq_data)!= T_CHANGER) break; Modified: stable/10/sys/cam/scsi/scsi_da.c ============================================================================== --- stable/10/sys/cam/scsi/scsi_da.c Mon Sep 28 12:23:10 2015 (r288337) +++ stable/10/sys/cam/scsi/scsi_da.c Mon Sep 28 12:30:22 2015 (r288338) @@ -1667,7 +1667,8 @@ daasync(void *callback_arg, u_int32_t co if (cgd->protocol != PROTO_SCSI) break; - + if (SID_QUAL(&cgd->inq_data) != SID_QUAL_LU_CONNECTED) + break; if (SID_TYPE(&cgd->inq_data) != T_DIRECT && SID_TYPE(&cgd->inq_data) != T_RBC && SID_TYPE(&cgd->inq_data) != T_OPTICAL) Modified: stable/10/sys/cam/scsi/scsi_pt.c ============================================================================== --- stable/10/sys/cam/scsi/scsi_pt.c Mon Sep 28 12:23:10 2015 (r288337) +++ stable/10/sys/cam/scsi/scsi_pt.c Mon Sep 28 12:30:22 2015 (r288338) @@ -366,7 +366,8 @@ ptasync(void *callback_arg, u_int32_t co if (cgd->protocol != PROTO_SCSI) break; - + if (SID_QUAL(&cgd->inq_data) != SID_QUAL_LU_CONNECTED) + break; if (SID_TYPE(&cgd->inq_data) != T_PROCESSOR) break; Modified: stable/10/sys/cam/scsi/scsi_sa.c ============================================================================== --- stable/10/sys/cam/scsi/scsi_sa.c Mon Sep 28 12:23:10 2015 (r288337) +++ stable/10/sys/cam/scsi/scsi_sa.c Mon Sep 28 12:30:22 2015 (r288338) @@ -2254,7 +2254,8 @@ saasync(void *callback_arg, u_int32_t co if (cgd->protocol != PROTO_SCSI) break; - + if (SID_QUAL(&cgd->inq_data) != SID_QUAL_LU_CONNECTED) + break; if (SID_TYPE(&cgd->inq_data) != T_SEQUENTIAL) break; Modified: stable/10/sys/cam/scsi/scsi_xpt.c ============================================================================== --- stable/10/sys/cam/scsi/scsi_xpt.c Mon Sep 28 12:23:10 2015 (r288337) +++ stable/10/sys/cam/scsi/scsi_xpt.c Mon Sep 28 12:30:22 2015 (r288338) @@ -1126,6 +1126,7 @@ probedone(struct cam_periph *periph, uni { probe_softc *softc; struct cam_path *path; + struct scsi_inquiry_data *inq_buf; u_int32_t priority; CAM_DEBUG(done_ccb->ccb_h.path, CAM_DEBUG_TRACE, ("probedone\n")); @@ -1165,7 +1166,6 @@ out: case PROBE_FULL_INQUIRY: { if (cam_ccb_status(done_ccb) == CAM_REQ_CMP) { - struct scsi_inquiry_data *inq_buf; u_int8_t periph_qual; path->device->flags |= CAM_DEV_INQUIRY_DATA_VALID; @@ -1174,7 +1174,8 @@ out: periph_qual = SID_QUAL(inq_buf); - if (periph_qual == SID_QUAL_LU_CONNECTED) { + if (periph_qual == SID_QUAL_LU_CONNECTED || + periph_qual == SID_QUAL_LU_OFFLINE) { u_int8_t len; /* @@ -1350,10 +1351,10 @@ out: probe_purge_old(path, lp, softc->flags); lp = NULL; } + inq_buf = &path->device->inq_data; if (path->device->flags & CAM_DEV_INQUIRY_DATA_VALID && - SID_QUAL(&path->device->inq_data) == SID_QUAL_LU_CONNECTED) { - struct scsi_inquiry_data *inq_buf; - inq_buf = &path->device->inq_data; + (SID_QUAL(inq_buf) == SID_QUAL_LU_CONNECTED || + SID_QUAL(inq_buf) == SID_QUAL_LU_OFFLINE)) { if (INQ_DATA_TQ_ENABLED(inq_buf)) PROBE_SET_ACTION(softc, PROBE_MODE_SENSE); else From owner-svn-src-stable@freebsd.org Mon Sep 28 18:58:29 2015 Return-Path: Delivered-To: svn-src-stable@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id E7136A0B57C; Mon, 28 Sep 2015 18:58:28 +0000 (UTC) (envelope-from delphij@FreeBSD.org) Received: from repo.freebsd.org (repo.freebsd.org [IPv6:2001:1900:2254:2068::e6a:0]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id D6BD91354; Mon, 28 Sep 2015 18:58:28 +0000 (UTC) (envelope-from delphij@FreeBSD.org) Received: from repo.freebsd.org ([127.0.1.70]) by repo.freebsd.org (8.15.2/8.15.2) with ESMTP id t8SIwShi030146; Mon, 28 Sep 2015 18:58:28 GMT (envelope-from delphij@FreeBSD.org) Received: (from delphij@localhost) by repo.freebsd.org (8.15.2/8.15.2/Submit) id t8SIwSfD030144; Mon, 28 Sep 2015 18:58:28 GMT (envelope-from delphij@FreeBSD.org) Message-Id: <201509281858.t8SIwSfD030144@repo.freebsd.org> X-Authentication-Warning: repo.freebsd.org: delphij set sender to delphij@FreeBSD.org using -f From: Xin LI Date: Mon, 28 Sep 2015 18:58:28 +0000 (UTC) To: src-committers@freebsd.org, svn-src-all@freebsd.org, svn-src-stable@freebsd.org, svn-src-stable-10@freebsd.org Subject: svn commit: r288346 - stable/10/cddl/contrib/opensolaris/cmd/zfs X-SVN-Group: stable-10 MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit X-BeenThere: svn-src-stable@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: SVN commit messages for all the -stable branches of the src tree List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 28 Sep 2015 18:58:29 -0000 Author: delphij Date: Mon Sep 28 18:58:27 2015 New Revision: 288346 URL: https://svnweb.freebsd.org/changeset/base/288346 Log: MFC r287770: MFV r277429: Document -S option when zfs inherit fails on quota and in manual pages. Illumos ZFS issues: 5410 Document -S option to zfs inherit https://illumos.org/issues/5410 5412 Mention -S option when zfs inherit fails on quota https://illumos.org/issues/5412 illumos/illumos-gate@5ff8cfa92ec8ea0f8593ad21aa2a04829b0ef5ea Modified: stable/10/cddl/contrib/opensolaris/cmd/zfs/zfs.8 stable/10/cddl/contrib/opensolaris/cmd/zfs/zfs_main.c Directory Properties: stable/10/ (props changed) Modified: stable/10/cddl/contrib/opensolaris/cmd/zfs/zfs.8 ============================================================================== --- stable/10/cddl/contrib/opensolaris/cmd/zfs/zfs.8 Mon Sep 28 18:39:21 2015 (r288345) +++ stable/10/cddl/contrib/opensolaris/cmd/zfs/zfs.8 Mon Sep 28 18:58:27 2015 (r288346) @@ -31,7 +31,7 @@ .\" .\" $FreeBSD$ .\" -.Dd July 30, 2015 +.Dd September 14, 2015 .Dt ZFS 8 .Os .Sh NAME @@ -2126,7 +2126,8 @@ Property name .It value Property value .It source -Property source. Can either be local, default, temporary, inherited, or none +Property source. Can either be local, default, temporary, inherited, received, +or none (\&-). .El .Pp @@ -2192,8 +2193,11 @@ The default value is all sources. .Ar filesystem Ns | Ns Ar volume Ns | Ns Ar snapshot Ns ... .Xc .Pp -Clears the specified property, causing it to be inherited from an ancestor. If -no ancestor has the property set, then the default value is used. See the +Clears the specified property, causing it to be inherited from an ancestor, +restored to default if no ancestor has the property set, or with the +.Fl S +option reverted to the received value if one exists. +See the .Qq Sx Properties section for a listing of default values, and details on which properties can be inherited. @@ -2201,8 +2205,10 @@ inherited. .It Fl r Recursively inherit the given property for all children. .It Fl S -For properties with a received value, revert to this value. This flag has no -effect on properties that do not have a received value. +Revert the property to the received value if one exists; otherwise operate as +if the +.Fl S +option was not specified. .El .It Xo .Nm Modified: stable/10/cddl/contrib/opensolaris/cmd/zfs/zfs_main.c ============================================================================== --- stable/10/cddl/contrib/opensolaris/cmd/zfs/zfs_main.c Mon Sep 28 18:39:21 2015 (r288345) +++ stable/10/cddl/contrib/opensolaris/cmd/zfs/zfs_main.c Mon Sep 28 18:58:27 2015 (r288346) @@ -1927,9 +1927,13 @@ zfs_do_inherit(int argc, char **argv) if (prop == ZFS_PROP_QUOTA || prop == ZFS_PROP_RESERVATION || prop == ZFS_PROP_REFQUOTA || - prop == ZFS_PROP_REFRESERVATION) + prop == ZFS_PROP_REFRESERVATION) { (void) fprintf(stderr, gettext("use 'zfs set " "%s=none' to clear\n"), propname); + (void) fprintf(stderr, gettext("use 'zfs " + "inherit -S %s' to revert to received " + "value\n"), propname); + } return (1); } if (received && (prop == ZFS_PROP_VOLSIZE || From owner-svn-src-stable@freebsd.org Tue Sep 29 05:23:28 2015 Return-Path: Delivered-To: svn-src-stable@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 152C5A0BB2B; Tue, 29 Sep 2015 05:23:28 +0000 (UTC) (envelope-from mav@FreeBSD.org) Received: from repo.freebsd.org (repo.freebsd.org [IPv6:2001:1900:2254:2068::e6a:0]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id D4DBE1286; Tue, 29 Sep 2015 05:23:27 +0000 (UTC) (envelope-from mav@FreeBSD.org) Received: from repo.freebsd.org ([127.0.1.70]) by repo.freebsd.org (8.15.2/8.15.2) with ESMTP id t8T5NR72090550; Tue, 29 Sep 2015 05:23:27 GMT (envelope-from mav@FreeBSD.org) Received: (from mav@localhost) by repo.freebsd.org (8.15.2/8.15.2/Submit) id t8T5NRSq090546; Tue, 29 Sep 2015 05:23:27 GMT (envelope-from mav@FreeBSD.org) Message-Id: <201509290523.t8T5NRSq090546@repo.freebsd.org> X-Authentication-Warning: repo.freebsd.org: mav set sender to mav@FreeBSD.org using -f From: Alexander Motin Date: Tue, 29 Sep 2015 05:23:27 +0000 (UTC) To: src-committers@freebsd.org, svn-src-all@freebsd.org, svn-src-stable@freebsd.org, svn-src-stable-10@freebsd.org Subject: svn commit: r288354 - stable/10/sys/cam/scsi X-SVN-Group: stable-10 MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit X-BeenThere: svn-src-stable@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: SVN commit messages for all the -stable branches of the src tree List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 29 Sep 2015 05:23:28 -0000 Author: mav Date: Tue Sep 29 05:23:26 2015 New Revision: 288354 URL: https://svnweb.freebsd.org/changeset/base/288354 Log: MFC r287819: Make CAM log errors that make it wait. Waiting can take minutes, and it would be good for user to know what is going on. Modified: stable/10/sys/cam/scsi/scsi_all.c stable/10/sys/cam/scsi/scsi_all.h Directory Properties: stable/10/ (props changed) Modified: stable/10/sys/cam/scsi/scsi_all.c ============================================================================== --- stable/10/sys/cam/scsi/scsi_all.c Tue Sep 29 05:03:24 2015 (r288353) +++ stable/10/sys/cam/scsi/scsi_all.c Tue Sep 29 05:23:26 2015 (r288354) @@ -1079,7 +1079,7 @@ static struct asc_table_entry asc_table[ { SST(0x04, 0x00, SS_RDEF, "Logical unit not ready, cause not reportable") }, /* DTLPWROMAEBKVF */ - { SST(0x04, 0x01, SS_TUR | SSQ_MANY | SSQ_DECREMENT_COUNT | EBUSY, + { SST(0x04, 0x01, SS_WAIT | EBUSY, "Logical unit is in process of becoming ready") }, /* DTLPWROMAEBKVF */ { SST(0x04, 0x02, SS_START | SSQ_DECREMENT_COUNT | ENXIO, @@ -1106,7 +1106,7 @@ static struct asc_table_entry asc_table[ { SST(0x04, 0x09, SS_RDEF, /* XXX TBD */ "Logical unit not ready, self-test in progress") }, /* DTLPWROMAEBKVF */ - { SST(0x04, 0x0A, SS_TUR | SSQ_MANY | SSQ_DECREMENT_COUNT | ENXIO, + { SST(0x04, 0x0A, SS_WAIT | ENXIO, "Logical unit not accessible, asymmetric access state transition")}, /* DTLPWROMAEBKVF */ { SST(0x04, 0x0B, SS_FATAL | ENXIO, @@ -1121,7 +1121,7 @@ static struct asc_table_entry asc_table[ { SST(0x04, 0x10, SS_RDEF, /* XXX TBD */ "Logical unit not ready, auxiliary memory not accessible") }, /* DT WRO AEB VF */ - { SST(0x04, 0x11, SS_TUR | SSQ_MANY | SSQ_DECREMENT_COUNT | EBUSY, + { SST(0x04, 0x11, SS_WAIT | EBUSY, "Logical unit not ready, notify (enable spinup) required") }, /* M V */ { SST(0x04, 0x12, SS_RDEF, /* XXX TBD */ Modified: stable/10/sys/cam/scsi/scsi_all.h ============================================================================== --- stable/10/sys/cam/scsi/scsi_all.h Tue Sep 29 05:03:24 2015 (r288353) +++ stable/10/sys/cam/scsi/scsi_all.h Tue Sep 29 05:23:26 2015 (r288354) @@ -103,6 +103,9 @@ typedef enum { /* The retyable, error action, with table specified error code */ #define SS_RET SS_RETRY|SSQ_DECREMENT_COUNT|SSQ_PRINT_SENSE +/* Wait for transient error status to change */ +#define SS_WAIT SS_TUR|SSQ_MANY|SSQ_DECREMENT_COUNT|SSQ_PRINT_SENSE + /* Fatal error action, with table specified error code */ #define SS_FATAL SS_FAIL|SSQ_PRINT_SENSE From owner-svn-src-stable@freebsd.org Tue Sep 29 05:24:17 2015 Return-Path: Delivered-To: svn-src-stable@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 9C403A0BBFC; Tue, 29 Sep 2015 05:24:17 +0000 (UTC) (envelope-from mav@FreeBSD.org) Received: from repo.freebsd.org (repo.freebsd.org [IPv6:2001:1900:2254:2068::e6a:0]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 72D0513D0; Tue, 29 Sep 2015 05:24:17 +0000 (UTC) (envelope-from mav@FreeBSD.org) Received: from repo.freebsd.org ([127.0.1.70]) by repo.freebsd.org (8.15.2/8.15.2) with ESMTP id t8T5OH0k090651; Tue, 29 Sep 2015 05:24:17 GMT (envelope-from mav@FreeBSD.org) Received: (from mav@localhost) by repo.freebsd.org (8.15.2/8.15.2/Submit) id t8T5OHiH090649; Tue, 29 Sep 2015 05:24:17 GMT (envelope-from mav@FreeBSD.org) Message-Id: <201509290524.t8T5OHiH090649@repo.freebsd.org> X-Authentication-Warning: repo.freebsd.org: mav set sender to mav@FreeBSD.org using -f From: Alexander Motin Date: Tue, 29 Sep 2015 05:24:17 +0000 (UTC) To: src-committers@freebsd.org, svn-src-all@freebsd.org, svn-src-stable@freebsd.org, svn-src-stable-9@freebsd.org Subject: svn commit: r288355 - stable/9/sys/cam/scsi X-SVN-Group: stable-9 MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit X-BeenThere: svn-src-stable@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: SVN commit messages for all the -stable branches of the src tree List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 29 Sep 2015 05:24:17 -0000 Author: mav Date: Tue Sep 29 05:24:16 2015 New Revision: 288355 URL: https://svnweb.freebsd.org/changeset/base/288355 Log: MFC r287819: Make CAM log errors that make it wait. Waiting can take minutes, and it would be good for user to know what is going on. Modified: stable/9/sys/cam/scsi/scsi_all.c stable/9/sys/cam/scsi/scsi_all.h Directory Properties: stable/9/ (props changed) stable/9/sys/ (props changed) Modified: stable/9/sys/cam/scsi/scsi_all.c ============================================================================== --- stable/9/sys/cam/scsi/scsi_all.c Tue Sep 29 05:23:26 2015 (r288354) +++ stable/9/sys/cam/scsi/scsi_all.c Tue Sep 29 05:24:16 2015 (r288355) @@ -1076,7 +1076,7 @@ static struct asc_table_entry asc_table[ { SST(0x04, 0x00, SS_RDEF, "Logical unit not ready, cause not reportable") }, /* DTLPWROMAEBKVF */ - { SST(0x04, 0x01, SS_TUR | SSQ_MANY | SSQ_DECREMENT_COUNT | EBUSY, + { SST(0x04, 0x01, SS_WAIT | EBUSY, "Logical unit is in process of becoming ready") }, /* DTLPWROMAEBKVF */ { SST(0x04, 0x02, SS_START | SSQ_DECREMENT_COUNT | ENXIO, @@ -1103,7 +1103,7 @@ static struct asc_table_entry asc_table[ { SST(0x04, 0x09, SS_RDEF, /* XXX TBD */ "Logical unit not ready, self-test in progress") }, /* DTLPWROMAEBKVF */ - { SST(0x04, 0x0A, SS_TUR | SSQ_MANY | SSQ_DECREMENT_COUNT | ENXIO, + { SST(0x04, 0x0A, SS_WAIT | ENXIO, "Logical unit not accessible, asymmetric access state transition")}, /* DTLPWROMAEBKVF */ { SST(0x04, 0x0B, SS_FATAL | ENXIO, @@ -1118,7 +1118,7 @@ static struct asc_table_entry asc_table[ { SST(0x04, 0x10, SS_RDEF, /* XXX TBD */ "Logical unit not ready, auxiliary memory not accessible") }, /* DT WRO AEB VF */ - { SST(0x04, 0x11, SS_TUR | SSQ_MANY | SSQ_DECREMENT_COUNT | EBUSY, + { SST(0x04, 0x11, SS_WAIT | EBUSY, "Logical unit not ready, notify (enable spinup) required") }, /* M V */ { SST(0x04, 0x12, SS_RDEF, /* XXX TBD */ Modified: stable/9/sys/cam/scsi/scsi_all.h ============================================================================== --- stable/9/sys/cam/scsi/scsi_all.h Tue Sep 29 05:23:26 2015 (r288354) +++ stable/9/sys/cam/scsi/scsi_all.h Tue Sep 29 05:24:16 2015 (r288355) @@ -103,6 +103,9 @@ typedef enum { /* The retyable, error action, with table specified error code */ #define SS_RET SS_RETRY|SSQ_DECREMENT_COUNT|SSQ_PRINT_SENSE +/* Wait for transient error status to change */ +#define SS_WAIT SS_TUR|SSQ_MANY|SSQ_DECREMENT_COUNT|SSQ_PRINT_SENSE + /* Fatal error action, with table specified error code */ #define SS_FATAL SS_FAIL|SSQ_PRINT_SENSE From owner-svn-src-stable@freebsd.org Tue Sep 29 05:25:35 2015 Return-Path: Delivered-To: svn-src-stable@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id B5435A0BD0C; Tue, 29 Sep 2015 05:25:35 +0000 (UTC) (envelope-from mav@FreeBSD.org) Received: from repo.freebsd.org (repo.freebsd.org [IPv6:2001:1900:2254:2068::e6a:0]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id A5BA215C3; Tue, 29 Sep 2015 05:25:35 +0000 (UTC) (envelope-from mav@FreeBSD.org) Received: from repo.freebsd.org ([127.0.1.70]) by repo.freebsd.org (8.15.2/8.15.2) with ESMTP id t8T5PZ4h090771; Tue, 29 Sep 2015 05:25:35 GMT (envelope-from mav@FreeBSD.org) Received: (from mav@localhost) by repo.freebsd.org (8.15.2/8.15.2/Submit) id t8T5PZjt090770; Tue, 29 Sep 2015 05:25:35 GMT (envelope-from mav@FreeBSD.org) Message-Id: <201509290525.t8T5PZjt090770@repo.freebsd.org> X-Authentication-Warning: repo.freebsd.org: mav set sender to mav@FreeBSD.org using -f From: Alexander Motin Date: Tue, 29 Sep 2015 05:25:35 +0000 (UTC) To: src-committers@freebsd.org, svn-src-all@freebsd.org, svn-src-stable@freebsd.org, svn-src-stable-10@freebsd.org Subject: svn commit: r288356 - stable/10/sys/dev/ahci X-SVN-Group: stable-10 MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit X-BeenThere: svn-src-stable@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: SVN commit messages for all the -stable branches of the src tree List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 29 Sep 2015 05:25:35 -0000 Author: mav Date: Tue Sep 29 05:25:34 2015 New Revision: 288356 URL: https://svnweb.freebsd.org/changeset/base/288356 Log: MFC r288111: Allow AHCI driver attach to all known chips reporting RAID class. Reported by: Michael BlackHeart Modified: stable/10/sys/dev/ahci/ahci_pci.c Directory Properties: stable/10/ (props changed) Modified: stable/10/sys/dev/ahci/ahci_pci.c ============================================================================== --- stable/10/sys/dev/ahci/ahci_pci.c Tue Sep 29 05:24:16 2015 (r288355) +++ stable/10/sys/dev/ahci/ahci_pci.c Tue Sep 29 05:25:34 2015 (r288356) @@ -326,6 +326,9 @@ ahci_probe(device_t dev) pci_get_subclass(dev) == PCIS_STORAGE_SATA && pci_get_progif(dev) == PCIP_STORAGE_SATA_AHCI_1_0) valid = 1; + else if (pci_get_class(dev) == PCIC_STORAGE && + pci_get_subclass(dev) == PCIS_STORAGE_RAID) + valid = 2; /* Is this a known AHCI chip? */ for (i = 0; ahci_ids[i].id != 0; i++) { if (ahci_ids[i].id == devid && @@ -342,7 +345,7 @@ ahci_probe(device_t dev) return (BUS_PROBE_DEFAULT); } } - if (!valid) + if (valid != 1) return (ENXIO); device_set_desc_copy(dev, "AHCI SATA controller"); return (BUS_PROBE_DEFAULT); From owner-svn-src-stable@freebsd.org Tue Sep 29 16:54:23 2015 Return-Path: Delivered-To: svn-src-stable@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 0A730A0B0AD; Tue, 29 Sep 2015 16:54:23 +0000 (UTC) (envelope-from bdrewery@FreeBSD.org) Received: from repo.freebsd.org (repo.freebsd.org [IPv6:2001:1900:2254:2068::e6a:0]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id EF1221EDC; Tue, 29 Sep 2015 16:54:22 +0000 (UTC) (envelope-from bdrewery@FreeBSD.org) Received: from repo.freebsd.org ([127.0.1.70]) by repo.freebsd.org (8.15.2/8.15.2) with ESMTP id t8TGsMVs076785; Tue, 29 Sep 2015 16:54:22 GMT (envelope-from bdrewery@FreeBSD.org) Received: (from bdrewery@localhost) by repo.freebsd.org (8.15.2/8.15.2/Submit) id t8TGsMCI076784; Tue, 29 Sep 2015 16:54:22 GMT (envelope-from bdrewery@FreeBSD.org) Message-Id: <201509291654.t8TGsMCI076784@repo.freebsd.org> X-Authentication-Warning: repo.freebsd.org: bdrewery set sender to bdrewery@FreeBSD.org using -f From: Bryan Drewery Date: Tue, 29 Sep 2015 16:54:22 +0000 (UTC) To: src-committers@freebsd.org, svn-src-all@freebsd.org, svn-src-stable@freebsd.org, svn-src-stable-10@freebsd.org Subject: svn commit: r288375 - stable/10/share/mk X-SVN-Group: stable-10 MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit X-BeenThere: svn-src-stable@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: SVN commit messages for all the -stable branches of the src tree List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 29 Sep 2015 16:54:23 -0000 Author: bdrewery Date: Tue Sep 29 16:54:22 2015 New Revision: 288375 URL: https://svnweb.freebsd.org/changeset/base/288375 Log: MFC r287978: Fix LIBRARIES_ONLY Modified: stable/10/share/mk/bsd.lib.mk Directory Properties: stable/10/ (props changed) Modified: stable/10/share/mk/bsd.lib.mk ============================================================================== --- stable/10/share/mk/bsd.lib.mk Tue Sep 29 16:09:59 2015 (r288374) +++ stable/10/share/mk/bsd.lib.mk Tue Sep 29 16:54:22 2015 (r288375) @@ -258,7 +258,7 @@ ${LINTLIB}: ${LINTOBJS} all: ${_LIBS} -.if ${MK_MAN} != "no" +.if ${MK_MAN} != "no" && !defined(LIBRARIES_ONLY) all: _manpages .endif From owner-svn-src-stable@freebsd.org Tue Sep 29 16:56:29 2015 Return-Path: Delivered-To: svn-src-stable@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 03A9AA0B20A; Tue, 29 Sep 2015 16:56:29 +0000 (UTC) (envelope-from bdrewery@FreeBSD.org) Received: from repo.freebsd.org (repo.freebsd.org [IPv6:2001:1900:2254:2068::e6a:0]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id E811A110F; Tue, 29 Sep 2015 16:56:28 +0000 (UTC) (envelope-from bdrewery@FreeBSD.org) Received: from repo.freebsd.org ([127.0.1.70]) by repo.freebsd.org (8.15.2/8.15.2) with ESMTP id t8TGuS4b076946; Tue, 29 Sep 2015 16:56:28 GMT (envelope-from bdrewery@FreeBSD.org) Received: (from bdrewery@localhost) by repo.freebsd.org (8.15.2/8.15.2/Submit) id t8TGuSVJ076945; Tue, 29 Sep 2015 16:56:28 GMT (envelope-from bdrewery@FreeBSD.org) Message-Id: <201509291656.t8TGuSVJ076945@repo.freebsd.org> X-Authentication-Warning: repo.freebsd.org: bdrewery set sender to bdrewery@FreeBSD.org using -f From: Bryan Drewery Date: Tue, 29 Sep 2015 16:56:28 +0000 (UTC) To: src-committers@freebsd.org, svn-src-all@freebsd.org, svn-src-stable@freebsd.org, svn-src-stable-10@freebsd.org Subject: svn commit: r288376 - stable/10/lib/libc/sys X-SVN-Group: stable-10 MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit X-BeenThere: svn-src-stable@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: SVN commit messages for all the -stable branches of the src tree List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 29 Sep 2015 16:56:29 -0000 Author: bdrewery Date: Tue Sep 29 16:56:28 2015 New Revision: 288376 URL: https://svnweb.freebsd.org/changeset/base/288376 Log: MFC r288092: Avoid adding duplicates into OBJS. bsd.lib.mk already handles adding entries to OBJS based on SRCS. Modified: stable/10/lib/libc/sys/Makefile.inc Directory Properties: stable/10/ (props changed) Modified: stable/10/lib/libc/sys/Makefile.inc ============================================================================== --- stable/10/lib/libc/sys/Makefile.inc Tue Sep 29 16:54:22 2015 (r288375) +++ stable/10/lib/libc/sys/Makefile.inc Tue Sep 29 16:56:28 2015 (r288376) @@ -100,8 +100,6 @@ ASM+=$(_asm) .endif .endfor -OBJS+= ${ASM} ${PSEUDO} - SASM= ${ASM:S/.o/.S/} SPSEUDO= ${PSEUDO:S/.o/.S/} From owner-svn-src-stable@freebsd.org Tue Sep 29 17:02:28 2015 Return-Path: Delivered-To: svn-src-stable@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 3EF2CA0B7E6; Tue, 29 Sep 2015 17:02:28 +0000 (UTC) (envelope-from bdrewery@FreeBSD.org) Received: from repo.freebsd.org (repo.freebsd.org [IPv6:2001:1900:2254:2068::e6a:0]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 2EFFC182F; Tue, 29 Sep 2015 17:02:28 +0000 (UTC) (envelope-from bdrewery@FreeBSD.org) Received: from repo.freebsd.org ([127.0.1.70]) by repo.freebsd.org (8.15.2/8.15.2) with ESMTP id t8TH2SLM080951; Tue, 29 Sep 2015 17:02:28 GMT (envelope-from bdrewery@FreeBSD.org) Received: (from bdrewery@localhost) by repo.freebsd.org (8.15.2/8.15.2/Submit) id t8TH2S3d080950; Tue, 29 Sep 2015 17:02:28 GMT (envelope-from bdrewery@FreeBSD.org) Message-Id: <201509291702.t8TH2S3d080950@repo.freebsd.org> X-Authentication-Warning: repo.freebsd.org: bdrewery set sender to bdrewery@FreeBSD.org using -f From: Bryan Drewery Date: Tue, 29 Sep 2015 17:02:28 +0000 (UTC) To: src-committers@freebsd.org, svn-src-all@freebsd.org, svn-src-stable@freebsd.org, svn-src-stable-9@freebsd.org Subject: svn commit: r288377 - stable/9/lib/libc/sys X-SVN-Group: stable-9 MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit X-BeenThere: svn-src-stable@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: SVN commit messages for all the -stable branches of the src tree List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 29 Sep 2015 17:02:28 -0000 Author: bdrewery Date: Tue Sep 29 17:02:27 2015 New Revision: 288377 URL: https://svnweb.freebsd.org/changeset/base/288377 Log: MFC r288092: Avoid adding duplicates into OBJS. bsd.lib.mk already handles adding entries to OBJS based on SRCS. Modified: stable/9/lib/libc/sys/Makefile.inc Directory Properties: stable/9/lib/libc/ (props changed) stable/9/lib/libc/sys/ (props changed) Modified: stable/9/lib/libc/sys/Makefile.inc ============================================================================== --- stable/9/lib/libc/sys/Makefile.inc Tue Sep 29 16:56:28 2015 (r288376) +++ stable/9/lib/libc/sys/Makefile.inc Tue Sep 29 17:02:27 2015 (r288377) @@ -47,8 +47,6 @@ ASM+=$(_asm) .endif .endfor -OBJS+= ${ASM} ${PSEUDO} - SASM= ${ASM:S/.o/.S/} SPSEUDO= ${PSEUDO:S/.o/.S/} From owner-svn-src-stable@freebsd.org Tue Sep 29 17:05:13 2015 Return-Path: Delivered-To: svn-src-stable@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id D702AA0B9C8; Tue, 29 Sep 2015 17:05:13 +0000 (UTC) (envelope-from bdrewery@FreeBSD.org) Received: from repo.freebsd.org (repo.freebsd.org [IPv6:2001:1900:2254:2068::e6a:0]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id C78FF1B24; Tue, 29 Sep 2015 17:05:13 +0000 (UTC) (envelope-from bdrewery@FreeBSD.org) Received: from repo.freebsd.org ([127.0.1.70]) by repo.freebsd.org (8.15.2/8.15.2) with ESMTP id t8TH5DpW081158; Tue, 29 Sep 2015 17:05:13 GMT (envelope-from bdrewery@FreeBSD.org) Received: (from bdrewery@localhost) by repo.freebsd.org (8.15.2/8.15.2/Submit) id t8TH5DK5081157; Tue, 29 Sep 2015 17:05:13 GMT (envelope-from bdrewery@FreeBSD.org) Message-Id: <201509291705.t8TH5DK5081157@repo.freebsd.org> X-Authentication-Warning: repo.freebsd.org: bdrewery set sender to bdrewery@FreeBSD.org using -f From: Bryan Drewery Date: Tue, 29 Sep 2015 17:05:13 +0000 (UTC) To: src-committers@freebsd.org, svn-src-all@freebsd.org, svn-src-stable@freebsd.org, svn-src-stable-9@freebsd.org Subject: svn commit: r288379 - stable/9/share/mk X-SVN-Group: stable-9 MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit X-BeenThere: svn-src-stable@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: SVN commit messages for all the -stable branches of the src tree List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 29 Sep 2015 17:05:13 -0000 Author: bdrewery Date: Tue Sep 29 17:05:12 2015 New Revision: 288379 URL: https://svnweb.freebsd.org/changeset/base/288379 Log: MFC r287978: Fix LIBRARIES_ONLY Modified: stable/9/share/mk/bsd.lib.mk Directory Properties: stable/9/share/mk/ (props changed) Modified: stable/9/share/mk/bsd.lib.mk ============================================================================== --- stable/9/share/mk/bsd.lib.mk Tue Sep 29 17:04:20 2015 (r288378) +++ stable/9/share/mk/bsd.lib.mk Tue Sep 29 17:05:12 2015 (r288379) @@ -250,7 +250,7 @@ ${LINTLIB}: ${LINTOBJS} all: ${_LIBS} -.if ${MK_MAN} != "no" +.if ${MK_MAN} != "no" && !defined(LIBRARIES_ONLY) all: _manpages .endif From owner-svn-src-stable@freebsd.org Tue Sep 29 18:06:29 2015 Return-Path: Delivered-To: svn-src-stable@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id F0545A0B26E; Tue, 29 Sep 2015 18:06:28 +0000 (UTC) (envelope-from delphij@FreeBSD.org) Received: from repo.freebsd.org (repo.freebsd.org [IPv6:2001:1900:2254:2068::e6a:0]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id D4DF41EF4; Tue, 29 Sep 2015 18:06:28 +0000 (UTC) (envelope-from delphij@FreeBSD.org) Received: from repo.freebsd.org ([127.0.1.70]) by repo.freebsd.org (8.15.2/8.15.2) with ESMTP id t8TI6SXb006064; Tue, 29 Sep 2015 18:06:28 GMT (envelope-from delphij@FreeBSD.org) Received: (from delphij@localhost) by repo.freebsd.org (8.15.2/8.15.2/Submit) id t8TI6SKt006063; Tue, 29 Sep 2015 18:06:28 GMT (envelope-from delphij@FreeBSD.org) Message-Id: <201509291806.t8TI6SKt006063@repo.freebsd.org> X-Authentication-Warning: repo.freebsd.org: delphij set sender to delphij@FreeBSD.org using -f From: Xin LI Date: Tue, 29 Sep 2015 18:06:28 +0000 (UTC) To: src-committers@freebsd.org, svn-src-all@freebsd.org, svn-src-stable@freebsd.org, svn-src-stable-10@freebsd.org Subject: svn commit: r288384 - in stable: 10/usr.sbin/rpcbind 9/usr.sbin/rpcbind X-SVN-Group: stable-10 MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit X-BeenThere: svn-src-stable@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: SVN commit messages for all the -stable branches of the src tree List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 29 Sep 2015 18:06:29 -0000 Author: delphij Date: Tue Sep 29 18:06:27 2015 New Revision: 288384 URL: https://svnweb.freebsd.org/changeset/base/288384 Log: The Sun RPC framework uses a netbuf structure to represent the transport specific form of a universal transport address. The structure is expected to be opaque to consumers. In the current implementation, the structure contains a pointer to a buffer that holds the actual address. In rpcbind(8), netbuf structures are copied directly, which would result in two netbuf structures that reference to one shared address buffer. When one of the two netbuf structures is freed, access to the other netbuf structure would result in an undefined result that may crash the rpcbind(8) daemon. Fix this by making a copy of the buffer that is going to be freed instead of doing a shallow copy. Security: FreeBSD-SA-15:24.rpcbind Security: CVE-2015-7236 Modified: stable/10/usr.sbin/rpcbind/rpcb_svc_com.c Changes in other areas also in this revision: Modified: stable/9/usr.sbin/rpcbind/rpcb_svc_com.c Modified: stable/10/usr.sbin/rpcbind/rpcb_svc_com.c ============================================================================== --- stable/10/usr.sbin/rpcbind/rpcb_svc_com.c Tue Sep 29 18:05:54 2015 (r288383) +++ stable/10/usr.sbin/rpcbind/rpcb_svc_com.c Tue Sep 29 18:06:27 2015 (r288384) @@ -48,6 +48,7 @@ #include #include #include +#include #include #include #include @@ -1048,19 +1049,31 @@ netbufcmp(struct netbuf *n1, struct netb return ((n1->len != n2->len) || memcmp(n1->buf, n2->buf, n1->len)); } +static bool_t +netbuf_copybuf(struct netbuf *dst, const struct netbuf *src) +{ + + assert(dst->buf == NULL); + + if ((dst->buf = malloc(src->len)) == NULL) + return (FALSE); + + dst->maxlen = dst->len = src->len; + memcpy(dst->buf, src->buf, src->len); + return (TRUE); +} + static struct netbuf * netbufdup(struct netbuf *ap) { struct netbuf *np; - if ((np = malloc(sizeof(struct netbuf))) == NULL) + if ((np = calloc(1, sizeof(struct netbuf))) == NULL) return (NULL); - if ((np->buf = malloc(ap->len)) == NULL) { + if (netbuf_copybuf(np, ap) == FALSE) { free(np); return (NULL); } - np->maxlen = np->len = ap->len; - memcpy(np->buf, ap->buf, ap->len); return (np); } @@ -1068,6 +1081,7 @@ static void netbuffree(struct netbuf *ap) { free(ap->buf); + ap->buf = NULL; free(ap); } @@ -1185,7 +1199,7 @@ xprt_set_caller(SVCXPRT *xprt, struct fi { u_int32_t *xidp; - *(svc_getrpccaller(xprt)) = *(fi->caller_addr); + netbuf_copybuf(svc_getrpccaller(xprt), fi->caller_addr); xidp = __rpcb_get_dg_xidp(xprt); *xidp = fi->caller_xid; } From owner-svn-src-stable@freebsd.org Tue Sep 29 18:06:28 2015 Return-Path: Delivered-To: svn-src-stable@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 45536A0B26A; Tue, 29 Sep 2015 18:06:28 +0000 (UTC) (envelope-from delphij@FreeBSD.org) Received: from repo.freebsd.org (repo.freebsd.org [IPv6:2001:1900:2254:2068::e6a:0]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 27D2F1EF3; Tue, 29 Sep 2015 18:06:28 +0000 (UTC) (envelope-from delphij@FreeBSD.org) Received: from repo.freebsd.org ([127.0.1.70]) by repo.freebsd.org (8.15.2/8.15.2) with ESMTP id t8TI6SMu006056; Tue, 29 Sep 2015 18:06:28 GMT (envelope-from delphij@FreeBSD.org) Received: (from delphij@localhost) by repo.freebsd.org (8.15.2/8.15.2/Submit) id t8TI6SZR006055; Tue, 29 Sep 2015 18:06:28 GMT (envelope-from delphij@FreeBSD.org) Message-Id: <201509291806.t8TI6SZR006055@repo.freebsd.org> X-Authentication-Warning: repo.freebsd.org: delphij set sender to delphij@FreeBSD.org using -f From: Xin LI Date: Tue, 29 Sep 2015 18:06:28 +0000 (UTC) To: src-committers@freebsd.org, svn-src-all@freebsd.org, svn-src-stable@freebsd.org, svn-src-stable-9@freebsd.org Subject: svn commit: r288384 - in stable: 10/usr.sbin/rpcbind 9/usr.sbin/rpcbind X-SVN-Group: stable-9 MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit X-BeenThere: svn-src-stable@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: SVN commit messages for all the -stable branches of the src tree List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 29 Sep 2015 18:06:28 -0000 Author: delphij Date: Tue Sep 29 18:06:27 2015 New Revision: 288384 URL: https://svnweb.freebsd.org/changeset/base/288384 Log: The Sun RPC framework uses a netbuf structure to represent the transport specific form of a universal transport address. The structure is expected to be opaque to consumers. In the current implementation, the structure contains a pointer to a buffer that holds the actual address. In rpcbind(8), netbuf structures are copied directly, which would result in two netbuf structures that reference to one shared address buffer. When one of the two netbuf structures is freed, access to the other netbuf structure would result in an undefined result that may crash the rpcbind(8) daemon. Fix this by making a copy of the buffer that is going to be freed instead of doing a shallow copy. Security: FreeBSD-SA-15:24.rpcbind Security: CVE-2015-7236 Modified: stable/9/usr.sbin/rpcbind/rpcb_svc_com.c Changes in other areas also in this revision: Modified: stable/10/usr.sbin/rpcbind/rpcb_svc_com.c Modified: stable/9/usr.sbin/rpcbind/rpcb_svc_com.c ============================================================================== --- stable/9/usr.sbin/rpcbind/rpcb_svc_com.c Tue Sep 29 18:05:54 2015 (r288383) +++ stable/9/usr.sbin/rpcbind/rpcb_svc_com.c Tue Sep 29 18:06:27 2015 (r288384) @@ -48,6 +48,7 @@ #include #include #include +#include #include #include #include @@ -1048,19 +1049,31 @@ netbufcmp(struct netbuf *n1, struct netb return ((n1->len != n2->len) || memcmp(n1->buf, n2->buf, n1->len)); } +static bool_t +netbuf_copybuf(struct netbuf *dst, const struct netbuf *src) +{ + + assert(dst->buf == NULL); + + if ((dst->buf = malloc(src->len)) == NULL) + return (FALSE); + + dst->maxlen = dst->len = src->len; + memcpy(dst->buf, src->buf, src->len); + return (TRUE); +} + static struct netbuf * netbufdup(struct netbuf *ap) { struct netbuf *np; - if ((np = malloc(sizeof(struct netbuf))) == NULL) + if ((np = calloc(1, sizeof(struct netbuf))) == NULL) return (NULL); - if ((np->buf = malloc(ap->len)) == NULL) { + if (netbuf_copybuf(np, ap) == FALSE) { free(np); return (NULL); } - np->maxlen = np->len = ap->len; - memcpy(np->buf, ap->buf, ap->len); return (np); } @@ -1068,6 +1081,7 @@ static void netbuffree(struct netbuf *ap) { free(ap->buf); + ap->buf = NULL; free(ap); } @@ -1185,7 +1199,7 @@ xprt_set_caller(SVCXPRT *xprt, struct fi { u_int32_t *xidp; - *(svc_getrpccaller(xprt)) = *(fi->caller_addr); + netbuf_copybuf(svc_getrpccaller(xprt), fi->caller_addr); xidp = __rpcb_get_dg_xidp(xprt); *xidp = fi->caller_xid; } From owner-svn-src-stable@freebsd.org Tue Sep 29 18:33:18 2015 Return-Path: Delivered-To: svn-src-stable@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 527DBA0C5EB; Tue, 29 Sep 2015 18:33:18 +0000 (UTC) (envelope-from bdrewery@FreeBSD.org) Received: from repo.freebsd.org (repo.freebsd.org [IPv6:2001:1900:2254:2068::e6a:0]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 2F3B21302; Tue, 29 Sep 2015 18:33:18 +0000 (UTC) (envelope-from bdrewery@FreeBSD.org) Received: from repo.freebsd.org ([127.0.1.70]) by repo.freebsd.org (8.15.2/8.15.2) with ESMTP id t8TIXIEh019002; Tue, 29 Sep 2015 18:33:18 GMT (envelope-from bdrewery@FreeBSD.org) Received: (from bdrewery@localhost) by repo.freebsd.org (8.15.2/8.15.2/Submit) id t8TIXHX3019000; Tue, 29 Sep 2015 18:33:17 GMT (envelope-from bdrewery@FreeBSD.org) Message-Id: <201509291833.t8TIXHX3019000@repo.freebsd.org> X-Authentication-Warning: repo.freebsd.org: bdrewery set sender to bdrewery@FreeBSD.org using -f From: Bryan Drewery Date: Tue, 29 Sep 2015 18:33:17 +0000 (UTC) To: src-committers@freebsd.org, svn-src-all@freebsd.org, svn-src-stable@freebsd.org, svn-src-stable-9@freebsd.org Subject: svn commit: r288386 - stable/9/sys/dev/filemon X-SVN-Group: stable-9 MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit X-BeenThere: svn-src-stable@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: SVN commit messages for all the -stable branches of the src tree List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 29 Sep 2015 18:33:18 -0000 Author: bdrewery Date: Tue Sep 29 18:33:17 2015 New Revision: 288386 URL: https://svnweb.freebsd.org/changeset/base/288386 Log: MFC r284383,r284477,r284600,r284601: Latest clang uses openat(2). Bump the version since we now handle openat filemon_pid_check needs to hold proctree_lock sx_sunlock for sx_slock Modified: stable/9/sys/dev/filemon/filemon.h stable/9/sys/dev/filemon/filemon_wrapper.c Directory Properties: stable/9/sys/ (props changed) stable/9/sys/dev/ (props changed) Modified: stable/9/sys/dev/filemon/filemon.h ============================================================================== --- stable/9/sys/dev/filemon/filemon.h Tue Sep 29 18:07:18 2015 (r288385) +++ stable/9/sys/dev/filemon/filemon.h Tue Sep 29 18:33:17 2015 (r288386) @@ -30,5 +30,5 @@ #define FILEMON_SET_FD _IOWR('S', 1, int) #define FILEMON_SET_PID _IOWR('S', 2, pid_t) -#define FILEMON_VERSION 4 /* output format +#define FILEMON_VERSION 5 /* output format (bump when adding record types) */ Modified: stable/9/sys/dev/filemon/filemon_wrapper.c ============================================================================== --- stable/9/sys/dev/filemon/filemon_wrapper.c Tue Sep 29 18:07:18 2015 (r288385) +++ stable/9/sys/dev/filemon/filemon_wrapper.c Tue Sep 29 18:33:17 2015 (r288386) @@ -28,6 +28,8 @@ #include __FBSDID("$FreeBSD$"); +#include + #include "opt_compat.h" #if __FreeBSD_version > 800032 @@ -84,13 +86,17 @@ filemon_pid_check(struct proc *p) { struct filemon *filemon; - while (p->p_pptr) { + sx_slock(&proctree_lock); + while (p != initproc) { TAILQ_FOREACH(filemon, &filemons_inuse, link) { - if (p->p_pid == filemon->pid) + if (p->p_pid == filemon->pid) { + sx_sunlock(&proctree_lock); return (filemon); + } } - p = p->p_pptr; + p = proc_realparent(p); } + sx_sunlock(&proctree_lock); return (NULL); } @@ -316,6 +322,68 @@ filemon_wrapper_open(struct thread *td, } static int +filemon_wrapper_openat(struct thread *td, struct openat_args *uap) +{ + int ret; + size_t done; + size_t len; + struct filemon *filemon; + + if ((ret = sys_openat(td, uap)) == 0) { + /* Grab a read lock on the filemon inuse list. */ + filemon_lock_read(); + + if ((filemon = filemon_pid_check(curproc)) != NULL) { + /* Lock the found filemon structure. */ + filemon_filemon_lock(filemon); + + copyinstr(uap->path, filemon->fname1, + sizeof(filemon->fname1), &done); + + filemon->fname2[0] = '\0'; + if (filemon->fname1[0] != '/' && uap->fd != AT_FDCWD) { + /* + * rats - we cannot do too much about this. + * the trace should show a dir we read + * recently.. output an A record as a clue + * until we can do better. + */ + len = snprintf(filemon->msgbufr, + sizeof(filemon->msgbufr), "A %d %s\n", + curproc->p_pid, filemon->fname1); + filemon_output(filemon, filemon->msgbufr, len); + } + if (uap->flag & O_RDWR) { + /* + * We'll get the W record below, but need + * to also output an R to distingish from + * O_WRONLY. + */ + len = snprintf(filemon->msgbufr, + sizeof(filemon->msgbufr), "R %d %s%s\n", + curproc->p_pid, filemon->fname2, filemon->fname1); + filemon_output(filemon, filemon->msgbufr, len); + } + + + len = snprintf(filemon->msgbufr, + sizeof(filemon->msgbufr), "%c %d %s%s\n", + (uap->flag & O_ACCMODE) ? 'W':'R', + curproc->p_pid, filemon->fname2, filemon->fname1); + filemon_output(filemon, filemon->msgbufr, len); + + /* Unlock the found filemon structure. */ + filemon_filemon_unlock(filemon); + } + + /* Release the read lock. */ + filemon_unlock_read(); + } + + return (ret); +} + +static int filemon_wrapper_rename(struct thread *td, struct rename_args *uap) { int ret; @@ -671,6 +739,7 @@ filemon_wrapper_install(void) sv_table[SYS_execve].sy_call = (sy_call_t *) filemon_wrapper_execve; sv_table[SYS_fork].sy_call = (sy_call_t *) filemon_wrapper_fork; sv_table[SYS_open].sy_call = (sy_call_t *) filemon_wrapper_open; + sv_table[SYS_openat].sy_call = (sy_call_t *) filemon_wrapper_openat; sv_table[SYS_rename].sy_call = (sy_call_t *) filemon_wrapper_rename; sv_table[SYS_stat].sy_call = (sy_call_t *) filemon_wrapper_stat; sv_table[SYS_unlink].sy_call = (sy_call_t *) filemon_wrapper_unlink; @@ -689,6 +758,7 @@ filemon_wrapper_install(void) sv_table[FREEBSD32_SYS_freebsd32_execve].sy_call = (sy_call_t *) filemon_wrapper_freebsd32_execve; sv_table[FREEBSD32_SYS_fork].sy_call = (sy_call_t *) filemon_wrapper_fork; sv_table[FREEBSD32_SYS_open].sy_call = (sy_call_t *) filemon_wrapper_open; + sv_table[FREEBSD32_SYS_openat].sy_call = (sy_call_t *) filemon_wrapper_openat; sv_table[FREEBSD32_SYS_rename].sy_call = (sy_call_t *) filemon_wrapper_rename; sv_table[FREEBSD32_SYS_freebsd32_stat].sy_call = (sy_call_t *) filemon_wrapper_freebsd32_stat; sv_table[FREEBSD32_SYS_unlink].sy_call = (sy_call_t *) filemon_wrapper_unlink; @@ -717,6 +787,7 @@ filemon_wrapper_deinstall(void) sv_table[SYS_execve].sy_call = (sy_call_t *)sys_execve; sv_table[SYS_fork].sy_call = (sy_call_t *)sys_fork; sv_table[SYS_open].sy_call = (sy_call_t *)sys_open; + sv_table[SYS_openat].sy_call = (sy_call_t *)sys_openat; sv_table[SYS_rename].sy_call = (sy_call_t *)sys_rename; sv_table[SYS_stat].sy_call = (sy_call_t *)sys_stat; sv_table[SYS_unlink].sy_call = (sy_call_t *)sys_unlink; @@ -735,6 +806,7 @@ filemon_wrapper_deinstall(void) sv_table[FREEBSD32_SYS_freebsd32_execve].sy_call = (sy_call_t *)freebsd32_execve; sv_table[FREEBSD32_SYS_fork].sy_call = (sy_call_t *)sys_fork; sv_table[FREEBSD32_SYS_open].sy_call = (sy_call_t *)sys_open; + sv_table[FREEBSD32_SYS_openat].sy_call = (sy_call_t *)sys_openat; sv_table[FREEBSD32_SYS_rename].sy_call = (sy_call_t *)sys_rename; sv_table[FREEBSD32_SYS_freebsd32_stat].sy_call = (sy_call_t *)freebsd32_stat; sv_table[FREEBSD32_SYS_unlink].sy_call = (sy_call_t *)sys_unlink; From owner-svn-src-stable@freebsd.org Tue Sep 29 18:40:00 2015 Return-Path: Delivered-To: svn-src-stable@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 1E853A0C9AD; Tue, 29 Sep 2015 18:40:00 +0000 (UTC) (envelope-from jhb@FreeBSD.org) Received: from repo.freebsd.org (repo.freebsd.org [IPv6:2001:1900:2254:2068::e6a:0]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 0EF75189F; Tue, 29 Sep 2015 18:40:00 +0000 (UTC) (envelope-from jhb@FreeBSD.org) Received: from repo.freebsd.org ([127.0.1.70]) by repo.freebsd.org (8.15.2/8.15.2) with ESMTP id t8TIdxUg019336; Tue, 29 Sep 2015 18:39:59 GMT (envelope-from jhb@FreeBSD.org) Received: (from jhb@localhost) by repo.freebsd.org (8.15.2/8.15.2/Submit) id t8TIdxAh019334; Tue, 29 Sep 2015 18:39:59 GMT (envelope-from jhb@FreeBSD.org) Message-Id: <201509291839.t8TIdxAh019334@repo.freebsd.org> X-Authentication-Warning: repo.freebsd.org: jhb set sender to jhb@FreeBSD.org using -f From: John Baldwin Date: Tue, 29 Sep 2015 18:39:59 +0000 (UTC) To: src-committers@freebsd.org, svn-src-all@freebsd.org, svn-src-stable@freebsd.org, svn-src-stable-10@freebsd.org Subject: svn commit: r288387 - stable/10/usr.sbin/pmcstat X-SVN-Group: stable-10 MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit X-BeenThere: svn-src-stable@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: SVN commit messages for all the -stable branches of the src tree List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 29 Sep 2015 18:40:00 -0000 Author: jhb Date: Tue Sep 29 18:39:58 2015 New Revision: 288387 URL: https://svnweb.freebsd.org/changeset/base/288387 Log: MFC 283613,287374: Use the cpuset API more consistently: - Fetch the root set from cpuset_getaffinity() instead of assuming all CPUs from 0 to hw.ncpu are the root set. - Use CPU_SETSIZE and CPU_FFS. - The original notion of halted CPUs the manpage and code refers to is gone. Use the term "available" instead. Modified: stable/10/usr.sbin/pmcstat/pmcstat.8 stable/10/usr.sbin/pmcstat/pmcstat.c Directory Properties: stable/10/ (props changed) Modified: stable/10/usr.sbin/pmcstat/pmcstat.8 ============================================================================== --- stable/10/usr.sbin/pmcstat/pmcstat.8 Tue Sep 29 18:33:17 2015 (r288386) +++ stable/10/usr.sbin/pmcstat/pmcstat.8 Tue Sep 29 18:39:58 2015 (r288387) @@ -25,7 +25,7 @@ .\" .\" $FreeBSD$ .\" -.Dd May 8, 2015 +.Dd May 27, 2015 .Dt PMCSTAT 8 .Os .Sh NAME @@ -246,8 +246,8 @@ Argument .Ar cpu-spec is a comma separated list of CPU numbers, or the literal .Sq * -denoting all unhalted CPUs. -The default is to allocate system mode PMCs on all unhalted +denoting all available CPUs. +The default is to allocate system mode PMCs on all available CPUs. .It Fl d Toggle between process mode PMCs measuring events for the target Modified: stable/10/usr.sbin/pmcstat/pmcstat.c ============================================================================== --- stable/10/usr.sbin/pmcstat/pmcstat.c Tue Sep 29 18:33:17 2015 (r288386) +++ stable/10/usr.sbin/pmcstat/pmcstat.c Tue Sep 29 18:39:58 2015 (r288387) @@ -116,11 +116,10 @@ struct pmcstat_args args; static void pmcstat_clone_event_descriptor(struct pmcstat_ev *ev, const cpuset_t *cpumask) { - int cpu, mcpu; + int cpu; struct pmcstat_ev *ev_clone; - mcpu = sizeof(*cpumask) * NBBY; - for (cpu = 0; cpu < mcpu; cpu++) { + for (cpu = 0; cpu < CPU_SETSIZE; cpu++) { if (!CPU_ISSET(cpu, cpumask)) continue; @@ -161,6 +160,7 @@ pmcstat_get_cpumask(const char *cpuspec, CPU_SET(cpu, cpumask); s = end + strspn(end, ", \t"); } while (*s); + assert(!CPU_EMPTY(cpumask)); } void @@ -550,10 +550,10 @@ pmcstat_topexit(void) int main(int argc, char **argv) { - cpuset_t cpumask; + cpuset_t cpumask, rootmask; double interval; double duration; - int hcpu, option, npmc, ncpu; + int option, npmc; int c, check_driver_stats, current_sampling_count; int do_callchain, do_descendants, do_logproccsw, do_logprocexit; int do_print, do_read; @@ -618,14 +618,13 @@ main(int argc, char **argv) err(EX_OSERR, "ERROR: Cannot determine path of running kernel"); /* - * The initial CPU mask specifies all non-halted CPUS in the - * system. + * The initial CPU mask specifies the root mask of this process + * which is usually all CPUs in the system. */ - len = sizeof(int); - if (sysctlbyname("hw.ncpu", &ncpu, &len, NULL, 0) < 0) - err(EX_OSERR, "ERROR: Cannot determine the number of CPUs"); - for (hcpu = 0; hcpu < ncpu; hcpu++) - CPU_SET(hcpu, &cpumask); + if (cpuset_getaffinity(CPU_LEVEL_ROOT, CPU_WHICH_PID, -1, + sizeof(rootmask), &rootmask) == -1) + err(EX_OSERR, "ERROR: Cannot determine the root set of CPUs"); + CPU_COPY(&rootmask, &cpumask); while ((option = getopt(argc, argv, "CD:EF:G:M:NO:P:R:S:TWa:c:df:gk:l:m:n:o:p:qr:s:t:vw:z:")) != -1) @@ -642,11 +641,9 @@ main(int argc, char **argv) break; case 'c': /* CPU */ - - if (optarg[0] == '*' && optarg[1] == '\0') { - for (hcpu = 0; hcpu < ncpu; hcpu++) - CPU_SET(hcpu, &cpumask); - } else + if (optarg[0] == '*' && optarg[1] == '\0') + CPU_COPY(&rootmask, &cpumask); + else pmcstat_get_cpumask(optarg, &cpumask); args.pa_flags |= FLAGS_HAS_CPUMASK; @@ -771,13 +768,9 @@ main(int argc, char **argv) else ev->ev_count = -1; - if (option == 'S' || option == 's') { - hcpu = sizeof(cpumask) * NBBY; - for (hcpu--; hcpu >= 0; hcpu--) - if (CPU_ISSET(hcpu, &cpumask)) - break; - ev->ev_cpu = hcpu; - } else + if (option == 'S' || option == 's') + ev->ev_cpu = CPU_FFS(&cpumask) - 1; + else ev->ev_cpu = PMC_CPU_ANY; ev->ev_flags = 0; @@ -804,11 +797,9 @@ main(int argc, char **argv) STAILQ_INSERT_TAIL(&args.pa_events, ev, ev_next); if (option == 's' || option == 'S') { - hcpu = CPU_ISSET(ev->ev_cpu, &cpumask); CPU_CLR(ev->ev_cpu, &cpumask); pmcstat_clone_event_descriptor(ev, &cpumask); - if (hcpu != 0) - CPU_SET(ev->ev_cpu, &cpumask); + CPU_SET(ev->ev_cpu, &cpumask); } break; From owner-svn-src-stable@freebsd.org Tue Sep 29 18:46:40 2015 Return-Path: Delivered-To: svn-src-stable@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 7DF0AA0A25A; Tue, 29 Sep 2015 18:46:40 +0000 (UTC) (envelope-from bdrewery@FreeBSD.org) Received: from repo.freebsd.org (repo.freebsd.org [IPv6:2001:1900:2254:2068::e6a:0]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 6C7C21343; Tue, 29 Sep 2015 18:46:40 +0000 (UTC) (envelope-from bdrewery@FreeBSD.org) Received: from repo.freebsd.org ([127.0.1.70]) by repo.freebsd.org (8.15.2/8.15.2) with ESMTP id t8TIkeuf023370; Tue, 29 Sep 2015 18:46:40 GMT (envelope-from bdrewery@FreeBSD.org) Received: (from bdrewery@localhost) by repo.freebsd.org (8.15.2/8.15.2/Submit) id t8TIkdGX023365; Tue, 29 Sep 2015 18:46:39 GMT (envelope-from bdrewery@FreeBSD.org) Message-Id: <201509291846.t8TIkdGX023365@repo.freebsd.org> X-Authentication-Warning: repo.freebsd.org: bdrewery set sender to bdrewery@FreeBSD.org using -f From: Bryan Drewery Date: Tue, 29 Sep 2015 18:46:39 +0000 (UTC) To: src-committers@freebsd.org, svn-src-all@freebsd.org, svn-src-stable@freebsd.org, svn-src-stable-9@freebsd.org Subject: svn commit: r288388 - stable/9/sys/dev/filemon X-SVN-Group: stable-9 MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit X-BeenThere: svn-src-stable@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: SVN commit messages for all the -stable branches of the src tree List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 29 Sep 2015 18:46:40 -0000 Author: bdrewery Date: Tue Sep 29 18:46:39 2015 New Revision: 288388 URL: https://svnweb.freebsd.org/changeset/base/288388 Log: MFC r287151,r287152,r287153,r287155: r287151: Move common locking for filemon_inuse and struct filemon* to filemon_pid_check(). r287152: Remove unneeded inuse list locking in filemon_comment(). r287153: Avoid taking proctree_lock and searching parents in wrappers if not needed. r287155: Fix filemon locking races. Modified: stable/9/sys/dev/filemon/filemon.c stable/9/sys/dev/filemon/filemon_lock.c stable/9/sys/dev/filemon/filemon_wrapper.c Directory Properties: stable/9/sys/ (props changed) stable/9/sys/dev/ (props changed) Modified: stable/9/sys/dev/filemon/filemon.c ============================================================================== --- stable/9/sys/dev/filemon/filemon.c Tue Sep 29 18:39:58 2015 (r288387) +++ stable/9/sys/dev/filemon/filemon.c Tue Sep 29 18:46:39 2015 (r288388) @@ -1,6 +1,7 @@ /*- * Copyright (c) 2011, David E. O'Brien. * Copyright (c) 2009-2011, Juniper Networks, Inc. + * Copyright (c) 2015, EMC Corp. * All rights reserved. * * Redistribution and use in source and binary forms, with or without @@ -39,12 +40,14 @@ __FBSDID("$FreeBSD$"); #include #include #include +#include #include #include #include #include #include #include +#include #include #include #include @@ -85,12 +88,8 @@ MALLOC_DEFINE(M_FILEMON, "filemon", "Fil struct filemon { TAILQ_ENTRY(filemon) link; /* Link into the in-use list. */ - struct mtx mtx; /* Lock mutex for this filemon. */ - struct cv cv; /* Lock condition variable for this - filemon. */ + struct sx lock; /* Lock mutex for this filemon. */ struct file *fp; /* Output file pointer. */ - struct thread *locker; /* Ptr to the thread locking this - filemon. */ pid_t pid; /* The process ID being monitored. */ char fname1[MAXPATHLEN]; /* Temporary filename buffer. */ char fname2[MAXPATHLEN]; /* Temporary filename buffer. */ @@ -99,11 +98,7 @@ struct filemon { static TAILQ_HEAD(, filemon) filemons_inuse = TAILQ_HEAD_INITIALIZER(filemons_inuse); static TAILQ_HEAD(, filemon) filemons_free = TAILQ_HEAD_INITIALIZER(filemons_free); -static int n_readers = 0; -static struct mtx access_mtx; -static struct cv access_cv; -static struct thread *access_owner = NULL; -static struct thread *access_requester = NULL; +static struct sx access_lock; #if __FreeBSD_version < 701000 static struct clonedevs *filemon_clones; @@ -240,8 +235,7 @@ filemon_open(struct cdev *dev, int oflag filemon->fp = NULL; - mtx_init(&filemon->mtx, "filemon", "filemon", MTX_DEF); - cv_init(&filemon->cv, "filemon"); + sx_init(&filemon->lock, "filemon"); } filemon->pid = curproc->p_pid; @@ -283,8 +277,7 @@ filemon_close(struct cdev *dev __unused, static void filemon_load(void *dummy __unused) { - mtx_init(&access_mtx, "filemon", "filemon", MTX_DEF); - cv_init(&access_cv, "filemon"); + sx_init(&access_lock, "filemons_inuse"); /* Install the syscall wrappers. */ filemon_wrapper_install(); @@ -342,14 +335,12 @@ filemon_unload(void) filemon_lock_write(); while ((filemon = TAILQ_FIRST(&filemons_free)) != NULL) { TAILQ_REMOVE(&filemons_free, filemon, link); - mtx_destroy(&filemon->mtx); - cv_destroy(&filemon->cv); + sx_destroy(&filemon->lock); free(filemon, M_FILEMON); } filemon_unlock_write(); - mtx_destroy(&access_mtx); - cv_destroy(&access_cv); + sx_destroy(&access_lock); } return (error); Modified: stable/9/sys/dev/filemon/filemon_lock.c ============================================================================== --- stable/9/sys/dev/filemon/filemon_lock.c Tue Sep 29 18:39:58 2015 (r288387) +++ stable/9/sys/dev/filemon/filemon_lock.c Tue Sep 29 18:46:39 2015 (r288388) @@ -1,5 +1,6 @@ /*- * Copyright (c) 2009-2011, Juniper Networks, Inc. + * Copyright (c) 2015, EMC Corp. * All rights reserved. * * Redistribution and use in source and binary forms, with or without @@ -27,96 +28,44 @@ #include __FBSDID("$FreeBSD$"); -static void +static __inline void filemon_filemon_lock(struct filemon *filemon) { - mtx_lock(&filemon->mtx); - while (filemon->locker != NULL && filemon->locker != curthread) - cv_wait(&filemon->cv, &filemon->mtx); - - filemon->locker = curthread; - - mtx_unlock(&filemon->mtx); + sx_xlock(&filemon->lock); } -static void +static __inline void filemon_filemon_unlock(struct filemon *filemon) { - mtx_lock(&filemon->mtx); - - if (filemon->locker == curthread) - filemon->locker = NULL; - /* Wake up threads waiting. */ - cv_broadcast(&filemon->cv); - - mtx_unlock(&filemon->mtx); + sx_xunlock(&filemon->lock); } -static void +static __inline void filemon_lock_read(void) { - mtx_lock(&access_mtx); - - while (access_owner != NULL || access_requester != NULL) - cv_wait(&access_cv, &access_mtx); - - n_readers++; - /* Wake up threads waiting. */ - cv_broadcast(&access_cv); - - mtx_unlock(&access_mtx); + sx_slock(&access_lock); } -static void +static __inline void filemon_unlock_read(void) { - mtx_lock(&access_mtx); - - if (n_readers > 0) - n_readers--; - - /* Wake up a thread waiting. */ - cv_broadcast(&access_cv); - mtx_unlock(&access_mtx); + sx_sunlock(&access_lock); } -static void +static __inline void filemon_lock_write(void) { - mtx_lock(&access_mtx); - while (access_owner != curthread) { - if (access_owner == NULL && - (access_requester == NULL || - access_requester == curthread)) { - access_owner = curthread; - access_requester = NULL; - } else { - if (access_requester == NULL) - access_requester = curthread; - - cv_wait(&access_cv, &access_mtx); - } - } - - mtx_unlock(&access_mtx); + sx_xlock(&access_lock); } -static void +static __inline void filemon_unlock_write(void) { - mtx_lock(&access_mtx); - - /* Sanity check that the current thread actually has the write lock. */ - if (access_owner == curthread) - access_owner = NULL; - - /* Wake up a thread waiting. */ - cv_broadcast(&access_cv); - mtx_unlock(&access_mtx); + sx_xunlock(&access_lock); } Modified: stable/9/sys/dev/filemon/filemon_wrapper.c ============================================================================== --- stable/9/sys/dev/filemon/filemon_wrapper.c Tue Sep 29 18:39:58 2015 (r288387) +++ stable/9/sys/dev/filemon/filemon_wrapper.c Tue Sep 29 18:46:39 2015 (r288388) @@ -1,6 +1,7 @@ /*- * Copyright (c) 2011, David E. O'Brien. * Copyright (c) 2009-2011, Juniper Networks, Inc. + * Copyright (c) 2015, EMC Corp. * All rights reserved. * * Redistribution and use in source and binary forms, with or without @@ -86,17 +87,25 @@ filemon_pid_check(struct proc *p) { struct filemon *filemon; + filemon_lock_read(); + if (TAILQ_EMPTY(&filemons_inuse)) { + filemon_unlock_read(); + return (NULL); + } sx_slock(&proctree_lock); while (p != initproc) { TAILQ_FOREACH(filemon, &filemons_inuse, link) { if (p->p_pid == filemon->pid) { sx_sunlock(&proctree_lock); + filemon_filemon_lock(filemon); + filemon_unlock_read(); return (filemon); } } p = proc_realparent(p); } sx_sunlock(&proctree_lock); + filemon_unlock_read(); return (NULL); } @@ -109,9 +118,6 @@ filemon_comment(struct filemon *filemon) /* Load timestamp before locking. Less accurate but less contention. */ getmicrotime(&now); - /* Grab a read lock on the filemon inuse list. */ - filemon_lock_read(); - /* Lock the found filemon structure. */ filemon_filemon_lock(filemon); @@ -124,9 +130,6 @@ filemon_comment(struct filemon *filemon) /* Unlock the found filemon structure. */ filemon_filemon_unlock(filemon); - - /* Release the read lock. */ - filemon_unlock_read(); } static int @@ -138,13 +141,7 @@ filemon_wrapper_chdir(struct thread *td, struct filemon *filemon; if ((ret = sys_chdir(td, uap)) == 0) { - /* Grab a read lock on the filemon inuse list. */ - filemon_lock_read(); - if ((filemon = filemon_pid_check(curproc)) != NULL) { - /* Lock the found filemon structure. */ - filemon_filemon_lock(filemon); - copyinstr(uap->path, filemon->fname1, sizeof(filemon->fname1), &done); @@ -157,9 +154,6 @@ filemon_wrapper_chdir(struct thread *td, /* Unlock the found filemon structure. */ filemon_filemon_unlock(filemon); } - - /* Release the read lock. */ - filemon_unlock_read(); } return (ret); @@ -177,13 +171,7 @@ filemon_wrapper_execve(struct thread *td copyinstr(uap->fname, fname, sizeof(fname), &done); if ((ret = sys_execve(td, uap)) == 0) { - /* Grab a read lock on the filemon inuse list. */ - filemon_lock_read(); - if ((filemon = filemon_pid_check(curproc)) != NULL) { - /* Lock the found filemon structure. */ - filemon_filemon_lock(filemon); - len = snprintf(filemon->msgbufr, sizeof(filemon->msgbufr), "E %d %s\n", curproc->p_pid, fname); @@ -193,9 +181,6 @@ filemon_wrapper_execve(struct thread *td /* Unlock the found filemon structure. */ filemon_filemon_unlock(filemon); } - - /* Release the read lock. */ - filemon_unlock_read(); } return (ret); @@ -215,13 +200,7 @@ filemon_wrapper_freebsd32_execve(struct copyinstr(uap->fname, fname, sizeof(fname), &done); if ((ret = freebsd32_execve(td, uap)) == 0) { - /* Grab a read lock on the filemon inuse list. */ - filemon_lock_read(); - if ((filemon = filemon_pid_check(curproc)) != NULL) { - /* Lock the found filemon structure. */ - filemon_filemon_lock(filemon); - len = snprintf(filemon->msgbufr, sizeof(filemon->msgbufr), "E %d %s\n", curproc->p_pid, fname); @@ -231,9 +210,6 @@ filemon_wrapper_freebsd32_execve(struct /* Unlock the found filemon structure. */ filemon_filemon_unlock(filemon); } - - /* Release the read lock. */ - filemon_unlock_read(); } return (ret); @@ -248,13 +224,7 @@ filemon_wrapper_fork(struct thread *td, struct filemon *filemon; if ((ret = sys_fork(td, uap)) == 0) { - /* Grab a read lock on the filemon inuse list. */ - filemon_lock_read(); - if ((filemon = filemon_pid_check(curproc)) != NULL) { - /* Lock the found filemon structure. */ - filemon_filemon_lock(filemon); - len = snprintf(filemon->msgbufr, sizeof(filemon->msgbufr), "F %d %ld\n", curproc->p_pid, (long)curthread->td_retval[0]); @@ -264,9 +234,6 @@ filemon_wrapper_fork(struct thread *td, /* Unlock the found filemon structure. */ filemon_filemon_unlock(filemon); } - - /* Release the read lock. */ - filemon_unlock_read(); } return (ret); @@ -281,13 +248,7 @@ filemon_wrapper_open(struct thread *td, struct filemon *filemon; if ((ret = sys_open(td, uap)) == 0) { - /* Grab a read lock on the filemon inuse list. */ - filemon_lock_read(); - if ((filemon = filemon_pid_check(curproc)) != NULL) { - /* Lock the found filemon structure. */ - filemon_filemon_lock(filemon); - copyinstr(uap->path, filemon->fname1, sizeof(filemon->fname1), &done); @@ -313,9 +274,6 @@ filemon_wrapper_open(struct thread *td, /* Unlock the found filemon structure. */ filemon_filemon_unlock(filemon); } - - /* Release the read lock. */ - filemon_unlock_read(); } return (ret); @@ -330,13 +288,7 @@ filemon_wrapper_openat(struct thread *td struct filemon *filemon; if ((ret = sys_openat(td, uap)) == 0) { - /* Grab a read lock on the filemon inuse list. */ - filemon_lock_read(); - if ((filemon = filemon_pid_check(curproc)) != NULL) { - /* Lock the found filemon structure. */ - filemon_filemon_lock(filemon); - copyinstr(uap->path, filemon->fname1, sizeof(filemon->fname1), &done); @@ -375,9 +327,6 @@ filemon_wrapper_openat(struct thread *td /* Unlock the found filemon structure. */ filemon_filemon_unlock(filemon); } - - /* Release the read lock. */ - filemon_unlock_read(); } return (ret); @@ -392,13 +341,7 @@ filemon_wrapper_rename(struct thread *td struct filemon *filemon; if ((ret = sys_rename(td, uap)) == 0) { - /* Grab a read lock on the filemon inuse list. */ - filemon_lock_read(); - if ((filemon = filemon_pid_check(curproc)) != NULL) { - /* Lock the found filemon structure. */ - filemon_filemon_lock(filemon); - copyinstr(uap->from, filemon->fname1, sizeof(filemon->fname1), &done); copyinstr(uap->to, filemon->fname2, @@ -413,9 +356,6 @@ filemon_wrapper_rename(struct thread *td /* Unlock the found filemon structure. */ filemon_filemon_unlock(filemon); } - - /* Release the read lock. */ - filemon_unlock_read(); } return (ret); @@ -430,13 +370,7 @@ filemon_wrapper_link(struct thread *td, struct filemon *filemon; if ((ret = sys_link(td, uap)) == 0) { - /* Grab a read lock on the filemon inuse list. */ - filemon_lock_read(); - if ((filemon = filemon_pid_check(curproc)) != NULL) { - /* Lock the found filemon structure. */ - filemon_filemon_lock(filemon); - copyinstr(uap->path, filemon->fname1, sizeof(filemon->fname1), &done); copyinstr(uap->link, filemon->fname2, @@ -451,9 +385,6 @@ filemon_wrapper_link(struct thread *td, /* Unlock the found filemon structure. */ filemon_filemon_unlock(filemon); } - - /* Release the read lock. */ - filemon_unlock_read(); } return (ret); @@ -468,13 +399,7 @@ filemon_wrapper_symlink(struct thread *t struct filemon *filemon; if ((ret = sys_symlink(td, uap)) == 0) { - /* Grab a read lock on the filemon inuse list. */ - filemon_lock_read(); - if ((filemon = filemon_pid_check(curproc)) != NULL) { - /* Lock the found filemon structure. */ - filemon_filemon_lock(filemon); - copyinstr(uap->path, filemon->fname1, sizeof(filemon->fname1), &done); copyinstr(uap->link, filemon->fname2, @@ -489,9 +414,6 @@ filemon_wrapper_symlink(struct thread *t /* Unlock the found filemon structure. */ filemon_filemon_unlock(filemon); } - - /* Release the read lock. */ - filemon_unlock_read(); } return (ret); @@ -507,13 +429,7 @@ filemon_wrapper_linkat(struct thread *td struct filemon *filemon; if ((ret = sys_linkat(td, uap)) == 0) { - /* Grab a read lock on the filemon inuse list. */ - filemon_lock_read(); - if ((filemon = filemon_pid_check(curproc)) != NULL) { - /* Lock the found filemon structure. */ - filemon_filemon_lock(filemon); - copyinstr(uap->path1, filemon->fname1, sizeof(filemon->fname1), &done); copyinstr(uap->path2, filemon->fname2, @@ -528,9 +444,6 @@ filemon_wrapper_linkat(struct thread *td /* Unlock the found filemon structure. */ filemon_filemon_unlock(filemon); } - - /* Release the read lock. */ - filemon_unlock_read(); } return (ret); @@ -546,13 +459,7 @@ filemon_wrapper_stat(struct thread *td, struct filemon *filemon; if ((ret = sys_stat(td, uap)) == 0) { - /* Grab a read lock on the filemon inuse list. */ - filemon_lock_read(); - if ((filemon = filemon_pid_check(curproc)) != NULL) { - /* Lock the found filemon structure. */ - filemon_filemon_lock(filemon); - copyinstr(uap->path, filemon->fname1, sizeof(filemon->fname1), &done); @@ -565,9 +472,6 @@ filemon_wrapper_stat(struct thread *td, /* Unlock the found filemon structure. */ filemon_filemon_unlock(filemon); } - - /* Release the read lock. */ - filemon_unlock_read(); } return (ret); @@ -584,13 +488,7 @@ filemon_wrapper_freebsd32_stat(struct th struct filemon *filemon; if ((ret = freebsd32_stat(td, uap)) == 0) { - /* Grab a read lock on the filemon inuse list. */ - filemon_lock_read(); - if ((filemon = filemon_pid_check(curproc)) != NULL) { - /* Lock the found filemon structure. */ - filemon_filemon_lock(filemon); - copyinstr(uap->path, filemon->fname1, sizeof(filemon->fname1), &done); @@ -603,9 +501,6 @@ filemon_wrapper_freebsd32_stat(struct th /* Unlock the found filemon structure. */ filemon_filemon_unlock(filemon); } - - /* Release the read lock. */ - filemon_unlock_read(); } return (ret); @@ -622,13 +517,7 @@ filemon_wrapper_sys_exit(struct thread * /* Get timestamp before locking. */ getmicrotime(&now); - /* Grab a read lock on the filemon inuse list. */ - filemon_lock_read(); - if ((filemon = filemon_pid_check(curproc)) != NULL) { - /* Lock the found filemon structure. */ - filemon_filemon_lock(filemon); - len = snprintf(filemon->msgbufr, sizeof(filemon->msgbufr), "X %d %d\n", curproc->p_pid, uap->rval); @@ -649,9 +538,6 @@ filemon_wrapper_sys_exit(struct thread * filemon_filemon_unlock(filemon); } - /* Release the read lock. */ - filemon_unlock_read(); - sys_sys_exit(td, uap); } @@ -664,13 +550,7 @@ filemon_wrapper_unlink(struct thread *td struct filemon *filemon; if ((ret = sys_unlink(td, uap)) == 0) { - /* Grab a read lock on the filemon inuse list. */ - filemon_lock_read(); - if ((filemon = filemon_pid_check(curproc)) != NULL) { - /* Lock the found filemon structure. */ - filemon_filemon_lock(filemon); - copyinstr(uap->path, filemon->fname1, sizeof(filemon->fname1), &done); @@ -683,9 +563,6 @@ filemon_wrapper_unlink(struct thread *td /* Unlock the found filemon structure. */ filemon_filemon_unlock(filemon); } - - /* Release the read lock. */ - filemon_unlock_read(); } return (ret); @@ -699,13 +576,7 @@ filemon_wrapper_vfork(struct thread *td, struct filemon *filemon; if ((ret = sys_vfork(td, uap)) == 0) { - /* Grab a read lock on the filemon inuse list. */ - filemon_lock_read(); - if ((filemon = filemon_pid_check(curproc)) != NULL) { - /* Lock the found filemon structure. */ - filemon_filemon_lock(filemon); - len = snprintf(filemon->msgbufr, sizeof(filemon->msgbufr), "F %d %ld\n", curproc->p_pid, (long)curthread->td_retval[0]); @@ -715,9 +586,6 @@ filemon_wrapper_vfork(struct thread *td, /* Unlock the found filemon structure. */ filemon_filemon_unlock(filemon); } - - /* Release the read lock. */ - filemon_unlock_read(); } return (ret); From owner-svn-src-stable@freebsd.org Tue Sep 29 19:14:53 2015 Return-Path: Delivered-To: svn-src-stable@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id A0336A0BB70; Tue, 29 Sep 2015 19:14:53 +0000 (UTC) (envelope-from gjb@FreeBSD.org) Received: from repo.freebsd.org (repo.freebsd.org [IPv6:2001:1900:2254:2068::e6a:0]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 905421AEA; Tue, 29 Sep 2015 19:14:53 +0000 (UTC) (envelope-from gjb@FreeBSD.org) Received: from repo.freebsd.org ([127.0.1.70]) by repo.freebsd.org (8.15.2/8.15.2) with ESMTP id t8TJErSU035796; Tue, 29 Sep 2015 19:14:53 GMT (envelope-from gjb@FreeBSD.org) Received: (from gjb@localhost) by repo.freebsd.org (8.15.2/8.15.2/Submit) id t8TJErPD035795; Tue, 29 Sep 2015 19:14:53 GMT (envelope-from gjb@FreeBSD.org) Message-Id: <201509291914.t8TJErPD035795@repo.freebsd.org> X-Authentication-Warning: repo.freebsd.org: gjb set sender to gjb@FreeBSD.org using -f From: Glen Barber Date: Tue, 29 Sep 2015 19:14:53 +0000 (UTC) To: src-committers@freebsd.org, svn-src-all@freebsd.org, svn-src-stable@freebsd.org, svn-src-stable-9@freebsd.org Subject: svn commit: r288392 - in stable: 10/release/doc/share/xml 9/release/doc/share/xml X-SVN-Group: stable-9 MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit X-BeenThere: svn-src-stable@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: SVN commit messages for all the -stable branches of the src tree List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 29 Sep 2015 19:14:53 -0000 Author: gjb Date: Tue Sep 29 19:14:52 2015 New Revision: 288392 URL: https://svnweb.freebsd.org/changeset/base/288392 Log: Document SA-15:24. Sponsored by: The FreeBSD Foundation Modified: stable/9/release/doc/share/xml/security.xml Changes in other areas also in this revision: Modified: stable/10/release/doc/share/xml/security.xml Modified: stable/9/release/doc/share/xml/security.xml ============================================================================== --- stable/9/release/doc/share/xml/security.xml Tue Sep 29 18:57:30 2015 (r288391) +++ stable/9/release/doc/share/xml/security.xml Tue Sep 29 19:14:52 2015 (r288392) @@ -231,6 +231,13 @@ Remote denial of service vulnerability + + + FreeBSD-SA-15:24.rpcbind + 29 September 2015 + Remote denial of service + From owner-svn-src-stable@freebsd.org Tue Sep 29 19:14:52 2015 Return-Path: Delivered-To: svn-src-stable@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id EFFAEA0BB6C; Tue, 29 Sep 2015 19:14:52 +0000 (UTC) (envelope-from gjb@FreeBSD.org) Received: from repo.freebsd.org (repo.freebsd.org [IPv6:2001:1900:2254:2068::e6a:0]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id E00FA1AE9; Tue, 29 Sep 2015 19:14:52 +0000 (UTC) (envelope-from gjb@FreeBSD.org) Received: from repo.freebsd.org ([127.0.1.70]) by repo.freebsd.org (8.15.2/8.15.2) with ESMTP id t8TJEqD6035789; Tue, 29 Sep 2015 19:14:52 GMT (envelope-from gjb@FreeBSD.org) Received: (from gjb@localhost) by repo.freebsd.org (8.15.2/8.15.2/Submit) id t8TJEqbw035788; Tue, 29 Sep 2015 19:14:52 GMT (envelope-from gjb@FreeBSD.org) Message-Id: <201509291914.t8TJEqbw035788@repo.freebsd.org> X-Authentication-Warning: repo.freebsd.org: gjb set sender to gjb@FreeBSD.org using -f From: Glen Barber Date: Tue, 29 Sep 2015 19:14:52 +0000 (UTC) To: src-committers@freebsd.org, svn-src-all@freebsd.org, svn-src-stable@freebsd.org, svn-src-stable-10@freebsd.org Subject: svn commit: r288392 - in stable: 10/release/doc/share/xml 9/release/doc/share/xml X-SVN-Group: stable-10 MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit X-BeenThere: svn-src-stable@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: SVN commit messages for all the -stable branches of the src tree List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 29 Sep 2015 19:14:53 -0000 Author: gjb Date: Tue Sep 29 19:14:52 2015 New Revision: 288392 URL: https://svnweb.freebsd.org/changeset/base/288392 Log: Document SA-15:24. Sponsored by: The FreeBSD Foundation Modified: stable/10/release/doc/share/xml/security.xml Changes in other areas also in this revision: Modified: stable/9/release/doc/share/xml/security.xml Modified: stable/10/release/doc/share/xml/security.xml ============================================================================== --- stable/10/release/doc/share/xml/security.xml Tue Sep 29 18:57:30 2015 (r288391) +++ stable/10/release/doc/share/xml/security.xml Tue Sep 29 19:14:52 2015 (r288392) @@ -32,6 +32,13 @@ 25 August 2015 Multiple vulnerabilities + + + FreeBSD-SA-15:24.rpcbind + 29 September 2015 + Remote denial of service + From owner-svn-src-stable@freebsd.org Tue Sep 29 21:45:24 2015 Return-Path: Delivered-To: svn-src-stable@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 4A0ABA0BA0B; Tue, 29 Sep 2015 21:45:24 +0000 (UTC) (envelope-from bdrewery@FreeBSD.org) Received: from repo.freebsd.org (repo.freebsd.org [IPv6:2001:1900:2254:2068::e6a:0]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 215801E1B; Tue, 29 Sep 2015 21:45:24 +0000 (UTC) (envelope-from bdrewery@FreeBSD.org) Received: from repo.freebsd.org ([127.0.1.70]) by repo.freebsd.org (8.15.2/8.15.2) with ESMTP id t8TLjO1P096771; Tue, 29 Sep 2015 21:45:24 GMT (envelope-from bdrewery@FreeBSD.org) Received: (from bdrewery@localhost) by repo.freebsd.org (8.15.2/8.15.2/Submit) id t8TLjNUb096770; Tue, 29 Sep 2015 21:45:23 GMT (envelope-from bdrewery@FreeBSD.org) Message-Id: <201509292145.t8TLjNUb096770@repo.freebsd.org> X-Authentication-Warning: repo.freebsd.org: bdrewery set sender to bdrewery@FreeBSD.org using -f From: Bryan Drewery Date: Tue, 29 Sep 2015 21:45:23 +0000 (UTC) To: src-committers@freebsd.org, svn-src-all@freebsd.org, svn-src-stable@freebsd.org, svn-src-stable-10@freebsd.org Subject: svn commit: r288396 - stable/10/tools/build/options X-SVN-Group: stable-10 MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit X-BeenThere: svn-src-stable@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: SVN commit messages for all the -stable branches of the src tree List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 29 Sep 2015 21:45:24 -0000 Author: bdrewery Date: Tue Sep 29 21:45:23 2015 New Revision: 288396 URL: https://svnweb.freebsd.org/changeset/base/288396 Log: MFC r287935: Optimize makeman slightly by removing uneeded cat and extra test -s. Modified: stable/10/tools/build/options/makeman Directory Properties: stable/10/ (props changed) Modified: stable/10/tools/build/options/makeman ============================================================================== --- stable/10/tools/build/options/makeman Tue Sep 29 21:16:35 2015 (r288395) +++ stable/10/tools/build/options/makeman Tue Sep 29 21:45:23 2015 (r288396) @@ -59,7 +59,7 @@ show_options() fi done - cat $t/settings | while read opt targets ; do + while read opt targets ; do if [ "${targets}" = "${ALL_TARGETS}" ] ; then echo "WITHOUT_${opt}" elif [ -z "${targets}" ] ; then @@ -68,7 +68,7 @@ show_options() echo "WITHOUT_${opt}" $(no_targets "${ALL_TARGETS}" "${targets}") echo "WITH_${opt} ${targets}" fi - done + done < $t/settings } # @@ -250,31 +250,33 @@ EOF :> $t/deps2 fi + havedeps=0 if [ -s $t/deps ] ; then + havedeps=1 echo 'When set, it also enforces the following options:' echo '.Pp' echo '.Bl -item -compact' - cat $t/deps | while read opt2 ; do + while read opt2 ; do echo '.It' echo ".Va ${opt2}" - done + done < $t/deps echo '.El' fi if [ -s $t/deps2 ] ; then - if [ -s $t/deps ] ; then + if [ ${havedeps} -eq 1 ] ; then echo '.Pp' fi echo 'When set, the following options are also in effect:' echo '.Pp' echo '.Bl -inset -compact' - cat $t/deps2 | while read opt2 ; do + while read opt2 ; do echo ".It Va ${opt2}" noopt=$(echo ${opt2} | sed -e's/WITH_/WITHOUT_/;t' -e's/WITHOUT_/WITH_/') echo '(unless' echo ".Va ${noopt}" echo 'is set explicitly)' - done + done < $t/deps2 echo '.El' fi twiddle >&2 From owner-svn-src-stable@freebsd.org Tue Sep 29 21:47:51 2015 Return-Path: Delivered-To: svn-src-stable@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 989C3A0BB5C; Tue, 29 Sep 2015 21:47:51 +0000 (UTC) (envelope-from bdrewery@FreeBSD.org) Received: from repo.freebsd.org (repo.freebsd.org [IPv6:2001:1900:2254:2068::e6a:0]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 889501FBE; Tue, 29 Sep 2015 21:47:51 +0000 (UTC) (envelope-from bdrewery@FreeBSD.org) Received: from repo.freebsd.org ([127.0.1.70]) by repo.freebsd.org (8.15.2/8.15.2) with ESMTP id t8TLlppd096912; Tue, 29 Sep 2015 21:47:51 GMT (envelope-from bdrewery@FreeBSD.org) Received: (from bdrewery@localhost) by repo.freebsd.org (8.15.2/8.15.2/Submit) id t8TLlpOf096910; Tue, 29 Sep 2015 21:47:51 GMT (envelope-from bdrewery@FreeBSD.org) Message-Id: <201509292147.t8TLlpOf096910@repo.freebsd.org> X-Authentication-Warning: repo.freebsd.org: bdrewery set sender to bdrewery@FreeBSD.org using -f From: Bryan Drewery Date: Tue, 29 Sep 2015 21:47:51 +0000 (UTC) To: src-committers@freebsd.org, svn-src-all@freebsd.org, svn-src-stable@freebsd.org, svn-src-stable-10@freebsd.org Subject: svn commit: r288397 - stable/10/usr.sbin/sysrc X-SVN-Group: stable-10 MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit X-BeenThere: svn-src-stable@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: SVN commit messages for all the -stable branches of the src tree List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 29 Sep 2015 21:47:51 -0000 Author: bdrewery Date: Tue Sep 29 21:47:50 2015 New Revision: 288397 URL: https://svnweb.freebsd.org/changeset/base/288397 Log: MFC r287979: Remove redundant beforeinstall. Modified: stable/10/usr.sbin/sysrc/Makefile Directory Properties: stable/10/ (props changed) Modified: stable/10/usr.sbin/sysrc/Makefile ============================================================================== --- stable/10/usr.sbin/sysrc/Makefile Tue Sep 29 21:45:23 2015 (r288396) +++ stable/10/usr.sbin/sysrc/Makefile Tue Sep 29 21:47:50 2015 (r288397) @@ -4,8 +4,4 @@ SCRIPTS= sysrc MAN= sysrc.8 -beforeinstall: - mkdir -p ${DESTDIR}${SCRIPTSDIR} - mkdir -p ${DESTDIR}${MANDIR}8 - .include From owner-svn-src-stable@freebsd.org Tue Sep 29 21:48:52 2015 Return-Path: Delivered-To: svn-src-stable@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 8D454A0BC44; Tue, 29 Sep 2015 21:48:52 +0000 (UTC) (envelope-from bdrewery@FreeBSD.org) Received: from repo.freebsd.org (repo.freebsd.org [IPv6:2001:1900:2254:2068::e6a:0]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 58D50118D; Tue, 29 Sep 2015 21:48:52 +0000 (UTC) (envelope-from bdrewery@FreeBSD.org) Received: from repo.freebsd.org ([127.0.1.70]) by repo.freebsd.org (8.15.2/8.15.2) with ESMTP id t8TLmqvr097111; Tue, 29 Sep 2015 21:48:52 GMT (envelope-from bdrewery@FreeBSD.org) Received: (from bdrewery@localhost) by repo.freebsd.org (8.15.2/8.15.2/Submit) id t8TLmqCr097110; Tue, 29 Sep 2015 21:48:52 GMT (envelope-from bdrewery@FreeBSD.org) Message-Id: <201509292148.t8TLmqCr097110@repo.freebsd.org> X-Authentication-Warning: repo.freebsd.org: bdrewery set sender to bdrewery@FreeBSD.org using -f From: Bryan Drewery Date: Tue, 29 Sep 2015 21:48:52 +0000 (UTC) To: src-committers@freebsd.org, svn-src-all@freebsd.org, svn-src-stable@freebsd.org, svn-src-stable-9@freebsd.org Subject: svn commit: r288398 - stable/9/usr.sbin/sysrc X-SVN-Group: stable-9 MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit X-BeenThere: svn-src-stable@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: SVN commit messages for all the -stable branches of the src tree List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 29 Sep 2015 21:48:52 -0000 Author: bdrewery Date: Tue Sep 29 21:48:51 2015 New Revision: 288398 URL: https://svnweb.freebsd.org/changeset/base/288398 Log: MFC r287979: Remove redundant beforeinstall. Modified: stable/9/usr.sbin/sysrc/Makefile Directory Properties: stable/9/usr.sbin/sysrc/ (props changed) Modified: stable/9/usr.sbin/sysrc/Makefile ============================================================================== --- stable/9/usr.sbin/sysrc/Makefile Tue Sep 29 21:47:50 2015 (r288397) +++ stable/9/usr.sbin/sysrc/Makefile Tue Sep 29 21:48:51 2015 (r288398) @@ -4,8 +4,4 @@ SCRIPTS= sysrc MAN= sysrc.8 -beforeinstall: - mkdir -p ${DESTDIR}${SCRIPTSDIR} - mkdir -p ${DESTDIR}${MANDIR}8 - .include From owner-svn-src-stable@freebsd.org Tue Sep 29 21:52:33 2015 Return-Path: Delivered-To: svn-src-stable@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id A6177A0BFB7; Tue, 29 Sep 2015 21:52:33 +0000 (UTC) (envelope-from bdrewery@FreeBSD.org) Received: from repo.freebsd.org (repo.freebsd.org [IPv6:2001:1900:2254:2068::e6a:0]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 7C2D91627; Tue, 29 Sep 2015 21:52:33 +0000 (UTC) (envelope-from bdrewery@FreeBSD.org) Received: from repo.freebsd.org ([127.0.1.70]) by repo.freebsd.org (8.15.2/8.15.2) with ESMTP id t8TLqXjk001042; Tue, 29 Sep 2015 21:52:33 GMT (envelope-from bdrewery@FreeBSD.org) Received: (from bdrewery@localhost) by repo.freebsd.org (8.15.2/8.15.2/Submit) id t8TLqXpH001041; Tue, 29 Sep 2015 21:52:33 GMT (envelope-from bdrewery@FreeBSD.org) Message-Id: <201509292152.t8TLqXpH001041@repo.freebsd.org> X-Authentication-Warning: repo.freebsd.org: bdrewery set sender to bdrewery@FreeBSD.org using -f From: Bryan Drewery Date: Tue, 29 Sep 2015 21:52:33 +0000 (UTC) To: src-committers@freebsd.org, svn-src-all@freebsd.org, svn-src-stable@freebsd.org, svn-src-stable-9@freebsd.org Subject: svn commit: r288399 - stable/9/tools/build/options X-SVN-Group: stable-9 MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit X-BeenThere: svn-src-stable@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: SVN commit messages for all the -stable branches of the src tree List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 29 Sep 2015 21:52:33 -0000 Author: bdrewery Date: Tue Sep 29 21:52:32 2015 New Revision: 288399 URL: https://svnweb.freebsd.org/changeset/base/288399 Log: MFC r287935: Optimize makeman slightly by removing uneeded cat and extra test -s. Modified: stable/9/tools/build/options/makeman Directory Properties: stable/9/tools/build/options/ (props changed) Modified: stable/9/tools/build/options/makeman ============================================================================== --- stable/9/tools/build/options/makeman Tue Sep 29 21:48:51 2015 (r288398) +++ stable/9/tools/build/options/makeman Tue Sep 29 21:52:32 2015 (r288399) @@ -59,7 +59,7 @@ show_options() fi done - cat $t/settings | while read opt targets ; do + while read opt targets ; do if [ "${targets}" = "${ALL_TARGETS}" ] ; then echo "WITHOUT_${opt}" elif [ -z "${targets}" ] ; then @@ -68,7 +68,7 @@ show_options() echo "WITHOUT_${opt}" $(no_targets "${ALL_TARGETS}" "${targets}") echo "WITH_${opt} ${targets}" fi - done + done < $t/settings } # @@ -250,31 +250,33 @@ EOF :> $t/deps2 fi + havedeps=0 if [ -s $t/deps ] ; then + havedeps=1 echo 'When set, it also enforces the following options:' echo '.Pp' echo '.Bl -item -compact' - cat $t/deps | while read opt2 ; do + while read opt2 ; do echo '.It' echo ".Va ${opt2}" - done + done < $t/deps echo '.El' fi if [ -s $t/deps2 ] ; then - if [ -s $t/deps ] ; then + if [ ${havedeps} -eq 1 ] ; then echo '.Pp' fi echo 'When set, the following options are also in effect:' echo '.Pp' echo '.Bl -inset -compact' - cat $t/deps2 | while read opt2 ; do + while read opt2 ; do echo ".It Va ${opt2}" noopt=$(echo ${opt2} | sed -e's/WITH_/WITHOUT_/;t' -e's/WITHOUT_/WITH_/') echo '(unless' echo ".Va ${noopt}" echo 'is set explicitly)' - done + done < $t/deps2 echo '.El' fi twiddle >&2 From owner-svn-src-stable@freebsd.org Tue Sep 29 21:54:10 2015 Return-Path: Delivered-To: svn-src-stable@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id DE7E7A0C07E; Tue, 29 Sep 2015 21:54:10 +0000 (UTC) (envelope-from bdrewery@FreeBSD.org) Received: from repo.freebsd.org (repo.freebsd.org [IPv6:2001:1900:2254:2068::e6a:0]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id CE37318B5; Tue, 29 Sep 2015 21:54:10 +0000 (UTC) (envelope-from bdrewery@FreeBSD.org) Received: from repo.freebsd.org ([127.0.1.70]) by repo.freebsd.org (8.15.2/8.15.2) with ESMTP id t8TLsAeA001168; Tue, 29 Sep 2015 21:54:10 GMT (envelope-from bdrewery@FreeBSD.org) Received: (from bdrewery@localhost) by repo.freebsd.org (8.15.2/8.15.2/Submit) id t8TLsAh2001167; Tue, 29 Sep 2015 21:54:10 GMT (envelope-from bdrewery@FreeBSD.org) Message-Id: <201509292154.t8TLsAh2001167@repo.freebsd.org> X-Authentication-Warning: repo.freebsd.org: bdrewery set sender to bdrewery@FreeBSD.org using -f From: Bryan Drewery Date: Tue, 29 Sep 2015 21:54:10 +0000 (UTC) To: src-committers@freebsd.org, svn-src-all@freebsd.org, svn-src-stable@freebsd.org, svn-src-stable-10@freebsd.org Subject: svn commit: r288400 - stable/10/sys/kern X-SVN-Group: stable-10 MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit X-BeenThere: svn-src-stable@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: SVN commit messages for all the -stable branches of the src tree List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 29 Sep 2015 21:54:11 -0000 Author: bdrewery Date: Tue Sep 29 21:54:09 2015 New Revision: 288400 URL: https://svnweb.freebsd.org/changeset/base/288400 Log: MFC r288091: vfs_mountroot_shuffle() never returns non-zero. Modified: stable/10/sys/kern/vfs_mountroot.c Directory Properties: stable/10/ (props changed) Modified: stable/10/sys/kern/vfs_mountroot.c ============================================================================== --- stable/10/sys/kern/vfs_mountroot.c Tue Sep 29 21:52:32 2015 (r288399) +++ stable/10/sys/kern/vfs_mountroot.c Tue Sep 29 21:54:09 2015 (r288400) @@ -245,7 +245,7 @@ vfs_mountroot_devfs(struct thread *td, s return (error); } -static int +static void vfs_mountroot_shuffle(struct thread *td, struct mount *mpdevfs) { struct nameidata nd; @@ -355,8 +355,6 @@ vfs_mountroot_shuffle(struct thread *td, printf("mountroot: unable to unlink /dev/dev " "(error %d)\n", error); } - - return (0); } /* @@ -948,12 +946,10 @@ vfs_mountroot(void) while (!error) { error = vfs_mountroot_parse(sb, mp); if (!error) { - error = vfs_mountroot_shuffle(td, mp); - if (!error) { - sbuf_clear(sb); - error = vfs_mountroot_readconf(td, sb); - sbuf_finish(sb); - } + vfs_mountroot_shuffle(td, mp); + sbuf_clear(sb); + error = vfs_mountroot_readconf(td, sb); + sbuf_finish(sb); } } From owner-svn-src-stable@freebsd.org Tue Sep 29 21:55:46 2015 Return-Path: Delivered-To: svn-src-stable@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 0F79DA0C1A9; Tue, 29 Sep 2015 21:55:46 +0000 (UTC) (envelope-from bdrewery@FreeBSD.org) Received: from repo.freebsd.org (repo.freebsd.org [IPv6:2001:1900:2254:2068::e6a:0]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id E74751A3B; Tue, 29 Sep 2015 21:55:45 +0000 (UTC) (envelope-from bdrewery@FreeBSD.org) Received: from repo.freebsd.org ([127.0.1.70]) by repo.freebsd.org (8.15.2/8.15.2) with ESMTP id t8TLtjH4001303; Tue, 29 Sep 2015 21:55:45 GMT (envelope-from bdrewery@FreeBSD.org) Received: (from bdrewery@localhost) by repo.freebsd.org (8.15.2/8.15.2/Submit) id t8TLtjVR001302; Tue, 29 Sep 2015 21:55:45 GMT (envelope-from bdrewery@FreeBSD.org) Message-Id: <201509292155.t8TLtjVR001302@repo.freebsd.org> X-Authentication-Warning: repo.freebsd.org: bdrewery set sender to bdrewery@FreeBSD.org using -f From: Bryan Drewery Date: Tue, 29 Sep 2015 21:55:45 +0000 (UTC) To: src-committers@freebsd.org, svn-src-all@freebsd.org, svn-src-stable@freebsd.org, svn-src-stable-9@freebsd.org Subject: svn commit: r288401 - stable/9/share/man/man5 X-SVN-Group: stable-9 MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit X-BeenThere: svn-src-stable@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: SVN commit messages for all the -stable branches of the src tree List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 29 Sep 2015 21:55:46 -0000 Author: bdrewery Date: Tue Sep 29 21:55:45 2015 New Revision: 288401 URL: https://svnweb.freebsd.org/changeset/base/288401 Log: Regenerate after r280421. Modified: stable/9/share/man/man5/src.conf.5 Modified: stable/9/share/man/man5/src.conf.5 ============================================================================== --- stable/9/share/man/man5/src.conf.5 Tue Sep 29 21:54:09 2015 (r288400) +++ stable/9/share/man/man5/src.conf.5 Tue Sep 29 21:55:45 2015 (r288401) @@ -1,7 +1,7 @@ .\" DO NOT EDIT-- this file is automatically generated. -.\" from FreeBSD: stable/9/tools/build/options/makeman 263058 2014-03-11 23:04:32Z gjb +.\" from FreeBSD: stable/9/tools/build/options/makeman 288399 2015-09-29 21:52:32Z bdrewery .\" $FreeBSD$ -.Dd February 13, 2015 +.Dd September 29, 2015 .Dt SRC.CONF 5 .Os .Sh NAME @@ -220,6 +220,14 @@ Set to not build Bluetooth related kerne .It Va WITHOUT_BOOT .\" from FreeBSD: stable/9/tools/build/options/WITHOUT_BOOT 156932 2006-03-21 07:50:50Z ru Set to not build the boot blocks and loader. +.It Va WITHOUT_BOOTPARAMD +.\" from FreeBSD: stable/9/tools/build/options/WITHOUT_BOOTPARAMD 280421 2015-03-24 07:11:54Z ngie +Set to not build or install +.Xr bootparamd 8 . +.It Va WITHOUT_BOOTPD +.\" from FreeBSD: stable/9/tools/build/options/WITHOUT_BOOTPD 280421 2015-03-24 07:11:54Z ngie +Set to not build or install +.Xr bootpd 8 . .It Va WITHOUT_BSDINSTALL .\" from FreeBSD: stable/9/tools/build/options/WITHOUT_BSDINSTALL 278714 2015-02-13 21:25:56Z ngie Set to not build @@ -425,6 +433,17 @@ This includes the device tree compiler ( .Pp It is a default setting on arm/arm, arm/armeb, powerpc/powerpc and powerpc/powerpc64. +.It Va WITHOUT_FILE +.\" from FreeBSD: stable/9/tools/build/options/WITHOUT_FILE 280421 2015-03-24 07:11:54Z ngie +Set to not build +.Xr file 1 +and related programs. +.It Va WITHOUT_FINGER +.\" from FreeBSD: stable/9/tools/build/options/WITHOUT_FINGER 280421 2015-03-24 07:11:54Z ngie +Set to not build or install +.Xr finger 1 +and +.Xr fingerd 8 . .It Va WITHOUT_FLOPPY .\" from FreeBSD: stable/9/tools/build/options/WITHOUT_FLOPPY 221540 2011-05-06 19:13:03Z ru Set to not build or install programs @@ -441,6 +460,12 @@ without floating-point support. .\" from FreeBSD: stable/9/tools/build/options/WITHOUT_FREEBSD_UPDATE 183242 2008-09-21 22:02:26Z sam Set to not build .Xr freebsd-update 8 . +.It Va WITHOUT_FTP +.\" from FreeBSD: stable/9/tools/build/options/WITHOUT_FTP 280421 2015-03-24 07:11:54Z ngie +Set to not build or install +.Xr ftp 1 +and +.Xr ftpd 8 . .It Va WITHOUT_GAMES .\" from FreeBSD: stable/9/tools/build/options/WITHOUT_GAMES 156932 2006-03-21 07:50:50Z ru Set to not build games. @@ -539,6 +564,10 @@ When set, it also enforces the following .It Va WITHOUT_INET6_SUPPORT .\" from FreeBSD: stable/9/tools/build/options/WITHOUT_INET6_SUPPORT 156932 2006-03-21 07:50:50Z ru Set to build libraries, programs, and kernel modules without IPv6 support. +.It Va WITHOUT_INETD +.\" from FreeBSD: stable/9/tools/build/options/WITHOUT_INETD 280421 2015-03-24 07:11:54Z ngie +Set to not build +.Xr inetd 8 . .It Va WITHOUT_INET_SUPPORT .\" from FreeBSD: stable/9/tools/build/options/WITHOUT_INET_SUPPORT 221266 2011-04-30 17:58:28Z bz Set to build libraries, programs, and kernel modules without IPv4 support. @@ -640,9 +669,6 @@ and On amd64, set to not build 32-bit library set and a .Nm ld-elf32.so.1 runtime linker. -.It Va WITH_LIBCPLUSPLUS -.\" from FreeBSD: stable/9/tools/build/options/WITH_LIBCPLUSPLUS 246322 2013-02-04 16:13:55Z des -Set to build libcxxrt and libc++. .It Va WITHOUT_LIBPTHREAD .\" from FreeBSD: stable/9/tools/build/options/WITHOUT_LIBPTHREAD 188848 2009-02-20 11:09:55Z mtm Set to not build the @@ -928,6 +954,16 @@ Set to avoid compiling profiled librarie Set to not build .Xr quota 8 and related programs. +.It Va WITHOUT_RADIUS_SUPPORT +.\" from FreeBSD: stable/9/tools/build/options/WITHOUT_RADIUS_SUPPORT 280421 2015-03-24 07:11:54Z ngie +Set to not build radius support into various applications, like +.Xr pam_radius 8 +and +.Xr ppp 8 . +.It Va WITHOUT_RBOOTD +.\" from FreeBSD: stable/9/tools/build/options/WITHOUT_RBOOTD 280421 2015-03-24 07:11:54Z ngie +Set to not build or install +.Xr rbootd 8 . .It Va WITHOUT_RCMDS .\" from FreeBSD: stable/9/tools/build/options/WITHOUT_RCMDS 156932 2006-03-21 07:50:50Z ru Disable building of the @@ -1012,6 +1048,11 @@ Set to not build or install .Xr talk 1 and .Xr talkd 8 . +.It Va WITHOUT_TCP_WRAPPERS +.\" from FreeBSD: stable/9/tools/build/options/WITHOUT_TCP_WRAPPERS 280421 2015-03-24 07:11:54Z ngie +Set to not build or install +.Xr tcpd 8 , +and related utilities. .It Va WITHOUT_TCSH .\" from FreeBSD: stable/9/tools/build/options/WITHOUT_TCSH 156932 2006-03-21 07:50:50Z ru Set to not build and install @@ -1033,6 +1074,16 @@ When set, it also enforces the following .It .Va WITHOUT_GROFF .El +.It Va WITHOUT_TFTP +.\" from FreeBSD: stable/9/tools/build/options/WITHOUT_TFTP 280421 2015-03-24 07:11:54Z ngie +Set to not build or install +.Xr tftp 1 +and +.Xr tftpd 8 . +.It Va WITHOUT_TIMED +.\" from FreeBSD: stable/9/tools/build/options/WITHOUT_TIMED 280421 2015-03-24 07:11:54Z ngie +Set to not build or install +.Xr timed 8 . .It Va WITHOUT_TOOLCHAIN .\" from FreeBSD: stable/9/tools/build/options/WITHOUT_TOOLCHAIN 174550 2007-12-12 16:43:17Z ru Set to not install From owner-svn-src-stable@freebsd.org Tue Sep 29 22:00:04 2015 Return-Path: Delivered-To: svn-src-stable@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 7FE51A0C466; Tue, 29 Sep 2015 22:00:04 +0000 (UTC) (envelope-from bdrewery@FreeBSD.org) Received: from repo.freebsd.org (repo.freebsd.org [IPv6:2001:1900:2254:2068::e6a:0]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 6FE291E21; Tue, 29 Sep 2015 22:00:04 +0000 (UTC) (envelope-from bdrewery@FreeBSD.org) Received: from repo.freebsd.org ([127.0.1.70]) by repo.freebsd.org (8.15.2/8.15.2) with ESMTP id t8TM04VQ001764; Tue, 29 Sep 2015 22:00:04 GMT (envelope-from bdrewery@FreeBSD.org) Received: (from bdrewery@localhost) by repo.freebsd.org (8.15.2/8.15.2/Submit) id t8TM04iV001763; Tue, 29 Sep 2015 22:00:04 GMT (envelope-from bdrewery@FreeBSD.org) Message-Id: <201509292200.t8TM04iV001763@repo.freebsd.org> X-Authentication-Warning: repo.freebsd.org: bdrewery set sender to bdrewery@FreeBSD.org using -f From: Bryan Drewery Date: Tue, 29 Sep 2015 22:00:04 +0000 (UTC) To: src-committers@freebsd.org, svn-src-all@freebsd.org, svn-src-stable@freebsd.org, svn-src-stable-10@freebsd.org Subject: svn commit: r288403 - stable/10/usr.bin/bmake X-SVN-Group: stable-10 MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit X-BeenThere: svn-src-stable@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: SVN commit messages for all the -stable branches of the src tree List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 29 Sep 2015 22:00:04 -0000 Author: bdrewery Date: Tue Sep 29 22:00:03 2015 New Revision: 288403 URL: https://svnweb.freebsd.org/changeset/base/288403 Log: MFC r288154: Similar to r266147, don't define PROG in the test subdirs. Modified: stable/10/usr.bin/bmake/Makefile.inc Directory Properties: stable/10/ (props changed) Modified: stable/10/usr.bin/bmake/Makefile.inc ============================================================================== --- stable/10/usr.bin/bmake/Makefile.inc Tue Sep 29 21:57:52 2015 (r288402) +++ stable/10/usr.bin/bmake/Makefile.inc Tue Sep 29 22:00:03 2015 (r288403) @@ -13,7 +13,7 @@ MK_BMAKE= yes .endif -.if defined(MK_BMAKE) && ${MK_BMAKE} != "no" +.if defined(MK_BMAKE) && ${MK_BMAKE} != "no" && exists(${.CURDIR}/tests) PROG= make .endif From owner-svn-src-stable@freebsd.org Wed Sep 30 00:11:07 2015 Return-Path: Delivered-To: svn-src-stable@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 74609A0C4E4; Wed, 30 Sep 2015 00:11:07 +0000 (UTC) (envelope-from jhb@FreeBSD.org) Received: from repo.freebsd.org (repo.freebsd.org [IPv6:2001:1900:2254:2068::e6a:0]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 618EA1641; Wed, 30 Sep 2015 00:11:07 +0000 (UTC) (envelope-from jhb@FreeBSD.org) Received: from repo.freebsd.org ([127.0.1.70]) by repo.freebsd.org (8.15.2/8.15.2) with ESMTP id t8U0B7HU059284; Wed, 30 Sep 2015 00:11:07 GMT (envelope-from jhb@FreeBSD.org) Received: (from jhb@localhost) by repo.freebsd.org (8.15.2/8.15.2/Submit) id t8U0B6XC059282; Wed, 30 Sep 2015 00:11:06 GMT (envelope-from jhb@FreeBSD.org) Message-Id: <201509300011.t8U0B6XC059282@repo.freebsd.org> X-Authentication-Warning: repo.freebsd.org: jhb set sender to jhb@FreeBSD.org using -f From: John Baldwin Date: Wed, 30 Sep 2015 00:11:06 +0000 (UTC) To: src-committers@freebsd.org, svn-src-all@freebsd.org, svn-src-stable@freebsd.org, svn-src-stable-10@freebsd.org Subject: svn commit: r288407 - stable/10/usr.bin/vmstat X-SVN-Group: stable-10 MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit X-BeenThere: svn-src-stable@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: SVN commit messages for all the -stable branches of the src tree List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 30 Sep 2015 00:11:07 -0000 Author: jhb Date: Wed Sep 30 00:11:06 2015 New Revision: 288407 URL: https://svnweb.freebsd.org/changeset/base/288407 Log: MFC 269727: Update vmstat usage for last-argument count/wait parameters Correct the usage in both the manpage and in usage() to indicate that the wait interval and repetition count may be given either with the respective -w/-c arguments, or as the final positional arguments. [0] The corresponding code to implement the positional arguments has been conditional on the (always-enabled) BACKWARD_COMPATIBILITY macro since the original 4.4-lite import. It's no longer reasonable to remove the functionality, so remove the macro and conditional instead. Note that multiple disks may be given on the command line. While here, sort arguments and apply minor mdoc fixes. PR: 184755 [0] Modified: stable/10/usr.bin/vmstat/vmstat.8 stable/10/usr.bin/vmstat/vmstat.c Directory Properties: stable/10/ (props changed) Modified: stable/10/usr.bin/vmstat/vmstat.8 ============================================================================== --- stable/10/usr.bin/vmstat/vmstat.8 Wed Sep 30 00:08:24 2015 (r288406) +++ stable/10/usr.bin/vmstat/vmstat.8 Wed Sep 30 00:11:06 2015 (r288407) @@ -28,7 +28,7 @@ .\" @(#)vmstat.8 8.1 (Berkeley) 6/6/93 .\" $FreeBSD$ .\" -.Dd October 21, 2006 +.Dd August 8, 2014 .Dt VMSTAT 8 .Os .Sh NAME @@ -38,9 +38,8 @@ .Nm .\" .Op Fl fimst .Op Fl afHhimPsz -.Op Fl c Ar count .Op Fl M Ar core Op Fl N Ar system -.Op Fl w Ar wait +.Op Fl c Ar count .Op Fl n Ar devs .Oo .Fl p @@ -48,7 +47,9 @@ .Ar type , if , pass .Sm on .Oc -.Op Ar disks +.Op Fl w Ar wait +.Op Ar disks ... +.Op wait Op count .Sh DESCRIPTION The .Nm @@ -91,10 +92,12 @@ and system calls since system startup, and the number of pages of virtual memory involved in each. .It Fl h -Changes memory columns into more easily human readable form. Default if +Changes memory columns into more easily human readable form. +The default if standard output is a terminal device. .It Fl H -Changes memory columns into straight numbers. Default if standard output +Changes memory columns into straight numbers. +The default if standard output is not a terminal device (such as a script). .It Fl i Report on the number of interrupts taken by each device since system @@ -214,6 +217,21 @@ Report on memory used by the kernel zone by zone. .El .Pp +The +.Ar wait +and +.Ar count +arguments may be given after their respective flags at any point +on the command line before the +.Ar disks +argument(s), or without their flags, as the final argument(s). +The latter form is accepted for backwards compatibility, but it is +preferred to use the forms with +.Fl w +and +.Fl c +to avoid ambiguity. +.Pp By default, .Nm displays the following information: Modified: stable/10/usr.bin/vmstat/vmstat.c ============================================================================== --- stable/10/usr.bin/vmstat/vmstat.c Wed Sep 30 00:08:24 2015 (r288406) +++ stable/10/usr.bin/vmstat/vmstat.c Wed Sep 30 00:11:06 2015 (r288407) @@ -289,15 +289,12 @@ main(int argc, char *argv[]) argv = getdrivedata(argv); } -#define BACKWARD_COMPATIBILITY -#ifdef BACKWARD_COMPATIBILITY if (*argv) { f = atof(*argv); interval = f * 1000; if (*++argv) reps = atoi(*argv); } -#endif if (interval) { if (!reps) @@ -1354,7 +1351,7 @@ static void usage(void) { (void)fprintf(stderr, "%s%s", - "usage: vmstat [-afHhimPsz] [-c count] [-M core [-N system]] [-w wait]\n", - " [-n devs] [-p type,if,pass] [disks]\n"); + "usage: vmstat [-afHhimPsz] [-M core [-N system]] [-c count] [-n devs]\n", + " [-p type,if,pass] [-w wait] [disks] [wait [count]]\n"); exit(1); } From owner-svn-src-stable@freebsd.org Wed Sep 30 03:33:29 2015 Return-Path: Delivered-To: svn-src-stable@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 2456CA0C765; Wed, 30 Sep 2015 03:33:29 +0000 (UTC) (envelope-from markj@FreeBSD.org) Received: from repo.freebsd.org (repo.freebsd.org [IPv6:2001:1900:2254:2068::e6a:0]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 14C0B1ECE; Wed, 30 Sep 2015 03:33:29 +0000 (UTC) (envelope-from markj@FreeBSD.org) Received: from repo.freebsd.org ([127.0.1.70]) by repo.freebsd.org (8.15.2/8.15.2) with ESMTP id t8U3XSI7046323; Wed, 30 Sep 2015 03:33:28 GMT (envelope-from markj@FreeBSD.org) Received: (from markj@localhost) by repo.freebsd.org (8.15.2/8.15.2/Submit) id t8U3XSfr046322; Wed, 30 Sep 2015 03:33:28 GMT (envelope-from markj@FreeBSD.org) Message-Id: <201509300333.t8U3XSfr046322@repo.freebsd.org> X-Authentication-Warning: repo.freebsd.org: markj set sender to markj@FreeBSD.org using -f From: Mark Johnston Date: Wed, 30 Sep 2015 03:33:28 +0000 (UTC) To: src-committers@freebsd.org, svn-src-all@freebsd.org, svn-src-stable@freebsd.org, svn-src-stable-10@freebsd.org Subject: svn commit: r288409 - stable/10/sys/cam/scsi X-SVN-Group: stable-10 MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit X-BeenThere: svn-src-stable@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: SVN commit messages for all the -stable branches of the src tree List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 30 Sep 2015 03:33:29 -0000 Author: markj Date: Wed Sep 30 03:33:28 2015 New Revision: 288409 URL: https://svnweb.freebsd.org/changeset/base/288409 Log: MFC r287806: Preserve the device queue status before retrying a sense request in chdone(). Modified: stable/10/sys/cam/scsi/scsi_ch.c Directory Properties: stable/10/ (props changed) Modified: stable/10/sys/cam/scsi/scsi_ch.c ============================================================================== --- stable/10/sys/cam/scsi/scsi_ch.c Wed Sep 30 03:30:24 2015 (r288408) +++ stable/10/sys/cam/scsi/scsi_ch.c Wed Sep 30 03:33:28 2015 (r288409) @@ -655,11 +655,13 @@ chdone(struct cam_periph *periph, union */ return; } else if (error != 0) { - int retry_scheduled; struct scsi_mode_sense_6 *sms; + int frozen, retry_scheduled; sms = (struct scsi_mode_sense_6 *) done_ccb->csio.cdb_io.cdb_bytes; + frozen = (done_ccb->ccb_h.status & + CAM_DEV_QFRZN) != 0; /* * Check to see if block descriptors were @@ -670,7 +672,8 @@ chdone(struct cam_periph *periph, union * block descriptors were disabled, enable * them and re-send the command. */ - if (sms->byte2 & SMS_DBD) { + if ((sms->byte2 & SMS_DBD) != 0 && + (periph->flags & CAM_PERIPH_INVALID) == 0) { sms->byte2 &= ~SMS_DBD; xpt_action(done_ccb); softc->quirks |= CH_Q_NO_DBD; @@ -679,7 +682,7 @@ chdone(struct cam_periph *periph, union retry_scheduled = 0; /* Don't wedge this device's queue */ - if ((done_ccb->ccb_h.status & CAM_DEV_QFRZN) != 0) + if (frozen) cam_release_devq(done_ccb->ccb_h.path, /*relsim_flags*/0, /*reduction*/0, From owner-svn-src-stable@freebsd.org Wed Sep 30 03:35:59 2015 Return-Path: Delivered-To: svn-src-stable@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id D9E97A0C956; Wed, 30 Sep 2015 03:35:59 +0000 (UTC) (envelope-from markj@FreeBSD.org) Received: from repo.freebsd.org (repo.freebsd.org [IPv6:2001:1900:2254:2068::e6a:0]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id CA6A11084; Wed, 30 Sep 2015 03:35:59 +0000 (UTC) (envelope-from markj@FreeBSD.org) Received: from repo.freebsd.org ([127.0.1.70]) by repo.freebsd.org (8.15.2/8.15.2) with ESMTP id t8U3Zx4R046626; Wed, 30 Sep 2015 03:35:59 GMT (envelope-from markj@FreeBSD.org) Received: (from markj@localhost) by repo.freebsd.org (8.15.2/8.15.2/Submit) id t8U3Zxs6046625; Wed, 30 Sep 2015 03:35:59 GMT (envelope-from markj@FreeBSD.org) Message-Id: <201509300335.t8U3Zxs6046625@repo.freebsd.org> X-Authentication-Warning: repo.freebsd.org: markj set sender to markj@FreeBSD.org using -f From: Mark Johnston Date: Wed, 30 Sep 2015 03:35:59 +0000 (UTC) To: src-committers@freebsd.org, svn-src-all@freebsd.org, svn-src-stable@freebsd.org, svn-src-stable-10@freebsd.org Subject: svn commit: r288410 - stable/10/sys/kern X-SVN-Group: stable-10 MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit X-BeenThere: svn-src-stable@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: SVN commit messages for all the -stable branches of the src tree List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 30 Sep 2015 03:36:00 -0000 Author: markj Date: Wed Sep 30 03:35:58 2015 New Revision: 288410 URL: https://svnweb.freebsd.org/changeset/base/288410 Log: MFC r288276: Fix argument ordering in vn_printf(). Modified: stable/10/sys/kern/vfs_subr.c Directory Properties: stable/10/ (props changed) Modified: stable/10/sys/kern/vfs_subr.c ============================================================================== --- stable/10/sys/kern/vfs_subr.c Wed Sep 30 03:33:28 2015 (r288409) +++ stable/10/sys/kern/vfs_subr.c Wed Sep 30 03:35:58 2015 (r288410) @@ -3010,8 +3010,8 @@ vn_printf(struct vnode *vp, const char * "cleanbuf %d dirtybuf %d\n", vp->v_object, vp->v_object->ref_count, vp->v_object->resident_page_count, - vp->v_bufobj.bo_dirty.bv_cnt, - vp->v_bufobj.bo_clean.bv_cnt); + vp->v_bufobj.bo_clean.bv_cnt, + vp->v_bufobj.bo_dirty.bv_cnt); printf(" "); lockmgr_printinfo(vp->v_vnlock); if (vp->v_data != NULL) From owner-svn-src-stable@freebsd.org Wed Sep 30 03:36:42 2015 Return-Path: Delivered-To: svn-src-stable@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 4C5E6A0C9FE; Wed, 30 Sep 2015 03:36:42 +0000 (UTC) (envelope-from markj@FreeBSD.org) Received: from repo.freebsd.org (repo.freebsd.org [IPv6:2001:1900:2254:2068::e6a:0]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 3CA0911D8; Wed, 30 Sep 2015 03:36:42 +0000 (UTC) (envelope-from markj@FreeBSD.org) Received: from repo.freebsd.org ([127.0.1.70]) by repo.freebsd.org (8.15.2/8.15.2) with ESMTP id t8U3agnR046757; Wed, 30 Sep 2015 03:36:42 GMT (envelope-from markj@FreeBSD.org) Received: (from markj@localhost) by repo.freebsd.org (8.15.2/8.15.2/Submit) id t8U3ag5x046756; Wed, 30 Sep 2015 03:36:42 GMT (envelope-from markj@FreeBSD.org) Message-Id: <201509300336.t8U3ag5x046756@repo.freebsd.org> X-Authentication-Warning: repo.freebsd.org: markj set sender to markj@FreeBSD.org using -f From: Mark Johnston Date: Wed, 30 Sep 2015 03:36:42 +0000 (UTC) To: src-committers@freebsd.org, svn-src-all@freebsd.org, svn-src-stable@freebsd.org, svn-src-stable-10@freebsd.org Subject: svn commit: r288411 - stable/10/share/man/man9 X-SVN-Group: stable-10 MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit X-BeenThere: svn-src-stable@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: SVN commit messages for all the -stable branches of the src tree List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 30 Sep 2015 03:36:42 -0000 Author: markj Date: Wed Sep 30 03:36:41 2015 New Revision: 288411 URL: https://svnweb.freebsd.org/changeset/base/288411 Log: MFC r288278: Document the interface for applying advice up to the end of a file. Modified: stable/10/share/man/man9/VOP_ADVISE.9 Directory Properties: stable/10/ (props changed) Modified: stable/10/share/man/man9/VOP_ADVISE.9 ============================================================================== --- stable/10/share/man/man9/VOP_ADVISE.9 Wed Sep 30 03:35:58 2015 (r288410) +++ stable/10/share/man/man9/VOP_ADVISE.9 Wed Sep 30 03:36:41 2015 (r288411) @@ -27,7 +27,7 @@ .\" .\" $FreeBSD$ .\" -.Dd October 3, 2013 +.Dd September 26, 2015 .Dt VOP_ADVISE 9 .Os .Sh NAME @@ -52,6 +52,9 @@ The vnode of the file. The start of the range of file data. .It Fa end The end of the range of file data. +A value of +.Dv OFF_MAX +indicates that the advice is to be applied up to the end of the file. .It Fa advice The type of operation to apply to the file data. Possible values are: From owner-svn-src-stable@freebsd.org Thu Oct 1 00:44:45 2015 Return-Path: Delivered-To: svn-src-stable@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id E7535A0CA17; Thu, 1 Oct 2015 00:44:45 +0000 (UTC) (envelope-from delphij@FreeBSD.org) Received: from repo.freebsd.org (repo.freebsd.org [IPv6:2001:1900:2254:2068::e6a:0]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id D7EFF147C; Thu, 1 Oct 2015 00:44:45 +0000 (UTC) (envelope-from delphij@FreeBSD.org) Received: from repo.freebsd.org ([127.0.1.70]) by repo.freebsd.org (8.15.2/8.15.2) with ESMTP id t910ijvP084767; Thu, 1 Oct 2015 00:44:45 GMT (envelope-from delphij@FreeBSD.org) Received: (from delphij@localhost) by repo.freebsd.org (8.15.2/8.15.2/Submit) id t910ijCM084766; Thu, 1 Oct 2015 00:44:45 GMT (envelope-from delphij@FreeBSD.org) Message-Id: <201510010044.t910ijCM084766@repo.freebsd.org> X-Authentication-Warning: repo.freebsd.org: delphij set sender to delphij@FreeBSD.org using -f From: Xin LI Date: Thu, 1 Oct 2015 00:44:45 +0000 (UTC) To: src-committers@freebsd.org, svn-src-all@freebsd.org, svn-src-stable@freebsd.org, svn-src-stable-10@freebsd.org Subject: svn commit: r288434 - stable/10/usr.sbin/bhyve X-SVN-Group: stable-10 MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit X-BeenThere: svn-src-stable@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: SVN commit messages for all the -stable branches of the src tree List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 01 Oct 2015 00:44:46 -0000 Author: delphij Date: Thu Oct 1 00:44:45 2015 New Revision: 288434 URL: https://svnweb.freebsd.org/changeset/base/288434 Log: MFC r287927: Use strlcpy() instead of strncpy() because subsequent mkstemps expects the string be nul-terminated. Reviewed by: neel Modified: stable/10/usr.sbin/bhyve/acpi.c Directory Properties: stable/10/ (props changed) Modified: stable/10/usr.sbin/bhyve/acpi.c ============================================================================== --- stable/10/usr.sbin/bhyve/acpi.c Thu Oct 1 00:34:38 2015 (r288433) +++ stable/10/usr.sbin/bhyve/acpi.c Thu Oct 1 00:44:45 2015 (r288434) @@ -790,10 +790,10 @@ basl_open(struct basl_fio *bf, int suffi err = 0; if (suffix) { - strncpy(bf->f_name, basl_stemplate, MAXPATHLEN); + strlcpy(bf->f_name, basl_stemplate, MAXPATHLEN); bf->fd = mkstemps(bf->f_name, strlen(BHYVE_ASL_SUFFIX)); } else { - strncpy(bf->f_name, basl_template, MAXPATHLEN); + strlcpy(bf->f_name, basl_template, MAXPATHLEN); bf->fd = mkstemp(bf->f_name); } From owner-svn-src-stable@freebsd.org Thu Oct 1 00:47:31 2015 Return-Path: Delivered-To: svn-src-stable@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 1EEAFA0CD2C; Thu, 1 Oct 2015 00:47:31 +0000 (UTC) (envelope-from gjb@FreeBSD.org) Received: from repo.freebsd.org (repo.freebsd.org [IPv6:2001:1900:2254:2068::e6a:0]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 0FCB8178C; Thu, 1 Oct 2015 00:47:31 +0000 (UTC) (envelope-from gjb@FreeBSD.org) Received: from repo.freebsd.org ([127.0.1.70]) by repo.freebsd.org (8.15.2/8.15.2) with ESMTP id t910lUnj084937; Thu, 1 Oct 2015 00:47:30 GMT (envelope-from gjb@FreeBSD.org) Received: (from gjb@localhost) by repo.freebsd.org (8.15.2/8.15.2/Submit) id t910lUhP084936; Thu, 1 Oct 2015 00:47:30 GMT (envelope-from gjb@FreeBSD.org) Message-Id: <201510010047.t910lUhP084936@repo.freebsd.org> X-Authentication-Warning: repo.freebsd.org: gjb set sender to gjb@FreeBSD.org using -f From: Glen Barber Date: Thu, 1 Oct 2015 00:47:30 +0000 (UTC) To: src-committers@freebsd.org, svn-src-all@freebsd.org, svn-src-stable@freebsd.org, svn-src-stable-10@freebsd.org Subject: svn commit: r288435 - stable/10/release X-SVN-Group: stable-10 MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit X-BeenThere: svn-src-stable@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: SVN commit messages for all the -stable branches of the src tree List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 01 Oct 2015 00:47:31 -0000 Author: gjb Date: Thu Oct 1 00:47:30 2015 New Revision: 288435 URL: https://svnweb.freebsd.org/changeset/base/288435 Log: MFC r288341, r288345, r288347: r288341: Honor VMFORMATS and VMSIZE if set in release.conf. [1] r288345: In followup to r288341, ensure VMFORMATS and VMSIZE are not set to empty values, which would result in nonintuitive build errors. r288347: Append VMFORMATS and VMSIZE to RELEASE_RMAKEFLAGS only if WITH_VMIMAGES is set. PR: 203420 [1] Sponsored by: The FreeBSD Foundation Modified: stable/10/release/release.sh Directory Properties: stable/10/ (props changed) Modified: stable/10/release/release.sh ============================================================================== --- stable/10/release/release.sh Thu Oct 1 00:44:45 2015 (r288434) +++ stable/10/release/release.sh Thu Oct 1 00:47:30 2015 (r288435) @@ -311,6 +311,18 @@ chroot_build_target() { # chroot_build_release(): Invoke the 'make release' target. chroot_build_release() { load_target_env + if [ ! -z "${WITH_VMIMAGES}" ]; then + if [ -z "${VMFORMATS}" ]; then + VMFORMATS="$(eval chroot ${CHROOTDIR} \ + make -C /usr/src/release -V VMFORMATS)" + fi + if [ -z "${VMSIZE}" ]; then + VMSIZE="$(eval chroot ${CHROOTDIR} \ + make -C /usr/src/release -V VMSIZE)" + fi + RELEASE_RMAKEFLAGS="${RELEASE_RMAKEFLAGS} \ + VMFORMATS=\"${VMFORMATS}\" VMSIZE=${VMSIZE}" + fi eval chroot ${CHROOTDIR} make -C /usr/src/release \ ${RELEASE_RMAKEFLAGS} release eval chroot ${CHROOTDIR} make -C /usr/src/release \ From owner-svn-src-stable@freebsd.org Thu Oct 1 17:09:23 2015 Return-Path: Delivered-To: svn-src-stable@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id B8ED5A0C067; Thu, 1 Oct 2015 17:09:23 +0000 (UTC) (envelope-from jhb@FreeBSD.org) Received: from repo.freebsd.org (repo.freebsd.org [IPv6:2001:1900:2254:2068::e6a:0]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id A779A1ABE; Thu, 1 Oct 2015 17:09:23 +0000 (UTC) (envelope-from jhb@FreeBSD.org) Received: from repo.freebsd.org ([127.0.1.70]) by repo.freebsd.org (8.15.2/8.15.2) with ESMTP id t91H9NqZ098458; Thu, 1 Oct 2015 17:09:23 GMT (envelope-from jhb@FreeBSD.org) Received: (from jhb@localhost) by repo.freebsd.org (8.15.2/8.15.2/Submit) id t91H9Lbs098447; Thu, 1 Oct 2015 17:09:21 GMT (envelope-from jhb@FreeBSD.org) Message-Id: <201510011709.t91H9Lbs098447@repo.freebsd.org> X-Authentication-Warning: repo.freebsd.org: jhb set sender to jhb@FreeBSD.org using -f From: John Baldwin Date: Thu, 1 Oct 2015 17:09:21 +0000 (UTC) To: src-committers@freebsd.org, svn-src-all@freebsd.org, svn-src-stable@freebsd.org, svn-src-stable-10@freebsd.org Subject: svn commit: r288453 - in stable/10: lib/libutil sys/sys sys/vm usr.bin/vmstat X-SVN-Group: stable-10 MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit X-BeenThere: svn-src-stable@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: SVN commit messages for all the -stable branches of the src tree List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 01 Oct 2015 17:09:23 -0000 Author: jhb Date: Thu Oct 1 17:09:20 2015 New Revision: 288453 URL: https://svnweb.freebsd.org/changeset/base/288453 Log: MFC 283624,283630: Export a list of VM objects in the system via a sysctl. The list can be examined via 'vmstat -o'. It can be used to determine which files are using physical pages of memory and how much each is using. Added: stable/10/lib/libutil/kinfo_getvmobject.3 - copied unchanged from r283624, head/lib/libutil/kinfo_getvmobject.3 stable/10/lib/libutil/kinfo_getvmobject.c - copied unchanged from r283624, head/lib/libutil/kinfo_getvmobject.c Modified: stable/10/lib/libutil/Makefile stable/10/lib/libutil/libutil.h stable/10/sys/sys/user.h stable/10/sys/vm/vm_object.c stable/10/usr.bin/vmstat/vmstat.8 stable/10/usr.bin/vmstat/vmstat.c Directory Properties: stable/10/ (props changed) Modified: stable/10/lib/libutil/Makefile ============================================================================== --- stable/10/lib/libutil/Makefile Thu Oct 1 16:59:07 2015 (r288452) +++ stable/10/lib/libutil/Makefile Thu Oct 1 17:09:20 2015 (r288453) @@ -10,7 +10,8 @@ SHLIB_MAJOR= 9 SRCS= _secure_path.c auth.c expand_number.c flopen.c fparseln.c gr_util.c \ hexdump.c humanize_number.c kinfo_getfile.c kinfo_getfile.c \ - kinfo_getallproc.c kinfo_getproc.c kinfo_getvmmap.c kld.c \ + kinfo_getallproc.c kinfo_getproc.c kinfo_getvmmap.c \ + kinfo_getvmobject.c kld.c \ login_auth.c login_cap.c \ login_class.c login_crypt.c login_ok.c login_times.c login_tty.c \ pidfile.c property.c pty.c pw_util.c quotafile.c realhostname.c \ @@ -27,7 +28,8 @@ CFLAGS+= -I${.CURDIR} -I${.CURDIR}/../li MAN+= expand_number.3 flopen.3 fparseln.3 hexdump.3 \ humanize_number.3 kinfo_getallproc.3 kinfo_getfile.3 \ - kinfo_getproc.3 kinfo_getvmmap.3 kld.3 login_auth.3 login_cap.3 \ + kinfo_getproc.3 kinfo_getvmmap.3 kinfo_getvmobject.3 kld.3 \ + login_auth.3 login_cap.3 \ login_class.3 login_ok.3 login_times.3 login_tty.3 pidfile.3 \ property.3 pty.3 quotafile.3 realhostname.3 realhostname_sa.3 \ _secure_path.3 trimdomain.3 uucplock.3 pw_util.3 Copied: stable/10/lib/libutil/kinfo_getvmobject.3 (from r283624, head/lib/libutil/kinfo_getvmobject.3) ============================================================================== --- /dev/null 00:00:00 1970 (empty, because file is newly added) +++ stable/10/lib/libutil/kinfo_getvmobject.3 Thu Oct 1 17:09:20 2015 (r288453, copy of r283624, head/lib/libutil/kinfo_getvmobject.3) @@ -0,0 +1,74 @@ +.\" +.\" Copyright (c) 2015 John Baldwin +.\" All rights reserved. +.\" +.\" Redistribution and use in source and binary forms, with or without +.\" modification, are permitted provided that the following conditions +.\" are met: +.\" 1. Redistributions of source code must retain the above copyright +.\" notice, this list of conditions and the following disclaimer. +.\" 2. Redistributions in binary form must reproduce the above copyright +.\" notice, this list of conditions and the following disclaimer in the +.\" documentation and/or other materials provided with the distribution. +.\" +.\" THIS SOFTWARE IS PROVIDED BY THE AUTHOR AND CONTRIBUTORS ``AS IS'' AND +.\" ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE +.\" IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE +.\" ARE DISCLAIMED. IN NO EVENT SHALL THE AUTHOR OR CONTRIBUTORS BE LIABLE +.\" FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL +.\" DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS +.\" OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) +.\" HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT +.\" LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY +.\" OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF +.\" SUCH DAMAGE. +.\" +.\" $FreeBSD$ +.\" +.Dd May 27, 2015 +.Dt KINFO_GETVMOBJECT 3 +.Os +.Sh NAME +.Nm kinfo_getvmobject +.Nd function for getting system-wide memory information +.Sh LIBRARY +.Lb libutil +.Sh SYNOPSIS +.In sys/types.h +.In sys/user.h +.In libutil.h +.Ft struct kinfo_vmobject * +.Fn kinfo_getvmobject "int *cntp" +.Sh DESCRIPTION +This function is used to obtain information about the objects using memory +in the system. +.Pp +The +.Ar cntp +argument allows the caller to know how many records are returned. +.Pp +This function is a wrapper around the +.Dq vm.objects +.Xr sysctl 3 +MIB. +While the kernel returns a packed structure, this function expands the +data into a fixed record format. +.Sh RETURN VALUES +On success the +.Fn kinfo_getvmobject +function returns a pointer to an array of +.Vt struct kinfo_vmobject +structures as defined by +.In sys/user.h . +The array is allocated by an internal call to +.Xr malloc 3 +and must be freed by the caller with a call to +.Xr free 3 . +On failure the +.Fn kinfo_getvmobject +function returns +.Dv NULL . +.Sh SEE ALSO +.Xr free 3 , +.Xr kinfo_getvmmap 3 , +.Xr malloc 3 Copied: stable/10/lib/libutil/kinfo_getvmobject.c (from r283624, head/lib/libutil/kinfo_getvmobject.c) ============================================================================== --- /dev/null 00:00:00 1970 (empty, because file is newly added) +++ stable/10/lib/libutil/kinfo_getvmobject.c Thu Oct 1 17:09:20 2015 (r288453, copy of r283624, head/lib/libutil/kinfo_getvmobject.c) @@ -0,0 +1,94 @@ +/* + * Copyright (c) 2013 Hudson River Trading LLC + * Written by: John H. Baldwin + * All rights reserved. + * + * Redistribution and use in source and binary forms, with or without + * modification, are permitted provided that the following conditions + * are met: + * 1. Redistributions of source code must retain the above copyright + * notice, this list of conditions and the following disclaimer. + * 2. Redistributions in binary form must reproduce the above copyright + * notice, this list of conditions and the following disclaimer in the + * documentation and/or other materials provided with the distribution. + * + * THIS SOFTWARE IS PROVIDED BY THE AUTHOR AND CONTRIBUTORS ``AS IS'' AND + * ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE + * IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE + * ARE DISCLAIMED. IN NO EVENT SHALL THE AUTHOR OR CONTRIBUTORS BE LIABLE + * FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL + * DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS + * OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) + * HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT + * LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY + * OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF + * SUCH DAMAGE. + */ + +#include +__FBSDID("$FreeBSD$"); + +#include +#include +#include +#include +#include + +#include "libutil.h" + +struct kinfo_vmobject * +kinfo_getvmobject(int *cntp) +{ + char *buf, *bp, *ep; + struct kinfo_vmobject *kvo, *list, *kp; + size_t len; + int cnt, i; + + buf = NULL; + for (i = 0; i < 3; i++) { + if (sysctlbyname("vm.objects", NULL, &len, NULL, 0) < 0) + return (NULL); + buf = reallocf(buf, len); + if (buf == NULL) + return (NULL); + if (sysctlbyname("vm.objects", buf, &len, NULL, 0) == 0) + goto unpack; + if (errno != ENOMEM) { + free(buf); + return (NULL); + } + } + free(buf); + return (NULL); + +unpack: + /* Count items */ + cnt = 0; + bp = buf; + ep = buf + len; + while (bp < ep) { + kvo = (struct kinfo_vmobject *)(uintptr_t)bp; + bp += kvo->kvo_structsize; + cnt++; + } + + list = calloc(cnt, sizeof(*list)); + if (list == NULL) { + free(buf); + return (NULL); + } + + /* Unpack */ + bp = buf; + kp = list; + while (bp < ep) { + kvo = (struct kinfo_vmobject *)(uintptr_t)bp; + memcpy(kp, kvo, kvo->kvo_structsize); + bp += kvo->kvo_structsize; + kp->kvo_structsize = sizeof(*kp); + kp++; + } + free(buf); + *cntp = cnt; + return (list); +} Modified: stable/10/lib/libutil/libutil.h ============================================================================== --- stable/10/lib/libutil/libutil.h Thu Oct 1 16:59:07 2015 (r288452) +++ stable/10/lib/libutil/libutil.h Thu Oct 1 17:09:20 2015 (r288453) @@ -102,6 +102,8 @@ struct kinfo_file * kinfo_getfile(pid_t _pid, int *_cntp); struct kinfo_vmentry * kinfo_getvmmap(pid_t _pid, int *_cntp); +struct kinfo_vmobject * + kinfo_getvmobject(int *_cntp); struct kinfo_proc * kinfo_getallproc(int *_cntp); struct kinfo_proc * Modified: stable/10/sys/sys/user.h ============================================================================== --- stable/10/sys/sys/user.h Thu Oct 1 16:59:07 2015 (r288452) +++ stable/10/sys/sys/user.h Thu Oct 1 17:09:20 2015 (r288453) @@ -483,6 +483,27 @@ struct kinfo_vmentry { }; /* + * The "vm.objects" sysctl provides a list of all VM objects in the system + * via an array of these entries. + */ +struct kinfo_vmobject { + int kvo_structsize; /* Variable size of record. */ + int kvo_type; /* Object type: KVME_TYPE_*. */ + uint64_t kvo_size; /* Object size in pages. */ + uint64_t kvo_vn_fileid; /* inode number if vnode. */ + uint32_t kvo_vn_fsid; /* dev_t of vnode location. */ + int kvo_ref_count; /* Reference count. */ + int kvo_shadow_count; /* Shadow count. */ + int kvo_memattr; /* Memory attribute. */ + uint64_t kvo_resident; /* Number of resident pages. */ + uint64_t kvo_active; /* Number of active pages. */ + uint64_t kvo_inactive; /* Number of inactive pages. */ + uint64_t _kvo_qspare[8]; + uint32_t _kvo_ispare[8]; + char kvo_path[PATH_MAX]; /* Pathname, if any. */ +}; + +/* * The KERN_PROC_KSTACK sysctl allows a process to dump the kernel stacks of * another process as a series of entries. Each stack is represented by a * series of symbol names and offsets as generated by stack_sbuf_print(9). Modified: stable/10/sys/vm/vm_object.c ============================================================================== --- stable/10/sys/vm/vm_object.c Thu Oct 1 16:59:07 2015 (r288452) +++ stable/10/sys/vm/vm_object.c Thu Oct 1 17:09:20 2015 (r288453) @@ -79,6 +79,7 @@ __FBSDID("$FreeBSD$"); #include #include #include +#include #include #include #include @@ -2269,6 +2270,142 @@ next_page: } } +static int +sysctl_vm_object_list(SYSCTL_HANDLER_ARGS) +{ + struct kinfo_vmobject kvo; + char *fullpath, *freepath; + struct vnode *vp; + struct vattr va; + vm_object_t obj; + vm_page_t m; + int count, error; + + if (req->oldptr == NULL) { + /* + * If an old buffer has not been provided, generate an + * estimate of the space needed for a subsequent call. + */ + mtx_lock(&vm_object_list_mtx); + count = 0; + TAILQ_FOREACH(obj, &vm_object_list, object_list) { + if (obj->type == OBJT_DEAD) + continue; + count++; + } + mtx_unlock(&vm_object_list_mtx); + return (SYSCTL_OUT(req, NULL, sizeof(struct kinfo_vmobject) * + count * 11 / 10)); + } + + error = 0; + + /* + * VM objects are type stable and are never removed from the + * list once added. This allows us to safely read obj->object_list + * after reacquiring the VM object lock. + */ + mtx_lock(&vm_object_list_mtx); + TAILQ_FOREACH(obj, &vm_object_list, object_list) { + if (obj->type == OBJT_DEAD) + continue; + VM_OBJECT_RLOCK(obj); + if (obj->type == OBJT_DEAD) { + VM_OBJECT_RUNLOCK(obj); + continue; + } + mtx_unlock(&vm_object_list_mtx); + kvo.kvo_size = ptoa(obj->size); + kvo.kvo_resident = obj->resident_page_count; + kvo.kvo_ref_count = obj->ref_count; + kvo.kvo_shadow_count = obj->shadow_count; + kvo.kvo_memattr = obj->memattr; + kvo.kvo_active = 0; + kvo.kvo_inactive = 0; + TAILQ_FOREACH(m, &obj->memq, listq) { + /* + * A page may belong to the object but be + * dequeued and set to PQ_NONE while the + * object lock is not held. This makes the + * reads of m->queue below racy, and we do not + * count pages set to PQ_NONE. However, this + * sysctl is only meant to give an + * approximation of the system anyway. + */ + if (m->queue == PQ_ACTIVE) + kvo.kvo_active++; + else if (m->queue == PQ_INACTIVE) + kvo.kvo_inactive++; + } + + kvo.kvo_vn_fileid = 0; + kvo.kvo_vn_fsid = 0; + freepath = NULL; + fullpath = ""; + vp = NULL; + switch (obj->type) { + case OBJT_DEFAULT: + kvo.kvo_type = KVME_TYPE_DEFAULT; + break; + case OBJT_VNODE: + kvo.kvo_type = KVME_TYPE_VNODE; + vp = obj->handle; + vref(vp); + break; + case OBJT_SWAP: + kvo.kvo_type = KVME_TYPE_SWAP; + break; + case OBJT_DEVICE: + kvo.kvo_type = KVME_TYPE_DEVICE; + break; + case OBJT_PHYS: + kvo.kvo_type = KVME_TYPE_PHYS; + break; + case OBJT_DEAD: + kvo.kvo_type = KVME_TYPE_DEAD; + break; + case OBJT_SG: + kvo.kvo_type = KVME_TYPE_SG; + break; + case OBJT_MGTDEVICE: + kvo.kvo_type = KVME_TYPE_MGTDEVICE; + break; + default: + kvo.kvo_type = KVME_TYPE_UNKNOWN; + break; + } + VM_OBJECT_RUNLOCK(obj); + if (vp != NULL) { + vn_fullpath(curthread, vp, &fullpath, &freepath); + vn_lock(vp, LK_SHARED | LK_RETRY); + if (VOP_GETATTR(vp, &va, curthread->td_ucred) == 0) { + kvo.kvo_vn_fileid = va.va_fileid; + kvo.kvo_vn_fsid = va.va_fsid; + } + vput(vp); + } + + strlcpy(kvo.kvo_path, fullpath, sizeof(kvo.kvo_path)); + if (freepath != NULL) + free(freepath, M_TEMP); + + /* Pack record size down */ + kvo.kvo_structsize = offsetof(struct kinfo_vmobject, kvo_path) + + strlen(kvo.kvo_path) + 1; + kvo.kvo_structsize = roundup(kvo.kvo_structsize, + sizeof(uint64_t)); + error = SYSCTL_OUT(req, &kvo, kvo.kvo_structsize); + mtx_lock(&vm_object_list_mtx); + if (error) + break; + } + mtx_unlock(&vm_object_list_mtx); + return (error); +} +SYSCTL_PROC(_vm, OID_AUTO, objects, CTLTYPE_STRUCT | CTLFLAG_RW | CTLFLAG_SKIP | + CTLFLAG_MPSAFE, NULL, 0, sysctl_vm_object_list, "S,kinfo_vmobject", + "List of VM objects"); + #include "opt_ddb.h" #ifdef DDB #include Modified: stable/10/usr.bin/vmstat/vmstat.8 ============================================================================== --- stable/10/usr.bin/vmstat/vmstat.8 Thu Oct 1 16:59:07 2015 (r288452) +++ stable/10/usr.bin/vmstat/vmstat.8 Thu Oct 1 17:09:20 2015 (r288453) @@ -37,7 +37,7 @@ .Sh SYNOPSIS .Nm .\" .Op Fl fimst -.Op Fl afHhimPsz +.Op Fl afHhimoPsz .Op Fl M Ar core Op Fl N Ar system .Op Fl c Ar count .Op Fl n Ar devs @@ -119,6 +119,9 @@ Report on the usage of kernel dynamic me by type. .It Fl n Change the maximum number of disks to display from the default of 2. +.It Fl o +Display a list of virtual memory objects in the system and the resident +memory used by each object. .It Fl P Report per-cpu system/user/idle cpu statistics. .It Fl p Modified: stable/10/usr.bin/vmstat/vmstat.c ============================================================================== --- stable/10/usr.bin/vmstat/vmstat.c Thu Oct 1 16:59:07 2015 (r288452) +++ stable/10/usr.bin/vmstat/vmstat.c Thu Oct 1 17:09:20 2015 (r288453) @@ -53,6 +53,7 @@ __FBSDID("$FreeBSD$"); #include #include #include +#include #include #include @@ -143,12 +144,14 @@ static kvm_t *kd; #define TIMESTAT 0x10 #define VMSTAT 0x20 #define ZMEMSTAT 0x40 +#define OBJSTAT 0x80 static void cpustats(void); static void pcpustats(int, u_long, int); static void devstats(void); static void doforkst(void); static void dointr(void); +static void doobjstat(void); static void dosum(void); static void dovmstat(unsigned int, int); static void domemstat_malloc(void); @@ -181,7 +184,7 @@ main(int argc, char *argv[]) interval = reps = todo = 0; maxshowdevs = 2; hflag = isatty(1); - while ((c = getopt(argc, argv, "ac:fhHiM:mN:n:Pp:stw:z")) != -1) { + while ((c = getopt(argc, argv, "ac:fhHiM:mN:n:oPp:stw:z")) != -1) { switch (c) { case 'a': aflag++; @@ -220,6 +223,9 @@ main(int argc, char *argv[]) errx(1, "number of devices %d is < 0", maxshowdevs); break; + case 'o': + todo |= OBJSTAT; + break; case 'p': if (devstat_buildmatch(optarg, &matches, &num_matches) != 0) errx(1, "%s", devstat_errbuf); @@ -310,6 +316,8 @@ main(int argc, char *argv[]) domemstat_zone(); if (todo & SUMSTAT) dosum(); + if (todo & OBJSTAT) + doobjstat(); #ifdef notyet if (todo & TIMESTAT) dotimes(); @@ -1299,6 +1307,129 @@ domemstat_zone(void) printf("\n"); } +static void +display_object(struct kinfo_vmobject *kvo) +{ + const char *str; + + printf("%5jd ", (uintmax_t)kvo->kvo_resident); + printf("%5jd ", (uintmax_t)kvo->kvo_active); + printf("%5jd ", (uintmax_t)kvo->kvo_inactive); + printf("%3d ", kvo->kvo_ref_count); + printf("%3d ", kvo->kvo_shadow_count); + switch (kvo->kvo_memattr) { +#ifdef VM_MEMATTR_UNCACHEABLE + case VM_MEMATTR_UNCACHEABLE: + str = "UC"; + break; +#endif +#ifdef VM_MEMATTR_WRITE_COMBINING + case VM_MEMATTR_WRITE_COMBINING: + str = "WC"; + break; +#endif +#ifdef VM_MEMATTR_WRITE_THROUGH + case VM_MEMATTR_WRITE_THROUGH: + str = "WT"; + break; +#endif +#ifdef VM_MEMATTR_WRITE_PROTECTED + case VM_MEMATTR_WRITE_PROTECTED: + str = "WP"; + break; +#endif +#ifdef VM_MEMATTR_WRITE_BACK + case VM_MEMATTR_WRITE_BACK: + str = "WB"; + break; +#endif +#ifdef VM_MEMATTR_WEAK_UNCACHEABLE + case VM_MEMATTR_WEAK_UNCACHEABLE: + str = "UC-"; + break; +#endif +#ifdef VM_MEMATTR_WB_WA + case VM_MEMATTR_WB_WA: + str = "WB"; + break; +#endif +#ifdef VM_MEMATTR_NOCACHE + case VM_MEMATTR_NOCACHE: + str = "NC"; + break; +#endif +#ifdef VM_MEMATTR_DEVICE + case VM_MEMATTR_DEVICE: + str = "DEV"; + break; +#endif +#ifdef VM_MEMATTR_CACHEABLE + case VM_MEMATTR_CACHEABLE: + str = "C"; + break; +#endif +#ifdef VM_MEMATTR_PREFETCHABLE + case VM_MEMATTR_PREFETCHABLE: + str = "PRE"; + break; +#endif + default: + str = "??"; + break; + } + printf("%-3s ", str); + switch (kvo->kvo_type) { + case KVME_TYPE_NONE: + str = "--"; + break; + case KVME_TYPE_DEFAULT: + str = "df"; + break; + case KVME_TYPE_VNODE: + str = "vn"; + break; + case KVME_TYPE_SWAP: + str = "sw"; + break; + case KVME_TYPE_DEVICE: + str = "dv"; + break; + case KVME_TYPE_PHYS: + str = "ph"; + break; + case KVME_TYPE_DEAD: + str = "dd"; + break; + case KVME_TYPE_SG: + str = "sg"; + break; + case KVME_TYPE_UNKNOWN: + default: + str = "??"; + break; + } + printf("%-2s ", str); + printf("%-s\n", kvo->kvo_path); +} + +static void +doobjstat(void) +{ + struct kinfo_vmobject *kvo; + int cnt, i; + + kvo = kinfo_getvmobject(&cnt); + if (kvo == NULL) { + warn("Failed to fetch VM object list"); + return; + } + printf("%5s %5s %5s %3s %3s %3s %2s %s\n", "RES", "ACT", "INACT", + "REF", "SHD", "CM", "TP", "PATH"); + for (i = 0; i < cnt; i++) + display_object(&kvo[i]); + free(kvo); +} + /* * kread reads something from the kernel, given its nlist index. */ @@ -1351,7 +1482,7 @@ static void usage(void) { (void)fprintf(stderr, "%s%s", - "usage: vmstat [-afHhimPsz] [-M core [-N system]] [-c count] [-n devs]\n", + "usage: vmstat [-afHhimoPsz] [-M core [-N system]] [-c count] [-n devs]\n", " [-p type,if,pass] [-w wait] [disks] [wait [count]]\n"); exit(1); } From owner-svn-src-stable@freebsd.org Thu Oct 1 20:49:11 2015 Return-Path: Delivered-To: svn-src-stable@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id C7C77A0C93E; Thu, 1 Oct 2015 20:49:11 +0000 (UTC) (envelope-from jhb@FreeBSD.org) Received: from repo.freebsd.org (repo.freebsd.org [IPv6:2001:1900:2254:2068::e6a:0]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id B8AC518CF; Thu, 1 Oct 2015 20:49:11 +0000 (UTC) (envelope-from jhb@FreeBSD.org) Received: from repo.freebsd.org ([127.0.1.70]) by repo.freebsd.org (8.15.2/8.15.2) with ESMTP id t91KnBEu089772; Thu, 1 Oct 2015 20:49:11 GMT (envelope-from jhb@FreeBSD.org) Received: (from jhb@localhost) by repo.freebsd.org (8.15.2/8.15.2/Submit) id t91KnBSJ089771; Thu, 1 Oct 2015 20:49:11 GMT (envelope-from jhb@FreeBSD.org) Message-Id: <201510012049.t91KnBSJ089771@repo.freebsd.org> X-Authentication-Warning: repo.freebsd.org: jhb set sender to jhb@FreeBSD.org using -f From: John Baldwin Date: Thu, 1 Oct 2015 20:49:11 +0000 (UTC) To: src-committers@freebsd.org, svn-src-all@freebsd.org, svn-src-stable@freebsd.org, svn-src-stable-9@freebsd.org Subject: svn commit: r288460 - in stable: 10/sys/dev/hwpmc 9/sys/dev/hwpmc X-SVN-Group: stable-9 MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit X-BeenThere: svn-src-stable@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: SVN commit messages for all the -stable branches of the src tree List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 01 Oct 2015 20:49:11 -0000 Author: jhb Date: Thu Oct 1 20:49:10 2015 New Revision: 288460 URL: https://svnweb.freebsd.org/changeset/base/288460 Log: MFC 283121: Use the proper mask when reloading sampling PMCs for Core CPUs. Modified: stable/9/sys/dev/hwpmc/hwpmc_core.c Directory Properties: stable/9/sys/ (props changed) stable/9/sys/dev/ (props changed) Changes in other areas also in this revision: Modified: stable/10/sys/dev/hwpmc/hwpmc_core.c Directory Properties: stable/10/ (props changed) Modified: stable/9/sys/dev/hwpmc/hwpmc_core.c ============================================================================== --- stable/9/sys/dev/hwpmc/hwpmc_core.c Thu Oct 1 19:48:10 2015 (r288459) +++ stable/9/sys/dev/hwpmc/hwpmc_core.c Thu Oct 1 20:49:10 2015 (r288460) @@ -2458,7 +2458,7 @@ core_intr(int cpu, struct trapframe *tf) TRAPF_USERMODE(tf)); v = pm->pm_sc.pm_reloadcount; - v = iaf_reload_count_to_perfctr_value(v); + v = iap_reload_count_to_perfctr_value(v); /* * Stop the counter, reload it but only restart it if From owner-svn-src-stable@freebsd.org Thu Oct 1 20:49:11 2015 Return-Path: Delivered-To: svn-src-stable@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 40FA9A0C937; Thu, 1 Oct 2015 20:49:11 +0000 (UTC) (envelope-from jhb@FreeBSD.org) Received: from repo.freebsd.org (repo.freebsd.org [IPv6:2001:1900:2254:2068::e6a:0]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 31A8618CE; Thu, 1 Oct 2015 20:49:11 +0000 (UTC) (envelope-from jhb@FreeBSD.org) Received: from repo.freebsd.org ([127.0.1.70]) by repo.freebsd.org (8.15.2/8.15.2) with ESMTP id t91KnBKD089766; Thu, 1 Oct 2015 20:49:11 GMT (envelope-from jhb@FreeBSD.org) Received: (from jhb@localhost) by repo.freebsd.org (8.15.2/8.15.2/Submit) id t91KnBcg089765; Thu, 1 Oct 2015 20:49:11 GMT (envelope-from jhb@FreeBSD.org) Message-Id: <201510012049.t91KnBcg089765@repo.freebsd.org> X-Authentication-Warning: repo.freebsd.org: jhb set sender to jhb@FreeBSD.org using -f From: John Baldwin Date: Thu, 1 Oct 2015 20:49:11 +0000 (UTC) To: src-committers@freebsd.org, svn-src-all@freebsd.org, svn-src-stable@freebsd.org, svn-src-stable-10@freebsd.org Subject: svn commit: r288460 - in stable: 10/sys/dev/hwpmc 9/sys/dev/hwpmc X-SVN-Group: stable-10 MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit X-BeenThere: svn-src-stable@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: SVN commit messages for all the -stable branches of the src tree List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 01 Oct 2015 20:49:11 -0000 Author: jhb Date: Thu Oct 1 20:49:10 2015 New Revision: 288460 URL: https://svnweb.freebsd.org/changeset/base/288460 Log: MFC 283121: Use the proper mask when reloading sampling PMCs for Core CPUs. Modified: stable/10/sys/dev/hwpmc/hwpmc_core.c Directory Properties: stable/10/ (props changed) Changes in other areas also in this revision: Modified: stable/9/sys/dev/hwpmc/hwpmc_core.c Directory Properties: stable/9/sys/ (props changed) stable/9/sys/dev/ (props changed) Modified: stable/10/sys/dev/hwpmc/hwpmc_core.c ============================================================================== --- stable/10/sys/dev/hwpmc/hwpmc_core.c Thu Oct 1 19:48:10 2015 (r288459) +++ stable/10/sys/dev/hwpmc/hwpmc_core.c Thu Oct 1 20:49:10 2015 (r288460) @@ -2572,7 +2572,7 @@ core_intr(int cpu, struct trapframe *tf) TRAPF_USERMODE(tf)); v = pm->pm_sc.pm_reloadcount; - v = iaf_reload_count_to_perfctr_value(v); + v = iap_reload_count_to_perfctr_value(v); /* * Stop the counter, reload it but only restart it if From owner-svn-src-stable@freebsd.org Thu Oct 1 20:54:20 2015 Return-Path: Delivered-To: svn-src-stable@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 74EBCA0CE9F; Thu, 1 Oct 2015 20:54:20 +0000 (UTC) (envelope-from jhb@FreeBSD.org) Received: from repo.freebsd.org (repo.freebsd.org [IPv6:2001:1900:2254:2068::e6a:0]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 596EE1E9A; Thu, 1 Oct 2015 20:54:20 +0000 (UTC) (envelope-from jhb@FreeBSD.org) Received: from repo.freebsd.org ([127.0.1.70]) by repo.freebsd.org (8.15.2/8.15.2) with ESMTP id t91KsKkf093633; Thu, 1 Oct 2015 20:54:20 GMT (envelope-from jhb@FreeBSD.org) Received: (from jhb@localhost) by repo.freebsd.org (8.15.2/8.15.2/Submit) id t91KsKtk093632; Thu, 1 Oct 2015 20:54:20 GMT (envelope-from jhb@FreeBSD.org) Message-Id: <201510012054.t91KsKtk093632@repo.freebsd.org> X-Authentication-Warning: repo.freebsd.org: jhb set sender to jhb@FreeBSD.org using -f From: John Baldwin Date: Thu, 1 Oct 2015 20:54:20 +0000 (UTC) To: src-committers@freebsd.org, svn-src-all@freebsd.org, svn-src-stable@freebsd.org, svn-src-stable-10@freebsd.org Subject: svn commit: r288461 - stable/10/sys/x86/acpica X-SVN-Group: stable-10 MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit X-BeenThere: svn-src-stable@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: SVN commit messages for all the -stable branches of the src tree List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 01 Oct 2015 20:54:20 -0000 Author: jhb Date: Thu Oct 1 20:54:19 2015 New Revision: 288461 URL: https://svnweb.freebsd.org/changeset/base/288461 Log: MFC 284175: Handle X2APIC entries in the MADT for APICs with an ID < 255. At least one BIOS has been seen to include such entries even though the relevant specs require that X2APIC entries only be used for CPUs with an APIC ID >= 255. This was tested on a system with "plain" local APIC entries in the MADT to ensure no regressions, but it has not yet been tested on a system with X2APIC entries in the MADT. Currently such systems do not boot at all, and with this change they might now boot correctly. Modified: stable/10/sys/x86/acpica/madt.c Directory Properties: stable/10/ (props changed) Modified: stable/10/sys/x86/acpica/madt.c ============================================================================== --- stable/10/sys/x86/acpica/madt.c Thu Oct 1 20:49:10 2015 (r288460) +++ stable/10/sys/x86/acpica/madt.c Thu Oct 1 20:54:19 2015 (r288461) @@ -53,8 +53,8 @@ static struct { } *ioapics; static struct lapic_info { - u_int la_enabled:1; - u_int la_acpi_id:8; + u_int la_enabled; + u_int la_acpi_id; } lapics[MAX_APIC_ID + 1]; static int madt_found_sci_override; @@ -220,34 +220,48 @@ madt_walk_table(acpi_subtable_handler *h } static void +madt_add_cpu(u_int acpi_id, u_int apic_id, u_int flags) +{ + struct lapic_info *la; + + /* + * The MADT does not include a BSP flag, so we have to let the + * MP code figure out which CPU is the BSP on its own. + */ + if (bootverbose) + printf("MADT: Found CPU APIC ID %u ACPI ID %u: %s\n", + apic_id, acpi_id, flags & ACPI_MADT_ENABLED ? + "enabled" : "disabled"); + if (!(flags & ACPI_MADT_ENABLED)) + return; + if (apic_id > MAX_APIC_ID) { + printf("MADT: Ignoring local APIC ID %u (too high)\n", + apic_id); + return; + } + + la = &lapics[apic_id]; + KASSERT(la->la_enabled == 0, ("Duplicate local APIC ID %u", apic_id)); + la->la_enabled = 1; + la->la_acpi_id = acpi_id; + lapic_create(apic_id, 0); +} + +static void madt_probe_cpus_handler(ACPI_SUBTABLE_HEADER *entry, void *arg) { ACPI_MADT_LOCAL_APIC *proc; - struct lapic_info *la; + ACPI_MADT_LOCAL_X2APIC *x2apic; switch (entry->Type) { case ACPI_MADT_TYPE_LOCAL_APIC: - /* - * The MADT does not include a BSP flag, so we have to - * let the MP code figure out which CPU is the BSP on - * its own. - */ proc = (ACPI_MADT_LOCAL_APIC *)entry; - if (bootverbose) - printf("MADT: Found CPU APIC ID %u ACPI ID %u: %s\n", - proc->Id, proc->ProcessorId, - (proc->LapicFlags & ACPI_MADT_ENABLED) ? - "enabled" : "disabled"); - if (!(proc->LapicFlags & ACPI_MADT_ENABLED)) - break; - if (proc->Id > MAX_APIC_ID) - panic("%s: CPU ID %u too high", __func__, proc->Id); - la = &lapics[proc->Id]; - KASSERT(la->la_enabled == 0, - ("Duplicate local APIC ID %u", proc->Id)); - la->la_enabled = 1; - la->la_acpi_id = proc->ProcessorId; - lapic_create(proc->Id, 0); + madt_add_cpu(proc->ProcessorId, proc->Id, proc->LapicFlags); + break; + case ACPI_MADT_TYPE_LOCAL_X2APIC: + x2apic = (ACPI_MADT_LOCAL_X2APIC *)entry; + madt_add_cpu(x2apic->Uid, x2apic->LocalApicId, + x2apic->LapicFlags); break; } } @@ -503,29 +517,44 @@ madt_parse_nmi(ACPI_MADT_NMI_SOURCE *nmi * Parse an entry for an NMI routed to a local APIC LVT pin. */ static void -madt_parse_local_nmi(ACPI_MADT_LOCAL_APIC_NMI *nmi) +madt_handle_local_nmi(u_int acpi_id, UINT8 Lint, UINT16 IntiFlags) { u_int apic_id, pin; - if (nmi->ProcessorId == 0xff) + if (acpi_id == 0xffffffff) apic_id = APIC_ID_ALL; - else if (madt_find_cpu(nmi->ProcessorId, &apic_id) != 0) { + else if (madt_find_cpu(acpi_id, &apic_id) != 0) { if (bootverbose) printf("MADT: Ignoring local NMI routed to " - "ACPI CPU %u\n", nmi->ProcessorId); + "ACPI CPU %u\n", acpi_id); return; } - if (nmi->Lint == 0) + if (Lint == 0) pin = APIC_LVT_LINT0; else pin = APIC_LVT_LINT1; lapic_set_lvt_mode(apic_id, pin, APIC_LVT_DM_NMI); - if (!(nmi->IntiFlags & ACPI_MADT_TRIGGER_CONFORMS)) + if (!(IntiFlags & ACPI_MADT_TRIGGER_CONFORMS)) lapic_set_lvt_triggermode(apic_id, pin, - interrupt_trigger(nmi->IntiFlags, 0)); - if (!(nmi->IntiFlags & ACPI_MADT_POLARITY_CONFORMS)) + interrupt_trigger(IntiFlags, 0)); + if (!(IntiFlags & ACPI_MADT_POLARITY_CONFORMS)) lapic_set_lvt_polarity(apic_id, pin, - interrupt_polarity(nmi->IntiFlags, 0)); + interrupt_polarity(IntiFlags, 0)); +} + +static void +madt_parse_local_nmi(ACPI_MADT_LOCAL_APIC_NMI *nmi) +{ + + madt_handle_local_nmi(nmi->ProcessorId == 0xff ? 0xffffffff : + nmi->ProcessorId, nmi->Lint, nmi->IntiFlags); +} + +static void +madt_parse_local_x2apic_nmi(ACPI_MADT_LOCAL_X2APIC_NMI *nmi) +{ + + madt_handle_local_nmi(nmi->Uid, nmi->Lint, nmi->IntiFlags); } /* @@ -546,6 +575,10 @@ madt_parse_ints(ACPI_SUBTABLE_HEADER *en case ACPI_MADT_TYPE_LOCAL_APIC_NMI: madt_parse_local_nmi((ACPI_MADT_LOCAL_APIC_NMI *)entry); break; + case ACPI_MADT_TYPE_LOCAL_X2APIC_NMI: + madt_parse_local_x2apic_nmi( + (ACPI_MADT_LOCAL_X2APIC_NMI *)entry); + break; } } From owner-svn-src-stable@freebsd.org Thu Oct 1 21:28:59 2015 Return-Path: Delivered-To: svn-src-stable@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id E53B3A0E863; Thu, 1 Oct 2015 21:28:59 +0000 (UTC) (envelope-from jhb@freebsd.org) Received: from bigwig.baldwin.cx (bigwig.baldwin.cx [IPv6:2001:470:1f11:75::1]) (using TLSv1 with cipher DHE-RSA-CAMELLIA256-SHA (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id C1E84105D; Thu, 1 Oct 2015 21:28:59 +0000 (UTC) (envelope-from jhb@freebsd.org) Received: from ralph.baldwin.cx (c-73-231-226-104.hsd1.ca.comcast.net [73.231.226.104]) by bigwig.baldwin.cx (Postfix) with ESMTPSA id 9C7DDB913; Thu, 1 Oct 2015 17:28:57 -0400 (EDT) From: John Baldwin To: src-committers@freebsd.org Cc: svn-src-all@freebsd.org, svn-src-stable@freebsd.org, svn-src-stable-10@freebsd.org Subject: Re: svn commit: r288453 - in stable/10: lib/libutil sys/sys sys/vm usr.bin/vmstat Date: Thu, 01 Oct 2015 14:28:49 -0700 Message-ID: <2096646.C1FjOzzH6L@ralph.baldwin.cx> User-Agent: KMail/4.14.3 (FreeBSD/10.2-PRERELEASE; KDE/4.14.3; amd64; ; ) In-Reply-To: <201510011709.t91H9Lbs098447@repo.freebsd.org> References: <201510011709.t91H9Lbs098447@repo.freebsd.org> MIME-Version: 1.0 Content-Transfer-Encoding: 7Bit Content-Type: text/plain; charset="us-ascii" X-Greylist: Sender succeeded SMTP AUTH, not delayed by milter-greylist-4.2.7 (bigwig.baldwin.cx); Thu, 01 Oct 2015 17:28:57 -0400 (EDT) X-BeenThere: svn-src-stable@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: SVN commit messages for all the -stable branches of the src tree List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 01 Oct 2015 21:29:00 -0000 On Thursday, October 01, 2015 05:09:21 PM John Baldwin wrote: > Author: jhb > Date: Thu Oct 1 17:09:20 2015 > New Revision: 288453 > URL: https://svnweb.freebsd.org/changeset/base/288453 > > Log: > MFC 283624,283630: > Export a list of VM objects in the system via a sysctl. The list can be > examined via 'vmstat -o'. It can be used to determine which files are > using physical pages of memory and how much each is using. Note that userland sysctls in stable/10 are single-threaded. Thus, other programs that use sysctls will hang while this new sysctl runs. Resolving the pathnames of mapped files can take a while, especially if any NFS files are mapped (on the order of seconds). Programs like top(1), ps(1), fstat(1), vmstat(1), etc. all use sysctls to fetch information, so they will all hang while 'vmstat -o' runs. This does not mean the entire machine has locked up, but it can feel like it. User beware, etc. -- John Baldwin From owner-svn-src-stable@freebsd.org Thu Oct 1 21:54:45 2015 Return-Path: Delivered-To: svn-src-stable@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 1EBCDA0DCEE; Thu, 1 Oct 2015 21:54:45 +0000 (UTC) (envelope-from jhb@FreeBSD.org) Received: from repo.freebsd.org (repo.freebsd.org [IPv6:2001:1900:2254:2068::e6a:0]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id EAC3D13AA; Thu, 1 Oct 2015 21:54:44 +0000 (UTC) (envelope-from jhb@FreeBSD.org) Received: from repo.freebsd.org ([127.0.1.70]) by repo.freebsd.org (8.15.2/8.15.2) with ESMTP id t91LsiEG018839; Thu, 1 Oct 2015 21:54:44 GMT (envelope-from jhb@FreeBSD.org) Received: (from jhb@localhost) by repo.freebsd.org (8.15.2/8.15.2/Submit) id t91Lsise018837; Thu, 1 Oct 2015 21:54:44 GMT (envelope-from jhb@FreeBSD.org) Message-Id: <201510012154.t91Lsise018837@repo.freebsd.org> X-Authentication-Warning: repo.freebsd.org: jhb set sender to jhb@FreeBSD.org using -f From: John Baldwin Date: Thu, 1 Oct 2015 21:54:44 +0000 (UTC) To: src-committers@freebsd.org, svn-src-all@freebsd.org, svn-src-stable@freebsd.org, svn-src-stable-10@freebsd.org Subject: svn commit: r288463 - in stable: 10/sys/kern 9/sys/kern X-SVN-Group: stable-10 MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit X-BeenThere: svn-src-stable@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: SVN commit messages for all the -stable branches of the src tree List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 01 Oct 2015 21:54:45 -0000 Author: jhb Date: Thu Oct 1 21:54:43 2015 New Revision: 288463 URL: https://svnweb.freebsd.org/changeset/base/288463 Log: MFC 286256: kgdb uses td_oncpu to determine if a thread is running and should use a pcb from stoppcbs[] rather than the thread's PCB. However, exited threads retained td_oncpu from the last time they ran, and newborn threads had their CPU fields cleared to zero during fork and thread creation since they are in the set of fields zeroed when threads are setup. To fix, explicitly update the CPU fields for exiting threads in sched_throw() to reflect the switch out and reset the CPU fields for new threads in sched_fork_thread() to NOCPU. Modified: stable/10/sys/kern/sched_4bsd.c stable/10/sys/kern/sched_ule.c Directory Properties: stable/10/ (props changed) Changes in other areas also in this revision: Modified: stable/9/sys/kern/sched_4bsd.c stable/9/sys/kern/sched_ule.c Directory Properties: stable/9/sys/ (props changed) Modified: stable/10/sys/kern/sched_4bsd.c ============================================================================== --- stable/10/sys/kern/sched_4bsd.c Thu Oct 1 21:52:25 2015 (r288462) +++ stable/10/sys/kern/sched_4bsd.c Thu Oct 1 21:54:43 2015 (r288463) @@ -793,6 +793,8 @@ sched_fork_thread(struct thread *td, str { struct td_sched *ts; + childtd->td_oncpu = NOCPU; + childtd->td_lastcpu = NOCPU; childtd->td_estcpu = td->td_estcpu; childtd->td_lock = &sched_lock; childtd->td_cpuset = cpuset_ref(td->td_cpuset); @@ -1672,6 +1674,8 @@ sched_throw(struct thread *td) } else { lock_profile_release_lock(&sched_lock.lock_object); MPASS(td->td_lock == &sched_lock); + td->td_lastcpu = td->td_oncpu; + td->td_oncpu = NOCPU; } mtx_assert(&sched_lock, MA_OWNED); KASSERT(curthread->td_md.md_spinlock_count == 1, ("invalid count")); Modified: stable/10/sys/kern/sched_ule.c ============================================================================== --- stable/10/sys/kern/sched_ule.c Thu Oct 1 21:52:25 2015 (r288462) +++ stable/10/sys/kern/sched_ule.c Thu Oct 1 21:54:43 2015 (r288463) @@ -2071,6 +2071,8 @@ sched_fork_thread(struct thread *td, str */ ts = td->td_sched; ts2 = child->td_sched; + child->td_oncpu = NOCPU; + child->td_lastcpu = NOCPU; child->td_lock = TDQ_LOCKPTR(tdq); child->td_cpuset = cpuset_ref(td->td_cpuset); ts2->ts_cpu = ts->ts_cpu; @@ -2694,6 +2696,8 @@ sched_throw(struct thread *td) MPASS(td->td_lock == TDQ_LOCKPTR(tdq)); tdq_load_rem(tdq, td); lock_profile_release_lock(&TDQ_LOCKPTR(tdq)->lock_object); + td->td_lastcpu = td->td_oncpu; + td->td_oncpu = NOCPU; } KASSERT(curthread->td_md.md_spinlock_count == 1, ("invalid count")); newtd = choosethread(); From owner-svn-src-stable@freebsd.org Thu Oct 1 21:54:46 2015 Return-Path: Delivered-To: svn-src-stable@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 2402DA0DCF8; Thu, 1 Oct 2015 21:54:46 +0000 (UTC) (envelope-from jhb@FreeBSD.org) Received: from repo.freebsd.org (repo.freebsd.org [IPv6:2001:1900:2254:2068::e6a:0]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id EEB5813AB; Thu, 1 Oct 2015 21:54:45 +0000 (UTC) (envelope-from jhb@FreeBSD.org) Received: from repo.freebsd.org ([127.0.1.70]) by repo.freebsd.org (8.15.2/8.15.2) with ESMTP id t91LsjJv018846; Thu, 1 Oct 2015 21:54:45 GMT (envelope-from jhb@FreeBSD.org) Received: (from jhb@localhost) by repo.freebsd.org (8.15.2/8.15.2/Submit) id t91LsjM3018844; Thu, 1 Oct 2015 21:54:45 GMT (envelope-from jhb@FreeBSD.org) Message-Id: <201510012154.t91LsjM3018844@repo.freebsd.org> X-Authentication-Warning: repo.freebsd.org: jhb set sender to jhb@FreeBSD.org using -f From: John Baldwin Date: Thu, 1 Oct 2015 21:54:45 +0000 (UTC) To: src-committers@freebsd.org, svn-src-all@freebsd.org, svn-src-stable@freebsd.org, svn-src-stable-9@freebsd.org Subject: svn commit: r288463 - in stable: 10/sys/kern 9/sys/kern X-SVN-Group: stable-9 MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit X-BeenThere: svn-src-stable@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: SVN commit messages for all the -stable branches of the src tree List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 01 Oct 2015 21:54:46 -0000 Author: jhb Date: Thu Oct 1 21:54:43 2015 New Revision: 288463 URL: https://svnweb.freebsd.org/changeset/base/288463 Log: MFC 286256: kgdb uses td_oncpu to determine if a thread is running and should use a pcb from stoppcbs[] rather than the thread's PCB. However, exited threads retained td_oncpu from the last time they ran, and newborn threads had their CPU fields cleared to zero during fork and thread creation since they are in the set of fields zeroed when threads are setup. To fix, explicitly update the CPU fields for exiting threads in sched_throw() to reflect the switch out and reset the CPU fields for new threads in sched_fork_thread() to NOCPU. Modified: stable/9/sys/kern/sched_4bsd.c stable/9/sys/kern/sched_ule.c Directory Properties: stable/9/sys/ (props changed) Changes in other areas also in this revision: Modified: stable/10/sys/kern/sched_4bsd.c stable/10/sys/kern/sched_ule.c Directory Properties: stable/10/ (props changed) Modified: stable/9/sys/kern/sched_4bsd.c ============================================================================== --- stable/9/sys/kern/sched_4bsd.c Thu Oct 1 21:52:25 2015 (r288462) +++ stable/9/sys/kern/sched_4bsd.c Thu Oct 1 21:54:43 2015 (r288463) @@ -793,6 +793,8 @@ sched_fork_thread(struct thread *td, str { struct td_sched *ts; + childtd->td_oncpu = NOCPU; + childtd->td_lastcpu = NOCPU; childtd->td_estcpu = td->td_estcpu; childtd->td_lock = &sched_lock; childtd->td_cpuset = cpuset_ref(td->td_cpuset); @@ -1671,6 +1673,8 @@ sched_throw(struct thread *td) } else { lock_profile_release_lock(&sched_lock.lock_object); MPASS(td->td_lock == &sched_lock); + td->td_lastcpu = td->td_oncpu; + td->td_oncpu = NOCPU; } mtx_assert(&sched_lock, MA_OWNED); KASSERT(curthread->td_md.md_spinlock_count == 1, ("invalid count")); Modified: stable/9/sys/kern/sched_ule.c ============================================================================== --- stable/9/sys/kern/sched_ule.c Thu Oct 1 21:52:25 2015 (r288462) +++ stable/9/sys/kern/sched_ule.c Thu Oct 1 21:54:43 2015 (r288463) @@ -2035,6 +2035,8 @@ sched_fork_thread(struct thread *td, str */ ts = td->td_sched; ts2 = child->td_sched; + child->td_oncpu = NOCPU; + child->td_lastcpu = NOCPU; child->td_lock = TDQ_LOCKPTR(TDQ_SELF()); child->td_cpuset = cpuset_ref(td->td_cpuset); ts2->ts_cpu = ts->ts_cpu; @@ -2656,6 +2658,8 @@ sched_throw(struct thread *td) MPASS(td->td_lock == TDQ_LOCKPTR(tdq)); tdq_load_rem(tdq, td); lock_profile_release_lock(&TDQ_LOCKPTR(tdq)->lock_object); + td->td_lastcpu = td->td_oncpu; + td->td_oncpu = NOCPU; } KASSERT(curthread->td_md.md_spinlock_count == 1, ("invalid count")); newtd = choosethread(); From owner-svn-src-stable@freebsd.org Thu Oct 1 21:57:16 2015 Return-Path: Delivered-To: svn-src-stable@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 706F9A0E006; Thu, 1 Oct 2015 21:57:16 +0000 (UTC) (envelope-from jhb@FreeBSD.org) Received: from repo.freebsd.org (repo.freebsd.org [IPv6:2001:1900:2254:2068::e6a:0]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 60D641886; Thu, 1 Oct 2015 21:57:16 +0000 (UTC) (envelope-from jhb@FreeBSD.org) Received: from repo.freebsd.org ([127.0.1.70]) by repo.freebsd.org (8.15.2/8.15.2) with ESMTP id t91LvGKT019036; Thu, 1 Oct 2015 21:57:16 GMT (envelope-from jhb@FreeBSD.org) Received: (from jhb@localhost) by repo.freebsd.org (8.15.2/8.15.2/Submit) id t91LvGih019035; Thu, 1 Oct 2015 21:57:16 GMT (envelope-from jhb@FreeBSD.org) Message-Id: <201510012157.t91LvGih019035@repo.freebsd.org> X-Authentication-Warning: repo.freebsd.org: jhb set sender to jhb@FreeBSD.org using -f From: John Baldwin Date: Thu, 1 Oct 2015 21:57:16 +0000 (UTC) To: src-committers@freebsd.org, svn-src-all@freebsd.org, svn-src-stable@freebsd.org, svn-src-stable-10@freebsd.org Subject: svn commit: r288464 - in stable: 10/lib/libc/stdio 9/lib/libc/stdio X-SVN-Group: stable-10 MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit X-BeenThere: svn-src-stable@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: SVN commit messages for all the -stable branches of the src tree List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 01 Oct 2015 21:57:16 -0000 Author: jhb Date: Thu Oct 1 21:57:15 2015 New Revision: 288464 URL: https://svnweb.freebsd.org/changeset/base/288464 Log: MFC 286177: Fix a couple of markup typos. Modified: stable/10/lib/libc/stdio/open_memstream.3 Directory Properties: stable/10/ (props changed) Changes in other areas also in this revision: Modified: stable/9/lib/libc/stdio/open_memstream.3 (contents, props changed) Directory Properties: stable/9/lib/libc/ (props changed) Modified: stable/10/lib/libc/stdio/open_memstream.3 ============================================================================== --- stable/10/lib/libc/stdio/open_memstream.3 Thu Oct 1 21:54:43 2015 (r288463) +++ stable/10/lib/libc/stdio/open_memstream.3 Thu Oct 1 21:57:15 2015 (r288464) @@ -25,7 +25,7 @@ .\" .\" $FreeBSD$ .\" -.Dd October 28, 2014 +.Dd August 1, 2015 .Dt OPEN_MEMSTREAM 3 .Os .Sh NAME @@ -86,13 +86,13 @@ will contain the start of the memory buf will contain the smaller of the current position and the current buffer length. .Pp After a successful call to -.Xr fflush 3, +.Xr fflush 3 , the pointer referenced by .Fa bufp and the variable referenced by .Fa sizep are only valid until the next write operation or a call to -.Xr fclose 3. +.Xr fclose 3 . .Pp Once a stream is closed, the allocated buffer referenced by From owner-svn-src-stable@freebsd.org Thu Oct 1 21:57:17 2015 Return-Path: Delivered-To: svn-src-stable@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 29395A0E019; Thu, 1 Oct 2015 21:57:17 +0000 (UTC) (envelope-from jhb@FreeBSD.org) Received: from repo.freebsd.org (repo.freebsd.org [IPv6:2001:1900:2254:2068::e6a:0]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 19AC61887; Thu, 1 Oct 2015 21:57:17 +0000 (UTC) (envelope-from jhb@FreeBSD.org) Received: from repo.freebsd.org ([127.0.1.70]) by repo.freebsd.org (8.15.2/8.15.2) with ESMTP id t91LvGn7019043; Thu, 1 Oct 2015 21:57:16 GMT (envelope-from jhb@FreeBSD.org) Received: (from jhb@localhost) by repo.freebsd.org (8.15.2/8.15.2/Submit) id t91LvGZe019042; Thu, 1 Oct 2015 21:57:16 GMT (envelope-from jhb@FreeBSD.org) Message-Id: <201510012157.t91LvGZe019042@repo.freebsd.org> X-Authentication-Warning: repo.freebsd.org: jhb set sender to jhb@FreeBSD.org using -f From: John Baldwin Date: Thu, 1 Oct 2015 21:57:16 +0000 (UTC) To: src-committers@freebsd.org, svn-src-all@freebsd.org, svn-src-stable@freebsd.org, svn-src-stable-9@freebsd.org Subject: svn commit: r288464 - in stable: 10/lib/libc/stdio 9/lib/libc/stdio X-SVN-Group: stable-9 MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit X-BeenThere: svn-src-stable@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: SVN commit messages for all the -stable branches of the src tree List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 01 Oct 2015 21:57:17 -0000 Author: jhb Date: Thu Oct 1 21:57:15 2015 New Revision: 288464 URL: https://svnweb.freebsd.org/changeset/base/288464 Log: MFC 286177: Fix a couple of markup typos. Modified: stable/9/lib/libc/stdio/open_memstream.3 (contents, props changed) Directory Properties: stable/9/lib/libc/ (props changed) Changes in other areas also in this revision: Modified: stable/10/lib/libc/stdio/open_memstream.3 Directory Properties: stable/10/ (props changed) Modified: stable/9/lib/libc/stdio/open_memstream.3 ============================================================================== --- stable/9/lib/libc/stdio/open_memstream.3 Thu Oct 1 21:54:43 2015 (r288463) +++ stable/9/lib/libc/stdio/open_memstream.3 Thu Oct 1 21:57:15 2015 (r288464) @@ -25,7 +25,7 @@ .\" .\" $FreeBSD$ .\" -.Dd October 28, 2014 +.Dd August 1, 2015 .Dt OPEN_MEMSTREAM 3 .Os .Sh NAME @@ -86,13 +86,13 @@ will contain the start of the memory buf will contain the smaller of the current position and the current buffer length. .Pp After a successful call to -.Xr fflush 3, +.Xr fflush 3 , the pointer referenced by .Fa bufp and the variable referenced by .Fa sizep are only valid until the next write operation or a call to -.Xr fclose 3. +.Xr fclose 3 . .Pp Once a stream is closed, the allocated buffer referenced by From owner-svn-src-stable@freebsd.org Thu Oct 1 22:08:18 2015 Return-Path: Delivered-To: svn-src-stable@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id E48B8A0E7E1; Thu, 1 Oct 2015 22:08:17 +0000 (UTC) (envelope-from jhb@FreeBSD.org) Received: from repo.freebsd.org (repo.freebsd.org [IPv6:2001:1900:2254:2068::e6a:0]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id D4BAA1121; Thu, 1 Oct 2015 22:08:17 +0000 (UTC) (envelope-from jhb@FreeBSD.org) Received: from repo.freebsd.org ([127.0.1.70]) by repo.freebsd.org (8.15.2/8.15.2) with ESMTP id t91M8Hwt023305; Thu, 1 Oct 2015 22:08:17 GMT (envelope-from jhb@FreeBSD.org) Received: (from jhb@localhost) by repo.freebsd.org (8.15.2/8.15.2/Submit) id t91M8HXl023303; Thu, 1 Oct 2015 22:08:17 GMT (envelope-from jhb@FreeBSD.org) Message-Id: <201510012208.t91M8HXl023303@repo.freebsd.org> X-Authentication-Warning: repo.freebsd.org: jhb set sender to jhb@FreeBSD.org using -f From: John Baldwin Date: Thu, 1 Oct 2015 22:08:17 +0000 (UTC) To: src-committers@freebsd.org, svn-src-all@freebsd.org, svn-src-stable@freebsd.org, svn-src-stable-10@freebsd.org Subject: svn commit: r288465 - stable/10/tests/sys/vm X-SVN-Group: stable-10 MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit X-BeenThere: svn-src-stable@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: SVN commit messages for all the -stable branches of the src tree List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 01 Oct 2015 22:08:18 -0000 Author: jhb Date: Thu Oct 1 22:08:16 2015 New Revision: 288465 URL: https://svnweb.freebsd.org/changeset/base/288465 Log: MFC 286369: Convert the map_at_zero test case to ATF. In particular, this will facilitate adding more mmap() tests. Modified: stable/10/tests/sys/vm/Makefile stable/10/tests/sys/vm/mmap_test.c Directory Properties: stable/10/ (props changed) Modified: stable/10/tests/sys/vm/Makefile ============================================================================== --- stable/10/tests/sys/vm/Makefile Thu Oct 1 21:57:15 2015 (r288464) +++ stable/10/tests/sys/vm/Makefile Thu Oct 1 22:08:16 2015 (r288465) @@ -2,6 +2,6 @@ TESTSDIR= ${TESTSBASE}/sys/vm -TAP_TESTS_C+= mmap_test +ATF_TESTS_C+= mmap_test .include Modified: stable/10/tests/sys/vm/mmap_test.c ============================================================================== --- stable/10/tests/sys/vm/mmap_test.c Thu Oct 1 21:57:15 2015 (r288464) +++ stable/10/tests/sys/vm/mmap_test.c Thu Oct 1 22:08:16 2015 (r288465) @@ -29,16 +29,14 @@ #include #include #include -#include +#include #include -#include -#include static const struct { void *addr; int ok[2]; /* Depending on security.bsd.map_at_zero {0, !=0}. */ -} tests[] = { +} map_at_zero_tests[] = { { (void *)0, { 0, 1 } }, /* Test sysctl. */ { (void *)1, { 0, 0 } }, { (void *)(PAGE_SIZE - 1), { 0, 0 } }, @@ -52,54 +50,43 @@ static const struct { #define MAP_AT_ZERO "security.bsd.map_at_zero" -int -main(void) +ATF_TC_WITHOUT_HEAD(mmap__map_at_zero); +ATF_TC_BODY(mmap__map_at_zero, tc) { void *p; size_t len; - int i, error, mib[3], map_at_zero; - - error = 0; - - /* Get the current sysctl value of security.bsd.map_at_zero. */ - len = sizeof(mib) / sizeof(*mib); - if (sysctlnametomib(MAP_AT_ZERO, mib, &len) == -1) { - printf("1..0 # SKIP: sysctlnametomib(\"%s\") failed: %s\n", - MAP_AT_ZERO, strerror(errno)); - return (0); - } + unsigned int i; + int map_at_zero; len = sizeof(map_at_zero); - if (sysctl(mib, 3, &map_at_zero, &len, NULL, 0) == -1) { - printf("1..0 # SKIP: sysctl for %s failed: %s\n", MAP_AT_ZERO, + if (sysctlbyname(MAP_AT_ZERO, &map_at_zero, &len, NULL, 0) == -1) { + atf_tc_skip("sysctl for %s failed: %s\n", MAP_AT_ZERO, strerror(errno)); - return (0); + return; } /* Normalize to 0 or 1 for array access. */ map_at_zero = !!map_at_zero; - printf("1..%zu\n", nitems(tests)); - for (i = 0; i < (int)nitems(tests); i++) { - p = mmap((void *)tests[i].addr, PAGE_SIZE, + for (i = 0; i < nitems(map_at_zero_tests); i++) { + p = mmap((void *)map_at_zero_tests[i].addr, PAGE_SIZE, PROT_READ | PROT_WRITE | PROT_EXEC, MAP_ANON | MAP_FIXED, -1, 0); if (p == MAP_FAILED) { - if (tests[i].ok[map_at_zero] != 0) - error++; - printf("%sok %d # mmap(%p, ...) failed\n", - tests[i].ok[map_at_zero] == 0 ? "" : "not ", - i + 1, - tests[i].addr); + ATF_CHECK_MSG(map_at_zero_tests[i].ok[map_at_zero] == 0, + "mmap(%p, ...) failed", map_at_zero_tests[i].addr); } else { - if (tests[i].ok[map_at_zero] != 1) - error++; - printf("%sok %d # mmap(%p, ...) succeeded: p=%p\n", - tests[i].ok[map_at_zero] == 1 ? "" : "not ", - i + 1, - tests[i].addr, p); + ATF_CHECK_MSG(map_at_zero_tests[i].ok[map_at_zero] == 1, + "mmap(%p, ...) succeeded: p=%p\n", + map_at_zero_tests[i].addr, p); } } +} + +ATF_TP_ADD_TCS(tp) +{ + + ATF_TP_ADD_TC(tp, mmap__map_at_zero); - return (error != 0); + return (atf_no_error()); } From owner-svn-src-stable@freebsd.org Thu Oct 1 22:17:28 2015 Return-Path: Delivered-To: svn-src-stable@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 6A8B5A0ECC0; Thu, 1 Oct 2015 22:17:28 +0000 (UTC) (envelope-from jhb@FreeBSD.org) Received: from repo.freebsd.org (repo.freebsd.org [IPv6:2001:1900:2254:2068::e6a:0]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 5AEE0177A; Thu, 1 Oct 2015 22:17:28 +0000 (UTC) (envelope-from jhb@FreeBSD.org) Received: from repo.freebsd.org ([127.0.1.70]) by repo.freebsd.org (8.15.2/8.15.2) with ESMTP id t91MHSPh027342; Thu, 1 Oct 2015 22:17:28 GMT (envelope-from jhb@FreeBSD.org) Received: (from jhb@localhost) by repo.freebsd.org (8.15.2/8.15.2/Submit) id t91MHSe1027341; Thu, 1 Oct 2015 22:17:28 GMT (envelope-from jhb@FreeBSD.org) Message-Id: <201510012217.t91MHSe1027341@repo.freebsd.org> X-Authentication-Warning: repo.freebsd.org: jhb set sender to jhb@FreeBSD.org using -f From: John Baldwin Date: Thu, 1 Oct 2015 22:17:28 +0000 (UTC) To: src-committers@freebsd.org, svn-src-all@freebsd.org, svn-src-stable@freebsd.org, svn-src-stable-10@freebsd.org Subject: svn commit: r288466 - stable/10/tests/sys/vm X-SVN-Group: stable-10 MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit X-BeenThere: svn-src-stable@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: SVN commit messages for all the -stable branches of the src tree List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 01 Oct 2015 22:17:28 -0000 Author: jhb Date: Thu Oct 1 22:17:27 2015 New Revision: 288466 URL: https://svnweb.freebsd.org/changeset/base/288466 Log: MFC 286370: Add various tests to ensure that invalid arguments passed to mmap() trigger failures. Note: most of the tests that should provoke an EINVAL error do not pass on stable/10 as stable/10 does not have the changes that added more strict parameter checking to mmap(). Those changes will not be merged to stable/10, so I have disabled them via #if 0 with a comment explaining why. Modified: stable/10/tests/sys/vm/mmap_test.c Directory Properties: stable/10/ (props changed) Modified: stable/10/tests/sys/vm/mmap_test.c ============================================================================== --- stable/10/tests/sys/vm/mmap_test.c Thu Oct 1 22:08:16 2015 (r288465) +++ stable/10/tests/sys/vm/mmap_test.c Thu Oct 1 22:17:27 2015 (r288466) @@ -32,6 +32,10 @@ #include #include +#include +#include +#include +#include static const struct { void *addr; @@ -83,10 +87,86 @@ ATF_TC_BODY(mmap__map_at_zero, tc) } } +static void +checked_mmap(int prot, int flags, int fd, int error, const char *msg) +{ + void *p; + + p = mmap(NULL, getpagesize(), prot, flags, fd, 0); + if (p == MAP_FAILED) { + if (error == 0) + ATF_CHECK_MSG(0, "%s failed with errno %d", msg, + errno); + else + ATF_CHECK_EQ_MSG(error, errno, + "%s failed with wrong errno %d (expected %d)", msg, + errno, error); + } else { + ATF_CHECK_MSG(error == 0, "%s succeeded", msg); + munmap(p, getpagesize()); + } +} + +ATF_TC_WITHOUT_HEAD(mmap__bad_arguments); +ATF_TC_BODY(mmap__bad_arguments, tc) +{ + int fd; + + ATF_REQUIRE((fd = shm_open(SHM_ANON, O_RDWR, 0644)) >= 0); + ATF_REQUIRE(ftruncate(fd, getpagesize()) == 0); + + /* These should work. */ + checked_mmap(PROT_READ | PROT_WRITE, MAP_ANON, -1, 0, + "simple MAP_ANON"); + checked_mmap(PROT_READ | PROT_WRITE, MAP_SHARED, fd, 0, + "simple shm fd shared"); + checked_mmap(PROT_READ | PROT_WRITE, MAP_PRIVATE, fd, 0, + "simple shm fd private"); + +#if 0 + /* + * These tests do not fail without r271635 and followup fixes. + * Those changes will not be merged to stable/10 since they + * are potentially disruptive. + */ + + /* Extra PROT flags. */ + checked_mmap(PROT_READ | PROT_WRITE | 0x100000, MAP_ANON, -1, EINVAL, + "MAP_ANON with extra PROT flags"); + checked_mmap(0xffff, MAP_SHARED, fd, EINVAL, + "shm fd with garbage PROT"); + + /* Undefined flag. */ + checked_mmap(PROT_READ | PROT_WRITE, MAP_ANON | MAP_RESERVED0080, -1, + EINVAL, "Undefined flag"); + + /* Both MAP_SHARED and MAP_PRIVATE */ + checked_mmap(PROT_READ | PROT_WRITE, MAP_ANON | MAP_PRIVATE | + MAP_SHARED, -1, EINVAL, "MAP_ANON with both SHARED and PRIVATE"); + checked_mmap(PROT_READ | PROT_WRITE, MAP_PRIVATE | MAP_SHARED, fd, + EINVAL, "shm fd with both SHARED and PRIVATE"); + + /* At least one of MAP_SHARED or MAP_PRIVATE without ANON */ + checked_mmap(PROT_READ | PROT_WRITE, 0, fd, EINVAL, + "shm fd without sharing flag"); +#endif + + /* MAP_ANON with either sharing flag (impacts fork). */ + checked_mmap(PROT_READ | PROT_WRITE, MAP_ANON | MAP_SHARED, -1, 0, + "shared MAP_ANON"); + checked_mmap(PROT_READ | PROT_WRITE, MAP_ANON | MAP_PRIVATE, -1, 0, + "private MAP_ANON"); + + /* MAP_ANON should require an fd of -1. */ + checked_mmap(PROT_READ | PROT_WRITE, MAP_ANON | MAP_PRIVATE, 0, EINVAL, + "MAP_ANON with fd != -1"); +} + ATF_TP_ADD_TCS(tp) { ATF_TP_ADD_TC(tp, mmap__map_at_zero); + ATF_TP_ADD_TC(tp, mmap__bad_arguments); return (atf_no_error()); } From owner-svn-src-stable@freebsd.org Thu Oct 1 22:19:42 2015 Return-Path: Delivered-To: svn-src-stable@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id B9C87A0EDE5; Thu, 1 Oct 2015 22:19:42 +0000 (UTC) (envelope-from jhb@FreeBSD.org) Received: from repo.freebsd.org (repo.freebsd.org [IPv6:2001:1900:2254:2068::e6a:0]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 9D8EF191C; Thu, 1 Oct 2015 22:19:42 +0000 (UTC) (envelope-from jhb@FreeBSD.org) Received: from repo.freebsd.org ([127.0.1.70]) by repo.freebsd.org (8.15.2/8.15.2) with ESMTP id t91MJgQp027498; Thu, 1 Oct 2015 22:19:42 GMT (envelope-from jhb@FreeBSD.org) Received: (from jhb@localhost) by repo.freebsd.org (8.15.2/8.15.2/Submit) id t91MJgH4027497; Thu, 1 Oct 2015 22:19:42 GMT (envelope-from jhb@FreeBSD.org) Message-Id: <201510012219.t91MJgH4027497@repo.freebsd.org> X-Authentication-Warning: repo.freebsd.org: jhb set sender to jhb@FreeBSD.org using -f From: John Baldwin Date: Thu, 1 Oct 2015 22:19:42 +0000 (UTC) To: src-committers@freebsd.org, svn-src-all@freebsd.org, svn-src-stable@freebsd.org, svn-src-stable-10@freebsd.org Subject: svn commit: r288467 - stable/10/tests/sys/vm X-SVN-Group: stable-10 MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit X-BeenThere: svn-src-stable@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: SVN commit messages for all the -stable branches of the src tree List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 01 Oct 2015 22:19:42 -0000 Author: jhb Date: Thu Oct 1 22:19:41 2015 New Revision: 288467 URL: https://svnweb.freebsd.org/changeset/base/288467 Log: MFC 287448: Add more mmap tests related to character devices. - Add cdev-related tests for bad args. - Add two simple tests cases for mapping /dev/zero that test for MAP_ANON-like behavior. Modified: stable/10/tests/sys/vm/mmap_test.c Directory Properties: stable/10/ (props changed) Modified: stable/10/tests/sys/vm/mmap_test.c ============================================================================== --- stable/10/tests/sys/vm/mmap_test.c Thu Oct 1 22:17:27 2015 (r288466) +++ stable/10/tests/sys/vm/mmap_test.c Thu Oct 1 22:19:41 2015 (r288467) @@ -110,18 +110,26 @@ checked_mmap(int prot, int flags, int fd ATF_TC_WITHOUT_HEAD(mmap__bad_arguments); ATF_TC_BODY(mmap__bad_arguments, tc) { - int fd; + int devstatfd, shmfd, zerofd; - ATF_REQUIRE((fd = shm_open(SHM_ANON, O_RDWR, 0644)) >= 0); - ATF_REQUIRE(ftruncate(fd, getpagesize()) == 0); + ATF_REQUIRE((devstatfd = open("/dev/devstat", O_RDONLY)) >= 0); + ATF_REQUIRE((shmfd = shm_open(SHM_ANON, O_RDWR, 0644)) >= 0); + ATF_REQUIRE(ftruncate(shmfd, getpagesize()) == 0); + ATF_REQUIRE((zerofd = open("/dev/zero", O_RDONLY)) >= 0); /* These should work. */ checked_mmap(PROT_READ | PROT_WRITE, MAP_ANON, -1, 0, "simple MAP_ANON"); - checked_mmap(PROT_READ | PROT_WRITE, MAP_SHARED, fd, 0, + checked_mmap(PROT_READ | PROT_WRITE, MAP_SHARED, shmfd, 0, "simple shm fd shared"); - checked_mmap(PROT_READ | PROT_WRITE, MAP_PRIVATE, fd, 0, + checked_mmap(PROT_READ | PROT_WRITE, MAP_PRIVATE, shmfd, 0, "simple shm fd private"); + checked_mmap(PROT_READ, MAP_SHARED, zerofd, 0, + "simple /dev/zero shared"); + checked_mmap(PROT_READ | PROT_WRITE, MAP_PRIVATE, zerofd, 0, + "simple /dev/zero private"); + checked_mmap(PROT_READ, MAP_SHARED, devstatfd, 0, + "simple /dev/devstat shared"); #if 0 /* @@ -133,7 +141,7 @@ ATF_TC_BODY(mmap__bad_arguments, tc) /* Extra PROT flags. */ checked_mmap(PROT_READ | PROT_WRITE | 0x100000, MAP_ANON, -1, EINVAL, "MAP_ANON with extra PROT flags"); - checked_mmap(0xffff, MAP_SHARED, fd, EINVAL, + checked_mmap(0xffff, MAP_SHARED, shmfd, EINVAL, "shm fd with garbage PROT"); /* Undefined flag. */ @@ -143,11 +151,11 @@ ATF_TC_BODY(mmap__bad_arguments, tc) /* Both MAP_SHARED and MAP_PRIVATE */ checked_mmap(PROT_READ | PROT_WRITE, MAP_ANON | MAP_PRIVATE | MAP_SHARED, -1, EINVAL, "MAP_ANON with both SHARED and PRIVATE"); - checked_mmap(PROT_READ | PROT_WRITE, MAP_PRIVATE | MAP_SHARED, fd, + checked_mmap(PROT_READ | PROT_WRITE, MAP_PRIVATE | MAP_SHARED, shmfd, EINVAL, "shm fd with both SHARED and PRIVATE"); /* At least one of MAP_SHARED or MAP_PRIVATE without ANON */ - checked_mmap(PROT_READ | PROT_WRITE, 0, fd, EINVAL, + checked_mmap(PROT_READ | PROT_WRITE, 0, shmfd, EINVAL, "shm fd without sharing flag"); #endif @@ -160,6 +168,91 @@ ATF_TC_BODY(mmap__bad_arguments, tc) /* MAP_ANON should require an fd of -1. */ checked_mmap(PROT_READ | PROT_WRITE, MAP_ANON | MAP_PRIVATE, 0, EINVAL, "MAP_ANON with fd != -1"); + + /* Writable MAP_SHARED should fail on read-only descriptors. */ + checked_mmap(PROT_READ | PROT_WRITE, MAP_SHARED, zerofd, EACCES, + "MAP_SHARED of read-only /dev/zero"); + + /* + * Character devices other than /dev/zero do not support private + * mappings. + */ + checked_mmap(PROT_READ, MAP_PRIVATE, devstatfd, EINVAL, + "MAP_PRIVATE of /dev/devstat"); +} + +ATF_TC_WITHOUT_HEAD(mmap__dev_zero_private); +ATF_TC_BODY(mmap__dev_zero_private, tc) +{ + char *p1, *p2, *p3; + size_t i; + int fd; + + ATF_REQUIRE((fd = open("/dev/zero", O_RDONLY)) >= 0); + + p1 = mmap(NULL, getpagesize(), PROT_READ | PROT_WRITE, MAP_PRIVATE, fd, + 0); + ATF_REQUIRE(p1 != MAP_FAILED); + + p2 = mmap(NULL, getpagesize(), PROT_READ | PROT_WRITE, MAP_PRIVATE, fd, + 0); + ATF_REQUIRE(p2 != MAP_FAILED); + + for (i = 0; i < getpagesize(); i++) + ATF_REQUIRE_EQ_MSG(0, p1[i], "byte at p1[%zu] is %x", i, p1[i]); + + ATF_REQUIRE(memcmp(p1, p2, getpagesize()) == 0); + + p1[0] = 1; + + ATF_REQUIRE(p2[0] == 0); + + p2[0] = 2; + + ATF_REQUIRE(p1[0] == 1); + + p3 = mmap(NULL, getpagesize(), PROT_READ | PROT_WRITE, MAP_PRIVATE, fd, + 0); + ATF_REQUIRE(p3 != MAP_FAILED); + + ATF_REQUIRE(p3[0] == 0); +} + +ATF_TC_WITHOUT_HEAD(mmap__dev_zero_shared); +ATF_TC_BODY(mmap__dev_zero_shared, tc) +{ + char *p1, *p2, *p3; + size_t i; + int fd; + + ATF_REQUIRE((fd = open("/dev/zero", O_RDWR)) >= 0); + + p1 = mmap(NULL, getpagesize(), PROT_READ | PROT_WRITE, MAP_SHARED, fd, + 0); + ATF_REQUIRE(p1 != MAP_FAILED); + + p2 = mmap(NULL, getpagesize(), PROT_READ | PROT_WRITE, MAP_SHARED, fd, + 0); + ATF_REQUIRE(p2 != MAP_FAILED); + + for (i = 0; i < getpagesize(); i++) + ATF_REQUIRE_EQ_MSG(0, p1[i], "byte at p1[%zu] is %x", i, p1[i]); + + ATF_REQUIRE(memcmp(p1, p2, getpagesize()) == 0); + + p1[0] = 1; + + ATF_REQUIRE(p2[0] == 0); + + p2[0] = 2; + + ATF_REQUIRE(p1[0] == 1); + + p3 = mmap(NULL, getpagesize(), PROT_READ | PROT_WRITE, MAP_SHARED, fd, + 0); + ATF_REQUIRE(p3 != MAP_FAILED); + + ATF_REQUIRE(p3[0] == 0); } ATF_TP_ADD_TCS(tp) @@ -167,6 +260,8 @@ ATF_TP_ADD_TCS(tp) ATF_TP_ADD_TC(tp, mmap__map_at_zero); ATF_TP_ADD_TC(tp, mmap__bad_arguments); + ATF_TP_ADD_TC(tp, mmap__dev_zero_private); + ATF_TP_ADD_TC(tp, mmap__dev_zero_shared); return (atf_no_error()); } From owner-svn-src-stable@freebsd.org Fri Oct 2 02:06:37 2015 Return-Path: Delivered-To: svn-src-stable@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 0EF29A0E9F9; Fri, 2 Oct 2015 02:06:37 +0000 (UTC) (envelope-from gjb@FreeBSD.org) Received: from repo.freebsd.org (repo.freebsd.org [IPv6:2001:1900:2254:2068::e6a:0]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id F0E80156D; Fri, 2 Oct 2015 02:06:36 +0000 (UTC) (envelope-from gjb@FreeBSD.org) Received: from repo.freebsd.org ([127.0.1.70]) by repo.freebsd.org (8.15.2/8.15.2) with ESMTP id t9226ada021373; Fri, 2 Oct 2015 02:06:36 GMT (envelope-from gjb@FreeBSD.org) Received: (from gjb@localhost) by repo.freebsd.org (8.15.2/8.15.2/Submit) id t9226adI021372; Fri, 2 Oct 2015 02:06:36 GMT (envelope-from gjb@FreeBSD.org) Message-Id: <201510020206.t9226adI021372@repo.freebsd.org> X-Authentication-Warning: repo.freebsd.org: gjb set sender to gjb@FreeBSD.org using -f From: Glen Barber Date: Fri, 2 Oct 2015 02:06:36 +0000 (UTC) To: src-committers@freebsd.org, svn-src-all@freebsd.org, svn-src-stable@freebsd.org, svn-src-stable-10@freebsd.org Subject: svn commit: r288468 - stable/10/release/tools X-SVN-Group: stable-10 MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit X-BeenThere: svn-src-stable@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: SVN commit messages for all the -stable branches of the src tree List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 02 Oct 2015 02:06:37 -0000 Author: gjb Date: Fri Oct 2 02:06:36 2015 New Revision: 288468 URL: https://svnweb.freebsd.org/changeset/base/288468 Log: MFC r288370: In vm_copy_base(), turn off SU+J on the resultant filesystem, leaving only SU enabled. Sponsored by: The FreeBSD Foundation Modified: stable/10/release/tools/vmimage.subr Directory Properties: stable/10/ (props changed) Modified: stable/10/release/tools/vmimage.subr ============================================================================== --- stable/10/release/tools/vmimage.subr Thu Oct 1 22:19:41 2015 (r288467) +++ stable/10/release/tools/vmimage.subr Fri Oct 2 02:06:36 2015 (r288468) @@ -102,7 +102,7 @@ vm_copy_base() { umount_loop /dev/${mdnew} rmdir ${DESTDIR}/new - tunefs -j enable /dev/${mdnew} + tunefs -n enable /dev/${mdnew} mdconfig -d -u ${mdnew} mv ${VMBASE}.tmp ${VMBASE} } From owner-svn-src-stable@freebsd.org Fri Oct 2 02:08:43 2015 Return-Path: Delivered-To: svn-src-stable@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 19790A0EBB3; Fri, 2 Oct 2015 02:08:43 +0000 (UTC) (envelope-from gjb@FreeBSD.org) Received: from repo.freebsd.org (repo.freebsd.org [IPv6:2001:1900:2254:2068::e6a:0]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id F27DF1807; Fri, 2 Oct 2015 02:08:42 +0000 (UTC) (envelope-from gjb@FreeBSD.org) Received: from repo.freebsd.org ([127.0.1.70]) by repo.freebsd.org (8.15.2/8.15.2) with ESMTP id t9228gAK021521; Fri, 2 Oct 2015 02:08:42 GMT (envelope-from gjb@FreeBSD.org) Received: (from gjb@localhost) by repo.freebsd.org (8.15.2/8.15.2/Submit) id t9228fLG021511; Fri, 2 Oct 2015 02:08:41 GMT (envelope-from gjb@FreeBSD.org) Message-Id: <201510020208.t9228fLG021511@repo.freebsd.org> X-Authentication-Warning: repo.freebsd.org: gjb set sender to gjb@FreeBSD.org using -f From: Glen Barber Date: Fri, 2 Oct 2015 02:08:41 +0000 (UTC) To: src-committers@freebsd.org, svn-src-all@freebsd.org, svn-src-stable@freebsd.org, svn-src-stable-10@freebsd.org Subject: svn commit: r288469 - stable/10/release/arm X-SVN-Group: stable-10 MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit X-BeenThere: svn-src-stable@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: SVN commit messages for all the -stable branches of the src tree List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 02 Oct 2015 02:08:43 -0000 Author: gjb Date: Fri Oct 2 02:08:40 2015 New Revision: 288469 URL: https://svnweb.freebsd.org/changeset/base/288469 Log: MFC r288374: In addition to the ubldr file, also copy ubldr.bin to the MS-DOS partition. This will help with transitioning to a single arm/armv6 userland build which could be used for all FreeBSD/armv6 images without UBLDR_LOADADDR being set for each board (ultimately requiring a separate buildworld for each currently). Sponsored by: The FreeBSD Foundation Modified: stable/10/release/arm/BEAGLEBONE.conf stable/10/release/arm/CUBOX-HUMMINGBOARD.conf stable/10/release/arm/GUMSTIX.conf stable/10/release/arm/PANDABOARD.conf stable/10/release/arm/RPI-B.conf stable/10/release/arm/RPI2.conf stable/10/release/arm/WANDBOARD.conf Directory Properties: stable/10/ (props changed) Modified: stable/10/release/arm/BEAGLEBONE.conf ============================================================================== --- stable/10/release/arm/BEAGLEBONE.conf Fri Oct 2 02:06:36 2015 (r288468) +++ stable/10/release/arm/BEAGLEBONE.conf Fri Oct 2 02:08:40 2015 (r288469) @@ -26,6 +26,8 @@ arm_install_uboot() { chroot ${CHROOTDIR} cp -p ${UBOOT_DIR}/MLO ${FATMOUNT}/MLO chroot ${CHROOTDIR} cp -p ${UBOOT_DIR}/u-boot.img ${FATMOUNT}/u-boot.img chroot ${CHROOTDIR} cp -p ${UFSMOUNT}/boot/ubldr ${FATMOUNT}/ubldr + chroot ${CHROOTDIR} cp -p ${UFSMOUNT}/boot/ubldr.bin \ + ${FATMOUNT}/ubldr.bin chroot ${CHROOTDIR} touch ${UFSMOUNT}/firstboot sync umount_loop ${CHROOTDIR}/${FATMOUNT} Modified: stable/10/release/arm/CUBOX-HUMMINGBOARD.conf ============================================================================== --- stable/10/release/arm/CUBOX-HUMMINGBOARD.conf Fri Oct 2 02:06:36 2015 (r288468) +++ stable/10/release/arm/CUBOX-HUMMINGBOARD.conf Fri Oct 2 02:08:40 2015 (r288469) @@ -28,6 +28,8 @@ arm_install_uboot() { chroot ${CHROOTDIR} mount_msdosfs /dev/${mddev}s1 ${FATMOUNT} chroot ${CHROOTDIR} mount /dev/${mddev}s2a ${UFSMOUNT} chroot ${CHROOTDIR} cp -p ${UFSMOUNT}/boot/ubldr ${FATMOUNT}/ubldr + chroot ${CHROOTDIR} cp -p ${UFSMOUNT}/boot/ubldr.bin \ + ${FATMOUNT}/ubldr.bin chroot ${CHROOTDIR} touch ${UFSMOUNT}/firstboot sync umount_loop ${CHROOTDIR}/${FATMOUNT} Modified: stable/10/release/arm/GUMSTIX.conf ============================================================================== --- stable/10/release/arm/GUMSTIX.conf Fri Oct 2 02:06:36 2015 (r288468) +++ stable/10/release/arm/GUMSTIX.conf Fri Oct 2 02:08:40 2015 (r288469) @@ -26,6 +26,8 @@ arm_install_uboot() { chroot ${CHROOTDIR} cp -p ${UBOOT_DIR}/MLO ${FATMOUNT}/MLO chroot ${CHROOTDIR} cp -p ${UBOOT_DIR}/u-boot.img ${FATMOUNT}/u-boot.img chroot ${CHROOTDIR} cp -p ${UFSMOUNT}/boot/ubldr ${FATMOUNT}/ubldr + chroot ${CHROOTDIR} cp -p ${UFSMOUNT}/boot/ubldr.bin \ + ${FATMOUNT}/ubldr.bin chroot ${CHROOTDIR} touch ${UFSMOUNT}/firstboot sync umount_loop ${CHROOTDIR}/${FATMOUNT} Modified: stable/10/release/arm/PANDABOARD.conf ============================================================================== --- stable/10/release/arm/PANDABOARD.conf Fri Oct 2 02:06:36 2015 (r288468) +++ stable/10/release/arm/PANDABOARD.conf Fri Oct 2 02:08:40 2015 (r288469) @@ -26,6 +26,8 @@ arm_install_uboot() { chroot ${CHROOTDIR} cp -p ${UBOOT_DIR}/MLO ${FATMOUNT}/MLO chroot ${CHROOTDIR} cp -p ${UBOOT_DIR}/u-boot.img ${FATMOUNT}/u-boot.img chroot ${CHROOTDIR} cp -p ${UFSMOUNT}/boot/ubldr ${FATMOUNT}/ubldr + chroot ${CHROOTDIR} cp -p ${UFSMOUNT}/boot/ubldr.bin \ + ${FATMOUNT}/ubldr.bin chroot ${CHROOTDIR} touch ${UFSMOUNT}/firstboot sync umount_loop ${CHROOTDIR}/${FATMOUNT} Modified: stable/10/release/arm/RPI-B.conf ============================================================================== --- stable/10/release/arm/RPI-B.conf Fri Oct 2 02:06:36 2015 (r288468) +++ stable/10/release/arm/RPI-B.conf Fri Oct 2 02:08:40 2015 (r288469) @@ -30,6 +30,8 @@ arm_install_uboot() { ${FATMOUNT}/${_UF} done chroot ${CHROOTDIR} cp -p ${UFSMOUNT}/boot/ubldr ${FATMOUNT}/ubldr + chroot ${CHROOTDIR} cp -p ${UFSMOUNT}/boot/ubldr.bin \ + ${FATMOUNT}/ubldr.bin chroot ${CHROOTDIR} cp -p ${UFSMOUNT}/boot/dtb/rpi.dtb \ ${FATMOUNT}/rpi.dtb chroot ${CHROOTDIR} touch ${UFSMOUNT}/firstboot Modified: stable/10/release/arm/RPI2.conf ============================================================================== --- stable/10/release/arm/RPI2.conf Fri Oct 2 02:06:36 2015 (r288468) +++ stable/10/release/arm/RPI2.conf Fri Oct 2 02:08:40 2015 (r288469) @@ -31,6 +31,8 @@ arm_install_uboot() { ${FATMOUNT}/${_UF} done chroot ${CHROOTDIR} cp -p ${UFSMOUNT}/boot/ubldr ${FATMOUNT}/ubldr + chroot ${CHROOTDIR} cp -p ${UFSMOUNT}/boot/ubldr.bin \ + ${FATMOUNT}/ubldr.bin chroot ${CHROOTDIR} cp -p ${UFSMOUNT}/boot/dtb/rpi2.dtb \ ${FATMOUNT}/rpi2.dtb chroot ${CHROOTDIR} touch ${UFSMOUNT}/firstboot Modified: stable/10/release/arm/WANDBOARD.conf ============================================================================== --- stable/10/release/arm/WANDBOARD.conf Fri Oct 2 02:06:36 2015 (r288468) +++ stable/10/release/arm/WANDBOARD.conf Fri Oct 2 02:08:40 2015 (r288469) @@ -29,6 +29,8 @@ arm_install_uboot() { chroot ${CHROOTDIR} mount_msdosfs /dev/${mddev}s1 ${FATMOUNT} chroot ${CHROOTDIR} mount /dev/${mddev}s2a ${UFSMOUNT} chroot ${CHROOTDIR} cp -p ${UFSMOUNT}/boot/ubldr ${FATMOUNT}/ubldr + chroot ${CHROOTDIR} cp -p ${UFSMOUNT}/boot/ubldr.bin \ + ${FATMOUNT}/ubldr.bin chroot ${CHROOTDIR} touch ${UFSMOUNT}/firstboot sync umount_loop ${CHROOTDIR}/${FATMOUNT} From owner-svn-src-stable@freebsd.org Fri Oct 2 05:27:13 2015 Return-Path: Delivered-To: svn-src-stable@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 8C310A0D361; Fri, 2 Oct 2015 05:27:13 +0000 (UTC) (envelope-from kib@FreeBSD.org) Received: from repo.freebsd.org (repo.freebsd.org [IPv6:2001:1900:2254:2068::e6a:0]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 7C3D71FF0; Fri, 2 Oct 2015 05:27:13 +0000 (UTC) (envelope-from kib@FreeBSD.org) Received: from repo.freebsd.org ([127.0.1.70]) by repo.freebsd.org (8.15.2/8.15.2) with ESMTP id t925RDol003516; Fri, 2 Oct 2015 05:27:13 GMT (envelope-from kib@FreeBSD.org) Received: (from kib@localhost) by repo.freebsd.org (8.15.2/8.15.2/Submit) id t925RDgQ003515; Fri, 2 Oct 2015 05:27:13 GMT (envelope-from kib@FreeBSD.org) Message-Id: <201510020527.t925RDgQ003515@repo.freebsd.org> X-Authentication-Warning: repo.freebsd.org: kib set sender to kib@FreeBSD.org using -f From: Konstantin Belousov Date: Fri, 2 Oct 2015 05:27:13 +0000 (UTC) To: src-committers@freebsd.org, svn-src-all@freebsd.org, svn-src-stable@freebsd.org, svn-src-stable-10@freebsd.org Subject: svn commit: r288473 - stable/10/sys/kern X-SVN-Group: stable-10 MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit X-BeenThere: svn-src-stable@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: SVN commit messages for all the -stable branches of the src tree List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 02 Oct 2015 05:27:13 -0000 Author: kib Date: Fri Oct 2 05:27:12 2015 New Revision: 288473 URL: https://svnweb.freebsd.org/changeset/base/288473 Log: MFC r288216: Use per-cpu values for base and last in tc_cpu_ticks(). Modified: stable/10/sys/kern/kern_tc.c Directory Properties: stable/10/ (props changed) Modified: stable/10/sys/kern/kern_tc.c ============================================================================== --- stable/10/sys/kern/kern_tc.c Fri Oct 2 05:04:17 2015 (r288472) +++ stable/10/sys/kern/kern_tc.c Fri Oct 2 05:27:12 2015 (r288473) @@ -1888,20 +1888,27 @@ SYSINIT(timecounter, SI_SUB_CLOCKS, SI_O static int cpu_tick_variable; static uint64_t cpu_tick_frequency; +static DPCPU_DEFINE(uint64_t, tc_cpu_ticks_base); +static DPCPU_DEFINE(unsigned, tc_cpu_ticks_last); + static uint64_t tc_cpu_ticks(void) { - static uint64_t base; - static unsigned last; - unsigned u; struct timecounter *tc; + uint64_t res, *base; + unsigned u, *last; + critical_enter(); + base = DPCPU_PTR(tc_cpu_ticks_base); + last = DPCPU_PTR(tc_cpu_ticks_last); tc = timehands->th_counter; u = tc->tc_get_timecount(tc) & tc->tc_counter_mask; - if (u < last) - base += (uint64_t)tc->tc_counter_mask + 1; - last = u; - return (u + base); + if (u < *last) + *base += (uint64_t)tc->tc_counter_mask + 1; + *last = u; + res = u + *base; + critical_exit(); + return (res); } void From owner-svn-src-stable@freebsd.org Fri Oct 2 13:48:33 2015 Return-Path: Delivered-To: svn-src-stable@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 0D057A0D226; Fri, 2 Oct 2015 13:48:33 +0000 (UTC) (envelope-from vangyzen@FreeBSD.org) Received: from repo.freebsd.org (repo.freebsd.org [IPv6:2001:1900:2254:2068::e6a:0]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id E5E4E10E7; Fri, 2 Oct 2015 13:48:32 +0000 (UTC) (envelope-from vangyzen@FreeBSD.org) Received: from repo.freebsd.org ([127.0.1.70]) by repo.freebsd.org (8.15.2/8.15.2) with ESMTP id t92DmWOt011528; Fri, 2 Oct 2015 13:48:32 GMT (envelope-from vangyzen@FreeBSD.org) Received: (from vangyzen@localhost) by repo.freebsd.org (8.15.2/8.15.2/Submit) id t92DmWrS011527; Fri, 2 Oct 2015 13:48:32 GMT (envelope-from vangyzen@FreeBSD.org) Message-Id: <201510021348.t92DmWrS011527@repo.freebsd.org> X-Authentication-Warning: repo.freebsd.org: vangyzen set sender to vangyzen@FreeBSD.org using -f From: Eric van Gyzen Date: Fri, 2 Oct 2015 13:48:32 +0000 (UTC) To: src-committers@freebsd.org, svn-src-all@freebsd.org, svn-src-stable@freebsd.org, svn-src-stable-10@freebsd.org Subject: svn commit: r288494 - stable/10/sys/kern X-SVN-Group: stable-10 MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit X-BeenThere: svn-src-stable@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: SVN commit messages for all the -stable branches of the src tree List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 02 Oct 2015 13:48:33 -0000 Author: vangyzen Date: Fri Oct 2 13:48:32 2015 New Revision: 288494 URL: https://svnweb.freebsd.org/changeset/base/288494 Log: MFC r280792 Clean up some cosmetic nits in kern_umtx.c, found during recent work in this area and by the Clang static analyzer. Remove some dead assignments. Fix a typo in a panic string. Use umtx_pi_disown() instead of duplicate code. Use an existing variable instead of curthread. Approved by: kib (mentor until recently) Sponsored by: Dell Inc. Modified: stable/10/sys/kern/kern_umtx.c Directory Properties: stable/10/ (props changed) Modified: stable/10/sys/kern/kern_umtx.c ============================================================================== --- stable/10/sys/kern/kern_umtx.c Fri Oct 2 13:30:56 2015 (r288493) +++ stable/10/sys/kern/kern_umtx.c Fri Oct 2 13:48:32 2015 (r288494) @@ -1273,7 +1273,7 @@ kern_umtx_wake(struct thread *td, void * is_private ? THREAD_SHARE : AUTO_SHARE, &key)) != 0) return (ret); umtxq_lock(&key); - ret = umtxq_signal(&key, n_wake); + umtxq_signal(&key, n_wake); umtxq_unlock(&key); umtx_key_release(&key); return (0); @@ -1805,7 +1805,7 @@ umtx_pi_setowner(struct umtx_pi *pi, str uq_owner = owner->td_umtxq; mtx_assert(&umtx_lock, MA_OWNED); if (pi->pi_owner != NULL) - panic("pi_ower != NULL"); + panic("pi_owner != NULL"); pi->pi_owner = owner; TAILQ_INSERT_TAIL(&uq_owner->uq_pi_contested, pi, pi_link); } @@ -1829,9 +1829,8 @@ umtx_pi_disown(struct umtx_pi *pi) static int umtx_pi_claim(struct umtx_pi *pi, struct thread *owner) { - struct umtx_q *uq, *uq_owner; + struct umtx_q *uq; - uq_owner = owner->td_umtxq; mtx_lock(&umtx_lock); if (pi->pi_owner == owner) { mtx_unlock(&umtx_lock); @@ -1977,11 +1976,8 @@ umtx_pi_unref(struct umtx_pi *pi) KASSERT(pi->pi_refcount > 0, ("invalid reference count")); if (--pi->pi_refcount == 0) { mtx_lock(&umtx_lock); - if (pi->pi_owner != NULL) { - TAILQ_REMOVE(&pi->pi_owner->td_umtxq->uq_pi_contested, - pi, pi_link); - pi->pi_owner = NULL; - } + if (pi->pi_owner != NULL) + umtx_pi_disown(pi); KASSERT(TAILQ_EMPTY(&pi->pi_blocked), ("blocked queue not empty")); mtx_unlock(&umtx_lock); @@ -2241,7 +2237,7 @@ do_unlock_pi(struct thread *td, struct u mtx_lock(&umtx_lock); pi = uq_first->uq_pi_blocked; KASSERT(pi != NULL, ("pi == NULL?")); - if (pi->pi_owner != curthread) { + if (pi->pi_owner != td) { mtx_unlock(&umtx_lock); umtxq_unbusy(&key); umtxq_unlock(&key); @@ -2249,7 +2245,7 @@ do_unlock_pi(struct thread *td, struct u /* userland messed the mutex */ return (EPERM); } - uq_me = curthread->td_umtxq; + uq_me = td->td_umtxq; umtx_pi_disown(pi); /* get highest priority thread which is still sleeping. */ uq_first = TAILQ_FIRST(&pi->pi_blocked); @@ -2265,9 +2261,9 @@ do_unlock_pi(struct thread *td, struct u pri = UPRI(uq_first2->uq_thread); } } - thread_lock(curthread); - sched_lend_user_prio(curthread, pri); - thread_unlock(curthread); + thread_lock(td); + sched_lend_user_prio(td, pri); + thread_unlock(td); mtx_unlock(&umtx_lock); if (uq_first) umtxq_signal_thread(uq_first); From owner-svn-src-stable@freebsd.org Fri Oct 2 14:11:45 2015 Return-Path: Delivered-To: svn-src-stable@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 69BD1A0E5FC; Fri, 2 Oct 2015 14:11:45 +0000 (UTC) (envelope-from vangyzen@FreeBSD.org) Received: from repo.freebsd.org (repo.freebsd.org [IPv6:2001:1900:2254:2068::e6a:0]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 412B610E1; Fri, 2 Oct 2015 14:11:45 +0000 (UTC) (envelope-from vangyzen@FreeBSD.org) Received: from repo.freebsd.org ([127.0.1.70]) by repo.freebsd.org (8.15.2/8.15.2) with ESMTP id t92EBjgp023612; Fri, 2 Oct 2015 14:11:45 GMT (envelope-from vangyzen@FreeBSD.org) Received: (from vangyzen@localhost) by repo.freebsd.org (8.15.2/8.15.2/Submit) id t92EBiJf023608; Fri, 2 Oct 2015 14:11:44 GMT (envelope-from vangyzen@FreeBSD.org) Message-Id: <201510021411.t92EBiJf023608@repo.freebsd.org> X-Authentication-Warning: repo.freebsd.org: vangyzen set sender to vangyzen@FreeBSD.org using -f From: Eric van Gyzen Date: Fri, 2 Oct 2015 14:11:44 +0000 (UTC) To: src-committers@freebsd.org, svn-src-all@freebsd.org, svn-src-stable@freebsd.org, svn-src-stable-10@freebsd.org Subject: svn commit: r288495 - stable/10/usr.sbin/pmcstat X-SVN-Group: stable-10 MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit X-BeenThere: svn-src-stable@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: SVN commit messages for all the -stable branches of the src tree List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 02 Oct 2015 14:11:45 -0000 Author: vangyzen Date: Fri Oct 2 14:11:44 2015 New Revision: 288495 URL: https://svnweb.freebsd.org/changeset/base/288495 Log: MFC r280793 pmcstat.8: fix -a flag description; improve -m flag to match The -a flag reads a file saved by -O, not -o. The -m flag requires the -R flag. Copy that paragraph from -a. Sponsored by: Dell Inc. Modified: stable/10/usr.sbin/pmcstat/pmcstat.8 stable/10/usr.sbin/pmcstat/pmcstat.c Directory Properties: stable/10/ (props changed) Modified: stable/10/usr.sbin/pmcstat/pmcstat.8 ============================================================================== --- stable/10/usr.sbin/pmcstat/pmcstat.8 Fri Oct 2 13:48:32 2015 (r288494) +++ stable/10/usr.sbin/pmcstat/pmcstat.8 Fri Oct 2 14:11:44 2015 (r288495) @@ -236,7 +236,7 @@ This option requires the .Fl R option to read in samples that were previously collected and saved with the -.Fl o +.Fl O option. .It Fl c Ar cpu-spec Set the cpus for subsequent system mode PMCs specified on the @@ -302,6 +302,12 @@ is a this information is sent to the output file specified by the .Fl o option. +This option requires the +.Fl R +option to read in samples that were previously collected and +saved with the +.Fl O +option. .It Fl n Ar rate Set the default sampling rate for subsequent sampling mode PMCs specified on the command line. Modified: stable/10/usr.sbin/pmcstat/pmcstat.c ============================================================================== --- stable/10/usr.sbin/pmcstat/pmcstat.c Fri Oct 2 13:48:32 2015 (r288494) +++ stable/10/usr.sbin/pmcstat/pmcstat.c Fri Oct 2 14:11:44 2015 (r288495) @@ -938,7 +938,7 @@ main(int argc, char **argv) errx(EX_USAGE, "ERROR: options -T and -l are mutually " "exclusive."); - /* -m option is allowed with -R only. */ + /* -a and -m require -R */ if (args.pa_flags & FLAG_DO_ANNOTATE && args.pa_inputpath == NULL) errx(EX_USAGE, "ERROR: option %s requires an input file", args.pa_plugin == PMCSTAT_PL_ANNOTATE ? "-m" : "-a"); From owner-svn-src-stable@freebsd.org Fri Oct 2 14:16:39 2015 Return-Path: Delivered-To: svn-src-stable@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 1171DA0EA49; Fri, 2 Oct 2015 14:16:39 +0000 (UTC) (envelope-from vangyzen@FreeBSD.org) Received: from repo.freebsd.org (repo.freebsd.org [IPv6:2001:1900:2254:2068::e6a:0]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 0169714A5; Fri, 2 Oct 2015 14:16:39 +0000 (UTC) (envelope-from vangyzen@FreeBSD.org) Received: from repo.freebsd.org ([127.0.1.70]) by repo.freebsd.org (8.15.2/8.15.2) with ESMTP id t92EGc0Y023842; Fri, 2 Oct 2015 14:16:38 GMT (envelope-from vangyzen@FreeBSD.org) Received: (from vangyzen@localhost) by repo.freebsd.org (8.15.2/8.15.2/Submit) id t92EGc3V023840; Fri, 2 Oct 2015 14:16:38 GMT (envelope-from vangyzen@FreeBSD.org) Message-Id: <201510021416.t92EGc3V023840@repo.freebsd.org> X-Authentication-Warning: repo.freebsd.org: vangyzen set sender to vangyzen@FreeBSD.org using -f From: Eric van Gyzen Date: Fri, 2 Oct 2015 14:16:38 +0000 (UTC) To: src-committers@freebsd.org, svn-src-all@freebsd.org, svn-src-stable@freebsd.org, svn-src-stable-9@freebsd.org Subject: svn commit: r288496 - stable/9/usr.sbin/pmcstat X-SVN-Group: stable-9 MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit X-BeenThere: svn-src-stable@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: SVN commit messages for all the -stable branches of the src tree List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 02 Oct 2015 14:16:39 -0000 Author: vangyzen Date: Fri Oct 2 14:16:37 2015 New Revision: 288496 URL: https://svnweb.freebsd.org/changeset/base/288496 Log: MFC r280793 pmcstat.8: The -m flag requires the -R flag. Approved by: kib (mentor until recently) Sponsored by: Dell Inc. Modified: stable/9/usr.sbin/pmcstat/pmcstat.8 stable/9/usr.sbin/pmcstat/pmcstat.c Directory Properties: stable/9/usr.sbin/pmcstat/ (props changed) Modified: stable/9/usr.sbin/pmcstat/pmcstat.8 ============================================================================== --- stable/9/usr.sbin/pmcstat/pmcstat.8 Fri Oct 2 14:11:44 2015 (r288495) +++ stable/9/usr.sbin/pmcstat/pmcstat.8 Fri Oct 2 14:16:37 2015 (r288496) @@ -279,6 +279,12 @@ is a this information is sent to the output file specified by the .Fl o option. +This option requires the +.Fl R +option to read in samples that were previously collected and +saved with the +.Fl O +option. .It Fl n Ar rate Set the default sampling rate for subsequent sampling mode PMCs specified on the command line. Modified: stable/9/usr.sbin/pmcstat/pmcstat.c ============================================================================== --- stable/9/usr.sbin/pmcstat/pmcstat.c Fri Oct 2 14:11:44 2015 (r288495) +++ stable/9/usr.sbin/pmcstat/pmcstat.c Fri Oct 2 14:16:37 2015 (r288496) @@ -935,7 +935,7 @@ main(int argc, char **argv) errx(EX_USAGE, "ERROR: options -T and -l are mutually " "exclusive."); - /* -m option is allowed with -R only. */ + /* -m requires -R */ if (args.pa_flags & FLAG_DO_ANNOTATE && args.pa_inputpath == NULL) errx(EX_USAGE, "ERROR: option -m requires an input file"); From owner-svn-src-stable@freebsd.org Fri Oct 2 14:21:08 2015 Return-Path: Delivered-To: svn-src-stable@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 5603AA0EE01; Fri, 2 Oct 2015 14:21:08 +0000 (UTC) (envelope-from vangyzen@FreeBSD.org) Received: from repo.freebsd.org (repo.freebsd.org [IPv6:2001:1900:2254:2068::e6a:0]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 4647D1902; Fri, 2 Oct 2015 14:21:08 +0000 (UTC) (envelope-from vangyzen@FreeBSD.org) Received: from repo.freebsd.org ([127.0.1.70]) by repo.freebsd.org (8.15.2/8.15.2) with ESMTP id t92EL8S1026501; Fri, 2 Oct 2015 14:21:08 GMT (envelope-from vangyzen@FreeBSD.org) Received: (from vangyzen@localhost) by repo.freebsd.org (8.15.2/8.15.2/Submit) id t92EL7Gc026499; Fri, 2 Oct 2015 14:21:07 GMT (envelope-from vangyzen@FreeBSD.org) Message-Id: <201510021421.t92EL7Gc026499@repo.freebsd.org> X-Authentication-Warning: repo.freebsd.org: vangyzen set sender to vangyzen@FreeBSD.org using -f From: Eric van Gyzen Date: Fri, 2 Oct 2015 14:21:07 +0000 (UTC) To: src-committers@freebsd.org, svn-src-all@freebsd.org, svn-src-stable@freebsd.org, svn-src-stable-10@freebsd.org Subject: svn commit: r288497 - in stable/10: share/man/man9 sys/kern X-SVN-Group: stable-10 MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit X-BeenThere: svn-src-stable@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: SVN commit messages for all the -stable branches of the src tree List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 02 Oct 2015 14:21:08 -0000 Author: vangyzen Date: Fri Oct 2 14:21:07 2015 New Revision: 288497 URL: https://svnweb.freebsd.org/changeset/base/288497 Log: MFC r281785 Always send log(9) messages to the message buffer. It is truer to the semantics of logging for messages to *always* go to the message buffer, where they can eventually be collected and, in fact, be put into a log file. This restores the behavior prior to r70239, which seems to have changed it inadvertently. Submitted by: Eric Badger Obtained from: Dell Inc. Modified: stable/10/share/man/man9/printf.9 stable/10/sys/kern/subr_prf.c Directory Properties: stable/10/ (props changed) Modified: stable/10/share/man/man9/printf.9 ============================================================================== --- stable/10/share/man/man9/printf.9 Fri Oct 2 14:16:37 2015 (r288496) +++ stable/10/share/man/man9/printf.9 Fri Oct 2 14:21:07 2015 (r288497) @@ -67,7 +67,8 @@ The .Fn log function sends the message to the kernel logging facility, using the log level as indicated by -.Fa pri . +.Fa pri , +and to the console if no process is yet reading the log. .Pp Each of these related functions use the .Fa fmt Modified: stable/10/sys/kern/subr_prf.c ============================================================================== --- stable/10/sys/kern/subr_prf.c Fri Oct 2 14:16:37 2015 (r288496) +++ stable/10/sys/kern/subr_prf.c Fri Oct 2 14:21:07 2015 (r288497) @@ -305,7 +305,7 @@ log(int level, const char *fmt, ...) va_list ap; va_start(ap, fmt); - (void)_vprintf(level, log_open ? TOLOG : TOCONS, fmt, ap); + (void)_vprintf(level, log_open ? TOLOG : TOCONS | TOLOG, fmt, ap); va_end(ap); msgbuftrigger = 1; From owner-svn-src-stable@freebsd.org Fri Oct 2 14:24:40 2015 Return-Path: Delivered-To: svn-src-stable@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 1CA7EA0D03A; Fri, 2 Oct 2015 14:24:40 +0000 (UTC) (envelope-from vangyzen@FreeBSD.org) Received: from repo.freebsd.org (repo.freebsd.org [IPv6:2001:1900:2254:2068::e6a:0]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 0D1DC1C78; Fri, 2 Oct 2015 14:24:40 +0000 (UTC) (envelope-from vangyzen@FreeBSD.org) Received: from repo.freebsd.org ([127.0.1.70]) by repo.freebsd.org (8.15.2/8.15.2) with ESMTP id t92EOdbG027976; Fri, 2 Oct 2015 14:24:39 GMT (envelope-from vangyzen@FreeBSD.org) Received: (from vangyzen@localhost) by repo.freebsd.org (8.15.2/8.15.2/Submit) id t92EOdxp027975; Fri, 2 Oct 2015 14:24:39 GMT (envelope-from vangyzen@FreeBSD.org) Message-Id: <201510021424.t92EOdxp027975@repo.freebsd.org> X-Authentication-Warning: repo.freebsd.org: vangyzen set sender to vangyzen@FreeBSD.org using -f From: Eric van Gyzen Date: Fri, 2 Oct 2015 14:24:39 +0000 (UTC) To: src-committers@freebsd.org, svn-src-all@freebsd.org, svn-src-stable@freebsd.org, svn-src-stable-10@freebsd.org Subject: svn commit: r288498 - stable/10/sbin/dmesg X-SVN-Group: stable-10 MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit X-BeenThere: svn-src-stable@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: SVN commit messages for all the -stable branches of the src tree List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 02 Oct 2015 14:24:40 -0000 Author: vangyzen Date: Fri Oct 2 14:24:39 2015 New Revision: 288498 URL: https://svnweb.freebsd.org/changeset/base/288498 Log: MFC r281787 dmesg: accommodate message buffer growth between the sysctl calls Allocate 12.5% extra space to avoid ENOMEM when the message buffer is growing steadily. Reported by: Steve Wahl (and tested) Sponsored by: Dell Inc. Modified: stable/10/sbin/dmesg/dmesg.c Directory Properties: stable/10/ (props changed) Modified: stable/10/sbin/dmesg/dmesg.c ============================================================================== --- stable/10/sbin/dmesg/dmesg.c Fri Oct 2 14:21:07 2015 (r288497) +++ stable/10/sbin/dmesg/dmesg.c Fri Oct 2 14:24:39 2015 (r288498) @@ -116,6 +116,9 @@ main(int argc, char *argv[]) */ if (sysctlbyname("kern.msgbuf", NULL, &buflen, NULL, 0) == -1) err(1, "sysctl kern.msgbuf"); + /* Allocate extra room for growth between the sysctl calls. */ + buflen += buflen/8; + /* Allocate more than sysctl sees, for room to append \n\0. */ if ((bp = malloc(buflen + 2)) == NULL) errx(1, "malloc failed"); if (sysctlbyname("kern.msgbuf", bp, &buflen, NULL, 0) == -1) From owner-svn-src-stable@freebsd.org Fri Oct 2 14:36:44 2015 Return-Path: Delivered-To: svn-src-stable@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 0C7F7A0D9E1; Fri, 2 Oct 2015 14:36:44 +0000 (UTC) (envelope-from vangyzen@FreeBSD.org) Received: from repo.freebsd.org (repo.freebsd.org [IPv6:2001:1900:2254:2068::e6a:0]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id F0051155B; Fri, 2 Oct 2015 14:36:43 +0000 (UTC) (envelope-from vangyzen@FreeBSD.org) Received: from repo.freebsd.org ([127.0.1.70]) by repo.freebsd.org (8.15.2/8.15.2) with ESMTP id t92EahEl032467; Fri, 2 Oct 2015 14:36:43 GMT (envelope-from vangyzen@FreeBSD.org) Received: (from vangyzen@localhost) by repo.freebsd.org (8.15.2/8.15.2/Submit) id t92EafPt032456; Fri, 2 Oct 2015 14:36:41 GMT (envelope-from vangyzen@FreeBSD.org) Message-Id: <201510021436.t92EafPt032456@repo.freebsd.org> X-Authentication-Warning: repo.freebsd.org: vangyzen set sender to vangyzen@FreeBSD.org using -f From: Eric van Gyzen Date: Fri, 2 Oct 2015 14:36:41 +0000 (UTC) To: src-committers@freebsd.org, svn-src-all@freebsd.org, svn-src-stable@freebsd.org, svn-src-stable-10@freebsd.org Subject: svn commit: r288499 - in stable/10/sys: compat/linprocfs dev/hwpmc fs/procfs kern vm X-SVN-Group: stable-10 MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit X-BeenThere: svn-src-stable@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: SVN commit messages for all the -stable branches of the src tree List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 02 Oct 2015 14:36:44 -0000 Author: vangyzen Date: Fri Oct 2 14:36:41 2015 New Revision: 288499 URL: https://svnweb.freebsd.org/changeset/base/288499 Log: MFC r283924 Provide vnode in memory map info for files on tmpfs When providing memory map information to userland, populate the vnode pointer for tmpfs files. Set the memory mapping to appear as a vnode type, to match FreeBSD 9 behavior. This fixes the use of tmpfs files with the dtrace pid provider, procstat -v, procfs, linprocfs, pmc (pmcstat), and ptrace (PT_VM_ENTRY). Submitted by: Eric Badger (initial revision) Obtained from: Dell Inc. PR: 198431 Modified: stable/10/sys/compat/linprocfs/linprocfs.c stable/10/sys/dev/hwpmc/hwpmc_mod.c stable/10/sys/fs/procfs/procfs_map.c stable/10/sys/kern/kern_proc.c stable/10/sys/kern/sys_process.c stable/10/sys/vm/vm_object.c stable/10/sys/vm/vm_object.h Directory Properties: stable/10/ (props changed) Modified: stable/10/sys/compat/linprocfs/linprocfs.c ============================================================================== --- stable/10/sys/compat/linprocfs/linprocfs.c Fri Oct 2 14:24:39 2015 (r288498) +++ stable/10/sys/compat/linprocfs/linprocfs.c Fri Oct 2 14:36:41 2015 (r288499) @@ -1042,20 +1042,16 @@ linprocfs_doprocmaps(PFS_FILL_ARGS) ino = 0; if (lobj) { off = IDX_TO_OFF(lobj->size); - if (lobj->type == OBJT_VNODE) { - vp = lobj->handle; - if (vp) - vref(vp); - } - else - vp = NULL; + vp = vm_object_vnode(lobj); + if (vp != NULL) + vref(vp); if (lobj != obj) VM_OBJECT_RUNLOCK(lobj); flags = obj->flags; ref_count = obj->ref_count; shadow_count = obj->shadow_count; VM_OBJECT_RUNLOCK(obj); - if (vp) { + if (vp != NULL) { vn_fullpath(td, vp, &name, &freename); vn_lock(vp, LK_SHARED | LK_RETRY); VOP_GETATTR(vp, &vat, td->td_ucred); Modified: stable/10/sys/dev/hwpmc/hwpmc_mod.c ============================================================================== --- stable/10/sys/dev/hwpmc/hwpmc_mod.c Fri Oct 2 14:24:39 2015 (r288498) +++ stable/10/sys/dev/hwpmc/hwpmc_mod.c Fri Oct 2 14:36:41 2015 (r288499) @@ -1660,7 +1660,8 @@ pmc_log_process_mappings(struct pmc_owne continue; } - if (lobj->type != OBJT_VNODE || lobj->handle == NULL) { + vp = vm_object_vnode(lobj); + if (vp == NULL) { if (lobj != obj) VM_OBJECT_RUNLOCK(lobj); VM_OBJECT_RUNLOCK(obj); @@ -1672,7 +1673,7 @@ pmc_log_process_mappings(struct pmc_owne * vnode, so we don't emit redundant MAP-IN * directives. */ - if (entry->start == last_end && lobj->handle == last_vp) { + if (entry->start == last_end && vp == last_vp) { last_end = entry->end; if (lobj != obj) VM_OBJECT_RUNLOCK(lobj); @@ -1695,7 +1696,6 @@ pmc_log_process_mappings(struct pmc_owne last_timestamp = map->timestamp; vm_map_unlock_read(map); - vp = lobj->handle; vref(vp); if (lobj != obj) VM_OBJECT_RUNLOCK(lobj); Modified: stable/10/sys/fs/procfs/procfs_map.c ============================================================================== --- stable/10/sys/fs/procfs/procfs_map.c Fri Oct 2 14:24:39 2015 (r288498) +++ stable/10/sys/fs/procfs/procfs_map.c Fri Oct 2 14:36:41 2015 (r288499) @@ -159,11 +159,11 @@ procfs_doprocmap(PFS_FILL_ARGS) freepath = NULL; fullpath = "-"; if (lobj) { + vp = NULL; switch (lobj->type) { default: case OBJT_DEFAULT: type = "default"; - vp = NULL; break; case OBJT_VNODE: type = "vnode"; @@ -171,13 +171,19 @@ procfs_doprocmap(PFS_FILL_ARGS) vref(vp); break; case OBJT_SWAP: - type = "swap"; - vp = NULL; + if ((lobj->flags & OBJ_TMPFS_NODE) != 0) { + type = "vnode"; + if ((lobj->flags & OBJ_TMPFS) != 0) { + vp = lobj->un_pager.swp.swp_tmpfs; + vref(vp); + } + } else { + type = "swap"; + } break; case OBJT_SG: case OBJT_DEVICE: type = "device"; - vp = NULL; break; } if (lobj != obj) Modified: stable/10/sys/kern/kern_proc.c ============================================================================== --- stable/10/sys/kern/kern_proc.c Fri Oct 2 14:24:39 2015 (r288498) +++ stable/10/sys/kern/kern_proc.c Fri Oct 2 14:36:41 2015 (r288499) @@ -2074,7 +2074,15 @@ sysctl_kern_proc_ovmmap(SYSCTL_HANDLER_A vref(vp); break; case OBJT_SWAP: - kve->kve_type = KVME_TYPE_SWAP; + if ((lobj->flags & OBJ_TMPFS_NODE) != 0) { + kve->kve_type = KVME_TYPE_VNODE; + if ((lobj->flags & OBJ_TMPFS) != 0) { + vp = lobj->un_pager.swp.swp_tmpfs; + vref(vp); + } + } else { + kve->kve_type = KVME_TYPE_SWAP; + } break; case OBJT_DEVICE: kve->kve_type = KVME_TYPE_DEVICE; @@ -2300,7 +2308,15 @@ kern_proc_vmmap_out(struct proc *p, stru vref(vp); break; case OBJT_SWAP: - kve->kve_type = KVME_TYPE_SWAP; + if ((lobj->flags & OBJ_TMPFS_NODE) != 0) { + kve->kve_type = KVME_TYPE_VNODE; + if ((lobj->flags & OBJ_TMPFS) != 0) { + vp = lobj->un_pager.swp.swp_tmpfs; + vref(vp); + } + } else { + kve->kve_type = KVME_TYPE_SWAP; + } break; case OBJT_DEVICE: kve->kve_type = KVME_TYPE_DEVICE; Modified: stable/10/sys/kern/sys_process.c ============================================================================== --- stable/10/sys/kern/sys_process.c Fri Oct 2 14:24:39 2015 (r288498) +++ stable/10/sys/kern/sys_process.c Fri Oct 2 14:36:41 2015 (r288499) @@ -402,7 +402,7 @@ ptrace_vm_entry(struct thread *td, struc lobj = tobj; pve->pve_offset += tobj->backing_object_offset; } - vp = (lobj->type == OBJT_VNODE) ? lobj->handle : NULL; + vp = vm_object_vnode(lobj); if (vp != NULL) vref(vp); if (lobj != obj) Modified: stable/10/sys/vm/vm_object.c ============================================================================== --- stable/10/sys/vm/vm_object.c Fri Oct 2 14:24:39 2015 (r288498) +++ stable/10/sys/vm/vm_object.c Fri Oct 2 14:36:41 2015 (r288499) @@ -2270,6 +2270,18 @@ next_page: } } +struct vnode * +vm_object_vnode(vm_object_t object) +{ + + VM_OBJECT_ASSERT_LOCKED(object); + if (object->type == OBJT_VNODE) + return (object->handle); + if (object->type == OBJT_SWAP && (object->flags & OBJ_TMPFS) != 0) + return (object->un_pager.swp.swp_tmpfs); + return (NULL); +} + static int sysctl_vm_object_list(SYSCTL_HANDLER_ARGS) { Modified: stable/10/sys/vm/vm_object.h ============================================================================== --- stable/10/sys/vm/vm_object.h Fri Oct 2 14:24:39 2015 (r288498) +++ stable/10/sys/vm/vm_object.h Fri Oct 2 14:36:41 2015 (r288499) @@ -297,6 +297,7 @@ boolean_t vm_object_sync(vm_object_t, vm boolean_t); void vm_object_unwire(vm_object_t object, vm_ooffset_t offset, vm_size_t length, uint8_t queue); +struct vnode *vm_object_vnode(vm_object_t object); #endif /* _KERNEL */ #endif /* _VM_OBJECT_ */ From owner-svn-src-stable@freebsd.org Fri Oct 2 15:07:03 2015 Return-Path: Delivered-To: svn-src-stable@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 74EF1A0EDB8; Fri, 2 Oct 2015 15:07:03 +0000 (UTC) (envelope-from vangyzen@FreeBSD.org) Received: from repo.freebsd.org (repo.freebsd.org [IPv6:2001:1900:2254:2068::e6a:0]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 64C3F17B4; Fri, 2 Oct 2015 15:07:03 +0000 (UTC) (envelope-from vangyzen@FreeBSD.org) Received: from repo.freebsd.org ([127.0.1.70]) by repo.freebsd.org (8.15.2/8.15.2) with ESMTP id t92F73Nx044989; Fri, 2 Oct 2015 15:07:03 GMT (envelope-from vangyzen@FreeBSD.org) Received: (from vangyzen@localhost) by repo.freebsd.org (8.15.2/8.15.2/Submit) id t92F730p044988; Fri, 2 Oct 2015 15:07:03 GMT (envelope-from vangyzen@FreeBSD.org) Message-Id: <201510021507.t92F730p044988@repo.freebsd.org> X-Authentication-Warning: repo.freebsd.org: vangyzen set sender to vangyzen@FreeBSD.org using -f From: Eric van Gyzen Date: Fri, 2 Oct 2015 15:07:03 +0000 (UTC) To: src-committers@freebsd.org, svn-src-all@freebsd.org, svn-src-stable@freebsd.org, svn-src-stable-9@freebsd.org Subject: svn commit: r288500 - stable/9/sbin/dmesg X-SVN-Group: stable-9 MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit X-BeenThere: svn-src-stable@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: SVN commit messages for all the -stable branches of the src tree List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 02 Oct 2015 15:07:03 -0000 Author: vangyzen Date: Fri Oct 2 15:07:02 2015 New Revision: 288500 URL: https://svnweb.freebsd.org/changeset/base/288500 Log: MFC r281787 dmesg: accommodate message buffer growth between the sysctl calls Allocate 12.5% extra space to avoid ENOMEM when the message buffer is growing steadily. Reported by: Steve Wahl (and tested) Approved by: kib (mentor until recently) Obtained from: Dell Inc. Modified: stable/9/sbin/dmesg/dmesg.c Directory Properties: stable/9/sbin/dmesg/ (props changed) Modified: stable/9/sbin/dmesg/dmesg.c ============================================================================== --- stable/9/sbin/dmesg/dmesg.c Fri Oct 2 14:36:41 2015 (r288499) +++ stable/9/sbin/dmesg/dmesg.c Fri Oct 2 15:07:02 2015 (r288500) @@ -112,6 +112,9 @@ main(int argc, char *argv[]) */ if (sysctlbyname("kern.msgbuf", NULL, &buflen, NULL, 0) == -1) err(1, "sysctl kern.msgbuf"); + /* Allocate extra room for growth between the sysctl calls. */ + buflen += buflen/8; + /* Allocate more than sysctl sees, for room to append \n\0. */ if ((bp = malloc(buflen + 2)) == NULL) errx(1, "malloc failed"); if (sysctlbyname("kern.msgbuf", bp, &buflen, NULL, 0) == -1) From owner-svn-src-stable@freebsd.org Fri Oct 2 16:21:15 2015 Return-Path: Delivered-To: svn-src-stable@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id A3A48A0E2A2; Fri, 2 Oct 2015 16:21:15 +0000 (UTC) (envelope-from bdrewery@FreeBSD.org) Received: from repo.freebsd.org (repo.freebsd.org [IPv6:2001:1900:2254:2068::e6a:0]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 9327C1C34; Fri, 2 Oct 2015 16:21:15 +0000 (UTC) (envelope-from bdrewery@FreeBSD.org) Received: from repo.freebsd.org ([127.0.1.70]) by repo.freebsd.org (8.15.2/8.15.2) with ESMTP id t92GLF3e076655; Fri, 2 Oct 2015 16:21:15 GMT (envelope-from bdrewery@FreeBSD.org) Received: (from bdrewery@localhost) by repo.freebsd.org (8.15.2/8.15.2/Submit) id t92GLF8t076654; Fri, 2 Oct 2015 16:21:15 GMT (envelope-from bdrewery@FreeBSD.org) Message-Id: <201510021621.t92GLF8t076654@repo.freebsd.org> X-Authentication-Warning: repo.freebsd.org: bdrewery set sender to bdrewery@FreeBSD.org using -f From: Bryan Drewery Date: Fri, 2 Oct 2015 16:21:15 +0000 (UTC) To: src-committers@freebsd.org, svn-src-all@freebsd.org, svn-src-stable@freebsd.org, svn-src-stable-10@freebsd.org Subject: svn commit: r288505 - stable/10/usr.sbin/etcupdate X-SVN-Group: stable-10 MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit X-BeenThere: svn-src-stable@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: SVN commit messages for all the -stable branches of the src tree List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 02 Oct 2015 16:21:15 -0000 Author: bdrewery Date: Fri Oct 2 16:21:14 2015 New Revision: 288505 URL: https://svnweb.freebsd.org/changeset/base/288505 Log: MFC r288380: Document the post-merge actions of calling tzsetup(8) and services_mkdb(8) added in r259134. Modified: stable/10/usr.sbin/etcupdate/etcupdate.8 Directory Properties: stable/10/ (props changed) Modified: stable/10/usr.sbin/etcupdate/etcupdate.8 ============================================================================== --- stable/10/usr.sbin/etcupdate/etcupdate.8 Fri Oct 2 15:30:37 2015 (r288504) +++ stable/10/usr.sbin/etcupdate/etcupdate.8 Fri Oct 2 16:21:14 2015 (r288505) @@ -25,7 +25,7 @@ .\" .\" $FreeBSD$ .\" -.Dd October 29, 2014 +.Dd September 29, 2015 .Dt ETCUPDATE 8 .Os .Sh NAME @@ -233,6 +233,16 @@ is changed, is invoked if .Pa /etc/mail/aliases is changed, +.Xr services_mkdb 8 +is invoked if +.Pa /etc/services +is changed, +.Xr tzsetup 8 +is invoked if +.Pa /etc/localtime +is changed and if +.Fa /var/db/zoneinfo +exists, and .Pa /etc/rc.d/motd is invoked if @@ -843,7 +853,9 @@ but it has been removed in the destinati .Xr make 1 , .Xr newaliases 1 , .Xr sh 1 , -.Xr pwd_mkdb 8 +.Xr pwd_mkdb 8 , +.Xr services_mkdb 8 , +.Xr tzsetup 8 .Sh HISTORY The .Nm From owner-svn-src-stable@freebsd.org Fri Oct 2 16:21:26 2015 Return-Path: Delivered-To: svn-src-stable@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 49324A0E2DB; Fri, 2 Oct 2015 16:21:26 +0000 (UTC) (envelope-from bdrewery@FreeBSD.org) Received: from repo.freebsd.org (repo.freebsd.org [IPv6:2001:1900:2254:2068::e6a:0]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 391711D86; Fri, 2 Oct 2015 16:21:26 +0000 (UTC) (envelope-from bdrewery@FreeBSD.org) Received: from repo.freebsd.org ([127.0.1.70]) by repo.freebsd.org (8.15.2/8.15.2) with ESMTP id t92GLQ8W076706; Fri, 2 Oct 2015 16:21:26 GMT (envelope-from bdrewery@FreeBSD.org) Received: (from bdrewery@localhost) by repo.freebsd.org (8.15.2/8.15.2/Submit) id t92GLQE1076705; Fri, 2 Oct 2015 16:21:26 GMT (envelope-from bdrewery@FreeBSD.org) Message-Id: <201510021621.t92GLQE1076705@repo.freebsd.org> X-Authentication-Warning: repo.freebsd.org: bdrewery set sender to bdrewery@FreeBSD.org using -f From: Bryan Drewery Date: Fri, 2 Oct 2015 16:21:26 +0000 (UTC) To: src-committers@freebsd.org, svn-src-all@freebsd.org, svn-src-stable@freebsd.org, svn-src-stable-9@freebsd.org Subject: svn commit: r288506 - stable/9/usr.sbin/etcupdate X-SVN-Group: stable-9 MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit X-BeenThere: svn-src-stable@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: SVN commit messages for all the -stable branches of the src tree List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 02 Oct 2015 16:21:26 -0000 Author: bdrewery Date: Fri Oct 2 16:21:25 2015 New Revision: 288506 URL: https://svnweb.freebsd.org/changeset/base/288506 Log: MFC r288380: Document the post-merge actions of calling tzsetup(8) and services_mkdb(8) added in r259134. Modified: stable/9/usr.sbin/etcupdate/etcupdate.8 Directory Properties: stable/9/usr.sbin/etcupdate/ (props changed) Modified: stable/9/usr.sbin/etcupdate/etcupdate.8 ============================================================================== --- stable/9/usr.sbin/etcupdate/etcupdate.8 Fri Oct 2 16:21:14 2015 (r288505) +++ stable/9/usr.sbin/etcupdate/etcupdate.8 Fri Oct 2 16:21:25 2015 (r288506) @@ -25,7 +25,7 @@ .\" .\" $FreeBSD$ .\" -.Dd October 29, 2014 +.Dd September 29, 2015 .Dt ETCUPDATE 8 .Os .Sh NAME @@ -233,6 +233,16 @@ is changed, is invoked if .Pa /etc/mail/aliases is changed, +.Xr services_mkdb 8 +is invoked if +.Pa /etc/services +is changed, +.Xr tzsetup 8 +is invoked if +.Pa /etc/localtime +is changed and if +.Fa /var/db/zoneinfo +exists, and .Pa /etc/rc.d/motd is invoked if @@ -843,7 +853,9 @@ but it has been removed in the destinati .Xr make 1 , .Xr newaliases 1 , .Xr sh 1 , -.Xr pwd_mkdb 8 +.Xr pwd_mkdb 8 , +.Xr services_mkdb 8 , +.Xr tzsetup 8 .Sh HISTORY The .Nm From owner-svn-src-stable@freebsd.org Fri Oct 2 16:22:25 2015 Return-Path: Delivered-To: svn-src-stable@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 6A373A0E417; Fri, 2 Oct 2015 16:22:25 +0000 (UTC) (envelope-from bdrewery@FreeBSD.org) Received: from repo.freebsd.org (repo.freebsd.org [IPv6:2001:1900:2254:2068::e6a:0]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 59EB4101A; Fri, 2 Oct 2015 16:22:25 +0000 (UTC) (envelope-from bdrewery@FreeBSD.org) Received: from repo.freebsd.org ([127.0.1.70]) by repo.freebsd.org (8.15.2/8.15.2) with ESMTP id t92GMPeu078017; Fri, 2 Oct 2015 16:22:25 GMT (envelope-from bdrewery@FreeBSD.org) Received: (from bdrewery@localhost) by repo.freebsd.org (8.15.2/8.15.2/Submit) id t92GMPoR078016; Fri, 2 Oct 2015 16:22:25 GMT (envelope-from bdrewery@FreeBSD.org) Message-Id: <201510021622.t92GMPoR078016@repo.freebsd.org> X-Authentication-Warning: repo.freebsd.org: bdrewery set sender to bdrewery@FreeBSD.org using -f From: Bryan Drewery Date: Fri, 2 Oct 2015 16:22:25 +0000 (UTC) To: src-committers@freebsd.org, svn-src-all@freebsd.org, svn-src-stable@freebsd.org, svn-src-stable-9@freebsd.org Subject: svn commit: r288508 - stable/9/usr.sbin/mergemaster X-SVN-Group: stable-9 MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit X-BeenThere: svn-src-stable@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: SVN commit messages for all the -stable branches of the src tree List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 02 Oct 2015 16:22:25 -0000 Author: bdrewery Date: Fri Oct 2 16:22:24 2015 New Revision: 288508 URL: https://svnweb.freebsd.org/changeset/base/288508 Log: MFC r288381: All supported releases have the -m support from r186678, so remove the mention of it and reword this a bit to remove 'you'. Modified: stable/9/usr.sbin/mergemaster/mergemaster.8 Directory Properties: stable/9/usr.sbin/mergemaster/ (props changed) Modified: stable/9/usr.sbin/mergemaster/mergemaster.8 ============================================================================== --- stable/9/usr.sbin/mergemaster/mergemaster.8 Fri Oct 2 16:22:21 2015 (r288507) +++ stable/9/usr.sbin/mergemaster/mergemaster.8 Fri Oct 2 16:22:24 2015 (r288508) @@ -24,7 +24,7 @@ .\" .\" $FreeBSD$ .\" -.Dd November 1, 2011 +.Dd September 29, 2015 .Dt MERGEMASTER 8 .Os .Sh NAME @@ -257,14 +257,13 @@ Specify the path to the directory where .Xr make 1 . (In other words, where your sources are, but -s was already taken.) -In previous versions of +In older versions of .Nm -you needed to specify the path all the way to -.Pa src/etc . -Starting with r186678 you only need to specify the path to -.Pa src . +the path to +.Pa src/etc +was required. .Nm -will convert the path for you if you use the old method. +will convert the path if this older method is used. .It Fl t Ar /path/to/temp/root Create the temporary root environment in .Pa /path/to/temp/root From owner-svn-src-stable@freebsd.org Fri Oct 2 16:22:22 2015 Return-Path: Delivered-To: svn-src-stable@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 7B4F4A0E3F0; Fri, 2 Oct 2015 16:22:22 +0000 (UTC) (envelope-from bdrewery@FreeBSD.org) Received: from repo.freebsd.org (repo.freebsd.org [IPv6:2001:1900:2254:2068::e6a:0]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 6AFBD1014; Fri, 2 Oct 2015 16:22:22 +0000 (UTC) (envelope-from bdrewery@FreeBSD.org) Received: from repo.freebsd.org ([127.0.1.70]) by repo.freebsd.org (8.15.2/8.15.2) with ESMTP id t92GMMsj077973; Fri, 2 Oct 2015 16:22:22 GMT (envelope-from bdrewery@FreeBSD.org) Received: (from bdrewery@localhost) by repo.freebsd.org (8.15.2/8.15.2/Submit) id t92GMMnU077972; Fri, 2 Oct 2015 16:22:22 GMT (envelope-from bdrewery@FreeBSD.org) Message-Id: <201510021622.t92GMMnU077972@repo.freebsd.org> X-Authentication-Warning: repo.freebsd.org: bdrewery set sender to bdrewery@FreeBSD.org using -f From: Bryan Drewery Date: Fri, 2 Oct 2015 16:22:22 +0000 (UTC) To: src-committers@freebsd.org, svn-src-all@freebsd.org, svn-src-stable@freebsd.org, svn-src-stable-10@freebsd.org Subject: svn commit: r288507 - stable/10/usr.sbin/mergemaster X-SVN-Group: stable-10 MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit X-BeenThere: svn-src-stable@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: SVN commit messages for all the -stable branches of the src tree List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 02 Oct 2015 16:22:22 -0000 Author: bdrewery Date: Fri Oct 2 16:22:21 2015 New Revision: 288507 URL: https://svnweb.freebsd.org/changeset/base/288507 Log: MFC r288381: All supported releases have the -m support from r186678, so remove the mention of it and reword this a bit to remove 'you'. Modified: stable/10/usr.sbin/mergemaster/mergemaster.8 Directory Properties: stable/10/ (props changed) Modified: stable/10/usr.sbin/mergemaster/mergemaster.8 ============================================================================== --- stable/10/usr.sbin/mergemaster/mergemaster.8 Fri Oct 2 16:21:25 2015 (r288506) +++ stable/10/usr.sbin/mergemaster/mergemaster.8 Fri Oct 2 16:22:21 2015 (r288507) @@ -24,7 +24,7 @@ .\" .\" $FreeBSD$ .\" -.Dd November 1, 2011 +.Dd September 29, 2015 .Dt MERGEMASTER 8 .Os .Sh NAME @@ -257,14 +257,13 @@ Specify the path to the directory where .Xr make 1 . (In other words, where your sources are, but -s was already taken.) -In previous versions of +In older versions of .Nm -you needed to specify the path all the way to -.Pa src/etc . -Starting with r186678 you only need to specify the path to -.Pa src . +the path to +.Pa src/etc +was required. .Nm -will convert the path for you if you use the old method. +will convert the path if this older method is used. .It Fl t Ar /path/to/temp/root Create the temporary root environment in .Pa /path/to/temp/root From owner-svn-src-stable@freebsd.org Fri Oct 2 16:30:54 2015 Return-Path: Delivered-To: svn-src-stable@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id E8F79A0E9D2; Fri, 2 Oct 2015 16:30:54 +0000 (UTC) (envelope-from hiren@FreeBSD.org) Received: from repo.freebsd.org (repo.freebsd.org [IPv6:2001:1900:2254:2068::e6a:0]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id D8DD71A2C; Fri, 2 Oct 2015 16:30:54 +0000 (UTC) (envelope-from hiren@FreeBSD.org) Received: from repo.freebsd.org ([127.0.1.70]) by repo.freebsd.org (8.15.2/8.15.2) with ESMTP id t92GUsGB079033; Fri, 2 Oct 2015 16:30:54 GMT (envelope-from hiren@FreeBSD.org) Received: (from hiren@localhost) by repo.freebsd.org (8.15.2/8.15.2/Submit) id t92GUsin079032; Fri, 2 Oct 2015 16:30:54 GMT (envelope-from hiren@FreeBSD.org) Message-Id: <201510021630.t92GUsin079032@repo.freebsd.org> X-Authentication-Warning: repo.freebsd.org: hiren set sender to hiren@FreeBSD.org using -f From: Hiren Panchasara Date: Fri, 2 Oct 2015 16:30:54 +0000 (UTC) To: src-committers@freebsd.org, svn-src-all@freebsd.org, svn-src-stable@freebsd.org, svn-src-stable-10@freebsd.org Subject: svn commit: r288509 - stable/10/sys/netinet X-SVN-Group: stable-10 MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit X-BeenThere: svn-src-stable@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: SVN commit messages for all the -stable branches of the src tree List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 02 Oct 2015 16:30:55 -0000 Author: hiren Date: Fri Oct 2 16:30:54 2015 New Revision: 288509 URL: https://svnweb.freebsd.org/changeset/base/288509 Log: MFC r287830 Remove unnecessary tcp state transition call. Modified: stable/10/sys/netinet/tcp_usrreq.c Directory Properties: stable/10/ (props changed) Modified: stable/10/sys/netinet/tcp_usrreq.c ============================================================================== --- stable/10/sys/netinet/tcp_usrreq.c Fri Oct 2 16:22:24 2015 (r288508) +++ stable/10/sys/netinet/tcp_usrreq.c Fri Oct 2 16:30:54 2015 (r288509) @@ -1746,9 +1746,9 @@ tcp_usrclosed(struct tcpcb *tp) #ifdef TCP_OFFLOAD tcp_offload_listen_stop(tp); #endif + tcp_state_change(tp, TCPS_CLOSED); /* FALLTHROUGH */ case TCPS_CLOSED: - tcp_state_change(tp, TCPS_CLOSED); tp = tcp_close(tp); /* * tcp_close() should never return NULL here as the socket is From owner-svn-src-stable@freebsd.org Fri Oct 2 16:36:17 2015 Return-Path: Delivered-To: svn-src-stable@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id C8E76A0ED9D; Fri, 2 Oct 2015 16:36:17 +0000 (UTC) (envelope-from delphij@FreeBSD.org) Received: from repo.freebsd.org (repo.freebsd.org [IPv6:2001:1900:2254:2068::e6a:0]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id B88D6100B; Fri, 2 Oct 2015 16:36:17 +0000 (UTC) (envelope-from delphij@FreeBSD.org) Received: from repo.freebsd.org ([127.0.1.70]) by repo.freebsd.org (8.15.2/8.15.2) with ESMTP id t92GaHw1082522; Fri, 2 Oct 2015 16:36:17 GMT (envelope-from delphij@FreeBSD.org) Received: (from delphij@localhost) by repo.freebsd.org (8.15.2/8.15.2/Submit) id t92GaHDG082521; Fri, 2 Oct 2015 16:36:17 GMT (envelope-from delphij@FreeBSD.org) Message-Id: <201510021636.t92GaHDG082521@repo.freebsd.org> X-Authentication-Warning: repo.freebsd.org: delphij set sender to delphij@FreeBSD.org using -f From: Xin LI Date: Fri, 2 Oct 2015 16:36:17 +0000 (UTC) To: src-committers@freebsd.org, svn-src-all@freebsd.org, svn-src-stable@freebsd.org, svn-src-stable-10@freebsd.org Subject: svn commit: r288511 - in stable: 10/usr.sbin/rpcbind 9/usr.sbin/rpcbind X-SVN-Group: stable-10 MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit X-BeenThere: svn-src-stable@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: SVN commit messages for all the -stable branches of the src tree List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 02 Oct 2015 16:36:17 -0000 Author: delphij Date: Fri Oct 2 16:36:16 2015 New Revision: 288511 URL: https://svnweb.freebsd.org/changeset/base/288511 Log: Fix a regression with SA-15:24 patch that prevented NIS from working. Modified: stable/10/usr.sbin/rpcbind/rpcb_svc_com.c Changes in other areas also in this revision: Modified: stable/9/usr.sbin/rpcbind/rpcb_svc_com.c Modified: stable/10/usr.sbin/rpcbind/rpcb_svc_com.c ============================================================================== --- stable/10/usr.sbin/rpcbind/rpcb_svc_com.c Fri Oct 2 16:35:41 2015 (r288510) +++ stable/10/usr.sbin/rpcbind/rpcb_svc_com.c Fri Oct 2 16:36:16 2015 (r288511) @@ -1053,12 +1053,15 @@ static bool_t netbuf_copybuf(struct netbuf *dst, const struct netbuf *src) { - assert(dst->buf == NULL); + if (dst->len != src->len || dst->buf == NULL) { + if (dst->buf != NULL) + free(dst->buf); + if ((dst->buf = malloc(src->len)) == NULL) + return (FALSE); - if ((dst->buf = malloc(src->len)) == NULL) - return (FALSE); + dst->maxlen = dst->len = src->len; + } - dst->maxlen = dst->len = src->len; memcpy(dst->buf, src->buf, src->len); return (TRUE); } From owner-svn-src-stable@freebsd.org Fri Oct 2 16:36:17 2015 Return-Path: Delivered-To: svn-src-stable@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 29A76A0ED99; Fri, 2 Oct 2015 16:36:17 +0000 (UTC) (envelope-from delphij@FreeBSD.org) Received: from repo.freebsd.org (repo.freebsd.org [IPv6:2001:1900:2254:2068::e6a:0]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 19242100A; Fri, 2 Oct 2015 16:36:17 +0000 (UTC) (envelope-from delphij@FreeBSD.org) Received: from repo.freebsd.org ([127.0.1.70]) by repo.freebsd.org (8.15.2/8.15.2) with ESMTP id t92GaGUw082515; Fri, 2 Oct 2015 16:36:16 GMT (envelope-from delphij@FreeBSD.org) Received: (from delphij@localhost) by repo.freebsd.org (8.15.2/8.15.2/Submit) id t92GaGQM082514; Fri, 2 Oct 2015 16:36:16 GMT (envelope-from delphij@FreeBSD.org) Message-Id: <201510021636.t92GaGQM082514@repo.freebsd.org> X-Authentication-Warning: repo.freebsd.org: delphij set sender to delphij@FreeBSD.org using -f From: Xin LI Date: Fri, 2 Oct 2015 16:36:16 +0000 (UTC) To: src-committers@freebsd.org, svn-src-all@freebsd.org, svn-src-stable@freebsd.org, svn-src-stable-9@freebsd.org Subject: svn commit: r288511 - in stable: 10/usr.sbin/rpcbind 9/usr.sbin/rpcbind X-SVN-Group: stable-9 MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit X-BeenThere: svn-src-stable@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: SVN commit messages for all the -stable branches of the src tree List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 02 Oct 2015 16:36:17 -0000 Author: delphij Date: Fri Oct 2 16:36:16 2015 New Revision: 288511 URL: https://svnweb.freebsd.org/changeset/base/288511 Log: Fix a regression with SA-15:24 patch that prevented NIS from working. Modified: stable/9/usr.sbin/rpcbind/rpcb_svc_com.c Changes in other areas also in this revision: Modified: stable/10/usr.sbin/rpcbind/rpcb_svc_com.c Modified: stable/9/usr.sbin/rpcbind/rpcb_svc_com.c ============================================================================== --- stable/9/usr.sbin/rpcbind/rpcb_svc_com.c Fri Oct 2 16:35:41 2015 (r288510) +++ stable/9/usr.sbin/rpcbind/rpcb_svc_com.c Fri Oct 2 16:36:16 2015 (r288511) @@ -1053,12 +1053,15 @@ static bool_t netbuf_copybuf(struct netbuf *dst, const struct netbuf *src) { - assert(dst->buf == NULL); + if (dst->len != src->len || dst->buf == NULL) { + if (dst->buf != NULL) + free(dst->buf); + if ((dst->buf = malloc(src->len)) == NULL) + return (FALSE); - if ((dst->buf = malloc(src->len)) == NULL) - return (FALSE); + dst->maxlen = dst->len = src->len; + } - dst->maxlen = dst->len = src->len; memcpy(dst->buf, src->buf, src->len); return (TRUE); } From owner-svn-src-stable@freebsd.org Fri Oct 2 19:41:47 2015 Return-Path: Delivered-To: svn-src-stable@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id E9F74A0D547; Fri, 2 Oct 2015 19:41:47 +0000 (UTC) (envelope-from mav@FreeBSD.org) Received: from repo.freebsd.org (repo.freebsd.org [IPv6:2001:1900:2254:2068::e6a:0]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id D9AB11CC7; Fri, 2 Oct 2015 19:41:47 +0000 (UTC) (envelope-from mav@FreeBSD.org) Received: from repo.freebsd.org ([127.0.1.70]) by repo.freebsd.org (8.15.2/8.15.2) with ESMTP id t92Jfl7B068152; Fri, 2 Oct 2015 19:41:47 GMT (envelope-from mav@FreeBSD.org) Received: (from mav@localhost) by repo.freebsd.org (8.15.2/8.15.2/Submit) id t92JflSf068151; Fri, 2 Oct 2015 19:41:47 GMT (envelope-from mav@FreeBSD.org) Message-Id: <201510021941.t92JflSf068151@repo.freebsd.org> X-Authentication-Warning: repo.freebsd.org: mav set sender to mav@FreeBSD.org using -f From: Alexander Motin Date: Fri, 2 Oct 2015 19:41:47 +0000 (UTC) To: src-committers@freebsd.org, svn-src-all@freebsd.org, svn-src-stable@freebsd.org, svn-src-stable-10@freebsd.org Subject: svn commit: r288516 - stable/10/sys/cddl/contrib/opensolaris/uts/common/fs/zfs X-SVN-Group: stable-10 MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit X-BeenThere: svn-src-stable@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: SVN commit messages for all the -stable branches of the src tree List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 02 Oct 2015 19:41:48 -0000 Author: mav Date: Fri Oct 2 19:41:47 2015 New Revision: 288516 URL: https://svnweb.freebsd.org/changeset/base/288516 Log: MFC r269117: Make sysctls under vfs.zfs.zfetch writeable. I don't see any reason for them to be read-only, while tuning them without reboot is much more convenient for experiments. Modified: stable/10/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/dmu_zfetch.c Directory Properties: stable/10/ (props changed) Modified: stable/10/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/dmu_zfetch.c ============================================================================== --- stable/10/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/dmu_zfetch.c Fri Oct 2 17:58:16 2015 (r288515) +++ stable/10/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/dmu_zfetch.c Fri Oct 2 19:41:47 2015 (r288516) @@ -59,13 +59,13 @@ TUNABLE_INT("vfs.zfs.zfetch.max_streams" SYSCTL_UINT(_vfs_zfs_zfetch, OID_AUTO, max_streams, CTLFLAG_RW, &zfetch_max_streams, 0, "Max # of streams per zfetch"); TUNABLE_INT("vfs.zfs.zfetch.min_sec_reap", &zfetch_min_sec_reap); -SYSCTL_UINT(_vfs_zfs_zfetch, OID_AUTO, min_sec_reap, CTLFLAG_RDTUN, +SYSCTL_UINT(_vfs_zfs_zfetch, OID_AUTO, min_sec_reap, CTLFLAG_RWTUN, &zfetch_min_sec_reap, 0, "Min time before stream reclaim"); TUNABLE_INT("vfs.zfs.zfetch.block_cap", &zfetch_block_cap); -SYSCTL_UINT(_vfs_zfs_zfetch, OID_AUTO, block_cap, CTLFLAG_RDTUN, +SYSCTL_UINT(_vfs_zfs_zfetch, OID_AUTO, block_cap, CTLFLAG_RWTUN, &zfetch_block_cap, 0, "Max number of blocks to fetch at a time"); TUNABLE_QUAD("vfs.zfs.zfetch.array_rd_sz", &zfetch_array_rd_sz); -SYSCTL_UQUAD(_vfs_zfs_zfetch, OID_AUTO, array_rd_sz, CTLFLAG_RDTUN, +SYSCTL_UQUAD(_vfs_zfs_zfetch, OID_AUTO, array_rd_sz, CTLFLAG_RWTUN, &zfetch_array_rd_sz, 0, "Number of bytes in a array_read at which we stop prefetching"); From owner-svn-src-stable@freebsd.org Fri Oct 2 19:59:44 2015 Return-Path: Delivered-To: svn-src-stable@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 11806A0E6C6; Fri, 2 Oct 2015 19:59:44 +0000 (UTC) (envelope-from mav@FreeBSD.org) Received: from repo.freebsd.org (repo.freebsd.org [IPv6:2001:1900:2254:2068::e6a:0]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id EA90519E8; Fri, 2 Oct 2015 19:59:43 +0000 (UTC) (envelope-from mav@FreeBSD.org) Received: from repo.freebsd.org ([127.0.1.70]) by repo.freebsd.org (8.15.2/8.15.2) with ESMTP id t92JxhMK072614; Fri, 2 Oct 2015 19:59:43 GMT (envelope-from mav@FreeBSD.org) Received: (from mav@localhost) by repo.freebsd.org (8.15.2/8.15.2/Submit) id t92JxhtE072613; Fri, 2 Oct 2015 19:59:43 GMT (envelope-from mav@FreeBSD.org) Message-Id: <201510021959.t92JxhtE072613@repo.freebsd.org> X-Authentication-Warning: repo.freebsd.org: mav set sender to mav@FreeBSD.org using -f From: Alexander Motin Date: Fri, 2 Oct 2015 19:59:43 +0000 (UTC) To: src-committers@freebsd.org, svn-src-all@freebsd.org, svn-src-stable@freebsd.org, svn-src-stable-10@freebsd.org Subject: svn commit: r288517 - stable/10/sys/cddl/contrib/opensolaris/uts/common/fs/zfs X-SVN-Group: stable-10 MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit X-BeenThere: svn-src-stable@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: SVN commit messages for all the -stable branches of the src tree List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 02 Oct 2015 19:59:44 -0000 Author: mav Date: Fri Oct 2 19:59:43 2015 New Revision: 288517 URL: https://svnweb.freebsd.org/changeset/base/288517 Log: MFC r275780 (by delphij): Add a loader tunable, vfs.zfs.arc_meta_min, which controls how much metadata ZFS should keep in ARC at minimum. In arc_evict(), when doing recycle, take more factors into account by applying the following policy: 1. If no evictable data, evict metadata; 2. If no evictable metadata, evict data; 3. If we hit arc_meta_limit, evict metadata; 4. If we haven't hit arc_meta_min, evict data; 5* (Illumos only, not present in new FreeBSD code, yet) evict the oldest cached element from data and metadata. (FreeBSD) evict the data type specified by caller, which is the existing behavior. Note that because of our splitted locks (implemented in r205231 to improve scalability by reducing lock contention), implementing the fifth Illumos behavior will not be cheap, so for now just implement the 1-4 and fall back to current behavior for 5. Illumos issue: 5368 ARC should cache more metadata Modified: stable/10/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/arc.c Directory Properties: stable/10/ (props changed) Modified: stable/10/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/arc.c ============================================================================== --- stable/10/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/arc.c Fri Oct 2 19:41:47 2015 (r288516) +++ stable/10/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/arc.c Fri Oct 2 19:59:43 2015 (r288517) @@ -197,6 +197,7 @@ static boolean_t arc_warm; uint64_t zfs_arc_max; uint64_t zfs_arc_min; uint64_t zfs_arc_meta_limit = 0; +uint64_t zfs_arc_meta_min = 0; int zfs_arc_grow_retry = 0; int zfs_arc_shrink_shift = 0; int zfs_arc_p_min_shift = 0; @@ -220,6 +221,7 @@ SYSINIT(arc_free_target_init, SI_SUB_KTH TUNABLE_QUAD("vfs.zfs.arc_max", &zfs_arc_max); TUNABLE_QUAD("vfs.zfs.arc_min", &zfs_arc_min); TUNABLE_QUAD("vfs.zfs.arc_meta_limit", &zfs_arc_meta_limit); +TUNABLE_QUAD("vfs.zfs.arc_meta_min", &zfs_arc_meta_min); TUNABLE_QUAD("vfs.zfs.arc_average_blocksize", &zfs_arc_average_blocksize); TUNABLE_INT("vfs.zfs.arc_shrink_shift", &zfs_arc_shrink_shift); SYSCTL_DECL(_vfs_zfs); @@ -429,6 +431,7 @@ typedef struct arc_stats { kstat_named_t arcstat_meta_used; kstat_named_t arcstat_meta_limit; kstat_named_t arcstat_meta_max; + kstat_named_t arcstat_meta_min; } arc_stats_t; static arc_stats_t arc_stats = { @@ -509,7 +512,8 @@ static arc_stats_t arc_stats = { { "duplicate_reads", KSTAT_DATA_UINT64 }, { "arc_meta_used", KSTAT_DATA_UINT64 }, { "arc_meta_limit", KSTAT_DATA_UINT64 }, - { "arc_meta_max", KSTAT_DATA_UINT64 } + { "arc_meta_max", KSTAT_DATA_UINT64 }, + { "arc_meta_min", KSTAT_DATA_UINT64 } }; #define ARCSTAT(stat) (arc_stats.stat.value.ui64) @@ -572,6 +576,7 @@ static arc_state_t *arc_l2c_only; #define arc_c_min ARCSTAT(arcstat_c_min) /* min target cache size */ #define arc_c_max ARCSTAT(arcstat_c_max) /* max target cache size */ #define arc_meta_limit ARCSTAT(arcstat_meta_limit) /* max size for metadata */ +#define arc_meta_min ARCSTAT(arcstat_meta_min) /* min size for metadata */ #define arc_meta_used ARCSTAT(arcstat_meta_used) /* size of metadata */ #define arc_meta_max ARCSTAT(arcstat_meta_max) /* max size of metadata */ @@ -2041,6 +2046,49 @@ arc_evict(arc_state_t *state, uint64_t s evicted_state = (state == arc_mru) ? arc_mru_ghost : arc_mfu_ghost; + /* + * Decide which "type" (data vs metadata) to recycle from. + * + * If we are over the metadata limit, recycle from metadata. + * If we are under the metadata minimum, recycle from data. + * Otherwise, recycle from whichever type has the oldest (least + * recently accessed) header. This is not yet implemented. + */ + if (recycle) { + arc_buf_contents_t realtype; + if (state->arcs_lsize[ARC_BUFC_DATA] == 0) { + realtype = ARC_BUFC_METADATA; + } else if (state->arcs_lsize[ARC_BUFC_METADATA] == 0) { + realtype = ARC_BUFC_DATA; + } else if (arc_meta_used >= arc_meta_limit) { + realtype = ARC_BUFC_METADATA; + } else if (arc_meta_used <= arc_meta_min) { + realtype = ARC_BUFC_DATA; + } else { +#ifdef illumos + if (data_hdr->b_arc_access < + metadata_hdr->b_arc_access) { + realtype = ARC_BUFC_DATA; + } else { + realtype = ARC_BUFC_METADATA; + } +#else + /* TODO */ + realtype = type; +#endif + } + if (realtype != type) { + /* + * If we want to evict from a different list, + * we can not recycle, because DATA vs METADATA + * buffers are segregated into different kmem + * caches (and vmem arenas). + */ + type = realtype; + recycle = B_FALSE; + } + } + if (type == ARC_BUFC_METADATA) { offset = 0; list_count = ARC_BUFC_NUMMETADATALISTS; @@ -4194,6 +4242,12 @@ arc_init(void) if (arc_c_min < arc_meta_limit / 2 && zfs_arc_min == 0) arc_c_min = arc_meta_limit / 2; + if (zfs_arc_meta_min > 0) { + arc_meta_min = zfs_arc_meta_min; + } else { + arc_meta_min = arc_c_min / 2; + } + if (zfs_arc_grow_retry > 0) arc_grow_retry = zfs_arc_grow_retry; From owner-svn-src-stable@freebsd.org Fri Oct 2 20:03:53 2015 Return-Path: Delivered-To: svn-src-stable@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 7FC62A0EABA; Fri, 2 Oct 2015 20:03:53 +0000 (UTC) (envelope-from mav@FreeBSD.org) Received: from repo.freebsd.org (repo.freebsd.org [IPv6:2001:1900:2254:2068::e6a:0]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 6F6AD1E0B; Fri, 2 Oct 2015 20:03:53 +0000 (UTC) (envelope-from mav@FreeBSD.org) Received: from repo.freebsd.org ([127.0.1.70]) by repo.freebsd.org (8.15.2/8.15.2) with ESMTP id t92K3rCT076490; Fri, 2 Oct 2015 20:03:53 GMT (envelope-from mav@FreeBSD.org) Received: (from mav@localhost) by repo.freebsd.org (8.15.2/8.15.2/Submit) id t92K3r6a076489; Fri, 2 Oct 2015 20:03:53 GMT (envelope-from mav@FreeBSD.org) Message-Id: <201510022003.t92K3r6a076489@repo.freebsd.org> X-Authentication-Warning: repo.freebsd.org: mav set sender to mav@FreeBSD.org using -f From: Alexander Motin Date: Fri, 2 Oct 2015 20:03:53 +0000 (UTC) To: src-committers@freebsd.org, svn-src-all@freebsd.org, svn-src-stable@freebsd.org, svn-src-stable-10@freebsd.org Subject: svn commit: r288518 - stable/10/sys/cddl/contrib/opensolaris/uts/common/fs/zfs X-SVN-Group: stable-10 MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit X-BeenThere: svn-src-stable@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: SVN commit messages for all the -stable branches of the src tree List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 02 Oct 2015 20:03:53 -0000 Author: mav Date: Fri Oct 2 20:03:52 2015 New Revision: 288518 URL: https://svnweb.freebsd.org/changeset/base/288518 Log: MFC r277452 (by will): Fix arc__shrink DTrace probe's to_free argument. Remove the unnecessary #ifdef _KERNEL, which did not differ in the true or false cases. Actually set the value of to_free before using it. Modified: stable/10/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/arc.c Directory Properties: stable/10/ (props changed) Modified: stable/10/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/arc.c ============================================================================== --- stable/10/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/arc.c Fri Oct 2 19:59:43 2015 (r288517) +++ stable/10/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/arc.c Fri Oct 2 20:03:52 2015 (r288518) @@ -2543,13 +2543,9 @@ arc_shrink(void) if (arc_c > arc_c_min) { uint64_t to_free; + to_free = arc_c >> arc_shrink_shift; DTRACE_PROBE4(arc__shrink, uint64_t, arc_c, uint64_t, arc_c_min, uint64_t, arc_p, uint64_t, to_free); -#ifdef _KERNEL - to_free = arc_c >> arc_shrink_shift; -#else - to_free = arc_c >> arc_shrink_shift; -#endif if (arc_c > arc_c_min + to_free) atomic_add_64(&arc_c, -to_free); else From owner-svn-src-stable@freebsd.org Fri Oct 2 20:07:04 2015 Return-Path: Delivered-To: svn-src-stable@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 254ECA0ED0F; Fri, 2 Oct 2015 20:07:04 +0000 (UTC) (envelope-from mav@FreeBSD.org) Received: from repo.freebsd.org (repo.freebsd.org [IPv6:2001:1900:2254:2068::e6a:0]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 1523D1FD0; Fri, 2 Oct 2015 20:07:04 +0000 (UTC) (envelope-from mav@FreeBSD.org) Received: from repo.freebsd.org ([127.0.1.70]) by repo.freebsd.org (8.15.2/8.15.2) with ESMTP id t92K73Ln076683; Fri, 2 Oct 2015 20:07:03 GMT (envelope-from mav@FreeBSD.org) Received: (from mav@localhost) by repo.freebsd.org (8.15.2/8.15.2/Submit) id t92K73QQ076682; Fri, 2 Oct 2015 20:07:03 GMT (envelope-from mav@FreeBSD.org) Message-Id: <201510022007.t92K73QQ076682@repo.freebsd.org> X-Authentication-Warning: repo.freebsd.org: mav set sender to mav@FreeBSD.org using -f From: Alexander Motin Date: Fri, 2 Oct 2015 20:07:03 +0000 (UTC) To: src-committers@freebsd.org, svn-src-all@freebsd.org, svn-src-stable@freebsd.org, svn-src-stable-10@freebsd.org Subject: svn commit: r288519 - stable/10/sys/cddl/contrib/opensolaris/uts/common/fs/zfs X-SVN-Group: stable-10 MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit X-BeenThere: svn-src-stable@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: SVN commit messages for all the -stable branches of the src tree List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 02 Oct 2015 20:07:04 -0000 Author: mav Date: Fri Oct 2 20:07:03 2015 New Revision: 288519 URL: https://svnweb.freebsd.org/changeset/base/288519 Log: MFC r277826 (by delphij): Diff reduction with upstream. The actual change was merged in r272483 already. Modified: stable/10/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/arc.c Directory Properties: stable/10/ (props changed) Modified: stable/10/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/arc.c ============================================================================== --- stable/10/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/arc.c Fri Oct 2 20:03:52 2015 (r288518) +++ stable/10/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/arc.c Fri Oct 2 20:07:03 2015 (r288519) @@ -20,6 +20,7 @@ */ /* * Copyright (c) 2005, 2010, Oracle and/or its affiliates. All rights reserved. + * Copyright (c) 2012, Joyent, Inc. All rights reserved. * Copyright (c) 2011, 2014 by Delphix. All rights reserved. * Copyright (c) 2014 by Saso Kiselkov. All rights reserved. * Copyright 2014 Nexenta Systems, Inc. All rights reserved. From owner-svn-src-stable@freebsd.org Fri Oct 2 20:09:17 2015 Return-Path: Delivered-To: svn-src-stable@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 98E1BA0EE59; Fri, 2 Oct 2015 20:09:17 +0000 (UTC) (envelope-from mav@FreeBSD.org) Received: from repo.freebsd.org (repo.freebsd.org [IPv6:2001:1900:2254:2068::e6a:0]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 88A7B113E; Fri, 2 Oct 2015 20:09:17 +0000 (UTC) (envelope-from mav@FreeBSD.org) Received: from repo.freebsd.org ([127.0.1.70]) by repo.freebsd.org (8.15.2/8.15.2) with ESMTP id t92K9HMF076825; Fri, 2 Oct 2015 20:09:17 GMT (envelope-from mav@FreeBSD.org) Received: (from mav@localhost) by repo.freebsd.org (8.15.2/8.15.2/Submit) id t92K9Hjl076824; Fri, 2 Oct 2015 20:09:17 GMT (envelope-from mav@FreeBSD.org) Message-Id: <201510022009.t92K9Hjl076824@repo.freebsd.org> X-Authentication-Warning: repo.freebsd.org: mav set sender to mav@FreeBSD.org using -f From: Alexander Motin Date: Fri, 2 Oct 2015 20:09:17 +0000 (UTC) To: src-committers@freebsd.org, svn-src-all@freebsd.org, svn-src-stable@freebsd.org, svn-src-stable-10@freebsd.org Subject: svn commit: r288520 - stable/10/sys/cddl/contrib/opensolaris/uts/common/fs/zfs X-SVN-Group: stable-10 MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit X-BeenThere: svn-src-stable@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: SVN commit messages for all the -stable branches of the src tree List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 02 Oct 2015 20:09:17 -0000 Author: mav Date: Fri Oct 2 20:09:16 2015 New Revision: 288520 URL: https://svnweb.freebsd.org/changeset/base/288520 Log: MFC r279996 (by smh): Allow zvol_geom_worker to process BIO_DELETE's If zvol_geom_start is called with a BIO_DELETE from a thread which can sleep it queues it for later processing by the zvol_geom_worker. The zvol_geom_worker didn't have a delete case so would simply loose the bio hence preventing the original caller from every completing. In addition an other unknown types would suffer the same fate. Allow zvol_geom_worker to process BIO_DELETE's via zvol_strategy and return unsupported for all unknown bio types. Modified: stable/10/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zvol.c Directory Properties: stable/10/ (props changed) Modified: stable/10/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zvol.c ============================================================================== --- stable/10/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zvol.c Fri Oct 2 20:07:03 2015 (r288519) +++ stable/10/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zvol.c Fri Oct 2 20:09:16 2015 (r288520) @@ -2776,8 +2776,12 @@ zvol_geom_worker(void *arg) break; case BIO_READ: case BIO_WRITE: + case BIO_DELETE: zvol_strategy(bp); break; + default: + g_io_deliver(bp, EOPNOTSUPP); + break; } } } From owner-svn-src-stable@freebsd.org Fri Oct 2 20:13:57 2015 Return-Path: Delivered-To: svn-src-stable@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id CCA44A0E23D; Fri, 2 Oct 2015 20:13:57 +0000 (UTC) (envelope-from mav@FreeBSD.org) Received: from repo.freebsd.org (repo.freebsd.org [IPv6:2001:1900:2254:2068::e6a:0]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id BA80117A3; Fri, 2 Oct 2015 20:13:57 +0000 (UTC) (envelope-from mav@FreeBSD.org) Received: from repo.freebsd.org ([127.0.1.70]) by repo.freebsd.org (8.15.2/8.15.2) with ESMTP id t92KDvbH080814; Fri, 2 Oct 2015 20:13:57 GMT (envelope-from mav@FreeBSD.org) Received: (from mav@localhost) by repo.freebsd.org (8.15.2/8.15.2/Submit) id t92KDvt5080813; Fri, 2 Oct 2015 20:13:57 GMT (envelope-from mav@FreeBSD.org) Message-Id: <201510022013.t92KDvt5080813@repo.freebsd.org> X-Authentication-Warning: repo.freebsd.org: mav set sender to mav@FreeBSD.org using -f From: Alexander Motin Date: Fri, 2 Oct 2015 20:13:57 +0000 (UTC) To: src-committers@freebsd.org, svn-src-all@freebsd.org, svn-src-stable@freebsd.org, svn-src-stable-10@freebsd.org Subject: svn commit: r288521 - stable/10/sys/cddl/contrib/opensolaris/uts/common/sys X-SVN-Group: stable-10 MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit X-BeenThere: svn-src-stable@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: SVN commit messages for all the -stable branches of the src tree List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 02 Oct 2015 20:13:57 -0000 Author: mav Date: Fri Oct 2 20:13:56 2015 New Revision: 288521 URL: https://svnweb.freebsd.org/changeset/base/288521 Log: MFC r284591 (by avg): illums compat: use flsl/flsll for highbit/highbit64 Do that only when when fast inline versions are available. At the moment that can be the case only in the kernel and not for all platforms. The original code uses the binary search and that's kept as a fallback. This is a micro optimization. Modified: stable/10/sys/cddl/contrib/opensolaris/uts/common/sys/sysmacros.h Directory Properties: stable/10/ (props changed) Modified: stable/10/sys/cddl/contrib/opensolaris/uts/common/sys/sysmacros.h ============================================================================== --- stable/10/sys/cddl/contrib/opensolaris/uts/common/sys/sysmacros.h Fri Oct 2 20:09:16 2015 (r288520) +++ stable/10/sys/cddl/contrib/opensolaris/uts/common/sys/sysmacros.h Fri Oct 2 20:13:56 2015 (r288521) @@ -32,6 +32,9 @@ #include #include +#if defined(__FreeBSD__) && defined(_KERNEL) +#include +#endif #ifdef __cplusplus extern "C" { @@ -382,6 +385,9 @@ extern unsigned char bcd_to_byte[256]; static __inline int highbit(ulong_t i) { +#if defined(__FreeBSD__) && defined(_KERNEL) && defined(HAVE_INLINE_FLSL) + return (flsl(i)); +#else register int h = 1; if (i == 0) @@ -407,6 +413,7 @@ highbit(ulong_t i) h += 1; } return (h); +#endif } /* @@ -416,6 +423,9 @@ highbit(ulong_t i) static __inline int highbit64(uint64_t i) { +#if defined(__FreeBSD__) && defined(_KERNEL) && defined(HAVE_INLINE_FLSLL) + return (flsll(i)); +#else int h = 1; if (i == 0) @@ -439,6 +449,7 @@ highbit64(uint64_t i) h += 1; } return (h); +#endif } #ifdef __cplusplus From owner-svn-src-stable@freebsd.org Fri Oct 2 21:06:14 2015 Return-Path: Delivered-To: svn-src-stable@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id DC692A0EBE3; Fri, 2 Oct 2015 21:06:14 +0000 (UTC) (envelope-from avg@FreeBSD.org) Received: from citadel.icyb.net.ua (citadel.icyb.net.ua [212.40.38.140]) by mx1.freebsd.org (Postfix) with ESMTP id 568B914D2; Fri, 2 Oct 2015 21:06:06 +0000 (UTC) (envelope-from avg@FreeBSD.org) Received: from porto.starpoint.kiev.ua (porto-e.starpoint.kiev.ua [212.40.38.100]) by citadel.icyb.net.ua (8.8.8p3/ICyb-2.3exp) with ESMTP id AAA15900; Sat, 03 Oct 2015 00:06:04 +0300 (EEST) (envelope-from avg@FreeBSD.org) Received: from localhost ([127.0.0.1]) by porto.starpoint.kiev.ua with esmtp (Exim 4.34 (FreeBSD)) id 1Zi7Wl-0000My-OS; Sat, 03 Oct 2015 00:06:03 +0300 Subject: Re: svn commit: r288521 - stable/10/sys/cddl/contrib/opensolaris/uts/common/sys To: Alexander Motin , src-committers@FreeBSD.org, svn-src-all@FreeBSD.org, svn-src-stable@FreeBSD.org, svn-src-stable-10@FreeBSD.org References: <201510022013.t92KDvt5080813@repo.freebsd.org> From: Andriy Gapon Message-ID: <560EF182.60501@FreeBSD.org> Date: Sat, 3 Oct 2015 00:05:06 +0300 User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:38.0) Gecko/20100101 Thunderbird/38.2.0 MIME-Version: 1.0 In-Reply-To: <201510022013.t92KDvt5080813@repo.freebsd.org> Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 7bit X-BeenThere: svn-src-stable@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: SVN commit messages for all the -stable branches of the src tree List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 02 Oct 2015 21:06:15 -0000 On 02/10/2015 23:13, Alexander Motin wrote: > MFC r284591 (by avg): illums compat: use flsl/flsll for highbit/highbit64 Thank you! -- Andriy Gapon From owner-svn-src-stable@freebsd.org Sat Oct 3 07:17:46 2015 Return-Path: Delivered-To: svn-src-stable@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 37075A0D7E4; Sat, 3 Oct 2015 07:17:46 +0000 (UTC) (envelope-from mav@FreeBSD.org) Received: from repo.freebsd.org (repo.freebsd.org [IPv6:2001:1900:2254:2068::e6a:0]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 273581DE7; Sat, 3 Oct 2015 07:17:46 +0000 (UTC) (envelope-from mav@FreeBSD.org) Received: from repo.freebsd.org ([127.0.1.70]) by repo.freebsd.org (8.15.2/8.15.2) with ESMTP id t937Hk6q057960; Sat, 3 Oct 2015 07:17:46 GMT (envelope-from mav@FreeBSD.org) Received: (from mav@localhost) by repo.freebsd.org (8.15.2/8.15.2/Submit) id t937HkMu057959; Sat, 3 Oct 2015 07:17:46 GMT (envelope-from mav@FreeBSD.org) Message-Id: <201510030717.t937HkMu057959@repo.freebsd.org> X-Authentication-Warning: repo.freebsd.org: mav set sender to mav@FreeBSD.org using -f From: Alexander Motin Date: Sat, 3 Oct 2015 07:17:46 +0000 (UTC) To: src-committers@freebsd.org, svn-src-all@freebsd.org, svn-src-stable@freebsd.org, svn-src-stable-10@freebsd.org Subject: svn commit: r288536 - stable/10/sys/cddl/contrib/opensolaris/uts/common/fs/zfs X-SVN-Group: stable-10 MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit X-BeenThere: svn-src-stable@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: SVN commit messages for all the -stable branches of the src tree List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 03 Oct 2015 07:17:46 -0000 Author: mav Date: Sat Oct 3 07:17:45 2015 New Revision: 288536 URL: https://svnweb.freebsd.org/changeset/base/288536 Log: MFC r281109: Add DTrace probe to the new ARC reclaim cause added in r281026. Modified: stable/10/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/arc.c Directory Properties: stable/10/ (props changed) Modified: stable/10/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/arc.c ============================================================================== --- stable/10/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/arc.c Sat Oct 3 06:35:17 2015 (r288535) +++ stable/10/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/arc.c Sat Oct 3 07:17:45 2015 (r288536) @@ -2674,8 +2674,12 @@ arc_reclaim_needed(void) * Above limits know nothing about real level of KVA fragmentation. * Start aggressive reclamation if too little sequential KVA left. */ - if (vmem_size(heap_arena, VMEM_MAXFREE) < zfs_max_recordsize) + if (vmem_size(heap_arena, VMEM_MAXFREE) < zfs_max_recordsize) { + DTRACE_PROBE2(arc__reclaim_maxfree, uint64_t, + vmem_size(heap_arena, VMEM_MAXFREE), + uint64_t, zfs_max_recordsize); return (1); + } #else /* _KERNEL */ if (spa_get_random(100) == 0) From owner-svn-src-stable@freebsd.org Sat Oct 3 07:19:13 2015 Return-Path: Delivered-To: svn-src-stable@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 5E5CAA0D901; Sat, 3 Oct 2015 07:19:13 +0000 (UTC) (envelope-from mav@FreeBSD.org) Received: from repo.freebsd.org (repo.freebsd.org [IPv6:2001:1900:2254:2068::e6a:0]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 34BB41F38; Sat, 3 Oct 2015 07:19:13 +0000 (UTC) (envelope-from mav@FreeBSD.org) Received: from repo.freebsd.org ([127.0.1.70]) by repo.freebsd.org (8.15.2/8.15.2) with ESMTP id t937JD6O058174; Sat, 3 Oct 2015 07:19:13 GMT (envelope-from mav@FreeBSD.org) Received: (from mav@localhost) by repo.freebsd.org (8.15.2/8.15.2/Submit) id t937JD0R058173; Sat, 3 Oct 2015 07:19:13 GMT (envelope-from mav@FreeBSD.org) Message-Id: <201510030719.t937JD0R058173@repo.freebsd.org> X-Authentication-Warning: repo.freebsd.org: mav set sender to mav@FreeBSD.org using -f From: Alexander Motin Date: Sat, 3 Oct 2015 07:19:13 +0000 (UTC) To: src-committers@freebsd.org, svn-src-all@freebsd.org, svn-src-stable@freebsd.org, svn-src-stable-10@freebsd.org Subject: svn commit: r288537 - stable/10/sys/cddl/contrib/opensolaris/uts/common/fs/zfs X-SVN-Group: stable-10 MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit X-BeenThere: svn-src-stable@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: SVN commit messages for all the -stable branches of the src tree List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 03 Oct 2015 07:19:13 -0000 Author: mav Date: Sat Oct 3 07:19:12 2015 New Revision: 288537 URL: https://svnweb.freebsd.org/changeset/base/288537 Log: MFC r286539: 5562 ZFS sa_handle's violate kmem invariants, debug kernels panic on boot Reviewed by: Matthew Ahrens Reviewed by: Robert Mustacchi Reviewed by: George Wilson Reviewed by: Rich Lowe Approved by: Dan McDonald Author: Justin T. Gibbs illumos/illumos-gate@0fda3cc5c1c5a1d9bdea6d52637bef6e781549c9 Modified: stable/10/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/sa.c Directory Properties: stable/10/ (props changed) Modified: stable/10/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/sa.c ============================================================================== --- stable/10/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/sa.c Sat Oct 3 07:17:45 2015 (r288536) +++ stable/10/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/sa.c Sat Oct 3 07:19:12 2015 (r288537) @@ -210,12 +210,6 @@ sa_cache_constructor(void *buf, void *un { sa_handle_t *hdl = buf; - hdl->sa_bonus_tab = NULL; - hdl->sa_spill_tab = NULL; - hdl->sa_os = NULL; - hdl->sa_userp = NULL; - hdl->sa_bonus = NULL; - hdl->sa_spill = NULL; mutex_init(&hdl->sa_lock, NULL, MUTEX_DEFAULT, NULL); return (0); } @@ -1350,14 +1344,11 @@ sa_handle_destroy(sa_handle_t *hdl) (void) dmu_buf_update_user((dmu_buf_t *)hdl->sa_bonus, hdl, NULL, NULL); - if (hdl->sa_bonus_tab) { + if (hdl->sa_bonus_tab) sa_idx_tab_rele(hdl->sa_os, hdl->sa_bonus_tab); - hdl->sa_bonus_tab = NULL; - } - if (hdl->sa_spill_tab) { + + if (hdl->sa_spill_tab) sa_idx_tab_rele(hdl->sa_os, hdl->sa_spill_tab); - hdl->sa_spill_tab = NULL; - } dmu_buf_rele(hdl->sa_bonus, NULL); @@ -1392,6 +1383,8 @@ sa_handle_get_from_db(objset_t *os, dmu_ handle->sa_bonus = db; handle->sa_os = os; handle->sa_spill = NULL; + handle->sa_bonus_tab = NULL; + handle->sa_spill_tab = NULL; error = sa_build_index(handle, SA_BONUS); newhandle = (hdl_type == SA_HDL_SHARED) ? From owner-svn-src-stable@freebsd.org Sat Oct 3 07:20:29 2015 Return-Path: Delivered-To: svn-src-stable@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 396B3A0DA1A; Sat, 3 Oct 2015 07:20:29 +0000 (UTC) (envelope-from mav@FreeBSD.org) Received: from repo.freebsd.org (repo.freebsd.org [IPv6:2001:1900:2254:2068::e6a:0]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 1B19910E8; Sat, 3 Oct 2015 07:20:29 +0000 (UTC) (envelope-from mav@FreeBSD.org) Received: from repo.freebsd.org ([127.0.1.70]) by repo.freebsd.org (8.15.2/8.15.2) with ESMTP id t937KSDF058397; Sat, 3 Oct 2015 07:20:28 GMT (envelope-from mav@FreeBSD.org) Received: (from mav@localhost) by repo.freebsd.org (8.15.2/8.15.2/Submit) id t937KRkN058388; Sat, 3 Oct 2015 07:20:27 GMT (envelope-from mav@FreeBSD.org) Message-Id: <201510030720.t937KRkN058388@repo.freebsd.org> X-Authentication-Warning: repo.freebsd.org: mav set sender to mav@FreeBSD.org using -f From: Alexander Motin Date: Sat, 3 Oct 2015 07:20:27 +0000 (UTC) To: src-committers@freebsd.org, svn-src-all@freebsd.org, svn-src-stable@freebsd.org, svn-src-stable-10@freebsd.org Subject: svn commit: r288538 - in stable/10/sys/cddl/contrib/opensolaris/uts/common/fs/zfs: . sys X-SVN-Group: stable-10 MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit X-BeenThere: svn-src-stable@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: SVN commit messages for all the -stable branches of the src tree List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 03 Oct 2015 07:20:29 -0000 Author: mav Date: Sat Oct 3 07:20:26 2015 New Revision: 288538 URL: https://svnweb.freebsd.org/changeset/base/288538 Log: MFC r286541: 5531 NULL pointer dereference in dsl_prop_get_ds() Reviewed by: Matthew Ahrens Reviewed by: Dan McDonald Reviewed by: George Wilson Reviewed by: Bayard Bell Approved by: Robert Mustacchi Author: Justin T. Gibbs illumos/illumos-gate@e57a022b8f718889ffa92adbde47a8f08abcdb25 Modified: stable/10/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/dbuf.c stable/10/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/dnode_sync.c stable/10/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/dsl_dataset.c stable/10/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/dsl_prop.c stable/10/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/sys/dbuf.h stable/10/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/sys/dmu.h stable/10/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/sys/dsl_dataset.h Directory Properties: stable/10/ (props changed) Modified: stable/10/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/dbuf.c ============================================================================== --- stable/10/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/dbuf.c Sat Oct 3 07:19:12 2015 (r288537) +++ stable/10/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/dbuf.c Sat Oct 3 07:20:26 2015 (r288538) @@ -118,11 +118,9 @@ dbuf_hash(void *os, uint64_t obj, uint8_ (dbuf)->db_blkid == (blkid)) dmu_buf_impl_t * -dbuf_find(dnode_t *dn, uint8_t level, uint64_t blkid) +dbuf_find(objset_t *os, uint64_t obj, uint8_t level, uint64_t blkid) { dbuf_hash_table_t *h = &dbuf_hash_table; - objset_t *os = dn->dn_objset; - uint64_t obj = dn->dn_object; uint64_t hv = DBUF_HASH(os, obj, level, blkid); uint64_t idx = hv & h->hash_table_mask; dmu_buf_impl_t *db; @@ -142,6 +140,24 @@ dbuf_find(dnode_t *dn, uint8_t level, ui return (NULL); } +static dmu_buf_impl_t * +dbuf_find_bonus(objset_t *os, uint64_t object) +{ + dnode_t *dn; + dmu_buf_impl_t *db = NULL; + + if (dnode_hold(os, object, FTAG, &dn) == 0) { + rw_enter(&dn->dn_struct_rwlock, RW_READER); + if (dn->dn_bonus != NULL) { + db = dn->dn_bonus; + mutex_enter(&db->db_mtx); + } + rw_exit(&dn->dn_struct_rwlock); + dnode_rele(dn, FTAG); + } + return (db); +} + /* * Insert an entry into the hash table. If there is already an element * equal to elem in the hash table, then the already existing element @@ -1852,7 +1868,7 @@ dbuf_prefetch(dnode_t *dn, uint64_t blki return; /* dbuf_find() returns with db_mtx held */ - if (db = dbuf_find(dn, 0, blkid)) { + if (db = dbuf_find(dn->dn_objset, dn->dn_object, 0, blkid)) { /* * This dbuf is already in the cache. We assume that * it is already CACHED, or else about to be either @@ -1899,7 +1915,7 @@ dbuf_hold_impl(dnode_t *dn, uint8_t leve *dbp = NULL; top: /* dbuf_find() returns with db_mtx held */ - db = dbuf_find(dn, level, blkid); + db = dbuf_find(dn->dn_objset, dn->dn_object, level, blkid); if (db == NULL) { blkptr_t *bp = NULL; @@ -2035,6 +2051,30 @@ dbuf_add_ref(dmu_buf_impl_t *db, void *t ASSERT(holds > 1); } +#pragma weak dmu_buf_try_add_ref = dbuf_try_add_ref +boolean_t +dbuf_try_add_ref(dmu_buf_t *db_fake, objset_t *os, uint64_t obj, uint64_t blkid, + void *tag) +{ + dmu_buf_impl_t *db = (dmu_buf_impl_t *)db_fake; + dmu_buf_impl_t *found_db; + boolean_t result = B_FALSE; + + if (db->db_blkid == DMU_BONUS_BLKID) + found_db = dbuf_find_bonus(os, obj); + else + found_db = dbuf_find(os, obj, 0, blkid); + + if (found_db != NULL) { + if (db == found_db && dbuf_refcount(db) > db->db_dirtycnt) { + (void) refcount_add(&db->db_holds, tag); + result = B_TRUE; + } + mutex_exit(&db->db_mtx); + } + return (result); +} + /* * If you call dbuf_rele() you had better not be referencing the dnode handle * unless you have some other direct or indirect hold on the dnode. (An indirect Modified: stable/10/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/dnode_sync.c ============================================================================== --- stable/10/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/dnode_sync.c Sat Oct 3 07:19:12 2015 (r288537) +++ stable/10/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/dnode_sync.c Sat Oct 3 07:20:26 2015 (r288538) @@ -76,7 +76,8 @@ dnode_increase_indirection(dnode_t *dn, /* set dbuf's parent pointers to new indirect buf */ for (i = 0; i < nblkptr; i++) { - dmu_buf_impl_t *child = dbuf_find(dn, old_toplvl, i); + dmu_buf_impl_t *child = + dbuf_find(dn->dn_objset, dn->dn_object, old_toplvl, i); if (child == NULL) continue; Modified: stable/10/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/dsl_dataset.c ============================================================================== --- stable/10/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/dsl_dataset.c Sat Oct 3 07:19:12 2015 (r288537) +++ stable/10/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/dsl_dataset.c Sat Oct 3 07:20:26 2015 (r288538) @@ -369,6 +369,13 @@ dsl_dataset_snap_remove(dsl_dataset_t *d return (err); } +boolean_t +dsl_dataset_try_add_ref(dsl_pool_t *dp, dsl_dataset_t *ds, void *tag) +{ + return (dmu_buf_try_add_ref(ds->ds_dbuf, dp->dp_meta_objset, + ds->ds_object, DMU_BONUS_BLKID, tag)); +} + int dsl_dataset_hold_obj(dsl_pool_t *dp, uint64_t dsobj, void *tag, dsl_dataset_t **dsp) Modified: stable/10/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/dsl_prop.c ============================================================================== --- stable/10/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/dsl_prop.c Sat Oct 3 07:19:12 2015 (r288537) +++ stable/10/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/dsl_prop.c Sat Oct 3 07:20:26 2015 (r288538) @@ -442,9 +442,31 @@ dsl_prop_notify_all_cb(dsl_pool_t *dp, d cbr = list_next(&dd->dd_prop_cbs, cbr)) { uint64_t value; + /* + * Callback entries do not have holds on their datasets + * so that datasets with registered callbacks are still + * eligible for eviction. Unlike operations on callbacks + * for a single dataset, we are performing a recursive + * descent of related datasets and the calling context + * for this iteration only has a dataset hold on the root. + * Without a hold, the callback's pointer to the dataset + * could be invalidated by eviction at any time. + * + * Use dsl_dataset_try_add_ref() to verify that the + * dataset has not begun eviction processing and to + * prevent eviction from occurring for the duration + * of the callback. If the hold attempt fails, this + * object is already being evicted and the callback can + * be safely ignored. + */ + if (!dsl_dataset_try_add_ref(dp, cbr->cbr_ds, FTAG)) + continue; + if (dsl_prop_get_ds(cbr->cbr_ds, cbr->cbr_propname, sizeof (value), 1, &value, NULL) == 0) cbr->cbr_func(cbr->cbr_arg, value); + + dsl_dataset_rele(cbr->cbr_ds, FTAG); } mutex_exit(&dd->dd_lock); @@ -497,19 +519,28 @@ dsl_prop_changed_notify(dsl_pool_t *dp, mutex_enter(&dd->dd_lock); for (cbr = list_head(&dd->dd_prop_cbs); cbr; cbr = list_next(&dd->dd_prop_cbs, cbr)) { - uint64_t propobj = dsl_dataset_phys(cbr->cbr_ds)->ds_props_obj; + uint64_t propobj; - if (strcmp(cbr->cbr_propname, propname) != 0) + /* + * cbr->cbf_ds may be invalidated due to eviction, + * requiring the use of dsl_dataset_try_add_ref(). + * See comment block in dsl_prop_notify_all_cb() + * for details. + */ + if (strcmp(cbr->cbr_propname, propname) != 0 || + !dsl_dataset_try_add_ref(dp, cbr->cbr_ds, FTAG)) continue; + propobj = dsl_dataset_phys(cbr->cbr_ds)->ds_props_obj; + /* - * If the property is set on this ds, then it is not - * inherited here; don't call the callback. + * If the property is not set on this ds, then it is + * inherited here; call the callback. */ - if (propobj && 0 == zap_contains(mos, propobj, propname)) - continue; + if (propobj == 0 || zap_contains(mos, propobj, propname) != 0) + cbr->cbr_func(cbr->cbr_arg, value); - cbr->cbr_func(cbr->cbr_arg, value); + dsl_dataset_rele(cbr->cbr_ds, FTAG); } mutex_exit(&dd->dd_lock); Modified: stable/10/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/sys/dbuf.h ============================================================================== --- stable/10/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/sys/dbuf.h Sat Oct 3 07:19:12 2015 (r288537) +++ stable/10/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/sys/dbuf.h Sat Oct 3 07:20:26 2015 (r288538) @@ -264,12 +264,15 @@ int dbuf_hold_impl(struct dnode *dn, uin void dbuf_prefetch(struct dnode *dn, uint64_t blkid, zio_priority_t prio); void dbuf_add_ref(dmu_buf_impl_t *db, void *tag); +boolean_t dbuf_try_add_ref(dmu_buf_t *db, objset_t *os, uint64_t obj, + uint64_t blkid, void *tag); uint64_t dbuf_refcount(dmu_buf_impl_t *db); void dbuf_rele(dmu_buf_impl_t *db, void *tag); void dbuf_rele_and_unlock(dmu_buf_impl_t *db, void *tag); -dmu_buf_impl_t *dbuf_find(struct dnode *dn, uint8_t level, uint64_t blkid); +dmu_buf_impl_t *dbuf_find(struct objset *os, uint64_t object, uint8_t level, + uint64_t blkid); int dbuf_read(dmu_buf_impl_t *db, zio_t *zio, uint32_t flags); void dmu_buf_will_not_fill(dmu_buf_t *db, dmu_tx_t *tx); Modified: stable/10/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/sys/dmu.h ============================================================================== --- stable/10/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/sys/dmu.h Sat Oct 3 07:19:12 2015 (r288537) +++ stable/10/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/sys/dmu.h Sat Oct 3 07:20:26 2015 (r288538) @@ -462,7 +462,23 @@ int dmu_spill_hold_existing(dmu_buf_t *b */ int dmu_buf_hold(objset_t *os, uint64_t object, uint64_t offset, void *tag, dmu_buf_t **, int flags); + +/* + * Add a reference to a dmu buffer that has already been held via + * dmu_buf_hold() in the current context. + */ void dmu_buf_add_ref(dmu_buf_t *db, void* tag); + +/* + * Attempt to add a reference to a dmu buffer that is in an unknown state, + * using a pointer that may have been invalidated by eviction processing. + * The request will succeed if the passed in dbuf still represents the + * same os/object/blkid, is ineligible for eviction, and has at least + * one hold by a user other than the syncer. + */ +boolean_t dmu_buf_try_add_ref(dmu_buf_t *, objset_t *os, uint64_t object, + uint64_t blkid, void *tag); + void dmu_buf_rele(dmu_buf_t *db, void *tag); uint64_t dmu_buf_refcount(dmu_buf_t *db); Modified: stable/10/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/sys/dsl_dataset.h ============================================================================== --- stable/10/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/sys/dsl_dataset.h Sat Oct 3 07:19:12 2015 (r288537) +++ stable/10/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/sys/dsl_dataset.h Sat Oct 3 07:20:26 2015 (r288538) @@ -208,6 +208,8 @@ dsl_dataset_is_snapshot(dsl_dataset_t *d int dsl_dataset_hold(struct dsl_pool *dp, const char *name, void *tag, dsl_dataset_t **dsp); +boolean_t dsl_dataset_try_add_ref(struct dsl_pool *dp, dsl_dataset_t *ds, + void *tag); int dsl_dataset_hold_obj(struct dsl_pool *dp, uint64_t dsobj, void *tag, dsl_dataset_t **); void dsl_dataset_rele(dsl_dataset_t *ds, void *tag); From owner-svn-src-stable@freebsd.org Sat Oct 3 07:21:28 2015 Return-Path: Delivered-To: svn-src-stable@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 4D761A0DC5B; Sat, 3 Oct 2015 07:21:28 +0000 (UTC) (envelope-from mav@FreeBSD.org) Received: from repo.freebsd.org (repo.freebsd.org [IPv6:2001:1900:2254:2068::e6a:0]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 3DC0612FB; Sat, 3 Oct 2015 07:21:28 +0000 (UTC) (envelope-from mav@FreeBSD.org) Received: from repo.freebsd.org ([127.0.1.70]) by repo.freebsd.org (8.15.2/8.15.2) with ESMTP id t937LS03059560; Sat, 3 Oct 2015 07:21:28 GMT (envelope-from mav@FreeBSD.org) Received: (from mav@localhost) by repo.freebsd.org (8.15.2/8.15.2/Submit) id t937LSgw059559; Sat, 3 Oct 2015 07:21:28 GMT (envelope-from mav@FreeBSD.org) Message-Id: <201510030721.t937LSgw059559@repo.freebsd.org> X-Authentication-Warning: repo.freebsd.org: mav set sender to mav@FreeBSD.org using -f From: Alexander Motin Date: Sat, 3 Oct 2015 07:21:28 +0000 (UTC) To: src-committers@freebsd.org, svn-src-all@freebsd.org, svn-src-stable@freebsd.org, svn-src-stable-10@freebsd.org Subject: svn commit: r288539 - stable/10/sys/cddl/contrib/opensolaris/uts/common/fs/zfs X-SVN-Group: stable-10 MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit X-BeenThere: svn-src-stable@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: SVN commit messages for all the -stable branches of the src tree List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 03 Oct 2015 07:21:28 -0000 Author: mav Date: Sat Oct 3 07:21:27 2015 New Revision: 288539 URL: https://svnweb.freebsd.org/changeset/base/288539 Log: MFC r286543: 5592 NULL pointer dereference in dsl_prop_notify_all_cb() Reviewed by: Dan McDonald Reviewed by: Matthew Ahrens Reviewed by: George Wilson Reviewed by: Will Andrews Approved by: Robert Mustacchi illumos/illumos-gate@9d47dec0481d8cd53b2c1053c96bfa3f78357d6a Modified: stable/10/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/dsl_dataset.c Directory Properties: stable/10/ (props changed) Modified: stable/10/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/dsl_dataset.c ============================================================================== --- stable/10/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/dsl_dataset.c Sat Oct 3 07:20:26 2015 (r288538) +++ stable/10/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/dsl_dataset.c Sat Oct 3 07:21:27 2015 (r288539) @@ -372,8 +372,19 @@ dsl_dataset_snap_remove(dsl_dataset_t *d boolean_t dsl_dataset_try_add_ref(dsl_pool_t *dp, dsl_dataset_t *ds, void *tag) { - return (dmu_buf_try_add_ref(ds->ds_dbuf, dp->dp_meta_objset, - ds->ds_object, DMU_BONUS_BLKID, tag)); + dmu_buf_t *dbuf = ds->ds_dbuf; + boolean_t result = B_FALSE; + + if (dbuf != NULL && dmu_buf_try_add_ref(dbuf, dp->dp_meta_objset, + ds->ds_object, DMU_BONUS_BLKID, tag)) { + + if (ds == dmu_buf_get_user(dbuf)) + result = B_TRUE; + else + dmu_buf_rele(dbuf, tag); + } + + return (result); } int From owner-svn-src-stable@freebsd.org Sat Oct 3 07:22:26 2015 Return-Path: Delivered-To: svn-src-stable@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id A5F7BA0DD74; Sat, 3 Oct 2015 07:22:26 +0000 (UTC) (envelope-from mav@FreeBSD.org) Received: from repo.freebsd.org (repo.freebsd.org [IPv6:2001:1900:2254:2068::e6a:0]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 8B79C17E9; Sat, 3 Oct 2015 07:22:26 +0000 (UTC) (envelope-from mav@FreeBSD.org) Received: from repo.freebsd.org ([127.0.1.70]) by repo.freebsd.org (8.15.2/8.15.2) with ESMTP id t937MQC7062388; Sat, 3 Oct 2015 07:22:26 GMT (envelope-from mav@FreeBSD.org) Received: (from mav@localhost) by repo.freebsd.org (8.15.2/8.15.2/Submit) id t937MPDo062384; Sat, 3 Oct 2015 07:22:25 GMT (envelope-from mav@FreeBSD.org) Message-Id: <201510030722.t937MPDo062384@repo.freebsd.org> X-Authentication-Warning: repo.freebsd.org: mav set sender to mav@FreeBSD.org using -f From: Alexander Motin Date: Sat, 3 Oct 2015 07:22:25 +0000 (UTC) To: src-committers@freebsd.org, svn-src-all@freebsd.org, svn-src-stable@freebsd.org, svn-src-stable-10@freebsd.org Subject: svn commit: r288541 - in stable/10/sys/cddl/contrib/opensolaris/uts/common/fs/zfs: . sys X-SVN-Group: stable-10 MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit X-BeenThere: svn-src-stable@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: SVN commit messages for all the -stable branches of the src tree List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 03 Oct 2015 07:22:26 -0000 Author: mav Date: Sat Oct 3 07:22:24 2015 New Revision: 288541 URL: https://svnweb.freebsd.org/changeset/base/288541 Log: MFC r286545: 5630 stale bonus buffer in recycled dnode_t leads to data corruption Reviewed by: Matthew Ahrens Reviewed by: George Wilson Reviewed by: Will Andrews Approved by: Robert Mustacchi Author: Justin T. Gibbs Modified: stable/10/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/dbuf.c stable/10/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/dnode.c stable/10/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/dnode_sync.c stable/10/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/sys/dnode.h Directory Properties: stable/10/ (props changed) Modified: stable/10/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/dbuf.c ============================================================================== --- stable/10/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/dbuf.c Sat Oct 3 07:22:07 2015 (r288540) +++ stable/10/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/dbuf.c Sat Oct 3 07:22:24 2015 (r288541) @@ -2128,21 +2128,60 @@ dbuf_rele_and_unlock(dmu_buf_impl_t *db, if (holds == 0) { if (db->db_blkid == DMU_BONUS_BLKID) { - mutex_exit(&db->db_mtx); + dnode_t *dn; /* - * If the dnode moves here, we cannot cross this barrier - * until the move completes. + * If the dnode moves here, we cannot cross this + * barrier until the move completes. */ DB_DNODE_ENTER(db); - atomic_dec_32(&DB_DNODE(db)->dn_dbufs_count); + + dn = DB_DNODE(db); + atomic_dec_32(&dn->dn_dbufs_count); + + /* + * Decrementing the dbuf count means that the bonus + * buffer's dnode hold is no longer discounted in + * dnode_move(). The dnode cannot move until after + * the dnode_rele_and_unlock() below. + */ DB_DNODE_EXIT(db); + /* - * The bonus buffer's dnode hold is no longer discounted - * in dnode_move(). The dnode cannot move until after - * the dnode_rele(). + * Do not reference db after its lock is dropped. + * Another thread may evict it. */ - dnode_rele(DB_DNODE(db), db); + mutex_exit(&db->db_mtx); + + /* + * If the dnode has been freed, evict the bonus + * buffer immediately. The data in the bonus + * buffer is no longer relevant and this prevents + * a stale bonus buffer from being associated + * with this dnode_t should the dnode_t be reused + * prior to being destroyed. + */ + mutex_enter(&dn->dn_mtx); + if (dn->dn_type == DMU_OT_NONE || + dn->dn_free_txg != 0) { + /* + * Drop dn_mtx. It is a leaf lock and + * cannot be held when dnode_evict_bonus() + * acquires other locks in order to + * perform the eviction. + * + * Freed dnodes cannot be reused until the + * last hold is released. Since this bonus + * buffer has a hold, the dnode will remain + * in the free state, even without dn_mtx + * held, until the dnode_rele_and_unlock() + * below. + */ + mutex_exit(&dn->dn_mtx); + dnode_evict_bonus(dn); + mutex_enter(&dn->dn_mtx); + } + dnode_rele_and_unlock(dn, db); } else if (db->db_buf == NULL) { /* * This is a special case: we never associated this Modified: stable/10/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/dnode.c ============================================================================== --- stable/10/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/dnode.c Sat Oct 3 07:22:07 2015 (r288540) +++ stable/10/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/dnode.c Sat Oct 3 07:22:24 2015 (r288541) @@ -1205,12 +1205,18 @@ dnode_add_ref(dnode_t *dn, void *tag) void dnode_rele(dnode_t *dn, void *tag) { + mutex_enter(&dn->dn_mtx); + dnode_rele_and_unlock(dn, tag); +} + +void +dnode_rele_and_unlock(dnode_t *dn, void *tag) +{ uint64_t refs; /* Get while the hold prevents the dnode from moving. */ dmu_buf_impl_t *db = dn->dn_dbuf; dnode_handle_t *dnh = dn->dn_handle; - mutex_enter(&dn->dn_mtx); refs = refcount_remove(&dn->dn_holds, tag); mutex_exit(&dn->dn_mtx); Modified: stable/10/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/dnode_sync.c ============================================================================== --- stable/10/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/dnode_sync.c Sat Oct 3 07:22:07 2015 (r288540) +++ stable/10/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/dnode_sync.c Sat Oct 3 07:22:24 2015 (r288541) @@ -441,6 +441,12 @@ dnode_evict_dbufs(dnode_t *dn) ASSERT(pass < 100); /* sanity check */ } while (progress); + dnode_evict_bonus(dn); +} + +void +dnode_evict_bonus(dnode_t *dn) +{ rw_enter(&dn->dn_struct_rwlock, RW_WRITER); if (dn->dn_bonus && refcount_is_zero(&dn->dn_bonus->db_holds)) { mutex_enter(&dn->dn_bonus->db_mtx); Modified: stable/10/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/sys/dnode.h ============================================================================== --- stable/10/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/sys/dnode.h Sat Oct 3 07:22:07 2015 (r288540) +++ stable/10/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/sys/dnode.h Sat Oct 3 07:22:24 2015 (r288541) @@ -279,6 +279,7 @@ int dnode_hold_impl(struct objset *dd, u void *ref, dnode_t **dnp); boolean_t dnode_add_ref(dnode_t *dn, void *ref); void dnode_rele(dnode_t *dn, void *ref); +void dnode_rele_and_unlock(dnode_t *dn, void *tag); void dnode_setdirty(dnode_t *dn, dmu_tx_t *tx); void dnode_sync(dnode_t *dn, dmu_tx_t *tx); void dnode_allocate(dnode_t *dn, dmu_object_type_t ot, int blocksize, int ibs, @@ -300,6 +301,7 @@ void dnode_fini(void); int dnode_next_offset(dnode_t *dn, int flags, uint64_t *off, int minlvl, uint64_t blkfill, uint64_t txg); void dnode_evict_dbufs(dnode_t *dn); +void dnode_evict_bonus(dnode_t *dn); #ifdef ZFS_DEBUG From owner-svn-src-stable@freebsd.org Sat Oct 3 07:23:22 2015 Return-Path: Delivered-To: svn-src-stable@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 0A8C7A0DEDE; Sat, 3 Oct 2015 07:23:22 +0000 (UTC) (envelope-from mav@FreeBSD.org) Received: from repo.freebsd.org (repo.freebsd.org [IPv6:2001:1900:2254:2068::e6a:0]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id E31361958; Sat, 3 Oct 2015 07:23:21 +0000 (UTC) (envelope-from mav@FreeBSD.org) Received: from repo.freebsd.org ([127.0.1.70]) by repo.freebsd.org (8.15.2/8.15.2) with ESMTP id t937NLfM062545; Sat, 3 Oct 2015 07:23:21 GMT (envelope-from mav@FreeBSD.org) Received: (from mav@localhost) by repo.freebsd.org (8.15.2/8.15.2/Submit) id t937NKPu062539; Sat, 3 Oct 2015 07:23:20 GMT (envelope-from mav@FreeBSD.org) Message-Id: <201510030723.t937NKPu062539@repo.freebsd.org> X-Authentication-Warning: repo.freebsd.org: mav set sender to mav@FreeBSD.org using -f From: Alexander Motin Date: Sat, 3 Oct 2015 07:23:20 +0000 (UTC) To: src-committers@freebsd.org, svn-src-all@freebsd.org, svn-src-stable@freebsd.org, svn-src-stable-10@freebsd.org Subject: svn commit: r288542 - in stable/10: cddl/contrib/opensolaris/cmd/zfs sys/cddl/contrib/opensolaris/uts/common/fs/zfs sys/cddl/contrib/opensolaris/uts/common/fs/zfs/sys X-SVN-Group: stable-10 MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit X-BeenThere: svn-src-stable@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: SVN commit messages for all the -stable branches of the src tree List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 03 Oct 2015 07:23:22 -0000 Author: mav Date: Sat Oct 3 07:23:19 2015 New Revision: 288542 URL: https://svnweb.freebsd.org/changeset/base/288542 Log: MFC r286547: 5661 ZFS: "compression = on" should use lz4 if feature is enabled Reviewed by: Matthew Ahrens Reviewed by: Josef 'Jeff' Sipek Reviewed by: Xin LI Approved by: Robert Mustacchi Author: Justin T. Gibbs illumos/illumos-gate@db1741f555ec79def5e9846e6bfd132248514ffe Modified: stable/10/cddl/contrib/opensolaris/cmd/zfs/zfs.8 stable/10/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/dmu.c stable/10/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/dmu_objset.c stable/10/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/sys/zio.h stable/10/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio_compress.c Directory Properties: stable/10/ (props changed) Modified: stable/10/cddl/contrib/opensolaris/cmd/zfs/zfs.8 ============================================================================== --- stable/10/cddl/contrib/opensolaris/cmd/zfs/zfs.8 Sat Oct 3 07:22:24 2015 (r288541) +++ stable/10/cddl/contrib/opensolaris/cmd/zfs/zfs.8 Sat Oct 3 07:23:19 2015 (r288542) @@ -939,7 +939,23 @@ Disabling checksums is .Em NOT a recommended practice. .It Sy compression Ns = Ns Cm on | off | lzjb | gzip | gzip- Ns Ar N | Cm zle | Cm lz4 -Controls the compression algorithm used for this dataset. The +Controls the compression algorithm used for this dataset. +Setting compression to +.Cm on +indicates that the current default compression algorithm should be used. +The default balances compression and decompression speed, with compression +ratio and is expected to work well on a wide variety of workloads. +Unlike all other settings for this property, on does not select a fixed +compression type. +As new compression algorithms are added to ZFS and enabled on a pool, the +default compression algorithm may change. +The current default compression algorthm is either +.Cm lzjb +or, if the +.Sy lz4_compress +feature is enabled, +.Cm lz4 . +The .Cm lzjb compression algorithm is optimized for performance while providing decent data compression. Setting compression to Modified: stable/10/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/dmu.c ============================================================================== --- stable/10/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/dmu.c Sat Oct 3 07:22:24 2015 (r288541) +++ stable/10/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/dmu.c Sat Oct 3 07:23:19 2015 (r288542) @@ -1785,19 +1785,15 @@ dmu_write_policy(objset_t *os, dnode_t * * 3. all other level 0 blocks */ if (ismd) { - /* - * XXX -- we should design a compression algorithm - * that specializes in arrays of bps. - */ - boolean_t lz4_ac = spa_feature_is_active(os->os_spa, - SPA_FEATURE_LZ4_COMPRESS); - if (zfs_mdcomp_disable) { compress = ZIO_COMPRESS_EMPTY; - } else if (lz4_ac) { - compress = ZIO_COMPRESS_LZ4; } else { - compress = ZIO_COMPRESS_LZJB; + /* + * XXX -- we should design a compression algorithm + * that specializes in arrays of bps. + */ + compress = zio_compress_select(os->os_spa, + ZIO_COMPRESS_ON, ZIO_COMPRESS_ON); } /* @@ -1830,7 +1826,8 @@ dmu_write_policy(objset_t *os, dnode_t * compress = ZIO_COMPRESS_OFF; checksum = ZIO_CHECKSUM_NOPARITY; } else { - compress = zio_compress_select(dn->dn_compress, compress); + compress = zio_compress_select(os->os_spa, dn->dn_compress, + compress); checksum = (dedup_checksum == ZIO_CHECKSUM_OFF) ? zio_checksum_select(dn->dn_checksum, checksum) : Modified: stable/10/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/dmu_objset.c ============================================================================== --- stable/10/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/dmu_objset.c Sat Oct 3 07:22:24 2015 (r288541) +++ stable/10/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/dmu_objset.c Sat Oct 3 07:23:19 2015 (r288542) @@ -151,7 +151,8 @@ compression_changed_cb(void *arg, uint64 */ ASSERT(newval != ZIO_COMPRESS_INHERIT); - os->os_compress = zio_compress_select(newval, ZIO_COMPRESS_ON_VALUE); + os->os_compress = zio_compress_select(os->os_spa, newval, + ZIO_COMPRESS_ON); } static void @@ -408,7 +409,7 @@ dmu_objset_open_impl(spa_t *spa, dsl_dat } else { /* It's the meta-objset. */ os->os_checksum = ZIO_CHECKSUM_FLETCHER_4; - os->os_compress = ZIO_COMPRESS_LZJB; + os->os_compress = ZIO_COMPRESS_ON; os->os_copies = spa_max_replication(spa); os->os_dedup_checksum = ZIO_CHECKSUM_OFF; os->os_dedup_verify = B_FALSE; Modified: stable/10/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/sys/zio.h ============================================================================== --- stable/10/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/sys/zio.h Sat Oct 3 07:22:24 2015 (r288541) +++ stable/10/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/sys/zio.h Sat Oct 3 07:23:19 2015 (r288542) @@ -125,15 +125,19 @@ enum zio_compress { */ #define ZIO_COMPRESS_LEGACY_FUNCTIONS ZIO_COMPRESS_LZ4 -/* N.B. when altering this value, also change BOOTFS_COMPRESS_VALID below */ -#define ZIO_COMPRESS_ON_VALUE ZIO_COMPRESS_LZJB -#define ZIO_COMPRESS_DEFAULT ZIO_COMPRESS_OFF +/* + * The meaning of "compress = on" selected by the compression features enabled + * on a given pool. + */ +#define ZIO_COMPRESS_LEGACY_ON_VALUE ZIO_COMPRESS_LZJB +#define ZIO_COMPRESS_LZ4_ON_VALUE ZIO_COMPRESS_LZ4 + +#define ZIO_COMPRESS_DEFAULT ZIO_COMPRESS_OFF #define BOOTFS_COMPRESS_VALID(compress) \ ((compress) == ZIO_COMPRESS_LZJB || \ (compress) == ZIO_COMPRESS_LZ4 || \ - ((compress) == ZIO_COMPRESS_ON && \ - ZIO_COMPRESS_ON_VALUE == ZIO_COMPRESS_LZJB) || \ + (compress) == ZIO_COMPRESS_ON || \ (compress) == ZIO_COMPRESS_OFF) #define ZIO_FAILURE_MODE_WAIT 0 @@ -581,8 +585,8 @@ extern enum zio_checksum zio_checksum_se enum zio_checksum parent); extern enum zio_checksum zio_checksum_dedup_select(spa_t *spa, enum zio_checksum child, enum zio_checksum parent); -extern enum zio_compress zio_compress_select(enum zio_compress child, - enum zio_compress parent); +extern enum zio_compress zio_compress_select(spa_t *spa, + enum zio_compress child, enum zio_compress parent); extern void zio_suspend(spa_t *spa, zio_t *zio); extern int zio_resume(spa_t *spa); Modified: stable/10/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio_compress.c ============================================================================== --- stable/10/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio_compress.c Sat Oct 3 07:22:24 2015 (r288541) +++ stable/10/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio_compress.c Sat Oct 3 07:23:19 2015 (r288542) @@ -35,6 +35,7 @@ #include #include #include +#include #include #include @@ -81,19 +82,27 @@ zio_compress_info_t zio_compress_table[Z }; enum zio_compress -zio_compress_select(enum zio_compress child, enum zio_compress parent) +zio_compress_select(spa_t *spa, enum zio_compress child, + enum zio_compress parent) { + enum zio_compress result; + ASSERT(child < ZIO_COMPRESS_FUNCTIONS); ASSERT(parent < ZIO_COMPRESS_FUNCTIONS); - ASSERT(parent != ZIO_COMPRESS_INHERIT && parent != ZIO_COMPRESS_ON); - - if (child == ZIO_COMPRESS_INHERIT) - return (parent); + ASSERT(parent != ZIO_COMPRESS_INHERIT); - if (child == ZIO_COMPRESS_ON) - return (ZIO_COMPRESS_ON_VALUE); + result = child; + if (result == ZIO_COMPRESS_INHERIT) + result = parent; + + if (result == ZIO_COMPRESS_ON) { + if (spa_feature_is_active(spa, SPA_FEATURE_LZ4_COMPRESS)) + result = ZIO_COMPRESS_LZ4_ON_VALUE; + else + result = ZIO_COMPRESS_LEGACY_ON_VALUE; + } - return (child); + return (result); } size_t From owner-svn-src-stable@freebsd.org Sat Oct 3 07:24:13 2015 Return-Path: Delivered-To: svn-src-stable@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 59E6AA0DFA2; Sat, 3 Oct 2015 07:24:13 +0000 (UTC) (envelope-from mav@FreeBSD.org) Received: from repo.freebsd.org (repo.freebsd.org [IPv6:2001:1900:2254:2068::e6a:0]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 4A22B1AD0; Sat, 3 Oct 2015 07:24:13 +0000 (UTC) (envelope-from mav@FreeBSD.org) Received: from repo.freebsd.org ([127.0.1.70]) by repo.freebsd.org (8.15.2/8.15.2) with ESMTP id t937ODLS062697; Sat, 3 Oct 2015 07:24:13 GMT (envelope-from mav@FreeBSD.org) Received: (from mav@localhost) by repo.freebsd.org (8.15.2/8.15.2/Submit) id t937ODFQ062696; Sat, 3 Oct 2015 07:24:13 GMT (envelope-from mav@FreeBSD.org) Message-Id: <201510030724.t937ODFQ062696@repo.freebsd.org> X-Authentication-Warning: repo.freebsd.org: mav set sender to mav@FreeBSD.org using -f From: Alexander Motin Date: Sat, 3 Oct 2015 07:24:13 +0000 (UTC) To: src-committers@freebsd.org, svn-src-all@freebsd.org, svn-src-stable@freebsd.org, svn-src-stable-10@freebsd.org Subject: svn commit: r288543 - stable/10/sys/cddl/contrib/opensolaris/uts/common/fs/zfs X-SVN-Group: stable-10 MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit X-BeenThere: svn-src-stable@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: SVN commit messages for all the -stable branches of the src tree List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 03 Oct 2015 07:24:13 -0000 Author: mav Date: Sat Oct 3 07:24:12 2015 New Revision: 288543 URL: https://svnweb.freebsd.org/changeset/base/288543 Log: MFC r286549: 5693 ztest fails in dbuf_verify: buf[i] == 0, due to dedup and bp_override Reviewed by: George Wilson Reviewed by: Christopher Siden Reviewed by: Bayard Bell Approved by: Dan McDonald Author: Matthew Ahrens illumos/illumos-gate@7f7ace370074e350853da254c65688fd43ddc695 Modified: stable/10/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c Directory Properties: stable/10/ (props changed) Modified: stable/10/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c ============================================================================== --- stable/10/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c Sat Oct 3 07:23:19 2015 (r288542) +++ stable/10/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c Sat Oct 3 07:24:12 2015 (r288543) @@ -1216,8 +1216,6 @@ zio_write_bp_init(zio_t *zio) zio->io_pipeline |= ZIO_STAGE_DDT_WRITE; return (ZIO_PIPELINE_CONTINUE); } - zio->io_bp_override = NULL; - BP_ZERO(bp); } if (!BP_IS_HOLE(bp) && bp->blk_birth == zio->io_txg) { From owner-svn-src-stable@freebsd.org Sat Oct 3 07:25:06 2015 Return-Path: Delivered-To: svn-src-stable@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 05644A0E077; Sat, 3 Oct 2015 07:25:06 +0000 (UTC) (envelope-from mav@FreeBSD.org) Received: from repo.freebsd.org (repo.freebsd.org [IPv6:2001:1900:2254:2068::e6a:0]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id DE0C31C36; Sat, 3 Oct 2015 07:25:05 +0000 (UTC) (envelope-from mav@FreeBSD.org) Received: from repo.freebsd.org ([127.0.1.70]) by repo.freebsd.org (8.15.2/8.15.2) with ESMTP id t937P59s062852; Sat, 3 Oct 2015 07:25:05 GMT (envelope-from mav@FreeBSD.org) Received: (from mav@localhost) by repo.freebsd.org (8.15.2/8.15.2/Submit) id t937P5cu062851; Sat, 3 Oct 2015 07:25:05 GMT (envelope-from mav@FreeBSD.org) Message-Id: <201510030725.t937P5cu062851@repo.freebsd.org> X-Authentication-Warning: repo.freebsd.org: mav set sender to mav@FreeBSD.org using -f From: Alexander Motin Date: Sat, 3 Oct 2015 07:25:05 +0000 (UTC) To: src-committers@freebsd.org, svn-src-all@freebsd.org, svn-src-stable@freebsd.org, svn-src-stable-10@freebsd.org Subject: svn commit: r288544 - stable/10/sys/cddl/contrib/opensolaris/uts/common/fs/zfs X-SVN-Group: stable-10 MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit X-BeenThere: svn-src-stable@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: SVN commit messages for all the -stable branches of the src tree List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 03 Oct 2015 07:25:06 -0000 Author: mav Date: Sat Oct 3 07:25:05 2015 New Revision: 288544 URL: https://svnweb.freebsd.org/changeset/base/288544 Log: MFC r286551: 5694 traverse_prefetcher does not prefetch enough Reviewed by: Matthew Ahrens Reviewed by: Alex Reece Reviewed by: Christopher Siden Reviewed by: Josef 'Jeff' Sipek Reviewed by: Bayard Bell Approved by: Garrett D'Amore Author: George Wilson illumos/illumos-gate@34d7ce052c4565b078f73b95ccbd49274e98edaa Modified: stable/10/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/dmu_traverse.c Directory Properties: stable/10/ (props changed) Modified: stable/10/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/dmu_traverse.c ============================================================================== --- stable/10/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/dmu_traverse.c Sat Oct 3 07:24:12 2015 (r288543) +++ stable/10/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/dmu_traverse.c Sat Oct 3 07:25:05 2015 (r288544) @@ -39,13 +39,12 @@ #include #include -int zfs_pd_blks_max = 100; +int32_t zfs_pd_bytes_max = 50 * 1024 * 1024; /* 50MB */ typedef struct prefetch_data { kmutex_t pd_mtx; kcondvar_t pd_cv; - int pd_blks_max; - int pd_blks_fetched; + int32_t pd_bytes_fetched; int pd_flags; boolean_t pd_cancel; boolean_t pd_exited; @@ -250,11 +249,12 @@ traverse_visitbp(traverse_data_t *td, co } if (pd != NULL && !pd->pd_exited && prefetch_needed(pd, bp)) { + uint64_t size = BP_GET_LSIZE(bp); mutex_enter(&pd->pd_mtx); - ASSERT(pd->pd_blks_fetched >= 0); - while (pd->pd_blks_fetched == 0 && !pd->pd_exited) + ASSERT(pd->pd_bytes_fetched >= 0); + while (pd->pd_bytes_fetched < size && !pd->pd_exited) cv_wait(&pd->pd_cv, &pd->pd_mtx); - pd->pd_blks_fetched--; + pd->pd_bytes_fetched -= size; cv_broadcast(&pd->pd_cv); mutex_exit(&pd->pd_mtx); } @@ -447,7 +447,7 @@ traverse_prefetcher(spa_t *spa, zilog_t prefetch_data_t *pfd = arg; arc_flags_t aflags = ARC_FLAG_NOWAIT | ARC_FLAG_PREFETCH; - ASSERT(pfd->pd_blks_fetched >= 0); + ASSERT(pfd->pd_bytes_fetched >= 0); if (pfd->pd_cancel) return (SET_ERROR(EINTR)); @@ -455,9 +455,9 @@ traverse_prefetcher(spa_t *spa, zilog_t return (0); mutex_enter(&pfd->pd_mtx); - while (!pfd->pd_cancel && pfd->pd_blks_fetched >= pfd->pd_blks_max) + while (!pfd->pd_cancel && pfd->pd_bytes_fetched >= zfs_pd_bytes_max) cv_wait(&pfd->pd_cv, &pfd->pd_mtx); - pfd->pd_blks_fetched++; + pfd->pd_bytes_fetched += BP_GET_LSIZE(bp); cv_broadcast(&pfd->pd_cv); mutex_exit(&pfd->pd_mtx); @@ -529,7 +529,6 @@ traverse_impl(spa_t *spa, dsl_dataset_t td.td_hole_birth_enabled_txg = 0; } - pd.pd_blks_max = zfs_pd_blks_max; pd.pd_flags = flags; mutex_init(&pd.pd_mtx, NULL, MUTEX_DEFAULT, NULL); cv_init(&pd.pd_cv, NULL, CV_DEFAULT, NULL); From owner-svn-src-stable@freebsd.org Sat Oct 3 07:26:00 2015 Return-Path: Delivered-To: svn-src-stable@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 7D37FA0E179; Sat, 3 Oct 2015 07:26:00 +0000 (UTC) (envelope-from mav@FreeBSD.org) Received: from repo.freebsd.org (repo.freebsd.org [IPv6:2001:1900:2254:2068::e6a:0]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 6D6211DA6; Sat, 3 Oct 2015 07:26:00 +0000 (UTC) (envelope-from mav@FreeBSD.org) Received: from repo.freebsd.org ([127.0.1.70]) by repo.freebsd.org (8.15.2/8.15.2) with ESMTP id t937Q0ec063010; Sat, 3 Oct 2015 07:26:00 GMT (envelope-from mav@FreeBSD.org) Received: (from mav@localhost) by repo.freebsd.org (8.15.2/8.15.2/Submit) id t937Q0Di063009; Sat, 3 Oct 2015 07:26:00 GMT (envelope-from mav@FreeBSD.org) Message-Id: <201510030726.t937Q0Di063009@repo.freebsd.org> X-Authentication-Warning: repo.freebsd.org: mav set sender to mav@FreeBSD.org using -f From: Alexander Motin Date: Sat, 3 Oct 2015 07:26:00 +0000 (UTC) To: src-committers@freebsd.org, svn-src-all@freebsd.org, svn-src-stable@freebsd.org, svn-src-stable-10@freebsd.org Subject: svn commit: r288545 - stable/10/sys/cddl/contrib/opensolaris/uts/common/fs/zfs X-SVN-Group: stable-10 MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit X-BeenThere: svn-src-stable@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: SVN commit messages for all the -stable branches of the src tree List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 03 Oct 2015 07:26:00 -0000 Author: mav Date: Sat Oct 3 07:25:59 2015 New Revision: 288545 URL: https://svnweb.freebsd.org/changeset/base/288545 Log: MFC r286554: 5769 Cast 'zfs bad bloc' to ULL for x86 Reviewed by: Prakash Surya Reviewed by: Matthew Ahrens Reviewed by: Paul Dagnelie Reviewed by: Richard PALO Approved by: Dan McDonald illumos/illumos-gate@8c76e0763bcf0029556e106377da859f6492a7ee Modified: stable/10/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/dmu_send.c Directory Properties: stable/10/ (props changed) Modified: stable/10/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/dmu_send.c ============================================================================== --- stable/10/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/dmu_send.c Sat Oct 3 07:25:05 2015 (r288544) +++ stable/10/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/dmu_send.c Sat Oct 3 07:25:59 2015 (r288545) @@ -532,7 +532,7 @@ backup_cb(spa_t *spa, zilog_t *zilog, co for (ptr = abuf->b_data; (char *)ptr < (char *)abuf->b_data + blksz; ptr++) - *ptr = 0x2f5baddb10c; + *ptr = 0x2f5baddb10cULL; } else { return (SET_ERROR(EIO)); } From owner-svn-src-stable@freebsd.org Sat Oct 3 07:26:49 2015 Return-Path: Delivered-To: svn-src-stable@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id C4B9BA0E244; Sat, 3 Oct 2015 07:26:49 +0000 (UTC) (envelope-from mav@FreeBSD.org) Received: from repo.freebsd.org (repo.freebsd.org [IPv6:2001:1900:2254:2068::e6a:0]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id B50351EEA; Sat, 3 Oct 2015 07:26:49 +0000 (UTC) (envelope-from mav@FreeBSD.org) Received: from repo.freebsd.org ([127.0.1.70]) by repo.freebsd.org (8.15.2/8.15.2) with ESMTP id t937QnCG063152; Sat, 3 Oct 2015 07:26:49 GMT (envelope-from mav@FreeBSD.org) Received: (from mav@localhost) by repo.freebsd.org (8.15.2/8.15.2/Submit) id t937QnfM063150; Sat, 3 Oct 2015 07:26:49 GMT (envelope-from mav@FreeBSD.org) Message-Id: <201510030726.t937QnfM063150@repo.freebsd.org> X-Authentication-Warning: repo.freebsd.org: mav set sender to mav@FreeBSD.org using -f From: Alexander Motin Date: Sat, 3 Oct 2015 07:26:49 +0000 (UTC) To: src-committers@freebsd.org, svn-src-all@freebsd.org, svn-src-stable@freebsd.org, svn-src-stable-10@freebsd.org Subject: svn commit: r288546 - stable/10/sys/cddl/contrib/opensolaris/uts/common/fs/zfs X-SVN-Group: stable-10 MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit X-BeenThere: svn-src-stable@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: SVN commit messages for all the -stable branches of the src tree List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 03 Oct 2015 07:26:49 -0000 Author: mav Date: Sat Oct 3 07:26:48 2015 New Revision: 288546 URL: https://svnweb.freebsd.org/changeset/base/288546 Log: MFC r286556: Avoid 128K kmem allocations in mzap_upgrade() Reviewed by: Matthew Ahrens Reviewed by: Prakash Surya Reviewed by: George Wilson Reviewed by: Steven Hartland Approved by: Rich Lowe illumos/illumos-gate@be3e2ab906b80af79c7b22885f279e45ad8fb995 Modified: stable/10/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zap_micro.c Directory Properties: stable/10/ (props changed) Modified: stable/10/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zap_micro.c ============================================================================== --- stable/10/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zap_micro.c Sat Oct 3 07:25:59 2015 (r288545) +++ stable/10/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zap_micro.c Sat Oct 3 07:26:48 2015 (r288546) @@ -547,7 +547,7 @@ mzap_upgrade(zap_t **zapp, dmu_tx_t *tx, ASSERT(RW_WRITE_HELD(&zap->zap_rwlock)); sz = zap->zap_dbuf->db_size; - mzp = kmem_alloc(sz, KM_SLEEP); + mzp = zio_buf_alloc(sz); bcopy(zap->zap_dbuf->db_data, mzp, sz); nchunks = zap->zap_m.zap_num_chunks; @@ -555,7 +555,7 @@ mzap_upgrade(zap_t **zapp, dmu_tx_t *tx, err = dmu_object_set_blocksize(zap->zap_objset, zap->zap_object, 1ULL << fzap_default_block_shift, 0, tx); if (err) { - kmem_free(mzp, sz); + zio_buf_free(mzp, sz); return (err); } } @@ -581,7 +581,7 @@ mzap_upgrade(zap_t **zapp, dmu_tx_t *tx, if (err) break; } - kmem_free(mzp, sz); + zio_buf_free(mzp, sz); *zapp = zap; return (err); } From owner-svn-src-stable@freebsd.org Sat Oct 3 07:27:59 2015 Return-Path: Delivered-To: svn-src-stable@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 147DBA0E395; Sat, 3 Oct 2015 07:27:59 +0000 (UTC) (envelope-from mav@FreeBSD.org) Received: from repo.freebsd.org (repo.freebsd.org [IPv6:2001:1900:2254:2068::e6a:0]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 03FF01059; Sat, 3 Oct 2015 07:27:59 +0000 (UTC) (envelope-from mav@FreeBSD.org) Received: from repo.freebsd.org ([127.0.1.70]) by repo.freebsd.org (8.15.2/8.15.2) with ESMTP id t937RwxI063345; Sat, 3 Oct 2015 07:27:58 GMT (envelope-from mav@FreeBSD.org) Received: (from mav@localhost) by repo.freebsd.org (8.15.2/8.15.2/Submit) id t937RwuT063343; Sat, 3 Oct 2015 07:27:58 GMT (envelope-from mav@FreeBSD.org) Message-Id: <201510030727.t937RwuT063343@repo.freebsd.org> X-Authentication-Warning: repo.freebsd.org: mav set sender to mav@FreeBSD.org using -f From: Alexander Motin Date: Sat, 3 Oct 2015 07:27:58 +0000 (UTC) To: src-committers@freebsd.org, svn-src-all@freebsd.org, svn-src-stable@freebsd.org, svn-src-stable-10@freebsd.org Subject: svn commit: r288547 - in stable/10/sys/cddl/contrib/opensolaris/uts/common/fs/zfs: . sys X-SVN-Group: stable-10 MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit X-BeenThere: svn-src-stable@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: SVN commit messages for all the -stable branches of the src tree List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 03 Oct 2015 07:27:59 -0000 Author: mav Date: Sat Oct 3 07:27:58 2015 New Revision: 288547 URL: https://svnweb.freebsd.org/changeset/base/288547 Log: MFC r286570: 5408 managing ZFS cache devices requires lots of RAM Reviewed by: Christopher Siden Reviewed by: George Wilson Reviewed by: Matthew Ahrens Reviewed by: Don Brady Reviewed by: Josef 'Jeff' Sipek Approved by: Garrett D'Amore Author: Chris Williamson illumos/illumos-gate@89c86e32293a30cdd7af530c38b2073fee01411c Currently, every buffer cached in the L2ARC is accompanied by a 240-byte header in memory, leading to very high memory consumption when using very large cache devices. These changes significantly reduce this overhead. Currently: L1-only header = 176 bytes L1 + L2 or L2-only header = 176 bytes + 32 byte checksum + 32 byte l2hdr = 240 bytes Memory-optimized: L1-only header = 176 bytes L1 + L2 header = 176 bytes + 32 byte checksum = 208 bytes L2-only header = 96 bytes + 32 byte checksum = 128 bytes So overall: Trunk Optimized +-----------------+ L1-only | 176 B | 176 B | (same) +-----------------+ L1 & L2 | 240 B | 208 B | (saved 32 bytes) +-----------------+ L2-only | 240 B | 128 B | (saved 116 bytes) +-----------------+ For an average blocksize of 8KB, this means that for the L2ARC, the ratio of metadata to data has gone down from about 2.92% to 1.56%. For a 'storage optimized' EC2 instance with 1600GB of SSD and 60GB of RAM, this means that we expect a completely full L2ARC to use (1600 GB * 0.0156) / 60GB = 41% of the available memory, down from 78%. Relnotes: yes Modified: stable/10/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/arc.c stable/10/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/sys/arc.h Directory Properties: stable/10/ (props changed) Modified: stable/10/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/arc.c ============================================================================== --- stable/10/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/arc.c Sat Oct 3 07:26:48 2015 (r288546) +++ stable/10/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/arc.c Sat Oct 3 07:27:58 2015 (r288547) @@ -111,7 +111,7 @@ * Note that the majority of the performance stats are manipulated * with atomic operations. * - * The L2ARC uses the l2arc_buflist_mtx global mutex for the following: + * The L2ARC uses the l2ad_mtx on each vdev for the following: * * - L2ARC buflist creation * - L2ARC buflist eviction @@ -402,6 +402,7 @@ typedef struct arc_stats { kstat_named_t arcstat_l2_writes_hdr_miss; kstat_named_t arcstat_l2_evict_lock_retry; kstat_named_t arcstat_l2_evict_reading; + kstat_named_t arcstat_l2_evict_l1cached; kstat_named_t arcstat_l2_free_on_write; kstat_named_t arcstat_l2_cdata_free_on_write; kstat_named_t arcstat_l2_abort_lowmem; @@ -484,6 +485,7 @@ static arc_stats_t arc_stats = { { "l2_writes_hdr_miss", KSTAT_DATA_UINT64 }, { "l2_evict_lock_retry", KSTAT_DATA_UINT64 }, { "l2_evict_reading", KSTAT_DATA_UINT64 }, + { "l2_evict_l1cached", KSTAT_DATA_UINT64 }, { "l2_free_on_write", KSTAT_DATA_UINT64 }, { "l2_cdata_free_on_write", KSTAT_DATA_UINT64 }, { "l2_abort_lowmem", KSTAT_DATA_UINT64 }, @@ -588,8 +590,6 @@ static int arc_no_grow; /* Don't try to static uint64_t arc_tempreserve; static uint64_t arc_loaned_bytes; -typedef struct l2arc_buf_hdr l2arc_buf_hdr_t; - typedef struct arc_callback arc_callback_t; struct arc_callback { @@ -610,29 +610,53 @@ struct arc_write_callback { arc_buf_t *awcb_buf; }; -struct arc_buf_hdr { - /* protected by hash lock */ - dva_t b_dva; - uint64_t b_birth; - uint64_t b_cksum0; - +/* + * ARC buffers are separated into multiple structs as a memory saving measure: + * - Common fields struct, always defined, and embedded within it: + * - L2-only fields, always allocated but undefined when not in L2ARC + * - L1-only fields, only allocated when in L1ARC + * + * Buffer in L1 Buffer only in L2 + * +------------------------+ +------------------------+ + * | arc_buf_hdr_t | | arc_buf_hdr_t | + * | | | | + * | | | | + * | | | | + * +------------------------+ +------------------------+ + * | l2arc_buf_hdr_t | | l2arc_buf_hdr_t | + * | (undefined if L1-only) | | | + * +------------------------+ +------------------------+ + * | l1arc_buf_hdr_t | + * | | + * | | + * | | + * | | + * +------------------------+ + * + * Because it's possible for the L2ARC to become extremely large, we can wind + * up eating a lot of memory in L2ARC buffer headers, so the size of a header + * is minimized by only allocating the fields necessary for an L1-cached buffer + * when a header is actually in the L1 cache. The sub-headers (l1arc_buf_hdr and + * l2arc_buf_hdr) are embedded rather than allocated separately to save a couple + * words in pointers. arc_hdr_realloc() is used to switch a header between + * these two allocation states. + */ +typedef struct l1arc_buf_hdr { kmutex_t b_freeze_lock; - zio_cksum_t *b_freeze_cksum; +#ifdef ZFS_DEBUG + /* + * used for debugging wtih kmem_flags - by allocating and freeing + * b_thawed when the buffer is thawed, we get a record of the stack + * trace that thawed it. + */ void *b_thawed; +#endif - arc_buf_hdr_t *b_hash_next; arc_buf_t *b_buf; - arc_flags_t b_flags; uint32_t b_datacnt; - - arc_callback_t *b_acb; + /* for waiting on writes to complete */ kcondvar_t b_cv; - /* immutable */ - arc_buf_contents_t b_type; - uint64_t b_size; - uint64_t b_spa; - /* protected by arc state mutex */ arc_state_t *b_state; list_node_t b_arc_node; @@ -643,8 +667,46 @@ struct arc_buf_hdr { /* self protecting */ refcount_t b_refcnt; - l2arc_buf_hdr_t *b_l2hdr; + arc_callback_t *b_acb; + /* temporary buffer holder for in-flight compressed data */ + void *b_tmp_cdata; +} l1arc_buf_hdr_t; + +typedef struct l2arc_dev l2arc_dev_t; + +typedef struct l2arc_buf_hdr { + /* protected by arc_buf_hdr mutex */ + l2arc_dev_t *b_dev; /* L2ARC device */ + uint64_t b_daddr; /* disk address, offset byte */ + /* real alloc'd buffer size depending on b_compress applied */ + int32_t b_asize; + list_node_t b_l2node; +} l2arc_buf_hdr_t; + +struct arc_buf_hdr { + /* protected by hash lock */ + dva_t b_dva; + uint64_t b_birth; + /* + * Even though this checksum is only set/verified when a buffer is in + * the L1 cache, it needs to be in the set of common fields because it + * must be preserved from the time before a buffer is written out to + * L2ARC until after it is read back in. + */ + zio_cksum_t *b_freeze_cksum; + + arc_buf_hdr_t *b_hash_next; + arc_flags_t b_flags; + + /* immutable */ + int32_t b_size; + uint64_t b_spa; + + /* L2ARC fields. Undefined when not in L2ARC. */ + l2arc_buf_hdr_t b_l2hdr; + /* L1ARC fields. Undefined when in l2arc_only state */ + l1arc_buf_hdr_t b_l1hdr; }; #ifdef _KERNEL @@ -681,22 +743,38 @@ static arc_buf_hdr_t arc_eviction_hdr; #define HDR_PREFETCH(hdr) ((hdr)->b_flags & ARC_FLAG_PREFETCH) #define HDR_FREED_IN_READ(hdr) ((hdr)->b_flags & ARC_FLAG_FREED_IN_READ) #define HDR_BUF_AVAILABLE(hdr) ((hdr)->b_flags & ARC_FLAG_BUF_AVAILABLE) -#define HDR_FREE_IN_PROGRESS(hdr) \ - ((hdr)->b_flags & ARC_FLAG_FREE_IN_PROGRESS) + #define HDR_L2CACHE(hdr) ((hdr)->b_flags & ARC_FLAG_L2CACHE) +#define HDR_L2COMPRESS(hdr) ((hdr)->b_flags & ARC_FLAG_L2COMPRESS) #define HDR_L2_READING(hdr) \ - ((hdr)->b_flags & ARC_FLAG_IO_IN_PROGRESS && \ - (hdr)->b_l2hdr != NULL) + (((hdr)->b_flags & ARC_FLAG_IO_IN_PROGRESS) && \ + ((hdr)->b_flags & ARC_FLAG_HAS_L2HDR)) #define HDR_L2_WRITING(hdr) ((hdr)->b_flags & ARC_FLAG_L2_WRITING) #define HDR_L2_EVICTED(hdr) ((hdr)->b_flags & ARC_FLAG_L2_EVICTED) #define HDR_L2_WRITE_HEAD(hdr) ((hdr)->b_flags & ARC_FLAG_L2_WRITE_HEAD) +#define HDR_ISTYPE_METADATA(hdr) \ + ((hdr)->b_flags & ARC_FLAG_BUFC_METADATA) +#define HDR_ISTYPE_DATA(hdr) (!HDR_ISTYPE_METADATA(hdr)) + +#define HDR_HAS_L1HDR(hdr) ((hdr)->b_flags & ARC_FLAG_HAS_L1HDR) +#define HDR_HAS_L2HDR(hdr) ((hdr)->b_flags & ARC_FLAG_HAS_L2HDR) + +/* For storing compression mode in b_flags */ +#define HDR_COMPRESS_OFFSET 24 +#define HDR_COMPRESS_NBITS 7 + +#define HDR_GET_COMPRESS(hdr) ((enum zio_compress)BF32_GET(hdr->b_flags, \ + HDR_COMPRESS_OFFSET, HDR_COMPRESS_NBITS)) +#define HDR_SET_COMPRESS(hdr, cmp) BF32_SET(hdr->b_flags, \ + HDR_COMPRESS_OFFSET, HDR_COMPRESS_NBITS, (cmp)) + /* * Other sizes */ -#define HDR_SIZE ((int64_t)sizeof (arc_buf_hdr_t)) -#define L2HDR_SIZE ((int64_t)sizeof (l2arc_buf_hdr_t)) +#define HDR_FULL_SIZE ((int64_t)sizeof (arc_buf_hdr_t)) +#define HDR_L2ONLY_SIZE ((int64_t)offsetof(arc_buf_hdr_t, b_l1hdr)) /* * Hash table routines @@ -820,7 +898,7 @@ SYSCTL_UQUAD(_vfs_zfs, OID_AUTO, l2c_onl /* * L2ARC Internals */ -typedef struct l2arc_dev { +struct l2arc_dev { vdev_t *l2ad_vdev; /* vdev */ spa_t *l2ad_spa; /* spa */ uint64_t l2ad_hand; /* next write location */ @@ -829,15 +907,15 @@ typedef struct l2arc_dev { uint64_t l2ad_evict; /* last addr eviction reached */ boolean_t l2ad_first; /* first sweep through */ boolean_t l2ad_writing; /* currently writing */ - list_t *l2ad_buflist; /* buffer list */ + kmutex_t l2ad_mtx; /* lock for buffer list */ + list_t l2ad_buflist; /* buffer list */ list_node_t l2ad_node; /* device list node */ -} l2arc_dev_t; +}; static list_t L2ARC_dev_list; /* device list */ static list_t *l2arc_dev_list; /* device list pointer */ static kmutex_t l2arc_dev_mtx; /* device list mutex */ static l2arc_dev_t *l2arc_dev_last; /* last device used */ -static kmutex_t l2arc_buflist_mtx; /* mutex for all buflists */ static list_t L2ARC_free_on_write; /* free after write buf list */ static list_t *l2arc_free_on_write; /* free after write list ptr */ static kmutex_t l2arc_free_on_write_mtx; /* mutex for list */ @@ -857,18 +935,6 @@ typedef struct l2arc_write_callback { arc_buf_hdr_t *l2wcb_head; /* head of write buflist */ } l2arc_write_callback_t; -struct l2arc_buf_hdr { - /* protected by arc_buf_hdr mutex */ - l2arc_dev_t *b_dev; /* L2ARC device */ - uint64_t b_daddr; /* disk address, offset byte */ - /* compression applied to buffer data */ - enum zio_compress b_compress; - /* real alloc'd buffer size depending on b_compress applied */ - int b_asize; - /* temporary buffer holder for in-flight compressed data */ - void *b_tmp_cdata; -}; - typedef struct l2arc_data_free { /* protected by l2arc_free_on_write_mtx */ void *l2df_data; @@ -887,12 +953,13 @@ static int arc_evict_needed(arc_buf_cont static void arc_evict_ghost(arc_state_t *, uint64_t, int64_t); static void arc_buf_watch(arc_buf_t *); +static arc_buf_contents_t arc_buf_type(arc_buf_hdr_t *); +static uint32_t arc_bufc_to_flags(arc_buf_contents_t); + static boolean_t l2arc_write_eligible(uint64_t, arc_buf_hdr_t *); static void l2arc_read_done(zio_t *); -static void l2arc_hdr_stat_add(void); -static void l2arc_hdr_stat_remove(void); -static boolean_t l2arc_compress_buf(l2arc_buf_hdr_t *); +static boolean_t l2arc_compress_buf(arc_buf_hdr_t *); static void l2arc_decompress_zio(zio_t *, arc_buf_hdr_t *, enum zio_compress); static void l2arc_release_cdata_buf(arc_buf_hdr_t *); @@ -915,8 +982,7 @@ buf_hash(uint64_t spa, const dva_t *dva, #define BUF_EMPTY(buf) \ ((buf)->b_dva.dva_word[0] == 0 && \ - (buf)->b_dva.dva_word[1] == 0 && \ - (buf)->b_cksum0 == 0) + (buf)->b_dva.dva_word[1] == 0) #define BUF_EQUAL(spa, dva, birth, buf) \ ((buf)->b_dva.dva_word[0] == (dva)->dva_word[0]) && \ @@ -929,7 +995,6 @@ buf_discard_identity(arc_buf_hdr_t *hdr) hdr->b_dva.dva_word[0] = 0; hdr->b_dva.dva_word[1] = 0; hdr->b_birth = 0; - hdr->b_cksum0 = 0; } static arc_buf_hdr_t * @@ -959,6 +1024,7 @@ buf_hash_find(uint64_t spa, const blkptr * equal to elem in the hash table, then the already existing element * will be returned and the new element will not be inserted. * Otherwise returns NULL. + * If lockp == NULL, the caller is assumed to already hold the hash lock. */ static arc_buf_hdr_t * buf_hash_insert(arc_buf_hdr_t *hdr, kmutex_t **lockp) @@ -971,8 +1037,14 @@ buf_hash_insert(arc_buf_hdr_t *hdr, kmut ASSERT(!DVA_IS_EMPTY(&hdr->b_dva)); ASSERT(hdr->b_birth != 0); ASSERT(!HDR_IN_HASH_TABLE(hdr)); - *lockp = hash_lock; - mutex_enter(hash_lock); + + if (lockp != NULL) { + *lockp = hash_lock; + mutex_enter(hash_lock); + } else { + ASSERT(MUTEX_HELD(hash_lock)); + } + for (fhdr = buf_hash_table.ht_table[idx], i = 0; fhdr != NULL; fhdr = fhdr->b_hash_next, i++) { if (BUF_EQUAL(hdr->b_spa, &hdr->b_dva, hdr->b_birth, fhdr)) @@ -1027,7 +1099,8 @@ buf_hash_remove(arc_buf_hdr_t *hdr) /* * Global data structures and functions for the buf kmem cache. */ -static kmem_cache_t *hdr_cache; +static kmem_cache_t *hdr_full_cache; +static kmem_cache_t *hdr_l2only_cache; static kmem_cache_t *buf_cache; static void @@ -1039,7 +1112,8 @@ buf_fini(void) (buf_hash_table.ht_mask + 1) * sizeof (void *)); for (i = 0; i < BUF_LOCKS; i++) mutex_destroy(&buf_hash_table.ht_locks[i].ht_lock); - kmem_cache_destroy(hdr_cache); + kmem_cache_destroy(hdr_full_cache); + kmem_cache_destroy(hdr_l2only_cache); kmem_cache_destroy(buf_cache); } @@ -1049,15 +1123,27 @@ buf_fini(void) */ /* ARGSUSED */ static int -hdr_cons(void *vbuf, void *unused, int kmflag) +hdr_full_cons(void *vbuf, void *unused, int kmflag) +{ + arc_buf_hdr_t *hdr = vbuf; + + bzero(hdr, HDR_FULL_SIZE); + cv_init(&hdr->b_l1hdr.b_cv, NULL, CV_DEFAULT, NULL); + refcount_create(&hdr->b_l1hdr.b_refcnt); + mutex_init(&hdr->b_l1hdr.b_freeze_lock, NULL, MUTEX_DEFAULT, NULL); + arc_space_consume(HDR_FULL_SIZE, ARC_SPACE_HDRS); + + return (0); +} + +/* ARGSUSED */ +static int +hdr_l2only_cons(void *vbuf, void *unused, int kmflag) { arc_buf_hdr_t *hdr = vbuf; - bzero(hdr, sizeof (arc_buf_hdr_t)); - refcount_create(&hdr->b_refcnt); - cv_init(&hdr->b_cv, NULL, CV_DEFAULT, NULL); - mutex_init(&hdr->b_freeze_lock, NULL, MUTEX_DEFAULT, NULL); - arc_space_consume(sizeof (arc_buf_hdr_t), ARC_SPACE_HDRS); + bzero(hdr, HDR_L2ONLY_SIZE); + arc_space_consume(HDR_L2ONLY_SIZE, ARC_SPACE_L2HDRS); return (0); } @@ -1081,15 +1167,25 @@ buf_cons(void *vbuf, void *unused, int k */ /* ARGSUSED */ static void -hdr_dest(void *vbuf, void *unused) +hdr_full_dest(void *vbuf, void *unused) { arc_buf_hdr_t *hdr = vbuf; ASSERT(BUF_EMPTY(hdr)); - refcount_destroy(&hdr->b_refcnt); - cv_destroy(&hdr->b_cv); - mutex_destroy(&hdr->b_freeze_lock); - arc_space_return(sizeof (arc_buf_hdr_t), ARC_SPACE_HDRS); + cv_destroy(&hdr->b_l1hdr.b_cv); + refcount_destroy(&hdr->b_l1hdr.b_refcnt); + mutex_destroy(&hdr->b_l1hdr.b_freeze_lock); + arc_space_return(HDR_FULL_SIZE, ARC_SPACE_HDRS); +} + +/* ARGSUSED */ +static void +hdr_l2only_dest(void *vbuf, void *unused) +{ + arc_buf_hdr_t *hdr = vbuf; + + ASSERT(BUF_EMPTY(hdr)); + arc_space_return(HDR_L2ONLY_SIZE, ARC_SPACE_L2HDRS); } /* ARGSUSED */ @@ -1143,8 +1239,11 @@ retry: goto retry; } - hdr_cache = kmem_cache_create("arc_buf_hdr_t", sizeof (arc_buf_hdr_t), - 0, hdr_cons, hdr_dest, hdr_recl, NULL, NULL, 0); + hdr_full_cache = kmem_cache_create("arc_buf_hdr_t_full", HDR_FULL_SIZE, + 0, hdr_full_cons, hdr_full_dest, hdr_recl, NULL, NULL, 0); + hdr_l2only_cache = kmem_cache_create("arc_buf_hdr_t_l2only", + HDR_L2ONLY_SIZE, 0, hdr_l2only_cons, hdr_l2only_dest, hdr_recl, + NULL, NULL, 0); buf_cache = kmem_cache_create("arc_buf_t", sizeof (arc_buf_t), 0, buf_cons, buf_dest, NULL, NULL, NULL, 0); @@ -1158,6 +1257,81 @@ retry: } } +/* + * Transition between the two allocation states for the arc_buf_hdr struct. + * The arc_buf_hdr struct can be allocated with (hdr_full_cache) or without + * (hdr_l2only_cache) the fields necessary for the L1 cache - the smaller + * version is used when a cache buffer is only in the L2ARC in order to reduce + * memory usage. + */ +static arc_buf_hdr_t * +arc_hdr_realloc(arc_buf_hdr_t *hdr, kmem_cache_t *old, kmem_cache_t *new) +{ + ASSERT(HDR_HAS_L2HDR(hdr)); + + arc_buf_hdr_t *nhdr; + l2arc_dev_t *dev = hdr->b_l2hdr.b_dev; + + ASSERT((old == hdr_full_cache && new == hdr_l2only_cache) || + (old == hdr_l2only_cache && new == hdr_full_cache)); + + nhdr = kmem_cache_alloc(new, KM_PUSHPAGE); + + ASSERT(MUTEX_HELD(HDR_LOCK(hdr))); + buf_hash_remove(hdr); + + bcopy(hdr, nhdr, HDR_L2ONLY_SIZE); + if (new == hdr_full_cache) { + nhdr->b_flags |= ARC_FLAG_HAS_L1HDR; + /* + * arc_access and arc_change_state need to be aware that a + * header has just come out of L2ARC, so we set its state to + * l2c_only even though it's about to change. + */ + nhdr->b_l1hdr.b_state = arc_l2c_only; + } else { + ASSERT(hdr->b_l1hdr.b_buf == NULL); + ASSERT0(hdr->b_l1hdr.b_datacnt); + ASSERT(!list_link_active(&hdr->b_l1hdr.b_arc_node)); + /* + * We might be removing the L1hdr of a buffer which was just + * written out to L2ARC. If such a buffer is compressed then we + * need to free its b_tmp_cdata before destroying the header. + */ + if (hdr->b_l1hdr.b_tmp_cdata != NULL && + HDR_GET_COMPRESS(hdr) != ZIO_COMPRESS_OFF) + l2arc_release_cdata_buf(hdr); + nhdr->b_flags &= ~ARC_FLAG_HAS_L1HDR; + } + /* + * The header has been reallocated so we need to re-insert it into any + * lists it was on. + */ + (void) buf_hash_insert(nhdr, NULL); + + ASSERT(list_link_active(&hdr->b_l2hdr.b_l2node)); + + mutex_enter(&dev->l2ad_mtx); + + /* + * We must place the realloc'ed header back into the list at + * the same spot. Otherwise, if it's placed earlier in the list, + * l2arc_write_buffers() could find it during the function's + * write phase, and try to write it out to the l2arc. + */ + list_insert_after(&dev->l2ad_buflist, hdr, nhdr); + list_remove(&dev->l2ad_buflist, hdr); + + mutex_exit(&dev->l2ad_mtx); + + buf_discard_identity(hdr); + hdr->b_freeze_cksum = NULL; + kmem_cache_free(old, hdr); + + return (nhdr); +} + + #define ARC_MINTIME (hz>>4) /* 62 ms */ static void @@ -1168,16 +1342,15 @@ arc_cksum_verify(arc_buf_t *buf) if (!(zfs_flags & ZFS_DEBUG_MODIFY)) return; - mutex_enter(&buf->b_hdr->b_freeze_lock); - if (buf->b_hdr->b_freeze_cksum == NULL || - (buf->b_hdr->b_flags & ARC_FLAG_IO_ERROR)) { - mutex_exit(&buf->b_hdr->b_freeze_lock); + mutex_enter(&buf->b_hdr->b_l1hdr.b_freeze_lock); + if (buf->b_hdr->b_freeze_cksum == NULL || HDR_IO_ERROR(buf->b_hdr)) { + mutex_exit(&buf->b_hdr->b_l1hdr.b_freeze_lock); return; } fletcher_2_native(buf->b_data, buf->b_hdr->b_size, &zc); if (!ZIO_CHECKSUM_EQUAL(*buf->b_hdr->b_freeze_cksum, zc)) panic("buffer modified while frozen!"); - mutex_exit(&buf->b_hdr->b_freeze_lock); + mutex_exit(&buf->b_hdr->b_l1hdr.b_freeze_lock); } static int @@ -1186,10 +1359,10 @@ arc_cksum_equal(arc_buf_t *buf) zio_cksum_t zc; int equal; - mutex_enter(&buf->b_hdr->b_freeze_lock); + mutex_enter(&buf->b_hdr->b_l1hdr.b_freeze_lock); fletcher_2_native(buf->b_data, buf->b_hdr->b_size, &zc); equal = ZIO_CHECKSUM_EQUAL(*buf->b_hdr->b_freeze_cksum, zc); - mutex_exit(&buf->b_hdr->b_freeze_lock); + mutex_exit(&buf->b_hdr->b_l1hdr.b_freeze_lock); return (equal); } @@ -1200,15 +1373,15 @@ arc_cksum_compute(arc_buf_t *buf, boolea if (!force && !(zfs_flags & ZFS_DEBUG_MODIFY)) return; - mutex_enter(&buf->b_hdr->b_freeze_lock); + mutex_enter(&buf->b_hdr->b_l1hdr.b_freeze_lock); if (buf->b_hdr->b_freeze_cksum != NULL) { - mutex_exit(&buf->b_hdr->b_freeze_lock); + mutex_exit(&buf->b_hdr->b_l1hdr.b_freeze_lock); return; } buf->b_hdr->b_freeze_cksum = kmem_alloc(sizeof (zio_cksum_t), KM_SLEEP); fletcher_2_native(buf->b_data, buf->b_hdr->b_size, buf->b_hdr->b_freeze_cksum); - mutex_exit(&buf->b_hdr->b_freeze_lock); + mutex_exit(&buf->b_hdr->b_l1hdr.b_freeze_lock); #ifdef illumos arc_buf_watch(buf); #endif /* illumos */ @@ -1259,30 +1432,58 @@ arc_buf_watch(arc_buf_t *buf) } #endif /* illumos */ +static arc_buf_contents_t +arc_buf_type(arc_buf_hdr_t *hdr) +{ + if (HDR_ISTYPE_METADATA(hdr)) { + return (ARC_BUFC_METADATA); + } else { + return (ARC_BUFC_DATA); + } +} + +static uint32_t +arc_bufc_to_flags(arc_buf_contents_t type) +{ + switch (type) { + case ARC_BUFC_DATA: + /* metadata field is 0 if buffer contains normal data */ + return (0); + case ARC_BUFC_METADATA: + return (ARC_FLAG_BUFC_METADATA); + default: + break; + } + panic("undefined ARC buffer type!"); + return ((uint32_t)-1); +} + void arc_buf_thaw(arc_buf_t *buf) { if (zfs_flags & ZFS_DEBUG_MODIFY) { - if (buf->b_hdr->b_state != arc_anon) + if (buf->b_hdr->b_l1hdr.b_state != arc_anon) panic("modifying non-anon buffer!"); - if (buf->b_hdr->b_flags & ARC_FLAG_IO_IN_PROGRESS) + if (HDR_IO_IN_PROGRESS(buf->b_hdr)) panic("modifying buffer while i/o in progress!"); arc_cksum_verify(buf); } - mutex_enter(&buf->b_hdr->b_freeze_lock); + mutex_enter(&buf->b_hdr->b_l1hdr.b_freeze_lock); if (buf->b_hdr->b_freeze_cksum != NULL) { kmem_free(buf->b_hdr->b_freeze_cksum, sizeof (zio_cksum_t)); buf->b_hdr->b_freeze_cksum = NULL; } +#ifdef ZFS_DEBUG if (zfs_flags & ZFS_DEBUG_MODIFY) { - if (buf->b_hdr->b_thawed) - kmem_free(buf->b_hdr->b_thawed, 1); - buf->b_hdr->b_thawed = kmem_alloc(1, KM_SLEEP); + if (buf->b_hdr->b_l1hdr.b_thawed != NULL) + kmem_free(buf->b_hdr->b_l1hdr.b_thawed, 1); + buf->b_hdr->b_l1hdr.b_thawed = kmem_alloc(1, KM_SLEEP); } +#endif - mutex_exit(&buf->b_hdr->b_freeze_lock); + mutex_exit(&buf->b_hdr->b_l1hdr.b_freeze_lock); #ifdef illumos arc_buf_unwatch(buf); @@ -1301,7 +1502,7 @@ arc_buf_freeze(arc_buf_t *buf) mutex_enter(hash_lock); ASSERT(buf->b_hdr->b_freeze_cksum != NULL || - buf->b_hdr->b_state == arc_anon); + buf->b_hdr->b_l1hdr.b_state == arc_anon); arc_cksum_compute(buf, B_FALSE); mutex_exit(hash_lock); @@ -1312,7 +1513,7 @@ get_buf_info(arc_buf_hdr_t *hdr, arc_sta { uint64_t buf_hashid = buf_hash(hdr->b_spa, &hdr->b_dva, hdr->b_birth); - if (hdr->b_type == ARC_BUFC_METADATA) + if (arc_buf_type(hdr) == ARC_BUFC_METADATA) buf_hashid &= (ARC_BUFC_NUMMETADATALISTS - 1); else { buf_hashid &= (ARC_BUFC_NUMDATALISTS - 1); @@ -1327,32 +1528,36 @@ get_buf_info(arc_buf_hdr_t *hdr, arc_sta static void add_reference(arc_buf_hdr_t *hdr, kmutex_t *hash_lock, void *tag) { + ASSERT(HDR_HAS_L1HDR(hdr)); ASSERT(MUTEX_HELD(hash_lock)); + arc_state_t *state = hdr->b_l1hdr.b_state; - if ((refcount_add(&hdr->b_refcnt, tag) == 1) && - (hdr->b_state != arc_anon)) { - uint64_t delta = hdr->b_size * hdr->b_datacnt; - uint64_t *size = &hdr->b_state->arcs_lsize[hdr->b_type]; - list_t *list; - kmutex_t *lock; + if ((refcount_add(&hdr->b_l1hdr.b_refcnt, tag) == 1) && + (state != arc_anon)) { + /* We don't use the L2-only state list. */ + if (state != arc_l2c_only) { + uint64_t delta = hdr->b_size * hdr->b_l1hdr.b_datacnt; + uint64_t *size = &state->arcs_lsize[arc_buf_type(hdr)]; + list_t *list; + kmutex_t *lock; - get_buf_info(hdr, hdr->b_state, &list, &lock); - ASSERT(!MUTEX_HELD(lock)); - mutex_enter(lock); - ASSERT(list_link_active(&hdr->b_arc_node)); - list_remove(list, hdr); - if (GHOST_STATE(hdr->b_state)) { - ASSERT0(hdr->b_datacnt); - ASSERT3P(hdr->b_buf, ==, NULL); - delta = hdr->b_size; - } - ASSERT(delta > 0); - ASSERT3U(*size, >=, delta); - atomic_add_64(size, -delta); - mutex_exit(lock); + get_buf_info(hdr, state, &list, &lock); + ASSERT(!MUTEX_HELD(lock)); + mutex_enter(lock); + ASSERT(list_link_active(&hdr->b_l1hdr.b_arc_node)); + list_remove(list, hdr); + if (GHOST_STATE(state)) { + ASSERT0(hdr->b_l1hdr.b_datacnt); + ASSERT3P(hdr->b_l1hdr.b_buf, ==, NULL); + delta = hdr->b_size; + } + ASSERT(delta > 0); + ASSERT3U(*size, >=, delta); + atomic_add_64(size, -delta); + mutex_exit(lock); + } /* remove the prefetch flag if we get a reference */ - if (hdr->b_flags & ARC_FLAG_PREFETCH) - hdr->b_flags &= ~ARC_FLAG_PREFETCH; + hdr->b_flags &= ~ARC_FLAG_PREFETCH; } } @@ -1360,24 +1565,30 @@ static int remove_reference(arc_buf_hdr_t *hdr, kmutex_t *hash_lock, void *tag) { int cnt; - arc_state_t *state = hdr->b_state; + arc_state_t *state = hdr->b_l1hdr.b_state; + ASSERT(HDR_HAS_L1HDR(hdr)); ASSERT(state == arc_anon || MUTEX_HELD(hash_lock)); ASSERT(!GHOST_STATE(state)); - if (((cnt = refcount_remove(&hdr->b_refcnt, tag)) == 0) && + /* + * arc_l2c_only counts as a ghost state so we don't need to explicitly + * check to prevent usage of the arc_l2c_only list. + */ + if (((cnt = refcount_remove(&hdr->b_l1hdr.b_refcnt, tag)) == 0) && (state != arc_anon)) { - uint64_t *size = &state->arcs_lsize[hdr->b_type]; + uint64_t *size = &state->arcs_lsize[arc_buf_type(hdr)]; list_t *list; kmutex_t *lock; get_buf_info(hdr, state, &list, &lock); ASSERT(!MUTEX_HELD(lock)); mutex_enter(lock); - ASSERT(!list_link_active(&hdr->b_arc_node)); + ASSERT(!list_link_active(&hdr->b_l1hdr.b_arc_node)); list_insert_head(list, hdr); - ASSERT(hdr->b_datacnt > 0); - atomic_add_64(size, hdr->b_size * hdr->b_datacnt); + ASSERT(hdr->b_l1hdr.b_datacnt > 0); + atomic_add_64(size, hdr->b_size * + hdr->b_l1hdr.b_datacnt); mutex_exit(lock); } return (cnt); @@ -1391,44 +1602,64 @@ static void arc_change_state(arc_state_t *new_state, arc_buf_hdr_t *hdr, kmutex_t *hash_lock) { - arc_state_t *old_state = hdr->b_state; - int64_t refcnt = refcount_count(&hdr->b_refcnt); + arc_state_t *old_state; + int64_t refcnt; + uint32_t datacnt; uint64_t from_delta, to_delta; + arc_buf_contents_t buftype = arc_buf_type(hdr); list_t *list; kmutex_t *lock; + /* + * We almost always have an L1 hdr here, since we call arc_hdr_realloc() + * in arc_read() when bringing a buffer out of the L2ARC. However, the + * L1 hdr doesn't always exist when we change state to arc_anon before + * destroying a header, in which case reallocating to add the L1 hdr is + * pointless. + */ + if (HDR_HAS_L1HDR(hdr)) { + old_state = hdr->b_l1hdr.b_state; + refcnt = refcount_count(&hdr->b_l1hdr.b_refcnt); + datacnt = hdr->b_l1hdr.b_datacnt; + } else { + old_state = arc_l2c_only; + refcnt = 0; + datacnt = 0; + } + ASSERT(MUTEX_HELD(hash_lock)); ASSERT3P(new_state, !=, old_state); - ASSERT(refcnt == 0 || hdr->b_datacnt > 0); - ASSERT(hdr->b_datacnt == 0 || !GHOST_STATE(new_state)); - ASSERT(hdr->b_datacnt <= 1 || old_state != arc_anon); + ASSERT(refcnt == 0 || datacnt > 0); + ASSERT(!GHOST_STATE(new_state) || datacnt == 0); + ASSERT(old_state != arc_anon || datacnt <= 1); - from_delta = to_delta = hdr->b_datacnt * hdr->b_size; + from_delta = to_delta = datacnt * hdr->b_size; /* * If this buffer is evictable, transfer it from the * old state list to the new state list. */ if (refcnt == 0) { - if (old_state != arc_anon) { + if (old_state != arc_anon && old_state != arc_l2c_only) { int use_mutex; - uint64_t *size = &old_state->arcs_lsize[hdr->b_type]; + uint64_t *size = &old_state->arcs_lsize[buftype]; get_buf_info(hdr, old_state, &list, &lock); use_mutex = !MUTEX_HELD(lock); if (use_mutex) mutex_enter(lock); - ASSERT(list_link_active(&hdr->b_arc_node)); + ASSERT(HDR_HAS_L1HDR(hdr)); + ASSERT(list_link_active(&hdr->b_l1hdr.b_arc_node)); list_remove(list, hdr); /* * If prefetching out of the ghost cache, * we will have a non-zero datacnt. */ - if (GHOST_STATE(old_state) && hdr->b_datacnt == 0) { + if (GHOST_STATE(old_state) && datacnt == 0) { /* ghost elements have a ghost size */ - ASSERT(hdr->b_buf == NULL); + ASSERT(hdr->b_l1hdr.b_buf == NULL); from_delta = hdr->b_size; } ASSERT3U(*size, >=, from_delta); @@ -1437,10 +1668,17 @@ arc_change_state(arc_state_t *new_state, if (use_mutex) mutex_exit(lock); } - if (new_state != arc_anon) { + if (new_state != arc_anon && new_state != arc_l2c_only) { int use_mutex; - uint64_t *size = &new_state->arcs_lsize[hdr->b_type]; + uint64_t *size = &new_state->arcs_lsize[buftype]; + /* + * An L1 header always exists here, since if we're + * moving to some L1-cached state (i.e. not l2c_only or + * anonymous), we realloc the header to add an L1hdr + * beforehand. + */ + ASSERT(HDR_HAS_L1HDR(hdr)); get_buf_info(hdr, new_state, &list, &lock); use_mutex = !MUTEX_HELD(lock); if (use_mutex) @@ -1450,8 +1688,8 @@ arc_change_state(arc_state_t *new_state, /* ghost elements have a ghost size */ if (GHOST_STATE(new_state)) { - ASSERT(hdr->b_datacnt == 0); - ASSERT(hdr->b_buf == NULL); + ASSERT(datacnt == 0); + ASSERT(hdr->b_l1hdr.b_buf == NULL); to_delta = hdr->b_size; } atomic_add_64(size, to_delta); @@ -1465,20 +1703,22 @@ arc_change_state(arc_state_t *new_state, if (new_state == arc_anon && HDR_IN_HASH_TABLE(hdr)) buf_hash_remove(hdr); - /* adjust state sizes */ - if (to_delta) + /* adjust state sizes (ignore arc_l2c_only) */ + if (to_delta && new_state != arc_l2c_only) atomic_add_64(&new_state->arcs_size, to_delta); - if (from_delta) { + if (from_delta && old_state != arc_l2c_only) { ASSERT3U(old_state->arcs_size, >=, from_delta); atomic_add_64(&old_state->arcs_size, -from_delta); } - hdr->b_state = new_state; + if (HDR_HAS_L1HDR(hdr)) + hdr->b_l1hdr.b_state = new_state; - /* adjust l2arc hdr stats */ - if (new_state == arc_l2c_only) - l2arc_hdr_stat_add(); - else if (old_state == arc_l2c_only) - l2arc_hdr_stat_remove(); + /* + * L2 headers should never be on the L2 state list since they don't + * have L1 headers allocated. + */ + ASSERT(list_is_empty(&arc_l2c_only->arcs_list[ARC_BUFC_DATA]) && + list_is_empty(&arc_l2c_only->arcs_list[ARC_BUFC_METADATA])); } void @@ -1534,31 +1774,36 @@ arc_space_return(uint64_t space, arc_spa } arc_buf_t * -arc_buf_alloc(spa_t *spa, int size, void *tag, arc_buf_contents_t type) +arc_buf_alloc(spa_t *spa, int32_t size, void *tag, arc_buf_contents_t type) { arc_buf_hdr_t *hdr; arc_buf_t *buf; ASSERT3U(size, >, 0); - hdr = kmem_cache_alloc(hdr_cache, KM_PUSHPAGE); + hdr = kmem_cache_alloc(hdr_full_cache, KM_PUSHPAGE); ASSERT(BUF_EMPTY(hdr)); + ASSERT3P(hdr->b_freeze_cksum, ==, NULL); hdr->b_size = size; - hdr->b_type = type; hdr->b_spa = spa_load_guid(spa); - hdr->b_state = arc_anon; - hdr->b_arc_access = 0; + buf = kmem_cache_alloc(buf_cache, KM_PUSHPAGE); buf->b_hdr = hdr; buf->b_data = NULL; buf->b_efunc = NULL; buf->b_private = NULL; buf->b_next = NULL; - hdr->b_buf = buf; + + hdr->b_flags = arc_bufc_to_flags(type); + hdr->b_flags |= ARC_FLAG_HAS_L1HDR; + + hdr->b_l1hdr.b_buf = buf; + hdr->b_l1hdr.b_state = arc_anon; + hdr->b_l1hdr.b_arc_access = 0; + hdr->b_l1hdr.b_datacnt = 1; + arc_get_data_buf(buf); - hdr->b_datacnt = 1; - hdr->b_flags = 0; - ASSERT(refcount_is_zero(&hdr->b_refcnt)); - (void) refcount_add(&hdr->b_refcnt, tag); + ASSERT(refcount_is_zero(&hdr->b_l1hdr.b_refcnt)); + (void) refcount_add(&hdr->b_l1hdr.b_refcnt, tag); return (buf); } @@ -1591,8 +1836,9 @@ arc_return_buf(arc_buf_t *buf, void *tag arc_buf_hdr_t *hdr = buf->b_hdr; ASSERT(buf->b_data != NULL); - (void) refcount_add(&hdr->b_refcnt, tag); - (void) refcount_remove(&hdr->b_refcnt, arc_onloan_tag); + ASSERT(HDR_HAS_L1HDR(hdr)); + (void) refcount_add(&hdr->b_l1hdr.b_refcnt, tag); + (void) refcount_remove(&hdr->b_l1hdr.b_refcnt, arc_onloan_tag); atomic_add_64(&arc_loaned_bytes, -hdr->b_size); } @@ -1601,12 +1847,12 @@ arc_return_buf(arc_buf_t *buf, void *tag void arc_loan_inuse_buf(arc_buf_t *buf, void *tag) { - arc_buf_hdr_t *hdr; + arc_buf_hdr_t *hdr = buf->b_hdr; ASSERT(buf->b_data != NULL); - hdr = buf->b_hdr; - (void) refcount_add(&hdr->b_refcnt, arc_onloan_tag); - (void) refcount_remove(&hdr->b_refcnt, tag); + ASSERT(HDR_HAS_L1HDR(hdr)); + (void) refcount_add(&hdr->b_l1hdr.b_refcnt, arc_onloan_tag); + (void) refcount_remove(&hdr->b_l1hdr.b_refcnt, tag); buf->b_efunc = NULL; buf->b_private = NULL; @@ -1620,15 +1866,16 @@ arc_buf_clone(arc_buf_t *from) arc_buf_hdr_t *hdr = from->b_hdr; uint64_t size = hdr->b_size; - ASSERT(hdr->b_state != arc_anon); + ASSERT(HDR_HAS_L1HDR(hdr)); + ASSERT(hdr->b_l1hdr.b_state != arc_anon); buf = kmem_cache_alloc(buf_cache, KM_PUSHPAGE); buf->b_hdr = hdr; buf->b_data = NULL; buf->b_efunc = NULL; buf->b_private = NULL; - buf->b_next = hdr->b_buf; - hdr->b_buf = buf; + buf->b_next = hdr->b_l1hdr.b_buf; + hdr->b_l1hdr.b_buf = buf; arc_get_data_buf(buf); bcopy(from->b_data, buf->b_data, size); @@ -1638,11 +1885,11 @@ arc_buf_clone(arc_buf_t *from) * then track the size and number of duplicates. These stats will be * updated as duplicate buffers are created and destroyed. */ - if (hdr->b_type == ARC_BUFC_DATA) { + if (HDR_ISTYPE_DATA(hdr)) { ARCSTAT_BUMP(arcstat_duplicate_buffers); ARCSTAT_INCR(arcstat_duplicate_buffers_size, size); } - hdr->b_datacnt += 1; + hdr->b_l1hdr.b_datacnt += 1; return (buf); } @@ -1665,17 +1912,20 @@ arc_buf_add_ref(arc_buf_t *buf, void* ta hash_lock = HDR_LOCK(buf->b_hdr); mutex_enter(hash_lock); hdr = buf->b_hdr; + ASSERT(HDR_HAS_L1HDR(hdr)); ASSERT3P(hash_lock, ==, HDR_LOCK(hdr)); mutex_exit(&buf->b_evict_lock); - ASSERT(hdr->b_state == arc_mru || hdr->b_state == arc_mfu); + ASSERT(hdr->b_l1hdr.b_state == arc_mru || + hdr->b_l1hdr.b_state == arc_mfu); + add_reference(hdr, hash_lock, tag); DTRACE_PROBE1(arc__hit, arc_buf_hdr_t *, hdr); arc_access(hdr, hash_lock); mutex_exit(hash_lock); ARCSTAT_BUMP(arcstat_hits); - ARCSTAT_CONDSTAT(!(hdr->b_flags & ARC_FLAG_PREFETCH), - demand, prefetch, hdr->b_type != ARC_BUFC_METADATA, + ARCSTAT_CONDSTAT(!HDR_PREFETCH(hdr), + demand, prefetch, !HDR_ISTYPE_METADATA(hdr), data, metadata, hits); } @@ -1718,18 +1968,26 @@ arc_buf_data_free(arc_buf_t *buf, void ( static void arc_buf_l2_cdata_free(arc_buf_hdr_t *hdr) { - l2arc_buf_hdr_t *l2hdr = hdr->b_l2hdr; + ASSERT(HDR_HAS_L2HDR(hdr)); + ASSERT(MUTEX_HELD(&hdr->b_l2hdr.b_dev->l2ad_mtx)); *** DIFF OUTPUT TRUNCATED AT 1000 LINES *** From owner-svn-src-stable@freebsd.org Sat Oct 3 07:28:54 2015 Return-Path: Delivered-To: svn-src-stable@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 5FA93A0E44D; Sat, 3 Oct 2015 07:28:54 +0000 (UTC) (envelope-from mav@FreeBSD.org) Received: from repo.freebsd.org (repo.freebsd.org [IPv6:2001:1900:2254:2068::e6a:0]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 4E87711A3; Sat, 3 Oct 2015 07:28:54 +0000 (UTC) (envelope-from mav@FreeBSD.org) Received: from repo.freebsd.org ([127.0.1.70]) by repo.freebsd.org (8.15.2/8.15.2) with ESMTP id t937SsQX063499; Sat, 3 Oct 2015 07:28:54 GMT (envelope-from mav@FreeBSD.org) Received: (from mav@localhost) by repo.freebsd.org (8.15.2/8.15.2/Submit) id t937SrZt063496; Sat, 3 Oct 2015 07:28:53 GMT (envelope-from mav@FreeBSD.org) Message-Id: <201510030728.t937SrZt063496@repo.freebsd.org> X-Authentication-Warning: repo.freebsd.org: mav set sender to mav@FreeBSD.org using -f From: Alexander Motin Date: Sat, 3 Oct 2015 07:28:53 +0000 (UTC) To: src-committers@freebsd.org, svn-src-all@freebsd.org, svn-src-stable@freebsd.org, svn-src-stable-10@freebsd.org Subject: svn commit: r288548 - in stable/10/sys/cddl: compat/opensolaris/sys contrib/opensolaris/uts/common/fs/zfs contrib/opensolaris/uts/common/fs/zfs/sys X-SVN-Group: stable-10 MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit X-BeenThere: svn-src-stable@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: SVN commit messages for all the -stable branches of the src tree List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 03 Oct 2015 07:28:54 -0000 Author: mav Date: Sat Oct 3 07:28:52 2015 New Revision: 288548 URL: https://svnweb.freebsd.org/changeset/base/288548 Log: MFC r286574: 5445 Add more visibility via arcstats; specifically arc_state_t stats and differentiate between "data" and "metadata" Reviewed by: Basil Crow Reviewed by: George Wilson Reviewed by: Matthew Ahrens Reviewed by: Bayard Bell Approved by: Robert Mustacchi Author: Prakash Surya illumos/illumos-gate@4076b1bf41cfd9f968a33ed54a7ae76d9e996fe8 Modified: stable/10/sys/cddl/compat/opensolaris/sys/kstat.h stable/10/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/arc.c stable/10/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/sys/arc.h Directory Properties: stable/10/ (props changed) Modified: stable/10/sys/cddl/compat/opensolaris/sys/kstat.h ============================================================================== --- stable/10/sys/cddl/compat/opensolaris/sys/kstat.h Sat Oct 3 07:27:58 2015 (r288547) +++ stable/10/sys/cddl/compat/opensolaris/sys/kstat.h Sat Oct 3 07:28:52 2015 (r288548) @@ -35,12 +35,17 @@ #define KSTAT_FLAG_VIRTUAL 0x01 +#define KSTAT_READ 0 +#define KSTAT_WRITE 1 + typedef struct kstat { void *ks_data; u_int ks_ndata; #ifdef _KERNEL struct sysctl_ctx_list ks_sysctl_ctx; struct sysctl_oid *ks_sysctl_root; + int (*ks_update)(struct kstat *, int); /* dynamic update */ + void *ks_private; /* arbitrary provider-private data */ #endif } kstat_t; Modified: stable/10/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/arc.c ============================================================================== --- stable/10/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/arc.c Sat Oct 3 07:27:58 2015 (r288547) +++ stable/10/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/arc.c Sat Oct 3 07:28:52 2015 (r288548) @@ -387,9 +387,137 @@ typedef struct arc_stats { kstat_named_t arcstat_c_min; kstat_named_t arcstat_c_max; kstat_named_t arcstat_size; + /* + * Number of bytes consumed by internal ARC structures necessary + * for tracking purposes; these structures are not actually + * backed by ARC buffers. This includes arc_buf_hdr_t structures + * (allocated via arc_buf_hdr_t_full and arc_buf_hdr_t_l2only + * caches), and arc_buf_t structures (allocated via arc_buf_t + * cache). + */ kstat_named_t arcstat_hdr_size; + /* + * Number of bytes consumed by ARC buffers of type equal to + * ARC_BUFC_DATA. This is generally consumed by buffers backing + * on disk user data (e.g. plain file contents). + */ kstat_named_t arcstat_data_size; + /* + * Number of bytes consumed by ARC buffers of type equal to + * ARC_BUFC_METADATA. This is generally consumed by buffers + * backing on disk data that is used for internal ZFS + * structures (e.g. ZAP, dnode, indirect blocks, etc). + */ + kstat_named_t arcstat_metadata_size; + /* + * Number of bytes consumed by various buffers and structures + * not actually backed with ARC buffers. This includes bonus + * buffers (allocated directly via zio_buf_* functions), + * dmu_buf_impl_t structures (allocated via dmu_buf_impl_t + * cache), and dnode_t structures (allocated via dnode_t cache). + */ kstat_named_t arcstat_other_size; + /* + * Total number of bytes consumed by ARC buffers residing in the + * arc_anon state. This includes *all* buffers in the arc_anon + * state; e.g. data, metadata, evictable, and unevictable buffers + * are all included in this value. + */ + kstat_named_t arcstat_anon_size; + /* + * Number of bytes consumed by ARC buffers that meet the + * following criteria: backing buffers of type ARC_BUFC_DATA, + * residing in the arc_anon state, and are eligible for eviction + * (e.g. have no outstanding holds on the buffer). + */ + kstat_named_t arcstat_anon_evictable_data; + /* + * Number of bytes consumed by ARC buffers that meet the + * following criteria: backing buffers of type ARC_BUFC_METADATA, + * residing in the arc_anon state, and are eligible for eviction + * (e.g. have no outstanding holds on the buffer). + */ + kstat_named_t arcstat_anon_evictable_metadata; + /* + * Total number of bytes consumed by ARC buffers residing in the + * arc_mru state. This includes *all* buffers in the arc_mru + * state; e.g. data, metadata, evictable, and unevictable buffers + * are all included in this value. + */ + kstat_named_t arcstat_mru_size; + /* + * Number of bytes consumed by ARC buffers that meet the + * following criteria: backing buffers of type ARC_BUFC_DATA, + * residing in the arc_mru state, and are eligible for eviction + * (e.g. have no outstanding holds on the buffer). + */ + kstat_named_t arcstat_mru_evictable_data; + /* + * Number of bytes consumed by ARC buffers that meet the + * following criteria: backing buffers of type ARC_BUFC_METADATA, + * residing in the arc_mru state, and are eligible for eviction + * (e.g. have no outstanding holds on the buffer). + */ + kstat_named_t arcstat_mru_evictable_metadata; + /* + * Total number of bytes that *would have been* consumed by ARC + * buffers in the arc_mru_ghost state. The key thing to note + * here, is the fact that this size doesn't actually indicate + * RAM consumption. The ghost lists only consist of headers and + * don't actually have ARC buffers linked off of these headers. + * Thus, *if* the headers had associated ARC buffers, these + * buffers *would have* consumed this number of bytes. + */ + kstat_named_t arcstat_mru_ghost_size; + /* + * Number of bytes that *would have been* consumed by ARC + * buffers that are eligible for eviction, of type + * ARC_BUFC_DATA, and linked off the arc_mru_ghost state. + */ + kstat_named_t arcstat_mru_ghost_evictable_data; + /* + * Number of bytes that *would have been* consumed by ARC + * buffers that are eligible for eviction, of type + * ARC_BUFC_METADATA, and linked off the arc_mru_ghost state. + */ + kstat_named_t arcstat_mru_ghost_evictable_metadata; + /* + * Total number of bytes consumed by ARC buffers residing in the + * arc_mfu state. This includes *all* buffers in the arc_mfu + * state; e.g. data, metadata, evictable, and unevictable buffers + * are all included in this value. + */ + kstat_named_t arcstat_mfu_size; + /* + * Number of bytes consumed by ARC buffers that are eligible for + * eviction, of type ARC_BUFC_DATA, and reside in the arc_mfu + * state. + */ + kstat_named_t arcstat_mfu_evictable_data; + /* + * Number of bytes consumed by ARC buffers that are eligible for + * eviction, of type ARC_BUFC_METADATA, and reside in the + * arc_mfu state. + */ + kstat_named_t arcstat_mfu_evictable_metadata; + /* + * Total number of bytes that *would have been* consumed by ARC + * buffers in the arc_mfu_ghost state. See the comment above + * arcstat_mru_ghost_size for more details. + */ + kstat_named_t arcstat_mfu_ghost_size; + /* + * Number of bytes that *would have been* consumed by ARC + * buffers that are eligible for eviction, of type + * ARC_BUFC_DATA, and linked off the arc_mfu_ghost state. + */ + kstat_named_t arcstat_mfu_ghost_evictable_data; + /* + * Number of bytes that *would have been* consumed by ARC + * buffers that are eligible for eviction, of type + * ARC_BUFC_METADATA, and linked off the arc_mru_ghost state. + */ + kstat_named_t arcstat_mfu_ghost_evictable_metadata; kstat_named_t arcstat_l2_hits; kstat_named_t arcstat_l2_misses; kstat_named_t arcstat_l2_feeds; @@ -472,7 +600,23 @@ static arc_stats_t arc_stats = { { "size", KSTAT_DATA_UINT64 }, { "hdr_size", KSTAT_DATA_UINT64 }, { "data_size", KSTAT_DATA_UINT64 }, + { "metadata_size", KSTAT_DATA_UINT64 }, { "other_size", KSTAT_DATA_UINT64 }, + { "anon_size", KSTAT_DATA_UINT64 }, + { "anon_evictable_data", KSTAT_DATA_UINT64 }, + { "anon_evictable_metadata", KSTAT_DATA_UINT64 }, + { "mru_size", KSTAT_DATA_UINT64 }, + { "mru_evictable_data", KSTAT_DATA_UINT64 }, + { "mru_evictable_metadata", KSTAT_DATA_UINT64 }, + { "mru_ghost_size", KSTAT_DATA_UINT64 }, + { "mru_ghost_evictable_data", KSTAT_DATA_UINT64 }, + { "mru_ghost_evictable_metadata", KSTAT_DATA_UINT64 }, + { "mfu_size", KSTAT_DATA_UINT64 }, + { "mfu_evictable_data", KSTAT_DATA_UINT64 }, + { "mfu_evictable_metadata", KSTAT_DATA_UINT64 }, + { "mfu_ghost_size", KSTAT_DATA_UINT64 }, + { "mfu_ghost_evictable_data", KSTAT_DATA_UINT64 }, + { "mfu_ghost_evictable_metadata", KSTAT_DATA_UINT64 }, { "l2_hits", KSTAT_DATA_UINT64 }, { "l2_misses", KSTAT_DATA_UINT64 }, { "l2_feeds", KSTAT_DATA_UINT64 }, @@ -1730,6 +1874,9 @@ arc_space_consume(uint64_t space, arc_sp case ARC_SPACE_DATA: ARCSTAT_INCR(arcstat_data_size, space); break; + case ARC_SPACE_META: + ARCSTAT_INCR(arcstat_metadata_size, space); + break; case ARC_SPACE_OTHER: ARCSTAT_INCR(arcstat_other_size, space); break; @@ -1741,7 +1888,9 @@ arc_space_consume(uint64_t space, arc_sp break; } - ARCSTAT_INCR(arcstat_meta_used, space); + if (type != ARC_SPACE_DATA) + ARCSTAT_INCR(arcstat_meta_used, space); + atomic_add_64(&arc_size, space); } @@ -1754,6 +1903,9 @@ arc_space_return(uint64_t space, arc_spa case ARC_SPACE_DATA: ARCSTAT_INCR(arcstat_data_size, -space); break; + case ARC_SPACE_META: + ARCSTAT_INCR(arcstat_metadata_size, -space); + break; case ARC_SPACE_OTHER: ARCSTAT_INCR(arcstat_other_size, -space); break; @@ -1765,10 +1917,13 @@ arc_space_return(uint64_t space, arc_spa break; } - ASSERT(arc_meta_used >= space); - if (arc_meta_max < arc_meta_used) - arc_meta_max = arc_meta_used; - ARCSTAT_INCR(arcstat_meta_used, -space); + if (type != ARC_SPACE_DATA) { + ASSERT(arc_meta_used >= space); + if (arc_meta_max < arc_meta_used) + arc_meta_max = arc_meta_used; + ARCSTAT_INCR(arcstat_meta_used, -space); + } + ASSERT(arc_size >= space); atomic_add_64(&arc_size, -space); } @@ -2009,12 +2164,11 @@ arc_buf_destroy(arc_buf_t *buf, boolean_ if (!recycle) { if (type == ARC_BUFC_METADATA) { arc_buf_data_free(buf, zio_buf_free); - arc_space_return(size, ARC_SPACE_DATA); + arc_space_return(size, ARC_SPACE_META); } else { ASSERT(type == ARC_BUFC_DATA); arc_buf_data_free(buf, zio_data_buf_free); - ARCSTAT_INCR(arcstat_data_size, -size); - atomic_add_64(&arc_size, -size); + arc_space_return(size, ARC_SPACE_DATA); } } if (list_link_active(&buf->b_hdr->b_l1hdr.b_arc_node)) { @@ -3099,6 +3253,20 @@ arc_reclaim_thread(void *dummy __unused) } #endif + /* + * This is necessary in order for the mdb ::arc dcmd to + * show up to date information. Since the ::arc command + * does not call the kstat's update function, without + * this call, the command may show stale stats for the + * anon, mru, mru_ghost, mfu, and mfu_ghost lists. Even + * with this change, the data might be up to 1 second + * out of date; but that should suffice. The arc_state_t + * structures can be queried directly if more accurate + * information is needed. + */ + if (arc_ksp != NULL) + arc_ksp->ks_update(arc_ksp, KSTAT_READ); + /* block until needed, or one second, whichever is shorter */ CALLB_CPR_SAFE_BEGIN(&cpr); (void) cv_timedwait(&arc_reclaim_thr_cv, @@ -3236,12 +3404,11 @@ arc_get_data_buf(arc_buf_t *buf) if (!arc_evict_needed(type)) { if (type == ARC_BUFC_METADATA) { buf->b_data = zio_buf_alloc(size); - arc_space_consume(size, ARC_SPACE_DATA); + arc_space_consume(size, ARC_SPACE_META); } else { ASSERT(type == ARC_BUFC_DATA); buf->b_data = zio_data_buf_alloc(size); - ARCSTAT_INCR(arcstat_data_size, size); - atomic_add_64(&arc_size, size); + arc_space_consume(size, ARC_SPACE_DATA); } goto out; } @@ -3268,12 +3435,11 @@ arc_get_data_buf(arc_buf_t *buf) if ((buf->b_data = arc_evict(state, 0, size, TRUE, type)) == NULL) { if (type == ARC_BUFC_METADATA) { buf->b_data = zio_buf_alloc(size); - arc_space_consume(size, ARC_SPACE_DATA); + arc_space_consume(size, ARC_SPACE_META); } else { ASSERT(type == ARC_BUFC_DATA); buf->b_data = zio_data_buf_alloc(size); - ARCSTAT_INCR(arcstat_data_size, size); - atomic_add_64(&arc_size, size); + arc_space_consume(size, ARC_SPACE_DATA); } ARCSTAT_BUMP(arcstat_recycle_miss); } @@ -4434,6 +4600,48 @@ arc_memory_throttle(uint64_t reserve, ui return (0); } +static void +arc_kstat_update_state(arc_state_t *state, kstat_named_t *size, + kstat_named_t *evict_data, kstat_named_t *evict_metadata) +{ + size->value.ui64 = state->arcs_size; + evict_data->value.ui64 = state->arcs_lsize[ARC_BUFC_DATA]; + evict_metadata->value.ui64 = state->arcs_lsize[ARC_BUFC_METADATA]; +} + +static int +arc_kstat_update(kstat_t *ksp, int rw) +{ + arc_stats_t *as = ksp->ks_data; + + if (rw == KSTAT_WRITE) { + return (EACCES); + } else { + arc_kstat_update_state(arc_anon, + &as->arcstat_anon_size, + &as->arcstat_anon_evictable_data, + &as->arcstat_anon_evictable_metadata); + arc_kstat_update_state(arc_mru, + &as->arcstat_mru_size, + &as->arcstat_mru_evictable_data, + &as->arcstat_mru_evictable_metadata); + arc_kstat_update_state(arc_mru_ghost, + &as->arcstat_mru_ghost_size, + &as->arcstat_mru_ghost_evictable_data, + &as->arcstat_mru_ghost_evictable_metadata); + arc_kstat_update_state(arc_mfu, + &as->arcstat_mfu_size, + &as->arcstat_mfu_evictable_data, + &as->arcstat_mfu_evictable_metadata); + arc_kstat_update_state(arc_mfu_ghost, + &as->arcstat_mfu_ghost_size, + &as->arcstat_mfu_ghost_evictable_data, + &as->arcstat_mfu_ghost_evictable_metadata); + } + + return (0); +} + void arc_tempreserve_clear(uint64_t reserve) { @@ -4657,6 +4865,7 @@ arc_init(void) if (arc_ksp != NULL) { arc_ksp->ks_data = &arc_stats; + arc_ksp->ks_update = arc_kstat_update; kstat_install(arc_ksp); } Modified: stable/10/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/sys/arc.h ============================================================================== --- stable/10/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/sys/arc.h Sat Oct 3 07:27:58 2015 (r288547) +++ stable/10/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/sys/arc.h Sat Oct 3 07:28:52 2015 (r288548) @@ -115,6 +115,7 @@ typedef enum arc_buf_contents { */ typedef enum arc_space_type { ARC_SPACE_DATA, + ARC_SPACE_META, ARC_SPACE_HDRS, ARC_SPACE_L2HDRS, ARC_SPACE_OTHER, From owner-svn-src-stable@freebsd.org Sat Oct 3 07:29:58 2015 Return-Path: Delivered-To: svn-src-stable@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id EFB7FA0E521; Sat, 3 Oct 2015 07:29:58 +0000 (UTC) (envelope-from mav@FreeBSD.org) Received: from repo.freebsd.org (repo.freebsd.org [IPv6:2001:1900:2254:2068::e6a:0]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id DE4951309; Sat, 3 Oct 2015 07:29:58 +0000 (UTC) (envelope-from mav@FreeBSD.org) Received: from repo.freebsd.org ([127.0.1.70]) by repo.freebsd.org (8.15.2/8.15.2) with ESMTP id t937Tw8U063685; Sat, 3 Oct 2015 07:29:58 GMT (envelope-from mav@FreeBSD.org) Received: (from mav@localhost) by repo.freebsd.org (8.15.2/8.15.2/Submit) id t937Tu5k063670; Sat, 3 Oct 2015 07:29:56 GMT (envelope-from mav@FreeBSD.org) Message-Id: <201510030729.t937Tu5k063670@repo.freebsd.org> X-Authentication-Warning: repo.freebsd.org: mav set sender to mav@FreeBSD.org using -f From: Alexander Motin Date: Sat, 3 Oct 2015 07:29:56 +0000 (UTC) To: src-committers@freebsd.org, svn-src-all@freebsd.org, svn-src-stable@freebsd.org, svn-src-stable-10@freebsd.org Subject: svn commit: r288549 - in stable/10/sys/cddl/contrib/opensolaris/uts/common/fs/zfs: . sys X-SVN-Group: stable-10 MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit X-BeenThere: svn-src-stable@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: SVN commit messages for all the -stable branches of the src tree List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 03 Oct 2015 07:29:59 -0000 Author: mav Date: Sat Oct 3 07:29:56 2015 New Revision: 288549 URL: https://svnweb.freebsd.org/changeset/base/288549 Log: MFC r286575: 5056 ZFS deadlock on db_mtx and dn_holds Reviewed by: Will Andrews Reviewed by: Matt Ahrens Reviewed by: George Wilson Approved by: Dan McDonald Author: Justin Gibbs illumos/illumos-gate@bc9014e6a81272073b9854d9f65dd59e18d18c35 Modified: stable/10/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/dbuf.c stable/10/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/dmu_objset.c stable/10/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/dmu_send.c stable/10/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/dmu_traverse.c stable/10/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/dnode.c stable/10/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/dnode_sync.c stable/10/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/dsl_bookmark.c stable/10/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/dsl_dataset.c stable/10/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/dsl_deadlist.c stable/10/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/dsl_deleg.c stable/10/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/dsl_destroy.c stable/10/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/dsl_dir.c stable/10/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/dsl_pool.c stable/10/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/dsl_prop.c stable/10/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/dsl_scan.c stable/10/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/dsl_userhold.c stable/10/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/sa.c stable/10/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/spa.c stable/10/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/spa_misc.c stable/10/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/sys/dbuf.h stable/10/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/sys/dmu.h stable/10/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/sys/dmu_objset.h stable/10/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/sys/dnode.h stable/10/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/sys/dsl_dataset.h stable/10/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/sys/dsl_dir.h stable/10/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/sys/sa.h stable/10/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/sys/sa_impl.h stable/10/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/sys/spa.h stable/10/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/sys/spa_impl.h stable/10/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/sys/zap_impl.h stable/10/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/sys/zap_leaf.h stable/10/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zap.c stable/10/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zap_micro.c stable/10/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zfs_ioctl.c stable/10/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zfs_sa.c stable/10/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zil.c Directory Properties: stable/10/ (props changed) Modified: stable/10/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/dbuf.c ============================================================================== --- stable/10/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/dbuf.c Sat Oct 3 07:28:52 2015 (r288548) +++ stable/10/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/dbuf.c Sat Oct 3 07:29:56 2015 (r288549) @@ -24,6 +24,7 @@ * Copyright (c) 2012, 2015 by Delphix. All rights reserved. * Copyright (c) 2013 by Saso Kiselkov. All rights reserved. * Copyright (c) 2013, Joyent, Inc. All rights reserved. + * Copyright (c) 2014 Spectra Logic Corporation, All rights reserved. */ #include @@ -54,10 +55,16 @@ static void dbuf_destroy(dmu_buf_impl_t static boolean_t dbuf_undirty(dmu_buf_impl_t *db, dmu_tx_t *tx); static void dbuf_write(dbuf_dirty_record_t *dr, arc_buf_t *data, dmu_tx_t *tx); +#ifndef __lint +extern inline void dmu_buf_init_user(dmu_buf_user_t *dbu, + dmu_buf_evict_func_t *evict_func, dmu_buf_t **clear_on_evict_dbufp); +#endif /* ! __lint */ + /* * Global data structures and functions for the dbuf cache. */ static kmem_cache_t *dbuf_cache; +static taskq_t *dbu_evict_taskq; /* ARGSUSED */ static int @@ -231,17 +238,72 @@ dbuf_hash_remove(dmu_buf_impl_t *db) static arc_evict_func_t dbuf_do_evict; +typedef enum { + DBVU_EVICTING, + DBVU_NOT_EVICTING +} dbvu_verify_type_t; + +static void +dbuf_verify_user(dmu_buf_impl_t *db, dbvu_verify_type_t verify_type) +{ +#ifdef ZFS_DEBUG + int64_t holds; + + if (db->db_user == NULL) + return; + + /* Only data blocks support the attachment of user data. */ + ASSERT(db->db_level == 0); + + /* Clients must resolve a dbuf before attaching user data. */ + ASSERT(db->db.db_data != NULL); + ASSERT3U(db->db_state, ==, DB_CACHED); + + holds = refcount_count(&db->db_holds); + if (verify_type == DBVU_EVICTING) { + /* + * Immediate eviction occurs when holds == dirtycnt. + * For normal eviction buffers, holds is zero on + * eviction, except when dbuf_fix_old_data() calls + * dbuf_clear_data(). However, the hold count can grow + * during eviction even though db_mtx is held (see + * dmu_bonus_hold() for an example), so we can only + * test the generic invariant that holds >= dirtycnt. + */ + ASSERT3U(holds, >=, db->db_dirtycnt); + } else { + if (db->db_immediate_evict == TRUE) + ASSERT3U(holds, >=, db->db_dirtycnt); + else + ASSERT3U(holds, >, 0); + } +#endif +} + static void dbuf_evict_user(dmu_buf_impl_t *db) { + dmu_buf_user_t *dbu = db->db_user; + ASSERT(MUTEX_HELD(&db->db_mtx)); - if (db->db_level != 0 || db->db_evict_func == NULL) + if (dbu == NULL) return; - db->db_evict_func(&db->db, db->db_user_ptr); - db->db_user_ptr = NULL; - db->db_evict_func = NULL; + dbuf_verify_user(db, DBVU_EVICTING); + db->db_user = NULL; + +#ifdef ZFS_DEBUG + if (dbu->dbu_clear_on_evict_dbufp != NULL) + *dbu->dbu_clear_on_evict_dbufp = NULL; +#endif + + /* + * Invoke the callback from a taskq to avoid lock order reversals + * and limit stack depth. + */ + taskq_dispatch_ent(dbu_evict_taskq, dbu->dbu_evict_func, dbu, 0, + &dbu->dbu_tqent); } boolean_t @@ -302,6 +364,12 @@ retry: for (i = 0; i < DBUF_MUTEXES; i++) mutex_init(&h->hash_mutexes[i], NULL, MUTEX_DEFAULT, NULL); + + /* + * All entries are queued via taskq_dispatch_ent(), so min/maxalloc + * configuration is not required. + */ + dbu_evict_taskq = taskq_create("dbu_evict", 1, minclsyspri, 0, 0, 0); } void @@ -314,6 +382,7 @@ dbuf_fini(void) mutex_destroy(&h->hash_mutexes[i]); kmem_free(h->hash_table, (h->hash_table_mask + 1) * sizeof (void *)); kmem_cache_destroy(dbuf_cache); + taskq_destroy(dbu_evict_taskq); } /* @@ -431,21 +500,27 @@ dbuf_verify(dmu_buf_impl_t *db) #endif static void +dbuf_clear_data(dmu_buf_impl_t *db) +{ + ASSERT(MUTEX_HELD(&db->db_mtx)); + dbuf_evict_user(db); + db->db_buf = NULL; + db->db.db_data = NULL; + if (db->db_state != DB_NOFILL) + db->db_state = DB_UNCACHED; +} + +static void dbuf_set_data(dmu_buf_impl_t *db, arc_buf_t *buf) { ASSERT(MUTEX_HELD(&db->db_mtx)); + ASSERT(buf != NULL); + db->db_buf = buf; - if (buf != NULL) { - ASSERT(buf->b_data != NULL); - db->db.db_data = buf->b_data; - if (!arc_released(buf)) - arc_set_callback(buf, dbuf_do_evict, db); - } else { - dbuf_evict_user(db); - db->db.db_data = NULL; - if (db->db_state != DB_NOFILL) - db->db_state = DB_UNCACHED; - } + ASSERT(buf->b_data != NULL); + db->db.db_data = buf->b_data; + if (!arc_released(buf)) + arc_set_callback(buf, dbuf_do_evict, db); } /* @@ -467,7 +542,7 @@ dbuf_loan_arcbuf(dmu_buf_impl_t *db) } else { abuf = db->db_buf; arc_loan_inuse_buf(abuf, db); - dbuf_set_data(db, NULL); + dbuf_clear_data(db); mutex_exit(&db->db_mtx); } return (abuf); @@ -703,7 +778,7 @@ dbuf_noread(dmu_buf_impl_t *db) dbuf_set_data(db, arc_buf_alloc(spa, db->db.db_size, db, type)); db->db_state = DB_FILL; } else if (db->db_state == DB_NOFILL) { - dbuf_set_data(db, NULL); + dbuf_clear_data(db); } else { ASSERT3U(db->db_state, ==, DB_CACHED); } @@ -759,7 +834,7 @@ dbuf_fix_old_data(dmu_buf_impl_t *db, ui dr->dt.dl.dr_data = arc_buf_alloc(spa, size, db, type); bcopy(db->db.db_data, dr->dt.dl.dr_data->b_data, size); } else { - dbuf_set_data(db, NULL); + dbuf_clear_data(db); } } @@ -810,7 +885,8 @@ void dbuf_free_range(dnode_t *dn, uint64_t start_blkid, uint64_t end_blkid, dmu_tx_t *tx) { - dmu_buf_impl_t *db, *db_next, db_search; + dmu_buf_impl_t db_search; + dmu_buf_impl_t *db, *db_next; uint64_t txg = tx->tx_txg; avl_index_t where; @@ -1388,7 +1464,7 @@ dbuf_undirty(dmu_buf_impl_t *db, dmu_tx_ arc_buf_t *buf = db->db_buf; ASSERT(db->db_state == DB_NOFILL || arc_released(buf)); - dbuf_set_data(db, NULL); + dbuf_clear_data(db); VERIFY(arc_buf_remove_ref(buf, db)); dbuf_evict(db); return (B_TRUE); @@ -1728,8 +1804,7 @@ dbuf_create(dnode_t *dn, uint8_t level, db->db_parent = parent; db->db_blkptr = blkptr; - db->db_user_ptr = NULL; - db->db_evict_func = NULL; + db->db_user = NULL; db->db_immediate_evict = 0; db->db_freed_in_flight = 0; @@ -2195,7 +2270,7 @@ dbuf_rele_and_unlock(dmu_buf_impl_t *db, /* * This dbuf has anonymous data associated with it. */ - dbuf_set_data(db, NULL); + dbuf_clear_data(db); VERIFY(arc_buf_remove_ref(buf, db)); dbuf_evict(db); } else { @@ -2228,7 +2303,8 @@ dbuf_rele_and_unlock(dmu_buf_impl_t *db, } else { dbuf_clear(db); } - } else if (arc_buf_eviction_needed(db->db_buf)) { + } else if (db->db_objset->os_evicting || + arc_buf_eviction_needed(db->db_buf)) { dbuf_clear(db); } else { mutex_exit(&db->db_mtx); @@ -2247,51 +2323,57 @@ dbuf_refcount(dmu_buf_impl_t *db) } void * -dmu_buf_set_user(dmu_buf_t *db_fake, void *user_ptr, - dmu_buf_evict_func_t *evict_func) +dmu_buf_replace_user(dmu_buf_t *db_fake, dmu_buf_user_t *old_user, + dmu_buf_user_t *new_user) { - return (dmu_buf_update_user(db_fake, NULL, user_ptr, evict_func)); + dmu_buf_impl_t *db = (dmu_buf_impl_t *)db_fake; + + mutex_enter(&db->db_mtx); + dbuf_verify_user(db, DBVU_NOT_EVICTING); + if (db->db_user == old_user) + db->db_user = new_user; + else + old_user = db->db_user; + dbuf_verify_user(db, DBVU_NOT_EVICTING); + mutex_exit(&db->db_mtx); + + return (old_user); } void * -dmu_buf_set_user_ie(dmu_buf_t *db_fake, void *user_ptr, - dmu_buf_evict_func_t *evict_func) +dmu_buf_set_user(dmu_buf_t *db_fake, dmu_buf_user_t *user) { - dmu_buf_impl_t *db = (dmu_buf_impl_t *)db_fake; - - db->db_immediate_evict = TRUE; - return (dmu_buf_update_user(db_fake, NULL, user_ptr, evict_func)); + return (dmu_buf_replace_user(db_fake, NULL, user)); } void * -dmu_buf_update_user(dmu_buf_t *db_fake, void *old_user_ptr, void *user_ptr, - dmu_buf_evict_func_t *evict_func) +dmu_buf_set_user_ie(dmu_buf_t *db_fake, dmu_buf_user_t *user) { dmu_buf_impl_t *db = (dmu_buf_impl_t *)db_fake; - ASSERT(db->db_level == 0); - - ASSERT((user_ptr == NULL) == (evict_func == NULL)); - - mutex_enter(&db->db_mtx); - if (db->db_user_ptr == old_user_ptr) { - db->db_user_ptr = user_ptr; - db->db_evict_func = evict_func; - } else { - old_user_ptr = db->db_user_ptr; - } + db->db_immediate_evict = TRUE; + return (dmu_buf_set_user(db_fake, user)); +} - mutex_exit(&db->db_mtx); - return (old_user_ptr); +void * +dmu_buf_remove_user(dmu_buf_t *db_fake, dmu_buf_user_t *user) +{ + return (dmu_buf_replace_user(db_fake, user, NULL)); } void * dmu_buf_get_user(dmu_buf_t *db_fake) { dmu_buf_impl_t *db = (dmu_buf_impl_t *)db_fake; - ASSERT(!refcount_is_zero(&db->db_holds)); - return (db->db_user_ptr); + dbuf_verify_user(db, DBVU_NOT_EVICTING); + return (db->db_user); +} + +void +dmu_buf_user_evict_wait() +{ + taskq_wait(dbu_evict_taskq); } boolean_t Modified: stable/10/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/dmu_objset.c ============================================================================== --- stable/10/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/dmu_objset.c Sat Oct 3 07:28:52 2015 (r288548) +++ stable/10/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/dmu_objset.c Sat Oct 3 07:29:56 2015 (r288549) @@ -23,6 +23,7 @@ * Copyright (c) 2012, 2014 by Delphix. All rights reserved. * Copyright (c) 2013 by Saso Kiselkov. All rights reserved. * Copyright (c) 2013, Joyent, Inc. All rights reserved. + * Copyright (c) 2014 Spectra Logic Corporation, All rights reserved. * Copyright 2015 Nexenta Systems, Inc. All rights reserved. */ @@ -357,7 +358,7 @@ dmu_objset_open_impl(spa_t *spa, dsl_dat zfs_prop_to_name(ZFS_PROP_SECONDARYCACHE), secondary_cache_changed_cb, os); } - if (!dsl_dataset_is_snapshot(ds)) { + if (!ds->ds_is_snapshot) { if (err == 0) { err = dsl_prop_register(ds, zfs_prop_to_name(ZFS_PROP_CHECKSUM), @@ -419,7 +420,7 @@ dmu_objset_open_impl(spa_t *spa, dsl_dat os->os_secondary_cache = ZFS_CACHE_ALL; } - if (ds == NULL || !dsl_dataset_is_snapshot(ds)) + if (ds == NULL || !ds->ds_is_snapshot) os->os_zil_header = os->os_phys->os_zil_header; os->os_zil = zil_alloc(os, &os->os_zil_header); @@ -438,16 +439,13 @@ dmu_objset_open_impl(spa_t *spa, dsl_dat mutex_init(&os->os_obj_lock, NULL, MUTEX_DEFAULT, NULL); mutex_init(&os->os_user_ptr_lock, NULL, MUTEX_DEFAULT, NULL); - DMU_META_DNODE(os) = dnode_special_open(os, - &os->os_phys->os_meta_dnode, DMU_META_DNODE_OBJECT, - &os->os_meta_dnode); + dnode_special_open(os, &os->os_phys->os_meta_dnode, + DMU_META_DNODE_OBJECT, &os->os_meta_dnode); if (arc_buf_size(os->os_phys_buf) >= sizeof (objset_phys_t)) { - DMU_USERUSED_DNODE(os) = dnode_special_open(os, - &os->os_phys->os_userused_dnode, DMU_USERUSED_OBJECT, - &os->os_userused_dnode); - DMU_GROUPUSED_DNODE(os) = dnode_special_open(os, - &os->os_phys->os_groupused_dnode, DMU_GROUPUSED_OBJECT, - &os->os_groupused_dnode); + dnode_special_open(os, &os->os_phys->os_userused_dnode, + DMU_USERUSED_OBJECT, &os->os_userused_dnode); + dnode_special_open(os, &os->os_phys->os_groupused_dnode, + DMU_GROUPUSED_OBJECT, &os->os_groupused_dnode); } *osp = os; @@ -535,7 +533,7 @@ dmu_objset_own(const char *name, dmu_obj } else if (type != DMU_OST_ANY && type != (*osp)->os_phys->os_type) { dsl_dataset_disown(ds, tag); return (SET_ERROR(EINVAL)); - } else if (!readonly && dsl_dataset_is_snapshot(ds)) { + } else if (!readonly && ds->ds_is_snapshot) { dsl_dataset_disown(ds, tag); return (SET_ERROR(EROFS)); } @@ -591,41 +589,53 @@ dmu_objset_disown(objset_t *os, void *ta void dmu_objset_evict_dbufs(objset_t *os) { + dnode_t dn_marker; dnode_t *dn; mutex_enter(&os->os_lock); - - /* process the mdn last, since the other dnodes have holds on it */ - list_remove(&os->os_dnodes, DMU_META_DNODE(os)); - list_insert_tail(&os->os_dnodes, DMU_META_DNODE(os)); - - /* - * Find the first dnode with holds. We have to do this dance - * because dnode_add_ref() only works if you already have a - * hold. If there are no holds then it has no dbufs so OK to - * skip. - */ - for (dn = list_head(&os->os_dnodes); - dn && !dnode_add_ref(dn, FTAG); - dn = list_next(&os->os_dnodes, dn)) - continue; - - while (dn) { - dnode_t *next_dn = dn; - - do { - next_dn = list_next(&os->os_dnodes, next_dn); - } while (next_dn && !dnode_add_ref(next_dn, FTAG)); - - mutex_exit(&os->os_lock); - dnode_evict_dbufs(dn); - dnode_rele(dn, FTAG); - mutex_enter(&os->os_lock); - dn = next_dn; + dn = list_head(&os->os_dnodes); + while (dn != NULL) { + /* + * Skip dnodes without holds. We have to do this dance + * because dnode_add_ref() only works if there is already a + * hold. If the dnode has no holds, then it has no dbufs. + */ + if (dnode_add_ref(dn, FTAG)) { + list_insert_after(&os->os_dnodes, dn, &dn_marker); + mutex_exit(&os->os_lock); + + dnode_evict_dbufs(dn); + dnode_rele(dn, FTAG); + + mutex_enter(&os->os_lock); + dn = list_next(&os->os_dnodes, &dn_marker); + list_remove(&os->os_dnodes, &dn_marker); + } else { + dn = list_next(&os->os_dnodes, dn); + } } mutex_exit(&os->os_lock); + + if (DMU_USERUSED_DNODE(os) != NULL) { + dnode_evict_dbufs(DMU_GROUPUSED_DNODE(os)); + dnode_evict_dbufs(DMU_USERUSED_DNODE(os)); + } + dnode_evict_dbufs(DMU_META_DNODE(os)); } +/* + * Objset eviction processing is split into into two pieces. + * The first marks the objset as evicting, evicts any dbufs that + * have a refcount of zero, and then queues up the objset for the + * second phase of eviction. Once os->os_dnodes has been cleared by + * dnode_buf_pageout()->dnode_destroy(), the second phase is executed. + * The second phase closes the special dnodes, dequeues the objset from + * the list of those undergoing eviction, and finally frees the objset. + * + * NOTE: Due to asynchronous eviction processing (invocation of + * dnode_buf_pageout()), it is possible for the meta dnode for the + * objset to have no holds even though os->os_dnodes is not empty. + */ void dmu_objset_evict(objset_t *os) { @@ -635,7 +645,7 @@ dmu_objset_evict(objset_t *os) ASSERT(!dmu_objset_is_dirty(os, t)); if (ds) { - if (!dsl_dataset_is_snapshot(ds)) { + if (!ds->ds_is_snapshot) { VERIFY0(dsl_prop_unregister(ds, zfs_prop_to_name(ZFS_PROP_CHECKSUM), checksum_changed_cb, os)); @@ -672,8 +682,24 @@ dmu_objset_evict(objset_t *os) if (os->os_sa) sa_tear_down(os); + os->os_evicting = B_TRUE; dmu_objset_evict_dbufs(os); + mutex_enter(&os->os_lock); + spa_evicting_os_register(os->os_spa, os); + if (list_is_empty(&os->os_dnodes)) { + mutex_exit(&os->os_lock); + dmu_objset_evict_done(os); + } else { + mutex_exit(&os->os_lock); + } +} + +void +dmu_objset_evict_done(objset_t *os) +{ + ASSERT3P(list_head(&os->os_dnodes), ==, NULL); + dnode_special_close(&os->os_meta_dnode); if (DMU_USERUSED_DNODE(os)) { dnode_special_close(&os->os_userused_dnode); @@ -681,8 +707,6 @@ dmu_objset_evict(objset_t *os) } zil_free(os->os_zil); - ASSERT3P(list_head(&os->os_dnodes), ==, NULL); - VERIFY(arc_buf_remove_ref(os->os_phys_buf, &os->os_phys_buf)); /* @@ -697,6 +721,7 @@ dmu_objset_evict(objset_t *os) mutex_destroy(&os->os_lock); mutex_destroy(&os->os_obj_lock); mutex_destroy(&os->os_user_ptr_lock); + spa_evicting_os_deregister(os->os_spa, os); kmem_free(os, sizeof (objset_t)); } @@ -895,7 +920,7 @@ dmu_objset_clone_check(void *arg, dmu_tx return (error); /* You can only clone snapshots, not the head datasets. */ - if (!dsl_dataset_is_snapshot(origin)) { + if (!origin->ds_is_snapshot) { dsl_dataset_rele(origin, FTAG); return (SET_ERROR(EINVAL)); } @@ -1459,7 +1484,7 @@ int dmu_objset_is_snapshot(objset_t *os) { if (os->os_dsl_dataset != NULL) - return (dsl_dataset_is_snapshot(os->os_dsl_dataset)); + return (os->os_dsl_dataset->ds_is_snapshot); else return (B_FALSE); } Modified: stable/10/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/dmu_send.c ============================================================================== --- stable/10/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/dmu_send.c Sat Oct 3 07:28:52 2015 (r288548) +++ stable/10/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/dmu_send.c Sat Oct 3 07:29:56 2015 (r288549) @@ -636,7 +636,7 @@ dmu_send_impl(void *tag, dsl_pool_t *dp, fromtxg = fromzb->zbm_creation_txg; } dsl_dataset_name(ds, drr->drr_u.drr_begin.drr_toname); - if (!dsl_dataset_is_snapshot(ds)) { + if (!ds->ds_is_snapshot) { (void) strlcat(drr->drr_u.drr_begin.drr_toname, "@--head--", sizeof (drr->drr_u.drr_begin.drr_toname)); } @@ -852,11 +852,11 @@ dmu_send_estimate(dsl_dataset_t *ds, dsl ASSERT(dsl_pool_config_held(dp)); /* tosnap must be a snapshot */ - if (!dsl_dataset_is_snapshot(ds)) + if (!ds->ds_is_snapshot) return (SET_ERROR(EINVAL)); /* fromsnap, if provided, must be a snapshot */ - if (fromds != NULL && !dsl_dataset_is_snapshot(fromds)) + if (fromds != NULL && !fromds->ds_is_snapshot) return (SET_ERROR(EINVAL)); /* @@ -1105,7 +1105,7 @@ dmu_recv_begin_check(void *arg, dmu_tx_t dsl_dataset_rele(ds, FTAG); return (error); } - if (!dsl_dataset_is_snapshot(origin)) { + if (!origin->ds_is_snapshot) { dsl_dataset_rele(origin, FTAG); dsl_dataset_rele(ds, FTAG); return (SET_ERROR(EINVAL)); Modified: stable/10/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/dmu_traverse.c ============================================================================== --- stable/10/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/dmu_traverse.c Sat Oct 3 07:28:52 2015 (r288548) +++ stable/10/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/dmu_traverse.c Sat Oct 3 07:29:56 2015 (r288549) @@ -534,7 +534,7 @@ traverse_impl(spa_t *spa, dsl_dataset_t cv_init(&pd.pd_cv, NULL, CV_DEFAULT, NULL); /* See comment on ZIL traversal in dsl_scan_visitds. */ - if (ds != NULL && !dsl_dataset_is_snapshot(ds) && !BP_IS_HOLE(rootbp)) { + if (ds != NULL && !ds->ds_is_snapshot && !BP_IS_HOLE(rootbp)) { arc_flags_t flags = ARC_FLAG_WAIT; objset_phys_t *osp; arc_buf_t *buf; Modified: stable/10/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/dnode.c ============================================================================== --- stable/10/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/dnode.c Sat Oct 3 07:28:52 2015 (r288548) +++ stable/10/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/dnode.c Sat Oct 3 07:29:56 2015 (r288549) @@ -21,6 +21,7 @@ /* * Copyright (c) 2005, 2010, Oracle and/or its affiliates. All rights reserved. * Copyright (c) 2012, 2015 by Delphix. All rights reserved. + * Copyright (c) 2014 Spectra Logic Corporation, All rights reserved. */ #include @@ -405,8 +406,9 @@ static dnode_t * dnode_create(objset_t *os, dnode_phys_t *dnp, dmu_buf_impl_t *db, uint64_t object, dnode_handle_t *dnh) { - dnode_t *dn = kmem_cache_alloc(dnode_cache, KM_SLEEP); + dnode_t *dn; + dn = kmem_cache_alloc(dnode_cache, KM_SLEEP); ASSERT(!POINTER_IS_VALID(dn->dn_objset)); dn->dn_moved = 0; @@ -443,13 +445,31 @@ dnode_create(objset_t *os, dnode_phys_t ASSERT(DMU_OT_IS_VALID(dn->dn_phys->dn_type)); mutex_enter(&os->os_lock); - list_insert_head(&os->os_dnodes, dn); + if (dnh->dnh_dnode != NULL) { + /* Lost the allocation race. */ + mutex_exit(&os->os_lock); + kmem_cache_free(dnode_cache, dn); + return (dnh->dnh_dnode); + } + + /* + * Exclude special dnodes from os_dnodes so an empty os_dnodes + * signifies that the special dnodes have no references from + * their children (the entries in os_dnodes). This allows + * dnode_destroy() to easily determine if the last child has + * been removed and then complete eviction of the objset. + */ + if (!DMU_OBJECT_IS_SPECIAL(object)) + list_insert_head(&os->os_dnodes, dn); membar_producer(); + /* - * Everything else must be valid before assigning dn_objset makes the - * dnode eligible for dnode_move(). + * Everything else must be valid before assigning dn_objset + * makes the dnode eligible for dnode_move(). */ dn->dn_objset = os; + + dnh->dnh_dnode = dn; mutex_exit(&os->os_lock); arc_space_consume(sizeof (dnode_t), ARC_SPACE_OTHER); @@ -463,12 +483,18 @@ static void dnode_destroy(dnode_t *dn) { objset_t *os = dn->dn_objset; + boolean_t complete_os_eviction = B_FALSE; ASSERT((dn->dn_id_flags & DN_ID_NEW_EXIST) == 0); mutex_enter(&os->os_lock); POINTER_INVALIDATE(&dn->dn_objset); - list_remove(&os->os_dnodes, dn); + if (!DMU_OBJECT_IS_SPECIAL(dn->dn_object)) { + list_remove(&os->os_dnodes, dn); + complete_os_eviction = + list_is_empty(&os->os_dnodes) && + list_link_active(&os->os_evicting_node); + } mutex_exit(&os->os_lock); /* the dnode can no longer move, so we can release the handle */ @@ -503,6 +529,9 @@ dnode_destroy(dnode_t *dn) dmu_zfetch_rele(&dn->dn_zfetch); kmem_cache_free(dnode_cache, dn); arc_space_return(sizeof (dnode_t), ARC_SPACE_OTHER); + + if (complete_os_eviction) + dmu_objset_evict_done(os); } void @@ -971,33 +1000,32 @@ dnode_special_close(dnode_handle_t *dnh) */ while (refcount_count(&dn->dn_holds) > 0) delay(1); + ASSERT(dn->dn_dbuf == NULL || + dmu_buf_get_user(&dn->dn_dbuf->db) == NULL); zrl_add(&dnh->dnh_zrlock); dnode_destroy(dn); /* implicit zrl_remove() */ zrl_destroy(&dnh->dnh_zrlock); dnh->dnh_dnode = NULL; } -dnode_t * +void dnode_special_open(objset_t *os, dnode_phys_t *dnp, uint64_t object, dnode_handle_t *dnh) { - dnode_t *dn = dnode_create(os, dnp, NULL, object, dnh); - dnh->dnh_dnode = dn; + dnode_t *dn; + + dn = dnode_create(os, dnp, NULL, object, dnh); zrl_init(&dnh->dnh_zrlock); DNODE_VERIFY(dn); - return (dn); } static void -dnode_buf_pageout(dmu_buf_t *db, void *arg) +dnode_buf_pageout(void *dbu) { - dnode_children_t *children_dnodes = arg; + dnode_children_t *children_dnodes = dbu; int i; - int epb = db->db_size >> DNODE_SHIFT; - ASSERT(epb == children_dnodes->dnc_count); - - for (i = 0; i < epb; i++) { + for (i = 0; i < children_dnodes->dnc_count; i++) { dnode_handle_t *dnh = &children_dnodes->dnc_children[i]; dnode_t *dn; @@ -1027,7 +1055,7 @@ dnode_buf_pageout(dmu_buf_t *db, void *a dnh->dnh_dnode = NULL; } kmem_free(children_dnodes, sizeof (dnode_children_t) + - epb * sizeof (dnode_handle_t)); + children_dnodes->dnc_count * sizeof (dnode_handle_t)); } /* @@ -1117,10 +1145,11 @@ dnode_hold_impl(objset_t *os, uint64_t o dnh = &children_dnodes->dnc_children[0]; for (i = 0; i < epb; i++) { zrl_init(&dnh[i].dnh_zrlock); - dnh[i].dnh_dnode = NULL; } - if (winner = dmu_buf_set_user(&db->db, children_dnodes, - dnode_buf_pageout)) { + dmu_buf_init_user(&children_dnodes->dnc_dbu, + dnode_buf_pageout, NULL); + winner = dmu_buf_set_user(&db->db, &children_dnodes->dnc_dbu); + if (winner != NULL) { for (i = 0; i < epb; i++) { zrl_destroy(&dnh[i].dnh_zrlock); @@ -1135,17 +1164,11 @@ dnode_hold_impl(objset_t *os, uint64_t o dnh = &children_dnodes->dnc_children[idx]; zrl_add(&dnh->dnh_zrlock); - if ((dn = dnh->dnh_dnode) == NULL) { + dn = dnh->dnh_dnode; + if (dn == NULL) { dnode_phys_t *phys = (dnode_phys_t *)db->db.db_data+idx; - dnode_t *winner; dn = dnode_create(os, phys, db, object, dnh); - winner = atomic_cas_ptr(&dnh->dnh_dnode, NULL, dn); - if (winner != NULL) { - zrl_add(&dnh->dnh_zrlock); - dnode_destroy(dn); /* implicit zrl_remove() */ - dn = winner; - } } mutex_enter(&dn->dn_mtx); @@ -1159,10 +1182,10 @@ dnode_hold_impl(objset_t *os, uint64_t o dbuf_rele(db, FTAG); return (type == DMU_OT_NONE ? ENOENT : EEXIST); } - mutex_exit(&dn->dn_mtx); - if (refcount_add(&dn->dn_holds, tag) == 1) dbuf_add_ref(db, dnh); + mutex_exit(&dn->dn_mtx); + /* Now we can rely on the hold to prevent the dnode from moving. */ zrl_remove(&dnh->dnh_zrlock); Modified: stable/10/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/dnode_sync.c ============================================================================== --- stable/10/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/dnode_sync.c Sat Oct 3 07:28:52 2015 (r288548) +++ stable/10/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/dnode_sync.c Sat Oct 3 07:29:56 2015 (r288549) @@ -22,6 +22,7 @@ /* * Copyright (c) 2005, 2010, Oracle and/or its affiliates. All rights reserved. * Copyright (c) 2012, 2015 by Delphix. All rights reserved. + * Copyright (c) 2014 Spectra Logic Corporation, All rights reserved. */ #include @@ -397,49 +398,37 @@ dnode_sync_free_range(void *arg, uint64_ void dnode_evict_dbufs(dnode_t *dn) { - int progress; - int pass = 0; + dmu_buf_impl_t db_marker; + dmu_buf_impl_t *db, *db_next; + + mutex_enter(&dn->dn_dbufs_mtx); + for (db = avl_first(&dn->dn_dbufs); db != NULL; db = db_next) { - do { - dmu_buf_impl_t *db, *db_next; - int evicting = FALSE; - - progress = FALSE; - mutex_enter(&dn->dn_dbufs_mtx); - for (db = avl_first(&dn->dn_dbufs); db != NULL; db = db_next) { - db_next = AVL_NEXT(&dn->dn_dbufs, db); #ifdef DEBUG - DB_DNODE_ENTER(db); - ASSERT3P(DB_DNODE(db), ==, dn); - DB_DNODE_EXIT(db); + DB_DNODE_ENTER(db); + ASSERT3P(DB_DNODE(db), ==, dn); + DB_DNODE_EXIT(db); #endif /* DEBUG */ - mutex_enter(&db->db_mtx); - if (db->db_state == DB_EVICTING) { - progress = TRUE; - evicting = TRUE; - mutex_exit(&db->db_mtx); - } else if (refcount_is_zero(&db->db_holds)) { - progress = TRUE; - dbuf_clear(db); /* exits db_mtx for us */ - } else { - mutex_exit(&db->db_mtx); - } + mutex_enter(&db->db_mtx); + if (db->db_state != DB_EVICTING && + refcount_is_zero(&db->db_holds)) { + db_marker.db_level = db->db_level; + db_marker.db_blkid = db->db_blkid; + db_marker.db_state = DB_SEARCH; + avl_insert_here(&dn->dn_dbufs, &db_marker, db, + AVL_BEFORE); + + dbuf_clear(db); + db_next = AVL_NEXT(&dn->dn_dbufs, &db_marker); + avl_remove(&dn->dn_dbufs, &db_marker); + } else { + mutex_exit(&db->db_mtx); + db_next = AVL_NEXT(&dn->dn_dbufs, db); } - /* - * NB: we need to drop dn_dbufs_mtx between passes so - * that any DB_EVICTING dbufs can make progress. - * Ideally, we would have some cv we could wait on, but - * since we don't, just wait a bit to give the other - * thread a chance to run. - */ - mutex_exit(&dn->dn_dbufs_mtx); - if (evicting) - delay(1); - pass++; - ASSERT(pass < 100); /* sanity check */ - } while (progress); + } + mutex_exit(&dn->dn_dbufs_mtx); dnode_evict_bonus(dn); } @@ -504,7 +493,6 @@ dnode_sync_free(dnode_t *dn, dmu_tx_t *t dnode_undirty_dbufs(&dn->dn_dirty_records[txgoff]); dnode_evict_dbufs(dn); ASSERT(avl_is_empty(&dn->dn_dbufs)); - ASSERT3P(dn->dn_bonus, ==, NULL); /* * XXX - It would be nice to assert this, but we may still Modified: stable/10/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/dsl_bookmark.c ============================================================================== --- stable/10/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/dsl_bookmark.c Sat Oct 3 07:28:52 2015 (r288548) +++ stable/10/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/dsl_bookmark.c Sat Oct 3 07:29:56 2015 (r288549) @@ -120,7 +120,7 @@ dsl_bookmark_create_check_impl(dsl_datas int error; zfs_bookmark_phys_t bmark_phys; - if (!dsl_dataset_is_snapshot(snapds)) + if (!snapds->ds_is_snapshot) return (SET_ERROR(EINVAL)); error = dsl_bookmark_hold_ds(dp, bookmark_name, Modified: stable/10/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/dsl_dataset.c ============================================================================== --- stable/10/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/dsl_dataset.c Sat Oct 3 07:28:52 2015 (r288548) +++ stable/10/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/dsl_dataset.c Sat Oct 3 07:29:56 2015 (r288549) @@ -24,6 +24,7 @@ * Copyright (c) 2011, 2014 by Delphix. All rights reserved. * Copyright (c) 2014, Joyent, Inc. All rights reserved. * Copyright (c) 2014 RackTop Systems. + * Copyright (c) 2014 Spectra Logic Corporation, All rights reserved. */ #include @@ -77,7 +78,6 @@ SYSCTL_INT(_vfs_zfs, OID_AUTO, max_recor #define DS_REF_MAX (1ULL << 62) extern inline dsl_dataset_phys_t *dsl_dataset_phys(dsl_dataset_t *ds); -extern inline boolean_t dsl_dataset_is_snapshot(dsl_dataset_t *ds); /* * Figure out how much of this delta should be propogated to the dsl_dir @@ -161,7 +161,7 @@ dsl_dataset_block_kill(dsl_dataset_t *ds } ASSERT3P(tx->tx_pool, ==, ds->ds_dir->dd_pool); - ASSERT(!dsl_dataset_is_snapshot(ds)); + ASSERT(!ds->ds_is_snapshot); dmu_buf_will_dirty(ds->ds_dbuf, tx); if (bp->blk_birth > dsl_dataset_phys(ds)->ds_prev_snap_txg) { @@ -259,14 +259,15 @@ dsl_dataset_block_freeable(dsl_dataset_t return (B_TRUE); } -/* ARGSUSED */ static void -dsl_dataset_evict(dmu_buf_t *db, void *dsv) +dsl_dataset_evict(void *dbu) { - dsl_dataset_t *ds = dsv; + dsl_dataset_t *ds = dbu; ASSERT(ds->ds_owner == NULL); + ds->ds_dbuf = NULL; + unique_remove(ds->ds_fsid_guid); if (ds->ds_objset != NULL) @@ -278,10 +279,10 @@ dsl_dataset_evict(dmu_buf_t *db, void *d } bplist_destroy(&ds->ds_pending_deadlist); - if (dsl_dataset_phys(ds)->ds_deadlist_obj != 0) + if (ds->ds_deadlist.dl_os != NULL) dsl_deadlist_close(&ds->ds_deadlist); if (ds->ds_dir) - dsl_dir_rele(ds->ds_dir, ds); + dsl_dir_async_rele(ds->ds_dir, ds); ASSERT(!list_link_active(&ds->ds_synced_link)); @@ -417,6 +418,7 @@ dsl_dataset_hold_obj(dsl_pool_t *dp, uin ds = kmem_zalloc(sizeof (dsl_dataset_t), KM_SLEEP); ds->ds_dbuf = dbuf; ds->ds_object = dsobj; + ds->ds_is_snapshot = dsl_dataset_phys(ds)->ds_num_children != 0; mutex_init(&ds->ds_lock, NULL, MUTEX_DEFAULT, NULL); mutex_init(&ds->ds_opening_lock, NULL, MUTEX_DEFAULT, NULL); @@ -456,7 +458,7 @@ dsl_dataset_hold_obj(dsl_pool_t *dp, uin return (err); } - if (!dsl_dataset_is_snapshot(ds)) { + if (!ds->ds_is_snapshot) { ds->ds_snapname[0] = '\0'; if (dsl_dataset_phys(ds)->ds_prev_snap_obj != 0) { err = dsl_dataset_hold_obj(dp, @@ -483,7 +485,7 @@ dsl_dataset_hold_obj(dsl_pool_t *dp, uin } } - if (err == 0 && !dsl_dataset_is_snapshot(ds)) { + if (err == 0 && !ds->ds_is_snapshot) { err = dsl_prop_get_int_ds(ds, zfs_prop_to_name(ZFS_PROP_REFRESERVATION), &ds->ds_reserved); @@ -496,8 +498,11 @@ dsl_dataset_hold_obj(dsl_pool_t *dp, uin ds->ds_reserved = ds->ds_quota = 0; } - if (err != 0 || (winner = dmu_buf_set_user_ie(dbuf, ds, - dsl_dataset_evict)) != NULL) { + dmu_buf_init_user(&ds->ds_dbu, dsl_dataset_evict, &ds->ds_dbuf); + if (err == 0) + winner = dmu_buf_set_user_ie(dbuf, &ds->ds_dbu); + + if (err != 0 || winner != NULL) { bplist_destroy(&ds->ds_pending_deadlist); dsl_deadlist_close(&ds->ds_deadlist); if (ds->ds_prev) @@ -919,7 +924,7 @@ dsl_dataset_recalc_head_uniq(dsl_dataset uint64_t mrs_used; uint64_t dlused, dlcomp, dluncomp; - ASSERT(!dsl_dataset_is_snapshot(ds)); + ASSERT(!ds->ds_is_snapshot); if (dsl_dataset_phys(ds)->ds_prev_snap_obj != 0) mrs_used = dsl_dataset_phys(ds->ds_prev)->ds_referenced_bytes; @@ -1675,7 +1680,7 @@ dsl_dataset_stats(dsl_dataset_t *ds, nvl dsl_prop_nvlist_add_uint64(nv, ZFS_PROP_LOGICALREFERENCED, dsl_dataset_phys(ds)->ds_uncompressed_bytes); - if (dsl_dataset_is_snapshot(ds)) { + if (ds->ds_is_snapshot) { dsl_prop_nvlist_add_uint64(nv, ZFS_PROP_COMPRESSRATIO, ratio); dsl_prop_nvlist_add_uint64(nv, ZFS_PROP_USED, dsl_dataset_phys(ds)->ds_unique_bytes); @@ -1743,7 +1748,7 @@ dsl_dataset_fast_stat(dsl_dataset_t *ds, dsl_dataset_phys(ds)->ds_flags & DS_FLAG_INCONSISTENT; stat->dds_guid = dsl_dataset_phys(ds)->ds_guid; stat->dds_origin[0] = '\0'; - if (dsl_dataset_is_snapshot(ds)) { + if (ds->ds_is_snapshot) { stat->dds_is_snapshot = B_TRUE; stat->dds_num_clones = dsl_dataset_phys(ds)->ds_num_children - 1; @@ -2023,7 +2028,7 @@ dsl_dataset_rollback_check(void *arg, dm return (error); /* must not be a snapshot */ - if (dsl_dataset_is_snapshot(ds)) { *** DIFF OUTPUT TRUNCATED AT 1000 LINES *** From owner-svn-src-stable@freebsd.org Sat Oct 3 07:30:56 2015 Return-Path: Delivered-To: svn-src-stable@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id B41ACA0E65D; Sat, 3 Oct 2015 07:30:56 +0000 (UTC) (envelope-from mav@FreeBSD.org) Received: from repo.freebsd.org (repo.freebsd.org [IPv6:2001:1900:2254:2068::e6a:0]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id A4BFF18B2; Sat, 3 Oct 2015 07:30:56 +0000 (UTC) (envelope-from mav@FreeBSD.org) Received: from repo.freebsd.org ([127.0.1.70]) by repo.freebsd.org (8.15.2/8.15.2) with ESMTP id t937Uu4k067445; Sat, 3 Oct 2015 07:30:56 GMT (envelope-from mav@FreeBSD.org) Received: (from mav@localhost) by repo.freebsd.org (8.15.2/8.15.2/Submit) id t937UunL067444; Sat, 3 Oct 2015 07:30:56 GMT (envelope-from mav@FreeBSD.org) Message-Id: <201510030730.t937UunL067444@repo.freebsd.org> X-Authentication-Warning: repo.freebsd.org: mav set sender to mav@FreeBSD.org using -f From: Alexander Motin Date: Sat, 3 Oct 2015 07:30:56 +0000 (UTC) To: src-committers@freebsd.org, svn-src-all@freebsd.org, svn-src-stable@freebsd.org, svn-src-stable-10@freebsd.org Subject: svn commit: r288550 - stable/10/sys/cddl/contrib/opensolaris/uts/common/fs/zfs X-SVN-Group: stable-10 MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit X-BeenThere: svn-src-stable@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: SVN commit messages for all the -stable branches of the src tree List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 03 Oct 2015 07:30:56 -0000 Author: mav Date: Sat Oct 3 07:30:55 2015 New Revision: 288550 URL: https://svnweb.freebsd.org/changeset/base/288550 Log: MFC r286576: Fix r286570 build with debug. Modified: stable/10/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/arc.c Directory Properties: stable/10/ (props changed) Modified: stable/10/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/arc.c ============================================================================== --- stable/10/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/arc.c Sat Oct 3 07:29:56 2015 (r288549) +++ stable/10/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/arc.c Sat Oct 3 07:30:55 2015 (r288550) @@ -1861,8 +1861,10 @@ arc_change_state(arc_state_t *new_state, * L2 headers should never be on the L2 state list since they don't * have L1 headers allocated. */ +#ifdef illumos ASSERT(list_is_empty(&arc_l2c_only->arcs_list[ARC_BUFC_DATA]) && list_is_empty(&arc_l2c_only->arcs_list[ARC_BUFC_METADATA])); +#endif } void From owner-svn-src-stable@freebsd.org Sat Oct 3 07:31:43 2015 Return-Path: Delivered-To: svn-src-stable@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 3DBC1A0E813; Sat, 3 Oct 2015 07:31:43 +0000 (UTC) (envelope-from mav@FreeBSD.org) Received: from repo.freebsd.org (repo.freebsd.org [IPv6:2001:1900:2254:2068::e6a:0]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 2E7F01A90; Sat, 3 Oct 2015 07:31:43 +0000 (UTC) (envelope-from mav@FreeBSD.org) Received: from repo.freebsd.org ([127.0.1.70]) by repo.freebsd.org (8.15.2/8.15.2) with ESMTP id t937Vhl0067588; Sat, 3 Oct 2015 07:31:43 GMT (envelope-from mav@FreeBSD.org) Received: (from mav@localhost) by repo.freebsd.org (8.15.2/8.15.2/Submit) id t937Vhno067587; Sat, 3 Oct 2015 07:31:43 GMT (envelope-from mav@FreeBSD.org) Message-Id: <201510030731.t937Vhno067587@repo.freebsd.org> X-Authentication-Warning: repo.freebsd.org: mav set sender to mav@FreeBSD.org using -f From: Alexander Motin Date: Sat, 3 Oct 2015 07:31:43 +0000 (UTC) To: src-committers@freebsd.org, svn-src-all@freebsd.org, svn-src-stable@freebsd.org, svn-src-stable-10@freebsd.org Subject: svn commit: r288551 - stable/10/sys/cddl/compat/opensolaris/sys X-SVN-Group: stable-10 MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit X-BeenThere: svn-src-stable@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: SVN commit messages for all the -stable branches of the src tree List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 03 Oct 2015 07:31:43 -0000 Author: mav Date: Sat Oct 3 07:31:42 2015 New Revision: 288551 URL: https://svnweb.freebsd.org/changeset/base/288551 Log: MFC r286578: Fix r286574 build in user-space. Modified: stable/10/sys/cddl/compat/opensolaris/sys/kstat.h Directory Properties: stable/10/ (props changed) Modified: stable/10/sys/cddl/compat/opensolaris/sys/kstat.h ============================================================================== --- stable/10/sys/cddl/compat/opensolaris/sys/kstat.h Sat Oct 3 07:30:55 2015 (r288550) +++ stable/10/sys/cddl/compat/opensolaris/sys/kstat.h Sat Oct 3 07:31:42 2015 (r288551) @@ -44,9 +44,9 @@ typedef struct kstat { #ifdef _KERNEL struct sysctl_ctx_list ks_sysctl_ctx; struct sysctl_oid *ks_sysctl_root; +#endif int (*ks_update)(struct kstat *, int); /* dynamic update */ void *ks_private; /* arbitrary provider-private data */ -#endif } kstat_t; typedef struct kstat_named { From owner-svn-src-stable@freebsd.org Sat Oct 3 07:32:36 2015 Return-Path: Delivered-To: svn-src-stable@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 240EEA0E8EA; Sat, 3 Oct 2015 07:32:36 +0000 (UTC) (envelope-from mav@FreeBSD.org) Received: from repo.freebsd.org (repo.freebsd.org [IPv6:2001:1900:2254:2068::e6a:0]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 0864A1C3C; Sat, 3 Oct 2015 07:32:36 +0000 (UTC) (envelope-from mav@FreeBSD.org) Received: from repo.freebsd.org ([127.0.1.70]) by repo.freebsd.org (8.15.2/8.15.2) with ESMTP id t937WZ8W067742; Sat, 3 Oct 2015 07:32:35 GMT (envelope-from mav@FreeBSD.org) Received: (from mav@localhost) by repo.freebsd.org (8.15.2/8.15.2/Submit) id t937WZ5G067738; Sat, 3 Oct 2015 07:32:35 GMT (envelope-from mav@FreeBSD.org) Message-Id: <201510030732.t937WZ5G067738@repo.freebsd.org> X-Authentication-Warning: repo.freebsd.org: mav set sender to mav@FreeBSD.org using -f From: Alexander Motin Date: Sat, 3 Oct 2015 07:32:35 +0000 (UTC) To: src-committers@freebsd.org, svn-src-all@freebsd.org, svn-src-stable@freebsd.org, svn-src-stable-10@freebsd.org Subject: svn commit: r288552 - in stable/10/sys/cddl/contrib/opensolaris/uts/common/fs/zfs: . sys X-SVN-Group: stable-10 MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit X-BeenThere: svn-src-stable@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: SVN commit messages for all the -stable branches of the src tree List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 03 Oct 2015 07:32:36 -0000 Author: mav Date: Sat Oct 3 07:32:34 2015 New Revision: 288552 URL: https://svnweb.freebsd.org/changeset/base/288552 Log: MFC r286579: 5313 Allow I/Os to be aggregated across ZIO priority classes Reviewed by: Andriy Gapon Reviewed by: Will Andrews Reviewed by: Matt Ahrens Reviewed by: George Wilson Approved by: Robert Mustacchi Author: Justin T. Gibbs illumos/illumos-gate@fe319232d24f4ae183730a5a24a09423d8ab4429 Modified: stable/10/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/sys/vdev_impl.h stable/10/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/sys/zio.h stable/10/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c Directory Properties: stable/10/ (props changed) Modified: stable/10/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/sys/vdev_impl.h ============================================================================== --- stable/10/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/sys/vdev_impl.h Sat Oct 3 07:31:42 2015 (r288551) +++ stable/10/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/sys/vdev_impl.h Sat Oct 3 07:32:34 2015 (r288552) @@ -113,6 +113,8 @@ struct vdev_queue { vdev_t *vq_vdev; vdev_queue_class_t vq_class[ZIO_PRIORITY_NUM_QUEUEABLE]; avl_tree_t vq_active_tree; + avl_tree_t vq_read_offset_tree; + avl_tree_t vq_write_offset_tree; uint64_t vq_last_offset; hrtime_t vq_io_complete_ts; /* time last i/o completed */ kmutex_t vq_lock; Modified: stable/10/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/sys/zio.h ============================================================================== --- stable/10/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/sys/zio.h Sat Oct 3 07:31:42 2015 (r288551) +++ stable/10/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/sys/zio.h Sat Oct 3 07:32:34 2015 (r288552) @@ -461,6 +461,7 @@ struct zio { uint64_t io_offset; hrtime_t io_timestamp; avl_node_t io_queue_node; + avl_node_t io_offset_node; /* Internal pipeline state */ enum zio_flag io_flags; Modified: stable/10/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c ============================================================================== --- stable/10/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c Sat Oct 3 07:31:42 2015 (r288551) +++ stable/10/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c Sat Oct 3 07:32:34 2015 (r288552) @@ -302,6 +302,22 @@ vdev_queue_offset_compare(const void *x1 return (0); } +static inline avl_tree_t * +vdev_queue_class_tree(vdev_queue_t *vq, zio_priority_t p) +{ + return (&vq->vq_class[p].vqc_queued_tree); +} + +static inline avl_tree_t * +vdev_queue_type_tree(vdev_queue_t *vq, zio_type_t t) +{ + ASSERT(t == ZIO_TYPE_READ || t == ZIO_TYPE_WRITE); + if (t == ZIO_TYPE_READ) + return (&vq->vq_read_offset_tree); + else + return (&vq->vq_write_offset_tree); +} + int vdev_queue_timestamp_compare(const void *x1, const void *x2) { @@ -336,19 +352,27 @@ vdev_queue_init(vdev_t *vd) avl_create(&vq->vq_active_tree, vdev_queue_offset_compare, sizeof (zio_t), offsetof(struct zio, io_queue_node)); + avl_create(vdev_queue_type_tree(vq, ZIO_TYPE_READ), + vdev_queue_offset_compare, sizeof (zio_t), + offsetof(struct zio, io_offset_node)); + avl_create(vdev_queue_type_tree(vq, ZIO_TYPE_WRITE), + vdev_queue_offset_compare, sizeof (zio_t), + offsetof(struct zio, io_offset_node)); for (zio_priority_t p = 0; p < ZIO_PRIORITY_NUM_QUEUEABLE; p++) { + int (*compfn) (const void *, const void *); + /* - * The synchronous i/o queues are FIFO rather than LBA ordered. - * This provides more consistent latency for these i/os, and - * they tend to not be tightly clustered anyway so there is - * little to no throughput loss. + * The synchronous i/o queues are dispatched in FIFO rather + * than LBA order. This provides more consistent latency for + * these i/os. */ - boolean_t fifo = (p == ZIO_PRIORITY_SYNC_READ || - p == ZIO_PRIORITY_SYNC_WRITE); - avl_create(&vq->vq_class[p].vqc_queued_tree, - fifo ? vdev_queue_timestamp_compare : - vdev_queue_offset_compare, + if (p == ZIO_PRIORITY_SYNC_READ || p == ZIO_PRIORITY_SYNC_WRITE) + compfn = vdev_queue_timestamp_compare; + else + compfn = vdev_queue_offset_compare; + + avl_create(vdev_queue_class_tree(vq, p), compfn, sizeof (zio_t), offsetof(struct zio, io_queue_node)); } @@ -361,8 +385,10 @@ vdev_queue_fini(vdev_t *vd) vdev_queue_t *vq = &vd->vdev_queue; for (zio_priority_t p = 0; p < ZIO_PRIORITY_NUM_QUEUEABLE; p++) - avl_destroy(&vq->vq_class[p].vqc_queued_tree); + avl_destroy(vdev_queue_class_tree(vq, p)); avl_destroy(&vq->vq_active_tree); + avl_destroy(vdev_queue_type_tree(vq, ZIO_TYPE_READ)); + avl_destroy(vdev_queue_type_tree(vq, ZIO_TYPE_WRITE)); mutex_destroy(&vq->vq_lock); } @@ -373,7 +399,8 @@ vdev_queue_io_add(vdev_queue_t *vq, zio_ spa_t *spa = zio->io_spa; ASSERT(MUTEX_HELD(&vq->vq_lock)); ASSERT3U(zio->io_priority, <, ZIO_PRIORITY_NUM_QUEUEABLE); - avl_add(&vq->vq_class[zio->io_priority].vqc_queued_tree, zio); + avl_add(vdev_queue_class_tree(vq, zio->io_priority), zio); + avl_add(vdev_queue_type_tree(vq, zio->io_type), zio); #ifdef illumos mutex_enter(&spa->spa_iokstat_lock); @@ -390,7 +417,8 @@ vdev_queue_io_remove(vdev_queue_t *vq, z spa_t *spa = zio->io_spa; ASSERT(MUTEX_HELD(&vq->vq_lock)); ASSERT3U(zio->io_priority, <, ZIO_PRIORITY_NUM_QUEUEABLE); - avl_remove(&vq->vq_class[zio->io_priority].vqc_queued_tree, zio); + avl_remove(vdev_queue_class_tree(vq, zio->io_priority), zio); + avl_remove(vdev_queue_type_tree(vq, zio->io_type), zio); #ifdef illumos mutex_enter(&spa->spa_iokstat_lock); @@ -563,7 +591,7 @@ vdev_queue_class_to_issue(vdev_queue_t * /* find a queue that has not reached its minimum # outstanding i/os */ for (p = 0; p < ZIO_PRIORITY_NUM_QUEUEABLE; p++) { - if (avl_numnodes(&vq->vq_class[p].vqc_queued_tree) > 0 && + if (avl_numnodes(vdev_queue_class_tree(vq, p)) > 0 && vq->vq_class[p].vqc_active < vdev_queue_class_min_active(p)) return (p); @@ -574,7 +602,7 @@ vdev_queue_class_to_issue(vdev_queue_t * * maximum # outstanding i/os. */ for (p = 0; p < ZIO_PRIORITY_NUM_QUEUEABLE; p++) { - if (avl_numnodes(&vq->vq_class[p].vqc_queued_tree) > 0 && + if (avl_numnodes(vdev_queue_class_tree(vq, p)) > 0 && vq->vq_class[p].vqc_active < vdev_queue_class_max_active(spa, p)) return (p); @@ -642,7 +670,7 @@ vdev_queue_aggregate(vdev_queue_t *vq, z * recording the last non-option I/O. */ flags = zio->io_flags & ZIO_FLAG_AGG_INHERIT; - t = &vq->vq_class[zio->io_priority].vqc_queued_tree; + t = vdev_queue_type_tree(vq, zio->io_type); while ((dio = AVL_PREV(t, first)) != NULL && (dio->io_flags & ZIO_FLAG_AGG_INHERIT) == flags && IO_SPAN(dio, last) <= zfs_vdev_aggregation_limit && @@ -751,7 +779,7 @@ vdev_queue_io_to_issue(vdev_queue_t *vq) zio_t *zio, *aio; zio_priority_t p; avl_index_t idx; - vdev_queue_class_t *vqc; + avl_tree_t *tree; zio_t search; again: @@ -770,13 +798,13 @@ again: * * For FIFO queues (sync), issue the i/o with the lowest timestamp. */ - vqc = &vq->vq_class[p]; + tree = vdev_queue_class_tree(vq, p); search.io_timestamp = 0; search.io_offset = vq->vq_last_offset + 1; - VERIFY3P(avl_find(&vqc->vqc_queued_tree, &search, &idx), ==, NULL); - zio = avl_nearest(&vqc->vqc_queued_tree, idx, AVL_AFTER); + VERIFY3P(avl_find(tree, &search, &idx), ==, NULL); + zio = avl_nearest(tree, idx, AVL_AFTER); if (zio == NULL) - zio = avl_first(&vqc->vqc_queued_tree); + zio = avl_first(tree); ASSERT3U(zio->io_priority, ==, p); aio = vdev_queue_aggregate(vq, zio); From owner-svn-src-stable@freebsd.org Sat Oct 3 07:33:29 2015 Return-Path: Delivered-To: svn-src-stable@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 0A591A0EA21; Sat, 3 Oct 2015 07:33:29 +0000 (UTC) (envelope-from mav@FreeBSD.org) Received: from repo.freebsd.org (repo.freebsd.org [IPv6:2001:1900:2254:2068::e6a:0]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id EC1741E77; Sat, 3 Oct 2015 07:33:28 +0000 (UTC) (envelope-from mav@FreeBSD.org) Received: from repo.freebsd.org ([127.0.1.70]) by repo.freebsd.org (8.15.2/8.15.2) with ESMTP id t937XS7D068049; Sat, 3 Oct 2015 07:33:28 GMT (envelope-from mav@FreeBSD.org) Received: (from mav@localhost) by repo.freebsd.org (8.15.2/8.15.2/Submit) id t937XSUb068044; Sat, 3 Oct 2015 07:33:28 GMT (envelope-from mav@FreeBSD.org) Message-Id: <201510030733.t937XSUb068044@repo.freebsd.org> X-Authentication-Warning: repo.freebsd.org: mav set sender to mav@FreeBSD.org using -f From: Alexander Motin Date: Sat, 3 Oct 2015 07:33:28 +0000 (UTC) To: src-committers@freebsd.org, svn-src-all@freebsd.org, svn-src-stable@freebsd.org, svn-src-stable-10@freebsd.org Subject: svn commit: r288553 - in stable/10: cddl/contrib/opensolaris/cmd/zstreamdump cddl/contrib/opensolaris/lib/libzfs/common sys/cddl/contrib/opensolaris/uts/common/fs/zfs sys/cddl/contrib/opensolaris/u... X-SVN-Group: stable-10 MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit X-BeenThere: svn-src-stable@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: SVN commit messages for all the -stable branches of the src tree List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 03 Oct 2015 07:33:29 -0000 Author: mav Date: Sat Oct 3 07:33:27 2015 New Revision: 288553 URL: https://svnweb.freebsd.org/changeset/base/288553 Log: MFC r286587: 5746 more checksumming in zfs send Reviewed by: Christopher Siden Reviewed by: George Wilson Reviewed by: Bayard Bell Approved by: Albert Lee Author: Matthew Ahrens illumos/illumos-gate@98110f08fa182032082d98be2ddb9391fcd62bf1 Modified: stable/10/cddl/contrib/opensolaris/cmd/zstreamdump/zstreamdump.c stable/10/cddl/contrib/opensolaris/lib/libzfs/common/libzfs_sendrecv.c stable/10/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/dmu_send.c stable/10/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/sys/spa.h stable/10/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/sys/zfs_ioctl.h stable/10/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/sys/zio_checksum.h Directory Properties: stable/10/ (props changed) Modified: stable/10/cddl/contrib/opensolaris/cmd/zstreamdump/zstreamdump.c ============================================================================== --- stable/10/cddl/contrib/opensolaris/cmd/zstreamdump/zstreamdump.c Sat Oct 3 07:32:34 2015 (r288552) +++ stable/10/cddl/contrib/opensolaris/cmd/zstreamdump/zstreamdump.c Sat Oct 3 07:33:27 2015 (r288553) @@ -25,7 +25,7 @@ */ /* - * Copyright (c) 2013 by Delphix. All rights reserved. + * Copyright (c) 2013, 2014 by Delphix. All rights reserved. */ #include @@ -34,6 +34,7 @@ #include #include #include +#include #include #include @@ -83,7 +84,6 @@ safe_malloc(size_t size) * * Read while computing incremental checksum */ - static size_t ssread(void *buf, size_t len, zio_cksum_t *cksum) { @@ -92,7 +92,7 @@ ssread(void *buf, size_t len, zio_cksum_ if ((outlen = fread(buf, len, 1, send_stream)) == 0) return (0); - if (do_cksum && cksum) { + if (do_cksum) { if (do_byteswap) fletcher_4_incremental_byteswap(buf, len, cksum); else @@ -102,6 +102,34 @@ ssread(void *buf, size_t len, zio_cksum_ return (outlen); } +static size_t +read_hdr(dmu_replay_record_t *drr, zio_cksum_t *cksum) +{ + ASSERT3U(offsetof(dmu_replay_record_t, drr_u.drr_checksum.drr_checksum), + ==, sizeof (dmu_replay_record_t) - sizeof (zio_cksum_t)); + size_t r = ssread(drr, sizeof (*drr) - sizeof (zio_cksum_t), cksum); + if (r == 0) + return (0); + zio_cksum_t saved_cksum = *cksum; + r = ssread(&drr->drr_u.drr_checksum.drr_checksum, + sizeof (zio_cksum_t), cksum); + if (r == 0) + return (0); + if (!ZIO_CHECKSUM_IS_ZERO(&drr->drr_u.drr_checksum.drr_checksum) && + !ZIO_CHECKSUM_EQUAL(saved_cksum, + drr->drr_u.drr_checksum.drr_checksum)) { + fprintf(stderr, "invalid checksum\n"); + (void) printf("Incorrect checksum in record header.\n"); + (void) printf("Expected checksum = %llx/%llx/%llx/%llx\n", + saved_cksum.zc_word[0], + saved_cksum.zc_word[1], + saved_cksum.zc_word[2], + saved_cksum.zc_word[3]); + exit(1); + } + return (sizeof (*drr)); +} + /* * Print part of a block in ASCII characters */ @@ -183,8 +211,10 @@ main(int argc, char *argv[]) struct drr_free *drrf = &thedrr.drr_u.drr_free; struct drr_spill *drrs = &thedrr.drr_u.drr_spill; struct drr_write_embedded *drrwe = &thedrr.drr_u.drr_write_embedded; + struct drr_checksum *drrc = &thedrr.drr_u.drr_checksum; char c; boolean_t verbose = B_FALSE; + boolean_t very_verbose = B_FALSE; boolean_t first = B_TRUE; /* * dump flag controls whether the contents of any modified data blocks @@ -202,11 +232,14 @@ main(int argc, char *argv[]) do_cksum = B_FALSE; break; case 'v': + if (verbose) + very_verbose = B_TRUE; verbose = B_TRUE; break; case 'd': dump = B_TRUE; verbose = B_TRUE; + very_verbose = B_TRUE; break; case ':': (void) fprintf(stderr, @@ -230,7 +263,7 @@ main(int argc, char *argv[]) send_stream = stdin; pcksum = zc; - while (ssread(drr, sizeof (dmu_replay_record_t), &zc)) { + while (read_hdr(drr, &zc)) { /* * If this is the first DMU record being processed, check for @@ -432,7 +465,7 @@ main(int argc, char *argv[]) if (verbose) { (void) printf("WRITE object = %llu type = %u " "checksum type = %u\n" - "offset = %llu length = %llu " + " offset = %llu length = %llu " "props = %llx\n", (u_longlong_t)drrw->drr_object, drrw->drr_type, @@ -476,9 +509,9 @@ main(int argc, char *argv[]) if (verbose) { (void) printf("WRITE_BYREF object = %llu " "checksum type = %u props = %llx\n" - "offset = %llu length = %llu\n" + " offset = %llu length = %llu\n" "toguid = %llx refguid = %llx\n" - "refobject = %llu refoffset = %llu\n", + " refobject = %llu refoffset = %llu\n", (u_longlong_t)drrwbr->drr_object, drrwbr->drr_checksumtype, (u_longlong_t)drrwbr->drr_key.ddk_prop, @@ -538,7 +571,7 @@ main(int argc, char *argv[]) if (verbose) { (void) printf("WRITE_EMBEDDED object = %llu " "offset = %llu length = %llu\n" - "toguid = %llx comp = %u etype = %u " + " toguid = %llx comp = %u etype = %u " "lsize = %u psize = %u\n", (u_longlong_t)drrwe->drr_object, (u_longlong_t)drrwe->drr_offset, @@ -553,6 +586,13 @@ main(int argc, char *argv[]) P2ROUNDUP(drrwe->drr_psize, 8), &zc); break; } + if (drr->drr_type != DRR_BEGIN && very_verbose) { + (void) printf(" checksum = %llx/%llx/%llx/%llx\n", + (longlong_t)drrc->drr_checksum.zc_word[0], + (longlong_t)drrc->drr_checksum.zc_word[1], + (longlong_t)drrc->drr_checksum.zc_word[2], + (longlong_t)drrc->drr_checksum.zc_word[3]); + } pcksum = zc; } free(buf); Modified: stable/10/cddl/contrib/opensolaris/lib/libzfs/common/libzfs_sendrecv.c ============================================================================== --- stable/10/cddl/contrib/opensolaris/lib/libzfs/common/libzfs_sendrecv.c Sat Oct 3 07:32:34 2015 (r288552) +++ stable/10/cddl/contrib/opensolaris/lib/libzfs/common/libzfs_sendrecv.c Sat Oct 3 07:33:27 2015 (r288553) @@ -188,10 +188,28 @@ ddt_update(libzfs_handle_t *hdl, dedup_t } static int -cksum_and_write(const void *buf, uint64_t len, zio_cksum_t *zc, int outfd) +dump_record(dmu_replay_record_t *drr, void *payload, int payload_len, + zio_cksum_t *zc, int outfd) { - fletcher_4_incremental_native(buf, len, zc); - return (write(outfd, buf, len)); + ASSERT3U(offsetof(dmu_replay_record_t, drr_u.drr_checksum.drr_checksum), + ==, sizeof (dmu_replay_record_t) - sizeof (zio_cksum_t)); + fletcher_4_incremental_native(drr, + offsetof(dmu_replay_record_t, drr_u.drr_checksum.drr_checksum), zc); + if (drr->drr_type != DRR_BEGIN) { + ASSERT(ZIO_CHECKSUM_IS_ZERO(&drr->drr_u. + drr_checksum.drr_checksum)); + drr->drr_u.drr_checksum.drr_checksum = *zc; + } + fletcher_4_incremental_native(&drr->drr_u.drr_checksum.drr_checksum, + sizeof (zio_cksum_t), zc); + if (write(outfd, drr, sizeof (*drr)) == -1) + return (errno); + if (payload_len != 0) { + fletcher_4_incremental_native(payload, payload_len, zc); + if (write(outfd, payload, payload_len) == -1) + return (errno); + } + return (0); } /* @@ -218,26 +236,18 @@ cksummer(void *arg) char *buf = zfs_alloc(dda->dedup_hdl, SPA_MAXBLOCKSIZE); dmu_replay_record_t thedrr; dmu_replay_record_t *drr = &thedrr; - struct drr_begin *drrb = &thedrr.drr_u.drr_begin; - struct drr_end *drre = &thedrr.drr_u.drr_end; - struct drr_object *drro = &thedrr.drr_u.drr_object; - struct drr_write *drrw = &thedrr.drr_u.drr_write; - struct drr_spill *drrs = &thedrr.drr_u.drr_spill; - struct drr_write_embedded *drrwe = &thedrr.drr_u.drr_write_embedded; FILE *ofp; int outfd; - dmu_replay_record_t wbr_drr = {0}; - struct drr_write_byref *wbr_drrr = &wbr_drr.drr_u.drr_write_byref; dedup_table_t ddt; zio_cksum_t stream_cksum; uint64_t physmem = sysconf(_SC_PHYS_PAGES) * sysconf(_SC_PAGESIZE); uint64_t numbuckets; ddt.max_ddt_size = - MAX((physmem * MAX_DDT_PHYSMEM_PERCENT)/100, - SMALLEST_POSSIBLE_MAX_DDT_MB<<20); + MAX((physmem * MAX_DDT_PHYSMEM_PERCENT) / 100, + SMALLEST_POSSIBLE_MAX_DDT_MB << 20); - numbuckets = ddt.max_ddt_size/(sizeof (dedup_entry_t)); + numbuckets = ddt.max_ddt_size / (sizeof (dedup_entry_t)); /* * numbuckets must be a power of 2. Increase number to @@ -253,32 +263,29 @@ cksummer(void *arg) ddt.numhashbits = high_order_bit(numbuckets) - 1; ddt.ddt_full = B_FALSE; - /* Initialize the write-by-reference block. */ - wbr_drr.drr_type = DRR_WRITE_BYREF; - wbr_drr.drr_payloadlen = 0; - outfd = dda->outputfd; ofp = fdopen(dda->inputfd, "r"); - while (ssread(drr, sizeof (dmu_replay_record_t), ofp) != 0) { + while (ssread(drr, sizeof (*drr), ofp) != 0) { switch (drr->drr_type) { case DRR_BEGIN: { - int fflags; + struct drr_begin *drrb = &drr->drr_u.drr_begin; + int fflags; + int sz = 0; ZIO_SET_CHECKSUM(&stream_cksum, 0, 0, 0, 0); + ASSERT3U(drrb->drr_magic, ==, DMU_BACKUP_MAGIC); + /* set the DEDUP feature flag for this stream */ fflags = DMU_GET_FEATUREFLAGS(drrb->drr_versioninfo); fflags |= (DMU_BACKUP_FEATURE_DEDUP | DMU_BACKUP_FEATURE_DEDUPPROPS); DMU_SET_FEATUREFLAGS(drrb->drr_versioninfo, fflags); - if (cksum_and_write(drr, sizeof (dmu_replay_record_t), - &stream_cksum, outfd) == -1) - goto out; if (DMU_GET_STREAM_HDRTYPE(drrb->drr_versioninfo) == DMU_COMPOUNDSTREAM && drr->drr_payloadlen != 0) { - int sz = drr->drr_payloadlen; + sz = drr->drr_payloadlen; if (sz > SPA_MAXBLOCKSIZE) { buf = zfs_realloc(dda->dedup_hdl, buf, @@ -287,64 +294,60 @@ cksummer(void *arg) (void) ssread(buf, sz, ofp); if (ferror(stdin)) perror("fread"); - if (cksum_and_write(buf, sz, &stream_cksum, - outfd) == -1) - goto out; } + if (dump_record(drr, buf, sz, &stream_cksum, + outfd) != 0) + goto out; break; } case DRR_END: { + struct drr_end *drre = &drr->drr_u.drr_end; /* use the recalculated checksum */ - ZIO_SET_CHECKSUM(&drre->drr_checksum, - stream_cksum.zc_word[0], stream_cksum.zc_word[1], - stream_cksum.zc_word[2], stream_cksum.zc_word[3]); - if ((write(outfd, drr, - sizeof (dmu_replay_record_t))) == -1) + drre->drr_checksum = stream_cksum; + if (dump_record(drr, NULL, 0, &stream_cksum, + outfd) != 0) goto out; break; } case DRR_OBJECT: { - if (cksum_and_write(drr, sizeof (dmu_replay_record_t), - &stream_cksum, outfd) == -1) - goto out; + struct drr_object *drro = &drr->drr_u.drr_object; if (drro->drr_bonuslen > 0) { (void) ssread(buf, P2ROUNDUP((uint64_t)drro->drr_bonuslen, 8), ofp); - if (cksum_and_write(buf, - P2ROUNDUP((uint64_t)drro->drr_bonuslen, 8), - &stream_cksum, outfd) == -1) - goto out; } + if (dump_record(drr, buf, + P2ROUNDUP((uint64_t)drro->drr_bonuslen, 8), + &stream_cksum, outfd) != 0) + goto out; break; } case DRR_SPILL: { - if (cksum_and_write(drr, sizeof (dmu_replay_record_t), - &stream_cksum, outfd) == -1) - goto out; + struct drr_spill *drrs = &drr->drr_u.drr_spill; (void) ssread(buf, drrs->drr_length, ofp); - if (cksum_and_write(buf, drrs->drr_length, - &stream_cksum, outfd) == -1) + if (dump_record(drr, buf, drrs->drr_length, + &stream_cksum, outfd) != 0) goto out; break; } case DRR_FREEOBJECTS: { - if (cksum_and_write(drr, sizeof (dmu_replay_record_t), - &stream_cksum, outfd) == -1) + if (dump_record(drr, NULL, 0, &stream_cksum, + outfd) != 0) goto out; break; } case DRR_WRITE: { + struct drr_write *drrw = &drr->drr_u.drr_write; dataref_t dataref; (void) ssread(buf, drrw->drr_length, ofp); @@ -382,7 +385,13 @@ cksummer(void *arg) if (ddt_update(dda->dedup_hdl, &ddt, &drrw->drr_key.ddk_cksum, drrw->drr_key.ddk_prop, &dataref)) { + dmu_replay_record_t wbr_drr = {0}; + struct drr_write_byref *wbr_drrr = + &wbr_drr.drr_u.drr_write_byref; + /* block already present in stream */ + wbr_drr.drr_type = DRR_WRITE_BYREF; + wbr_drrr->drr_object = drrw->drr_object; wbr_drrr->drr_offset = drrw->drr_offset; wbr_drrr->drr_length = drrw->drr_length; @@ -402,19 +411,13 @@ cksummer(void *arg) wbr_drrr->drr_key.ddk_prop = drrw->drr_key.ddk_prop; - if (cksum_and_write(&wbr_drr, - sizeof (dmu_replay_record_t), &stream_cksum, - outfd) == -1) + if (dump_record(&wbr_drr, NULL, 0, + &stream_cksum, outfd) != 0) goto out; } else { /* block not previously seen */ - if (cksum_and_write(drr, - sizeof (dmu_replay_record_t), &stream_cksum, - outfd) == -1) - goto out; - if (cksum_and_write(buf, - drrw->drr_length, - &stream_cksum, outfd) == -1) + if (dump_record(drr, buf, drrw->drr_length, + &stream_cksum, outfd) != 0) goto out; } break; @@ -422,28 +425,27 @@ cksummer(void *arg) case DRR_WRITE_EMBEDDED: { - if (cksum_and_write(drr, sizeof (dmu_replay_record_t), - &stream_cksum, outfd) == -1) - goto out; + struct drr_write_embedded *drrwe = + &drr->drr_u.drr_write_embedded; (void) ssread(buf, P2ROUNDUP((uint64_t)drrwe->drr_psize, 8), ofp); - if (cksum_and_write(buf, + if (dump_record(drr, buf, P2ROUNDUP((uint64_t)drrwe->drr_psize, 8), - &stream_cksum, outfd) == -1) + &stream_cksum, outfd) != 0) goto out; break; } case DRR_FREE: { - if (cksum_and_write(drr, sizeof (dmu_replay_record_t), - &stream_cksum, outfd) == -1) + if (dump_record(drr, NULL, 0, &stream_cksum, + outfd) != 0) goto out; break; } default: - (void) printf("INVALID record type 0x%x\n", + (void) fprintf(stderr, "INVALID record type 0x%x\n", drr->drr_type); /* should never happen, so assert */ assert(B_FALSE); @@ -1470,18 +1472,11 @@ zfs_send(zfs_handle_t *zhp, const char * sizeof (drr.drr_u.drr_begin.drr_toname), "%s@%s", zhp->zfs_name, tosnap); drr.drr_payloadlen = buflen; - err = cksum_and_write(&drr, sizeof (drr), &zc, outfd); - /* write header nvlist */ - if (err != -1 && packbuf != NULL) { - err = cksum_and_write(packbuf, buflen, &zc, - outfd); - } + err = dump_record(&drr, packbuf, buflen, &zc, outfd); free(packbuf); - if (err == -1) { - err = errno; + if (err != 0) goto stderr_out; - } /* write end record */ bzero(&drr, sizeof (drr)); @@ -1714,6 +1709,8 @@ recv_read(libzfs_handle_t *hdl, int fd, int rv; int len = ilen; + assert(ilen <= SPA_MAXBLOCKSIZE); + do { rv = read(fd, cp, len); cp += rv; Modified: stable/10/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/dmu_send.c ============================================================================== --- stable/10/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/dmu_send.c Sat Oct 3 07:32:34 2015 (r288552) +++ stable/10/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/dmu_send.c Sat Oct 3 07:33:27 2015 (r288553) @@ -73,7 +73,6 @@ dump_bytes(dmu_sendarg_t *dsp, void *buf struct iovec aiov; ASSERT0(len % 8); - fletcher_4_incremental_native(buf, len, &dsp->dsa_zc); aiov.iov_base = buf; aiov.iov_len = len; auio.uio_iov = &aiov; @@ -99,6 +98,38 @@ dump_bytes(dmu_sendarg_t *dsp, void *buf return (dsp->dsa_err); } +/* + * For all record types except BEGIN, fill in the checksum (overlaid in + * drr_u.drr_checksum.drr_checksum). The checksum verifies everything + * up to the start of the checksum itself. + */ +static int +dump_record(dmu_sendarg_t *dsp, void *payload, int payload_len) +{ + ASSERT3U(offsetof(dmu_replay_record_t, drr_u.drr_checksum.drr_checksum), + ==, sizeof (dmu_replay_record_t) - sizeof (zio_cksum_t)); + fletcher_4_incremental_native(dsp->dsa_drr, + offsetof(dmu_replay_record_t, drr_u.drr_checksum.drr_checksum), + &dsp->dsa_zc); + if (dsp->dsa_drr->drr_type != DRR_BEGIN) { + ASSERT(ZIO_CHECKSUM_IS_ZERO(&dsp->dsa_drr->drr_u. + drr_checksum.drr_checksum)); + dsp->dsa_drr->drr_u.drr_checksum.drr_checksum = dsp->dsa_zc; + } + fletcher_4_incremental_native(&dsp->dsa_drr-> + drr_u.drr_checksum.drr_checksum, + sizeof (zio_cksum_t), &dsp->dsa_zc); + if (dump_bytes(dsp, dsp->dsa_drr, sizeof (dmu_replay_record_t)) != 0) + return (SET_ERROR(EINTR)); + if (payload_len != 0) { + fletcher_4_incremental_native(payload, payload_len, + &dsp->dsa_zc); + if (dump_bytes(dsp, payload, payload_len) != 0) + return (SET_ERROR(EINTR)); + } + return (0); +} + static int dump_free(dmu_sendarg_t *dsp, uint64_t object, uint64_t offset, uint64_t length) @@ -143,8 +174,7 @@ dump_free(dmu_sendarg_t *dsp, uint64_t o */ if (dsp->dsa_pending_op != PENDING_NONE && dsp->dsa_pending_op != PENDING_FREE) { - if (dump_bytes(dsp, dsp->dsa_drr, - sizeof (dmu_replay_record_t)) != 0) + if (dump_record(dsp, NULL, 0) != 0) return (SET_ERROR(EINTR)); dsp->dsa_pending_op = PENDING_NONE; } @@ -167,8 +197,7 @@ dump_free(dmu_sendarg_t *dsp, uint64_t o return (0); } else { /* not a continuation. Push out pending record */ - if (dump_bytes(dsp, dsp->dsa_drr, - sizeof (dmu_replay_record_t)) != 0) + if (dump_record(dsp, NULL, 0) != 0) return (SET_ERROR(EINTR)); dsp->dsa_pending_op = PENDING_NONE; } @@ -181,8 +210,7 @@ dump_free(dmu_sendarg_t *dsp, uint64_t o drrf->drr_length = length; drrf->drr_toguid = dsp->dsa_toguid; if (length == -1ULL) { - if (dump_bytes(dsp, dsp->dsa_drr, - sizeof (dmu_replay_record_t)) != 0) + if (dump_record(dsp, NULL, 0) != 0) return (SET_ERROR(EINTR)); } else { dsp->dsa_pending_op = PENDING_FREE; @@ -214,12 +242,11 @@ dump_write(dmu_sendarg_t *dsp, dmu_objec * of different types. */ if (dsp->dsa_pending_op != PENDING_NONE) { - if (dump_bytes(dsp, dsp->dsa_drr, - sizeof (dmu_replay_record_t)) != 0) + if (dump_record(dsp, NULL, 0) != 0) return (SET_ERROR(EINTR)); dsp->dsa_pending_op = PENDING_NONE; } - /* write a DATA record */ + /* write a WRITE record */ bzero(dsp->dsa_drr, sizeof (dmu_replay_record_t)); dsp->dsa_drr->drr_type = DRR_WRITE; drrw->drr_object = object; @@ -245,9 +272,7 @@ dump_write(dmu_sendarg_t *dsp, dmu_objec drrw->drr_key.ddk_cksum = bp->blk_cksum; } - if (dump_bytes(dsp, dsp->dsa_drr, sizeof (dmu_replay_record_t)) != 0) - return (SET_ERROR(EINTR)); - if (dump_bytes(dsp, data, blksz) != 0) + if (dump_record(dsp, data, blksz) != 0) return (SET_ERROR(EINTR)); return (0); } @@ -261,8 +286,7 @@ dump_write_embedded(dmu_sendarg_t *dsp, &(dsp->dsa_drr->drr_u.drr_write_embedded); if (dsp->dsa_pending_op != PENDING_NONE) { - if (dump_bytes(dsp, dsp->dsa_drr, - sizeof (dmu_replay_record_t)) != 0) + if (dump_record(dsp, NULL, 0) != 0) return (EINTR); dsp->dsa_pending_op = PENDING_NONE; } @@ -282,9 +306,7 @@ dump_write_embedded(dmu_sendarg_t *dsp, decode_embedded_bp_compressed(bp, buf); - if (dump_bytes(dsp, dsp->dsa_drr, sizeof (dmu_replay_record_t)) != 0) - return (EINTR); - if (dump_bytes(dsp, buf, P2ROUNDUP(drrw->drr_psize, 8)) != 0) + if (dump_record(dsp, buf, P2ROUNDUP(drrw->drr_psize, 8)) != 0) return (EINTR); return (0); } @@ -295,8 +317,7 @@ dump_spill(dmu_sendarg_t *dsp, uint64_t struct drr_spill *drrs = &(dsp->dsa_drr->drr_u.drr_spill); if (dsp->dsa_pending_op != PENDING_NONE) { - if (dump_bytes(dsp, dsp->dsa_drr, - sizeof (dmu_replay_record_t)) != 0) + if (dump_record(dsp, NULL, 0) != 0) return (SET_ERROR(EINTR)); dsp->dsa_pending_op = PENDING_NONE; } @@ -308,9 +329,7 @@ dump_spill(dmu_sendarg_t *dsp, uint64_t drrs->drr_length = blksz; drrs->drr_toguid = dsp->dsa_toguid; - if (dump_bytes(dsp, dsp->dsa_drr, sizeof (dmu_replay_record_t))) - return (SET_ERROR(EINTR)); - if (dump_bytes(dsp, data, blksz)) + if (dump_record(dsp, data, blksz) != 0) return (SET_ERROR(EINTR)); return (0); } @@ -333,8 +352,7 @@ dump_freeobjects(dmu_sendarg_t *dsp, uin */ if (dsp->dsa_pending_op != PENDING_NONE && dsp->dsa_pending_op != PENDING_FREEOBJECTS) { - if (dump_bytes(dsp, dsp->dsa_drr, - sizeof (dmu_replay_record_t)) != 0) + if (dump_record(dsp, NULL, 0) != 0) return (SET_ERROR(EINTR)); dsp->dsa_pending_op = PENDING_NONE; } @@ -348,8 +366,7 @@ dump_freeobjects(dmu_sendarg_t *dsp, uin return (0); } else { /* can't be aggregated. Push out pending record */ - if (dump_bytes(dsp, dsp->dsa_drr, - sizeof (dmu_replay_record_t)) != 0) + if (dump_record(dsp, NULL, 0) != 0) return (SET_ERROR(EINTR)); dsp->dsa_pending_op = PENDING_NONE; } @@ -376,8 +393,7 @@ dump_dnode(dmu_sendarg_t *dsp, uint64_t return (dump_freeobjects(dsp, object, 1)); if (dsp->dsa_pending_op != PENDING_NONE) { - if (dump_bytes(dsp, dsp->dsa_drr, - sizeof (dmu_replay_record_t)) != 0) + if (dump_record(dsp, NULL, 0) != 0) return (SET_ERROR(EINTR)); dsp->dsa_pending_op = PENDING_NONE; } @@ -398,11 +414,10 @@ dump_dnode(dmu_sendarg_t *dsp, uint64_t drro->drr_blksz > SPA_OLD_MAXBLOCKSIZE) drro->drr_blksz = SPA_OLD_MAXBLOCKSIZE; - if (dump_bytes(dsp, dsp->dsa_drr, sizeof (dmu_replay_record_t)) != 0) - return (SET_ERROR(EINTR)); - - if (dump_bytes(dsp, DN_BONUS(dnp), P2ROUNDUP(dnp->dn_bonuslen, 8)) != 0) + if (dump_record(dsp, DN_BONUS(dnp), + P2ROUNDUP(dnp->dn_bonuslen, 8)) != 0) { return (SET_ERROR(EINTR)); + } /* Free anything past the end of the file. */ if (dump_free(dsp, object, (dnp->dn_maxblkid + 1) * @@ -651,7 +666,6 @@ dmu_send_impl(void *tag, dsl_pool_t *dp, dsp->dsa_os = os; dsp->dsa_off = off; dsp->dsa_toguid = dsl_dataset_phys(ds)->ds_guid; - ZIO_SET_CHECKSUM(&dsp->dsa_zc, 0, 0, 0, 0); dsp->dsa_pending_op = PENDING_NONE; dsp->dsa_incremental = (fromzb != NULL); dsp->dsa_featureflags = featureflags; @@ -663,7 +677,7 @@ dmu_send_impl(void *tag, dsl_pool_t *dp, dsl_dataset_long_hold(ds, FTAG); dsl_pool_rele(dp, tag); - if (dump_bytes(dsp, drr, sizeof (dmu_replay_record_t)) != 0) { + if (dump_record(dsp, NULL, 0) != 0) { err = dsp->dsa_err; goto out; } @@ -672,7 +686,7 @@ dmu_send_impl(void *tag, dsl_pool_t *dp, backup_cb, dsp); if (dsp->dsa_pending_op != PENDING_NONE) - if (dump_bytes(dsp, drr, sizeof (dmu_replay_record_t)) != 0) + if (dump_record(dsp, NULL, 0) != 0) err = SET_ERROR(EINTR); if (err != 0) { @@ -686,7 +700,7 @@ dmu_send_impl(void *tag, dsl_pool_t *dp, drr->drr_u.drr_end.drr_checksum = dsp->dsa_zc; drr->drr_u.drr_end.drr_toguid = dsp->dsa_toguid; - if (dump_bytes(dsp, drr, sizeof (dmu_replay_record_t)) != 0) { + if (dump_record(dsp, NULL, 0) != 0) { err = dsp->dsa_err; goto out; } @@ -1251,14 +1265,20 @@ dmu_recv_begin(char *tofs, char *tosnap, } struct restorearg { + objset_t *os; int err; boolean_t byteswap; kthread_t *td; struct file *fp; - char *buf; uint64_t voff; int bufsize; /* amount of memory allocated for buf */ + + dmu_replay_record_t *drr; + dmu_replay_record_t *next_drr; + char *buf; zio_cksum_t cksum; + zio_cksum_t prev_cksum; + avl_tree_t *guid_to_ds_map; }; @@ -1323,14 +1343,11 @@ restore_bytes(struct restorearg *ra, voi return (error); } -static void * -restore_read(struct restorearg *ra, int len, char *buf) +static int +restore_read(struct restorearg *ra, int len, void *buf) { int done = 0; - if (buf == NULL) - buf = ra->buf; - /* some things will require 8-byte alignment, so everything must */ ASSERT0(len % 8); ASSERT3U(len, <=, ra->bufsize); @@ -1346,24 +1363,21 @@ restore_read(struct restorearg *ra, int ra->voff += len - done - resid; done = len - resid; if (ra->err != 0) - return (NULL); + return (ra->err); } ASSERT3U(done, ==, len); - if (ra->byteswap) - fletcher_4_incremental_byteswap(buf, len, &ra->cksum); - else - fletcher_4_incremental_native(buf, len, &ra->cksum); - return (buf); + return (0); } static void -backup_byteswap(dmu_replay_record_t *drr) +byteswap_record(dmu_replay_record_t *drr) { #define DO64(X) (drr->drr_u.X = BSWAP_64(drr->drr_u.X)) #define DO32(X) (drr->drr_u.X = BSWAP_32(drr->drr_u.X)) drr->drr_type = BSWAP_32(drr->drr_type); drr->drr_payloadlen = BSWAP_32(drr->drr_payloadlen); + switch (drr->drr_type) { case DRR_BEGIN: DO64(drr_begin.drr_magic); @@ -1393,10 +1407,7 @@ backup_byteswap(dmu_replay_record_t *drr DO64(drr_write.drr_offset); DO64(drr_write.drr_length); DO64(drr_write.drr_toguid); - DO64(drr_write.drr_key.ddk_cksum.zc_word[0]); - DO64(drr_write.drr_key.ddk_cksum.zc_word[1]); - DO64(drr_write.drr_key.ddk_cksum.zc_word[2]); - DO64(drr_write.drr_key.ddk_cksum.zc_word[3]); + ZIO_CHECKSUM_BSWAP(&drr->drr_u.drr_write.drr_key.ddk_cksum); DO64(drr_write.drr_key.ddk_prop); break; case DRR_WRITE_BYREF: @@ -1407,10 +1418,8 @@ backup_byteswap(dmu_replay_record_t *drr DO64(drr_write_byref.drr_refguid); DO64(drr_write_byref.drr_refobject); DO64(drr_write_byref.drr_refoffset); - DO64(drr_write_byref.drr_key.ddk_cksum.zc_word[0]); - DO64(drr_write_byref.drr_key.ddk_cksum.zc_word[1]); - DO64(drr_write_byref.drr_key.ddk_cksum.zc_word[2]); - DO64(drr_write_byref.drr_key.ddk_cksum.zc_word[3]); + ZIO_CHECKSUM_BSWAP(&drr->drr_u.drr_write_byref. + drr_key.ddk_cksum); DO64(drr_write_byref.drr_key.ddk_prop); break; case DRR_WRITE_EMBEDDED: @@ -1433,13 +1442,15 @@ backup_byteswap(dmu_replay_record_t *drr DO64(drr_spill.drr_toguid); break; case DRR_END: - DO64(drr_end.drr_checksum.zc_word[0]); - DO64(drr_end.drr_checksum.zc_word[1]); - DO64(drr_end.drr_checksum.zc_word[2]); - DO64(drr_end.drr_checksum.zc_word[3]); DO64(drr_end.drr_toguid); + ZIO_CHECKSUM_BSWAP(&drr->drr_u.drr_end.drr_checksum); break; } + + if (drr->drr_type != DRR_BEGIN) { + ZIO_CHECKSUM_BSWAP(&drr->drr_u.drr_checksum.drr_checksum); + } + #undef DO64 #undef DO32 } @@ -1456,11 +1467,10 @@ deduce_nblkptr(dmu_object_type_t bonus_t } static int -restore_object(struct restorearg *ra, objset_t *os, struct drr_object *drro) +restore_object(struct restorearg *ra, struct drr_object *drro, void *data) { dmu_object_info_t doi; dmu_tx_t *tx; - void *data = NULL; uint64_t object; int err; @@ -1471,23 +1481,17 @@ restore_object(struct restorearg *ra, ob drro->drr_compress >= ZIO_COMPRESS_FUNCTIONS || P2PHASE(drro->drr_blksz, SPA_MINBLOCKSIZE) || drro->drr_blksz < SPA_MINBLOCKSIZE || - drro->drr_blksz > spa_maxblocksize(dmu_objset_spa(os)) || + drro->drr_blksz > spa_maxblocksize(dmu_objset_spa(ra->os)) || drro->drr_bonuslen > DN_MAX_BONUSLEN) { return (SET_ERROR(EINVAL)); } - err = dmu_object_info(os, drro->drr_object, &doi); + err = dmu_object_info(ra->os, drro->drr_object, &doi); if (err != 0 && err != ENOENT) return (SET_ERROR(EINVAL)); object = err == 0 ? drro->drr_object : DMU_NEW_OBJECT; - if (drro->drr_bonuslen) { - data = restore_read(ra, P2ROUNDUP(drro->drr_bonuslen, 8), NULL); - if (ra->err != 0) - return (ra->err); - } - /* * If we are losing blkptrs or changing the block size this must * be a new file instance. We must clear out the previous file @@ -1501,14 +1505,14 @@ restore_object(struct restorearg *ra, ob if (drro->drr_blksz != doi.doi_data_block_size || nblkptr < doi.doi_nblkptr) { - err = dmu_free_long_range(os, drro->drr_object, + err = dmu_free_long_range(ra->os, drro->drr_object, 0, DMU_OBJECT_END); if (err != 0) return (SET_ERROR(EINVAL)); } } - tx = dmu_tx_create(os); + tx = dmu_tx_create(ra->os); dmu_tx_hold_bonus(tx, object); err = dmu_tx_assign(tx, TXG_WAIT); if (err != 0) { @@ -1518,7 +1522,7 @@ restore_object(struct restorearg *ra, ob if (object == DMU_NEW_OBJECT) { /* currently free, want to be allocated */ - err = dmu_object_claim(os, drro->drr_object, + err = dmu_object_claim(ra->os, drro->drr_object, drro->drr_type, drro->drr_blksz, drro->drr_bonustype, drro->drr_bonuslen, tx); } else if (drro->drr_type != doi.doi_type || @@ -1526,7 +1530,7 @@ restore_object(struct restorearg *ra, ob drro->drr_bonustype != doi.doi_bonus_type || drro->drr_bonuslen != doi.doi_bonus_size) { /* currently allocated, but with different properties */ - err = dmu_object_reclaim(os, drro->drr_object, + err = dmu_object_reclaim(ra->os, drro->drr_object, drro->drr_type, drro->drr_blksz, drro->drr_bonustype, drro->drr_bonuslen, tx); } @@ -1535,14 +1539,15 @@ restore_object(struct restorearg *ra, ob return (SET_ERROR(EINVAL)); } - dmu_object_set_checksum(os, drro->drr_object, drro->drr_checksumtype, - tx); - dmu_object_set_compress(os, drro->drr_object, drro->drr_compress, tx); + dmu_object_set_checksum(ra->os, drro->drr_object, + drro->drr_checksumtype, tx); + dmu_object_set_compress(ra->os, drro->drr_object, + drro->drr_compress, tx); if (data != NULL) { dmu_buf_t *db; - VERIFY(0 == dmu_bonus_hold(os, drro->drr_object, FTAG, &db)); + VERIFY0(dmu_bonus_hold(ra->os, drro->drr_object, FTAG, &db)); dmu_buf_will_dirty(db, tx); ASSERT3U(db->db_size, >=, drro->drr_bonuslen); @@ -1561,7 +1566,7 @@ restore_object(struct restorearg *ra, ob /* ARGSUSED */ static int -restore_freeobjects(struct restorearg *ra, objset_t *os, +restore_freeobjects(struct restorearg *ra, struct drr_freeobjects *drrfo) { uint64_t obj; @@ -1571,13 +1576,13 @@ restore_freeobjects(struct restorearg *r for (obj = drrfo->drr_firstobj; obj < drrfo->drr_firstobj + drrfo->drr_numobjs; - (void) dmu_object_next(os, &obj, FALSE, 0)) { + (void) dmu_object_next(ra->os, &obj, FALSE, 0)) { int err; - if (dmu_object_info(os, obj, NULL) != 0) + if (dmu_object_info(ra->os, obj, NULL) != 0) continue; - err = dmu_free_long_object(os, obj); + err = dmu_free_long_object(ra->os, obj); if (err != 0) return (err); } @@ -1585,49 +1590,37 @@ restore_freeobjects(struct restorearg *r } static int -restore_write(struct restorearg *ra, objset_t *os, - struct drr_write *drrw) +restore_write(struct restorearg *ra, struct drr_write *drrw, arc_buf_t *abuf) { dmu_tx_t *tx; - void *data; int err; if (drrw->drr_offset + drrw->drr_length < drrw->drr_offset || !DMU_OT_IS_VALID(drrw->drr_type)) return (SET_ERROR(EINVAL)); - if (dmu_object_info(os, drrw->drr_object, NULL) != 0) + if (dmu_object_info(ra->os, drrw->drr_object, NULL) != 0) return (SET_ERROR(EINVAL)); - dmu_buf_t *bonus; - if (dmu_bonus_hold(os, drrw->drr_object, FTAG, &bonus) != 0) - return (SET_ERROR(EINVAL)); - - arc_buf_t *abuf = dmu_request_arcbuf(bonus, drrw->drr_length); - - data = restore_read(ra, drrw->drr_length, abuf->b_data); - if (data == NULL) { - dmu_return_arcbuf(abuf); - dmu_buf_rele(bonus, FTAG); - return (ra->err); - } - - tx = dmu_tx_create(os); + tx = dmu_tx_create(ra->os); dmu_tx_hold_write(tx, drrw->drr_object, drrw->drr_offset, drrw->drr_length); err = dmu_tx_assign(tx, TXG_WAIT); if (err != 0) { - dmu_return_arcbuf(abuf); - dmu_buf_rele(bonus, FTAG); dmu_tx_abort(tx); return (err); } if (ra->byteswap) { dmu_object_byteswap_t byteswap = DMU_OT_BYTESWAP(drrw->drr_type); - dmu_ot_byteswap[byteswap].ob_func(data, drrw->drr_length); + dmu_ot_byteswap[byteswap].ob_func(abuf->b_data, + drrw->drr_length); } + + dmu_buf_t *bonus; + if (dmu_bonus_hold(ra->os, drrw->drr_object, FTAG, &bonus) != 0) + return (SET_ERROR(EINVAL)); dmu_assign_arcbuf(bonus, drrw->drr_offset, abuf, tx); dmu_tx_commit(tx); dmu_buf_rele(bonus, FTAG); @@ -1642,8 +1635,7 @@ restore_write(struct restorearg *ra, obj * data from the stream to fulfill this write. */ static int -restore_write_byref(struct restorearg *ra, objset_t *os, - struct drr_write_byref *drrwbr) +restore_write_byref(struct restorearg *ra, struct drr_write_byref *drrwbr) { dmu_tx_t *tx; int err; @@ -1669,7 +1661,7 @@ restore_write_byref(struct restorearg *r if (dmu_objset_from_ds(gmep->gme_ds, &ref_os)) return (SET_ERROR(EINVAL)); } else { - ref_os = os; + ref_os = ra->os; } err = dmu_buf_hold(ref_os, drrwbr->drr_refobject, @@ -1677,7 +1669,7 @@ restore_write_byref(struct restorearg *r if (err != 0) return (err); - tx = dmu_tx_create(os); + tx = dmu_tx_create(ra->os); dmu_tx_hold_write(tx, drrwbr->drr_object, drrwbr->drr_offset, drrwbr->drr_length); @@ -1686,7 +1678,7 @@ restore_write_byref(struct restorearg *r dmu_tx_abort(tx); return (err); } - dmu_write(os, drrwbr->drr_object, + dmu_write(ra->os, drrwbr->drr_object, drrwbr->drr_offset, drrwbr->drr_length, dbp->db_data, tx); dmu_buf_rele(dbp, FTAG); dmu_tx_commit(tx); @@ -1694,12 +1686,11 @@ restore_write_byref(struct restorearg *r } *** DIFF OUTPUT TRUNCATED AT 1000 LINES *** From owner-svn-src-stable@freebsd.org Sat Oct 3 07:34:22 2015 Return-Path: Delivered-To: svn-src-stable@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id A9CD8A0EACA; Sat, 3 Oct 2015 07:34:22 +0000 (UTC) (envelope-from mav@FreeBSD.org) Received: from repo.freebsd.org (repo.freebsd.org [IPv6:2001:1900:2254:2068::e6a:0]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 8FB661FDD; Sat, 3 Oct 2015 07:34:22 +0000 (UTC) (envelope-from mav@FreeBSD.org) Received: from repo.freebsd.org ([127.0.1.70]) by repo.freebsd.org (8.15.2/8.15.2) with ESMTP id t937YMP0068203; Sat, 3 Oct 2015 07:34:22 GMT (envelope-from mav@FreeBSD.org) Received: (from mav@localhost) by repo.freebsd.org (8.15.2/8.15.2/Submit) id t937YMlT068202; Sat, 3 Oct 2015 07:34:22 GMT (envelope-from mav@FreeBSD.org) Message-Id: <201510030734.t937YMlT068202@repo.freebsd.org> X-Authentication-Warning: repo.freebsd.org: mav set sender to mav@FreeBSD.org using -f From: Alexander Motin Date: Sat, 3 Oct 2015 07:34:22 +0000 (UTC) To: src-committers@freebsd.org, svn-src-all@freebsd.org, svn-src-stable@freebsd.org, svn-src-stable-10@freebsd.org Subject: svn commit: r288554 - stable/10/sys/cddl/contrib/opensolaris/uts/common/fs/zfs X-SVN-Group: stable-10 MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit X-BeenThere: svn-src-stable@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: SVN commit messages for all the -stable branches of the src tree List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 03 Oct 2015 07:34:22 -0000 Author: mav Date: Sat Oct 3 07:34:21 2015 New Revision: 288554 URL: https://svnweb.freebsd.org/changeset/base/288554 Log: MFC r286589: 5820 verify failed in zio_done(): BP_EQUAL(bp, io_bp_orig) Reviewed by: Alex Reece Reviewed by: George Wilson Reviewed by: Steven Hartland Approved by: Garrett D'Amore Author: Matthew Ahrens illumod/illumos-gate@34e8acef009195effafdcf6417aec385e241796e Modified: stable/10/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/dmu.c Directory Properties: stable/10/ (props changed) Modified: stable/10/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/dmu.c ============================================================================== --- stable/10/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/dmu.c Sat Oct 3 07:33:27 2015 (r288553) +++ stable/10/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/dmu.c Sat Oct 3 07:34:21 2015 (r288554) @@ -1653,19 +1653,32 @@ dmu_sync(zio_t *pio, uint64_t txg, dmu_s ASSERT(dr->dr_next == NULL || dr->dr_next->dr_txg < txg); /* - * Assume the on-disk data is X, the current syncing data is Y, - * and the current in-memory data is Z (currently in dmu_sync). - * X and Z are identical but Y is has been modified. Normally, - * when X and Z are the same we will perform a nopwrite but if Y - * is different we must disable nopwrite since the resulting write - * of Y to disk can free the block containing X. If we allowed a - * nopwrite to occur the block pointing to Z would reference a freed - * block. Since this is a rare case we simplify this by disabling - * nopwrite if the current dmu_sync-ing dbuf has been modified in - * a previous transaction. + * Assume the on-disk data is X, the current syncing data (in + * txg - 1) is Y, and the current in-memory data is Z (currently + * in dmu_sync). + * + * We usually want to perform a nopwrite if X and Z are the + * same. However, if Y is different (i.e. the BP is going to + * change before this write takes effect), then a nopwrite will + * be incorrect - we would override with X, which could have + * been freed when Y was written. + * + * (Note that this is not a concern when we are nop-writing from + * syncing context, because X and Y must be identical, because + * all previous txgs have been synced.) + * + * Therefore, we disable nopwrite if the current BP could change + * before this TXG. There are two ways it could change: by + * being dirty (dr_next is non-NULL), or by being freed + * (dnode_block_freed()). This behavior is verified by + * zio_done(), which VERIFYs that the override BP is identical + * to the on-disk BP. */ - if (dr->dr_next) + DB_DNODE_ENTER(db); + dn = DB_DNODE(db); + if (dr->dr_next != NULL || dnode_block_freed(dn, db->db_blkid)) zp.zp_nopwrite = B_FALSE; + DB_DNODE_EXIT(db); ASSERT(dr->dr_txg == txg); if (dr->dt.dl.dr_override_state == DR_IN_DMU_SYNC || From owner-svn-src-stable@freebsd.org Sat Oct 3 07:35:12 2015 Return-Path: Delivered-To: svn-src-stable@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 98C42A0EB79; Sat, 3 Oct 2015 07:35:12 +0000 (UTC) (envelope-from mav@FreeBSD.org) Received: from repo.freebsd.org (repo.freebsd.org [IPv6:2001:1900:2254:2068::e6a:0]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 7E2AD1131; Sat, 3 Oct 2015 07:35:12 +0000 (UTC) (envelope-from mav@FreeBSD.org) Received: from repo.freebsd.org ([127.0.1.70]) by repo.freebsd.org (8.15.2/8.15.2) with ESMTP id t937ZCE3068358; Sat, 3 Oct 2015 07:35:12 GMT (envelope-from mav@FreeBSD.org) Received: (from mav@localhost) by repo.freebsd.org (8.15.2/8.15.2/Submit) id t937ZCRK068357; Sat, 3 Oct 2015 07:35:12 GMT (envelope-from mav@FreeBSD.org) Message-Id: <201510030735.t937ZCRK068357@repo.freebsd.org> X-Authentication-Warning: repo.freebsd.org: mav set sender to mav@FreeBSD.org using -f From: Alexander Motin Date: Sat, 3 Oct 2015 07:35:12 +0000 (UTC) To: src-committers@freebsd.org, svn-src-all@freebsd.org, svn-src-stable@freebsd.org, svn-src-stable-10@freebsd.org Subject: svn commit: r288555 - stable/10/sys/cddl/contrib/opensolaris/uts/common/fs/zfs X-SVN-Group: stable-10 MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit X-BeenThere: svn-src-stable@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: SVN commit messages for all the -stable branches of the src tree List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 03 Oct 2015 07:35:12 -0000 Author: mav Date: Sat Oct 3 07:35:11 2015 New Revision: 288555 URL: https://svnweb.freebsd.org/changeset/base/288555 Log: MFC r286593: Local addition and mismerge fix for r286579. Modified: stable/10/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c Directory Properties: stable/10/ (props changed) Modified: stable/10/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c ============================================================================== --- stable/10/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c Sat Oct 3 07:34:21 2015 (r288554) +++ stable/10/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c Sat Oct 3 07:35:11 2015 (r288555) @@ -311,11 +311,12 @@ vdev_queue_class_tree(vdev_queue_t *vq, static inline avl_tree_t * vdev_queue_type_tree(vdev_queue_t *vq, zio_type_t t) { - ASSERT(t == ZIO_TYPE_READ || t == ZIO_TYPE_WRITE); if (t == ZIO_TYPE_READ) return (&vq->vq_read_offset_tree); - else + else if (t == ZIO_TYPE_WRITE) return (&vq->vq_write_offset_tree); + else + return (NULL); } int @@ -397,10 +398,13 @@ static void vdev_queue_io_add(vdev_queue_t *vq, zio_t *zio) { spa_t *spa = zio->io_spa; + avl_tree_t *qtt; ASSERT(MUTEX_HELD(&vq->vq_lock)); ASSERT3U(zio->io_priority, <, ZIO_PRIORITY_NUM_QUEUEABLE); avl_add(vdev_queue_class_tree(vq, zio->io_priority), zio); - avl_add(vdev_queue_type_tree(vq, zio->io_type), zio); + qtt = vdev_queue_type_tree(vq, zio->io_type); + if (qtt) + avl_add(qtt, zio); #ifdef illumos mutex_enter(&spa->spa_iokstat_lock); @@ -415,10 +419,13 @@ static void vdev_queue_io_remove(vdev_queue_t *vq, zio_t *zio) { spa_t *spa = zio->io_spa; + avl_tree_t *qtt; ASSERT(MUTEX_HELD(&vq->vq_lock)); ASSERT3U(zio->io_priority, <, ZIO_PRIORITY_NUM_QUEUEABLE); avl_remove(vdev_queue_class_tree(vq, zio->io_priority), zio); - avl_remove(vdev_queue_type_tree(vq, zio->io_type), zio); + qtt = vdev_queue_type_tree(vq, zio->io_type); + if (qtt) + avl_remove(qtt, zio); #ifdef illumos mutex_enter(&spa->spa_iokstat_lock); @@ -636,15 +643,6 @@ vdev_queue_aggregate(vdev_queue_t *vq, z if (zio->io_flags & ZIO_FLAG_DONT_AGGREGATE) return (NULL); - /* - * The synchronous i/o queues are not sorted by LBA, so we can't - * find adjacent i/os. These i/os tend to not be tightly clustered, - * or too large to aggregate, so this has little impact on performance. - */ - if (zio->io_priority == ZIO_PRIORITY_SYNC_READ || - zio->io_priority == ZIO_PRIORITY_SYNC_WRITE) - return (NULL); - first = last = zio; if (zio->io_type == ZIO_TYPE_READ) @@ -671,7 +669,7 @@ vdev_queue_aggregate(vdev_queue_t *vq, z */ flags = zio->io_flags & ZIO_FLAG_AGG_INHERIT; t = vdev_queue_type_tree(vq, zio->io_type); - while ((dio = AVL_PREV(t, first)) != NULL && + while (t != NULL && (dio = AVL_PREV(t, first)) != NULL && (dio->io_flags & ZIO_FLAG_AGG_INHERIT) == flags && IO_SPAN(dio, last) <= zfs_vdev_aggregation_limit && IO_GAP(dio, first) <= maxgap) { From owner-svn-src-stable@freebsd.org Sat Oct 3 07:43:34 2015 Return-Path: Delivered-To: svn-src-stable@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 76B63A0F29E; Sat, 3 Oct 2015 07:43:34 +0000 (UTC) (envelope-from alc@FreeBSD.org) Received: from repo.freebsd.org (repo.freebsd.org [IPv6:2001:1900:2254:2068::e6a:0]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 5929E17B5; Sat, 3 Oct 2015 07:43:34 +0000 (UTC) (envelope-from alc@FreeBSD.org) Received: from repo.freebsd.org ([127.0.1.70]) by repo.freebsd.org (8.15.2/8.15.2) with ESMTP id t937hYPD072945; Sat, 3 Oct 2015 07:43:34 GMT (envelope-from alc@FreeBSD.org) Received: (from alc@localhost) by repo.freebsd.org (8.15.2/8.15.2/Submit) id t937hY8Y072944; Sat, 3 Oct 2015 07:43:34 GMT (envelope-from alc@FreeBSD.org) Message-Id: <201510030743.t937hY8Y072944@repo.freebsd.org> X-Authentication-Warning: repo.freebsd.org: alc set sender to alc@FreeBSD.org using -f From: Alan Cox Date: Sat, 3 Oct 2015 07:43:34 +0000 (UTC) To: src-committers@freebsd.org, svn-src-all@freebsd.org, svn-src-stable@freebsd.org, svn-src-stable-10@freebsd.org Subject: svn commit: r288556 - stable/10/sys/vm X-SVN-Group: stable-10 MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit X-BeenThere: svn-src-stable@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: SVN commit messages for all the -stable branches of the src tree List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 03 Oct 2015 07:43:34 -0000 Author: alc Date: Sat Oct 3 07:43:33 2015 New Revision: 288556 URL: https://svnweb.freebsd.org/changeset/base/288556 Log: MFC r288281 The conversion of kmem_alloc_attr() from operating on a vm map to a vmem arena in r254025 introduced a bug in the case when an allocation is only partially successful. Specifically, the vm object lock was not being acquired before freeing the allocated pages. To address this bug, replace the existing code by a call to kmem_unback(). Change the type of a variable in kmem_alloc_attr() so that an allocation of two or more gigabytes won't fail. Replace the error handling code in kmem_back() by a call to kmem_unback(). Modified: stable/10/sys/vm/vm_kern.c Directory Properties: stable/10/ (props changed) Modified: stable/10/sys/vm/vm_kern.c ============================================================================== --- stable/10/sys/vm/vm_kern.c Sat Oct 3 07:35:11 2015 (r288555) +++ stable/10/sys/vm/vm_kern.c Sat Oct 3 07:43:33 2015 (r288556) @@ -159,11 +159,10 @@ kmem_alloc_attr(vmem_t *vmem, vm_size_t vm_paddr_t high, vm_memattr_t memattr) { vm_object_t object = vmem == kmem_arena ? kmem_object : kernel_object; - vm_offset_t addr; + vm_offset_t addr, i; vm_ooffset_t offset; vm_page_t m; int pflags, tries; - int i; size = round_page(size); if (vmem_alloc(vmem, size, M_BESTFIT | flags, &addr)) @@ -184,18 +183,7 @@ retry: tries++; goto retry; } - /* - * Unmap and free the pages. - */ - if (i != 0) - pmap_remove(kernel_pmap, addr, addr + i); - while (i != 0) { - i -= PAGE_SIZE; - m = vm_page_lookup(object, - OFF_TO_IDX(offset + i)); - vm_page_unwire(m, 0); - vm_page_free(m); - } + kmem_unback(object, addr, i); vmem_free(vmem, addr, size); return (0); } @@ -353,25 +341,13 @@ retry: * aren't on any queues. */ if (m == NULL) { + VM_OBJECT_WUNLOCK(object); if ((flags & M_NOWAIT) == 0) { - VM_OBJECT_WUNLOCK(object); VM_WAIT; VM_OBJECT_WLOCK(object); goto retry; } - /* - * Unmap and free the pages. - */ - if (i != 0) - pmap_remove(kernel_pmap, addr, addr + i); - while (i != 0) { - i -= PAGE_SIZE; - m = vm_page_lookup(object, - OFF_TO_IDX(offset + i)); - vm_page_unwire(m, 0); - vm_page_free(m); - } - VM_OBJECT_WUNLOCK(object); + kmem_unback(object, addr, i); return (KERN_NO_SPACE); } if (flags & M_ZERO && (m->flags & PG_ZERO) == 0) @@ -387,6 +363,15 @@ retry: return (KERN_SUCCESS); } +/* + * kmem_unback: + * + * Unmap and free the physical pages underlying the specified virtual + * address range. + * + * A physical page must exist within the specified object at each index + * that is being unmapped. + */ void kmem_unback(vm_object_t object, vm_offset_t addr, vm_size_t size) { From owner-svn-src-stable@freebsd.org Sat Oct 3 07:45:13 2015 Return-Path: Delivered-To: svn-src-stable@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 946E6A0F447; Sat, 3 Oct 2015 07:45:13 +0000 (UTC) (envelope-from mav@FreeBSD.org) Received: from repo.freebsd.org (repo.freebsd.org [IPv6:2001:1900:2254:2068::e6a:0]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 83C7F194C; Sat, 3 Oct 2015 07:45:13 +0000 (UTC) (envelope-from mav@FreeBSD.org) Received: from repo.freebsd.org ([127.0.1.70]) by repo.freebsd.org (8.15.2/8.15.2) with ESMTP id t937jDKD073205; Sat, 3 Oct 2015 07:45:13 GMT (envelope-from mav@FreeBSD.org) Received: (from mav@localhost) by repo.freebsd.org (8.15.2/8.15.2/Submit) id t937jDOg073204; Sat, 3 Oct 2015 07:45:13 GMT (envelope-from mav@FreeBSD.org) Message-Id: <201510030745.t937jDOg073204@repo.freebsd.org> X-Authentication-Warning: repo.freebsd.org: mav set sender to mav@FreeBSD.org using -f From: Alexander Motin Date: Sat, 3 Oct 2015 07:45:13 +0000 (UTC) To: src-committers@freebsd.org, svn-src-all@freebsd.org, svn-src-stable@freebsd.org, svn-src-stable-10@freebsd.org Subject: svn commit: r288557 - stable/10/sys/cddl/contrib/opensolaris/uts/common/fs/zfs X-SVN-Group: stable-10 MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit X-BeenThere: svn-src-stable@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: SVN commit messages for all the -stable branches of the src tree List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 03 Oct 2015 07:45:13 -0000 Author: mav Date: Sat Oct 3 07:45:12 2015 New Revision: 288557 URL: https://svnweb.freebsd.org/changeset/base/288557 Log: MFC r286598: 5701 zpool list reports incorrect "alloc" value for cache devices Modified: stable/10/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/arc.c Directory Properties: stable/10/ (props changed) Modified: stable/10/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/arc.c ============================================================================== --- stable/10/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/arc.c Sat Oct 3 07:43:33 2015 (r288556) +++ stable/10/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/arc.c Sat Oct 3 07:45:12 2015 (r288557) @@ -965,6 +965,16 @@ uint64_t zfs_crc64_table[256]; #define L2ARC_FEED_SECS 1 /* caching interval secs */ #define L2ARC_FEED_MIN_MS 200 /* min caching interval ms */ +/* + * Used to distinguish headers that are being process by + * l2arc_write_buffers(), but have yet to be assigned to a l2arc disk + * address. This can happen when the header is added to the l2arc's list + * of buffers to write in the first stage of l2arc_write_buffers(), but + * has not yet been written out which happens in the second stage of + * l2arc_write_buffers(). + */ +#define L2ARC_ADDR_UNSET ((uint64_t)(-1)) + #define l2arc_writes_sent ARCSTAT(arcstat_l2_writes_sent) #define l2arc_writes_done ARCSTAT(arcstat_l2_writes_done) @@ -1048,12 +1058,12 @@ struct l2arc_dev { uint64_t l2ad_hand; /* next write location */ uint64_t l2ad_start; /* first addr on device */ uint64_t l2ad_end; /* last addr on device */ - uint64_t l2ad_evict; /* last addr eviction reached */ boolean_t l2ad_first; /* first sweep through */ boolean_t l2ad_writing; /* currently writing */ kmutex_t l2ad_mtx; /* lock for buffer list */ list_t l2ad_buflist; /* buffer list */ list_node_t l2ad_node; /* device list node */ + refcount_t l2ad_alloc; /* allocated bytes */ }; static list_t L2ARC_dev_list; /* device list */ @@ -1425,6 +1435,7 @@ arc_hdr_realloc(arc_buf_hdr_t *hdr, kmem buf_hash_remove(hdr); bcopy(hdr, nhdr, HDR_L2ONLY_SIZE); + if (new == hdr_full_cache) { nhdr->b_flags |= ARC_FLAG_HAS_L1HDR; /* @@ -1468,6 +1479,20 @@ arc_hdr_realloc(arc_buf_hdr_t *hdr, kmem mutex_exit(&dev->l2ad_mtx); + /* + * Since we're using the pointer address as the tag when + * incrementing and decrementing the l2ad_alloc refcount, we + * must remove the old pointer (that we're about to destroy) and + * add the new pointer to the refcount. Otherwise we'd remove + * the wrong pointer address when calling arc_hdr_destroy() later. + */ + + (void) refcount_remove_many(&dev->l2ad_alloc, + hdr->b_l2hdr.b_asize, hdr); + + (void) refcount_add_many(&dev->l2ad_alloc, + nhdr->b_l2hdr.b_asize, nhdr); + buf_discard_identity(hdr); hdr->b_freeze_cksum = NULL; kmem_cache_free(old, hdr); @@ -2219,6 +2244,57 @@ arc_buf_destroy(arc_buf_t *buf, boolean_ } static void +arc_hdr_l2hdr_destroy(arc_buf_hdr_t *hdr) +{ + l2arc_buf_hdr_t *l2hdr = &hdr->b_l2hdr; + l2arc_dev_t *dev = l2hdr->b_dev; + + ASSERT(MUTEX_HELD(&dev->l2ad_mtx)); + ASSERT(HDR_HAS_L2HDR(hdr)); + + list_remove(&dev->l2ad_buflist, hdr); + + /* + * We don't want to leak the b_tmp_cdata buffer that was + * allocated in l2arc_write_buffers() + */ + arc_buf_l2_cdata_free(hdr); + + /* + * If the l2hdr's b_daddr is equal to L2ARC_ADDR_UNSET, then + * this header is being processed by l2arc_write_buffers() (i.e. + * it's in the first stage of l2arc_write_buffers()). + * Re-affirming that truth here, just to serve as a reminder. If + * b_daddr does not equal L2ARC_ADDR_UNSET, then the header may or + * may not have its HDR_L2_WRITING flag set. (the write may have + * completed, in which case HDR_L2_WRITING will be false and the + * b_daddr field will point to the address of the buffer on disk). + */ + IMPLY(l2hdr->b_daddr == L2ARC_ADDR_UNSET, HDR_L2_WRITING(hdr)); + + /* + * If b_daddr is equal to L2ARC_ADDR_UNSET, we're racing with + * l2arc_write_buffers(). Since we've just removed this header + * from the l2arc buffer list, this header will never reach the + * second stage of l2arc_write_buffers(), which increments the + * accounting stats for this header. Thus, we must be careful + * not to decrement them for this header either. + */ + if (l2hdr->b_daddr != L2ARC_ADDR_UNSET) { + ARCSTAT_INCR(arcstat_l2_asize, -l2hdr->b_asize); + ARCSTAT_INCR(arcstat_l2_size, -hdr->b_size); + + vdev_space_update(dev->l2ad_vdev, + -l2hdr->b_asize, 0, 0); + + (void) refcount_remove_many(&dev->l2ad_alloc, + l2hdr->b_asize, hdr); + } + + hdr->b_flags &= ~ARC_FLAG_HAS_L2HDR; +} + +static void arc_hdr_destroy(arc_buf_hdr_t *hdr) { if (HDR_HAS_L1HDR(hdr)) { @@ -2231,31 +2307,29 @@ arc_hdr_destroy(arc_buf_hdr_t *hdr) ASSERT(!HDR_IN_HASH_TABLE(hdr)); if (HDR_HAS_L2HDR(hdr)) { - l2arc_buf_hdr_t *l2hdr = &hdr->b_l2hdr; - boolean_t buflist_held = MUTEX_HELD(&l2hdr->b_dev->l2ad_mtx); - - if (!buflist_held) { - mutex_enter(&l2hdr->b_dev->l2ad_mtx); - l2hdr = &hdr->b_l2hdr; - } + l2arc_dev_t *dev = hdr->b_l2hdr.b_dev; + boolean_t buflist_held = MUTEX_HELD(&dev->l2ad_mtx); - trim_map_free(l2hdr->b_dev->l2ad_vdev, l2hdr->b_daddr, - l2hdr->b_asize, 0); - list_remove(&l2hdr->b_dev->l2ad_buflist, hdr); + if (!buflist_held) + mutex_enter(&dev->l2ad_mtx); /* - * We don't want to leak the b_tmp_cdata buffer that was - * allocated in l2arc_write_buffers() - */ - arc_buf_l2_cdata_free(hdr); - - ARCSTAT_INCR(arcstat_l2_size, -hdr->b_size); - ARCSTAT_INCR(arcstat_l2_asize, -l2hdr->b_asize); + * Even though we checked this conditional above, we + * need to check this again now that we have the + * l2ad_mtx. This is because we could be racing with + * another thread calling l2arc_evict() which might have + * destroyed this header's L2 portion as we were waiting + * to acquire the l2ad_mtx. If that happens, we don't + * want to re-destroy the header's L2 portion. + */ + if (HDR_HAS_L2HDR(hdr)) { + trim_map_free(dev->l2ad_vdev, hdr->b_l2hdr.b_daddr, + hdr->b_l2hdr.b_asize, 0); + arc_hdr_l2hdr_destroy(hdr); + } if (!buflist_held) - mutex_exit(&l2hdr->b_dev->l2ad_mtx); - - hdr->b_flags &= ~ARC_FLAG_HAS_L2HDR; + mutex_exit(&dev->l2ad_mtx); } if (!BUF_EMPTY(hdr)) @@ -4274,23 +4348,23 @@ arc_release(arc_buf_t *buf, void *tag) ASSERT(refcount_count(&hdr->b_l1hdr.b_refcnt) > 0); if (HDR_HAS_L2HDR(hdr)) { - ARCSTAT_INCR(arcstat_l2_asize, -hdr->b_l2hdr.b_asize); - ARCSTAT_INCR(arcstat_l2_size, -hdr->b_size); - mutex_enter(&hdr->b_l2hdr.b_dev->l2ad_mtx); - trim_map_free(hdr->b_l2hdr.b_dev->l2ad_vdev, - hdr->b_l2hdr.b_daddr, hdr->b_l2hdr.b_asize, 0); - list_remove(&hdr->b_l2hdr.b_dev->l2ad_buflist, hdr); /* - * We don't want to leak the b_tmp_cdata buffer that was - * allocated in l2arc_write_buffers() + * We have to recheck this conditional again now that + * we're holding the l2ad_mtx to prevent a race with + * another thread which might be concurrently calling + * l2arc_evict(). In that case, l2arc_evict() might have + * destroyed the header's L2 portion as we were waiting + * to acquire the l2ad_mtx. */ - arc_buf_l2_cdata_free(hdr); + if (HDR_HAS_L2HDR(hdr)) { + trim_map_free(hdr->b_l2hdr.b_dev->l2ad_vdev, + hdr->b_l2hdr.b_daddr, hdr->b_l2hdr.b_asize, 0); + arc_hdr_l2hdr_destroy(hdr); + } mutex_exit(&hdr->b_l2hdr.b_dev->l2ad_mtx); - - hdr->b_flags &= ~ARC_FLAG_HAS_L2HDR; } /* @@ -5358,6 +5432,10 @@ l2arc_write_done(zio_t *zio) ARCSTAT_INCR(arcstat_l2_asize, -hdr->b_l2hdr.b_asize); ARCSTAT_INCR(arcstat_l2_size, -hdr->b_size); + + bytes_dropped += hdr->b_l2hdr.b_asize; + (void) refcount_remove_many(&dev->l2ad_alloc, + hdr->b_l2hdr.b_asize, hdr); } /* @@ -5514,7 +5592,6 @@ l2arc_evict(l2arc_dev_t *dev, uint64_t d arc_buf_hdr_t *hdr, *hdr_prev; kmutex_t *hash_lock; uint64_t taddr; - int64_t bytes_evicted = 0; buflist = &dev->l2ad_buflist; @@ -5599,21 +5676,11 @@ top: hdr->b_flags |= ARC_FLAG_L2_EVICTED; } - /* Tell ARC this no longer exists in L2ARC. */ - ARCSTAT_INCR(arcstat_l2_asize, -hdr->b_l2hdr.b_asize); - ARCSTAT_INCR(arcstat_l2_size, -hdr->b_size); - hdr->b_flags &= ~ARC_FLAG_HAS_L2HDR; - list_remove(buflist, hdr); - - /* This may have been leftover after a failed write. */ - hdr->b_flags &= ~ARC_FLAG_L2_WRITING; + arc_hdr_l2hdr_destroy(hdr); } mutex_exit(hash_lock); } mutex_exit(&dev->l2ad_mtx); - - vdev_space_update(dev->l2ad_vdev, -bytes_evicted, 0, 0); - dev->l2ad_evict = taddr; } /* @@ -5773,6 +5840,28 @@ l2arc_write_buffers(spa_t *spa, l2arc_de hdr->b_l2hdr.b_asize = hdr->b_size; hdr->b_l1hdr.b_tmp_cdata = hdr->b_l1hdr.b_buf->b_data; + /* + * Explicitly set the b_daddr field to a known + * value which means "invalid address". This + * enables us to differentiate which stage of + * l2arc_write_buffers() the particular header + * is in (e.g. this loop, or the one below). + * ARC_FLAG_L2_WRITING is not enough to make + * this distinction, and we need to know in + * order to do proper l2arc vdev accounting in + * arc_release() and arc_hdr_destroy(). + * + * Note, we can't use a new flag to distinguish + * the two stages because we don't hold the + * header's hash_lock below, in the second stage + * of this function. Thus, we can't simply + * change the b_flags field to denote that the + * IO has been sent. We can change the b_daddr + * field of the L2 portion, though, since we'll + * be holding the l2ad_mtx; which is why we're + * using it to denote the header's state change. + */ + hdr->b_l2hdr.b_daddr = L2ARC_ADDR_UNSET; hdr->b_flags |= ARC_FLAG_HAS_L2HDR; list_insert_head(&dev->l2ad_buflist, hdr); @@ -5862,6 +5951,13 @@ l2arc_write_buffers(spa_t *spa, l2arc_de if (!L2ARC_IS_VALID_COMPRESS(HDR_GET_COMPRESS(hdr))) hdr->b_l1hdr.b_tmp_cdata = NULL; + /* + * We need to do this regardless if buf_sz is zero or + * not, otherwise, when this l2hdr is evicted we'll + * remove a reference that was never added. + */ + (void) refcount_add_many(&dev->l2ad_alloc, buf_sz, hdr); + /* Compression may have squashed the buffer to zero length. */ if (buf_sz != 0) { uint64_t buf_a_sz; @@ -5876,6 +5972,7 @@ l2arc_write_buffers(spa_t *spa, l2arc_de (void) zio_nowait(wzio); stats_size += buf_sz; + /* * Keep the clock hand suitably device-aligned. */ @@ -5900,7 +5997,6 @@ l2arc_write_buffers(spa_t *spa, l2arc_de */ if (dev->l2ad_hand >= (dev->l2ad_end - target_sz)) { dev->l2ad_hand = dev->l2ad_start; - dev->l2ad_evict = dev->l2ad_start; dev->l2ad_first = B_FALSE; } @@ -6207,7 +6303,6 @@ l2arc_add_vdev(spa_t *spa, vdev_t *vd) adddev->l2ad_start = VDEV_LABEL_START_SIZE; adddev->l2ad_end = VDEV_LABEL_START_SIZE + vdev_get_min_asize(vd); adddev->l2ad_hand = adddev->l2ad_start; - adddev->l2ad_evict = adddev->l2ad_start; adddev->l2ad_first = B_TRUE; adddev->l2ad_writing = B_FALSE; @@ -6220,6 +6315,7 @@ l2arc_add_vdev(spa_t *spa, vdev_t *vd) offsetof(arc_buf_hdr_t, b_l2hdr.b_l2node)); vdev_space_update(vd, 0, 0, adddev->l2ad_end - adddev->l2ad_hand); + refcount_create(&adddev->l2ad_alloc); /* * Add device to global list @@ -6265,6 +6361,7 @@ l2arc_remove_vdev(vdev_t *vd) l2arc_evict(remdev, 0, B_TRUE); list_destroy(&remdev->l2ad_buflist); mutex_destroy(&remdev->l2ad_mtx); + refcount_destroy(&remdev->l2ad_alloc); kmem_free(remdev, sizeof (l2arc_dev_t)); } From owner-svn-src-stable@freebsd.org Sat Oct 3 07:46:04 2015 Return-Path: Delivered-To: svn-src-stable@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 4C7BEA0F546; Sat, 3 Oct 2015 07:46:04 +0000 (UTC) (envelope-from mav@FreeBSD.org) Received: from repo.freebsd.org (repo.freebsd.org [IPv6:2001:1900:2254:2068::e6a:0]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 3CE801AAE; Sat, 3 Oct 2015 07:46:04 +0000 (UTC) (envelope-from mav@FreeBSD.org) Received: from repo.freebsd.org ([127.0.1.70]) by repo.freebsd.org (8.15.2/8.15.2) with ESMTP id t937k4QL073355; Sat, 3 Oct 2015 07:46:04 GMT (envelope-from mav@FreeBSD.org) Received: (from mav@localhost) by repo.freebsd.org (8.15.2/8.15.2/Submit) id t937k4Iu073354; Sat, 3 Oct 2015 07:46:04 GMT (envelope-from mav@FreeBSD.org) Message-Id: <201510030746.t937k4Iu073354@repo.freebsd.org> X-Authentication-Warning: repo.freebsd.org: mav set sender to mav@FreeBSD.org using -f From: Alexander Motin Date: Sat, 3 Oct 2015 07:46:04 +0000 (UTC) To: src-committers@freebsd.org, svn-src-all@freebsd.org, svn-src-stable@freebsd.org, svn-src-stable-10@freebsd.org Subject: svn commit: r288558 - stable/10/sys/cddl/contrib/opensolaris/uts/common/fs/zfs X-SVN-Group: stable-10 MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit X-BeenThere: svn-src-stable@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: SVN commit messages for all the -stable branches of the src tree List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 03 Oct 2015 07:46:04 -0000 Author: mav Date: Sat Oct 3 07:46:03 2015 New Revision: 288558 URL: https://svnweb.freebsd.org/changeset/base/288558 Log: MFC r286600: 5808 spa_check_logs is not necessary on readonly pools Reviewed by: George Wilson Reviewed by: Paul Dagnelie Reviewed by: Simon Klinkert Reviewed by: Will Andrews Approved by: Gordon Ross Author: Matthew Ahrens illumos/illumos-gate@23367a2f2caec1ccb4d918bdd0f2fc2c9cadcd06 Modified: stable/10/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/spa.c Directory Properties: stable/10/ (props changed) Modified: stable/10/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/spa.c ============================================================================== --- stable/10/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/spa.c Sat Oct 3 07:45:12 2015 (r288557) +++ stable/10/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/spa.c Sat Oct 3 07:46:03 2015 (r288558) @@ -2752,7 +2752,7 @@ spa_load_impl(spa_t *spa, uint64_t pool_ if (rvd->vdev_state <= VDEV_STATE_CANT_OPEN) return (SET_ERROR(ENXIO)); - if (spa_check_logs(spa)) { + if (spa_writeable(spa) && spa_check_logs(spa)) { *ereport = FM_EREPORT_ZFS_LOG_REPLAY; return (spa_vdev_err(rvd, VDEV_AUX_BAD_LOG, ENXIO)); } From owner-svn-src-stable@freebsd.org Sat Oct 3 07:47:35 2015 Return-Path: Delivered-To: svn-src-stable@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 0166BA0F69A; Sat, 3 Oct 2015 07:47:35 +0000 (UTC) (envelope-from mav@FreeBSD.org) Received: from repo.freebsd.org (repo.freebsd.org [IPv6:2001:1900:2254:2068::e6a:0]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id D9B4C1C2E; Sat, 3 Oct 2015 07:47:34 +0000 (UTC) (envelope-from mav@FreeBSD.org) Received: from repo.freebsd.org ([127.0.1.70]) by repo.freebsd.org (8.15.2/8.15.2) with ESMTP id t937lY8D073573; Sat, 3 Oct 2015 07:47:34 GMT (envelope-from mav@FreeBSD.org) Received: (from mav@localhost) by repo.freebsd.org (8.15.2/8.15.2/Submit) id t937lYco073568; Sat, 3 Oct 2015 07:47:34 GMT (envelope-from mav@FreeBSD.org) Message-Id: <201510030747.t937lYco073568@repo.freebsd.org> X-Authentication-Warning: repo.freebsd.org: mav set sender to mav@FreeBSD.org using -f From: Alexander Motin Date: Sat, 3 Oct 2015 07:47:34 +0000 (UTC) To: src-committers@freebsd.org, svn-src-all@freebsd.org, svn-src-stable@freebsd.org, svn-src-stable-10@freebsd.org Subject: svn commit: r288559 - in stable/10: cddl/contrib/opensolaris/cmd/zdb sys/cddl/contrib/opensolaris/uts/common/fs/zfs sys/cddl/contrib/opensolaris/uts/common/fs/zfs/sys X-SVN-Group: stable-10 MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit X-BeenThere: svn-src-stable@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: SVN commit messages for all the -stable branches of the src tree List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 03 Oct 2015 07:47:35 -0000 Author: mav Date: Sat Oct 3 07:47:33 2015 New Revision: 288559 URL: https://svnweb.freebsd.org/changeset/base/288559 Log: MFC r286603: 5810 zdb should print details of bpobj Reviewed by: Prakash Surya Reviewed by: Alex Reece Reviewed by: George Wilson Reviewed by: Will Andrews Reviewed by: Simon Klinkert Approved by: Gordon Ross Author: Matthew Ahrens illumos/illumos-gate@732885fca09e11183dd0ea69aaaab5588fb7dbff Modified: stable/10/cddl/contrib/opensolaris/cmd/zdb/zdb.c stable/10/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/bpobj.c stable/10/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/sys/bpobj.h Directory Properties: stable/10/ (props changed) Modified: stable/10/cddl/contrib/opensolaris/cmd/zdb/zdb.c ============================================================================== --- stable/10/cddl/contrib/opensolaris/cmd/zdb/zdb.c Sat Oct 3 07:46:03 2015 (r288558) +++ stable/10/cddl/contrib/opensolaris/cmd/zdb/zdb.c Sat Oct 3 07:47:33 2015 (r288559) @@ -95,6 +95,8 @@ int zopt_objects = 0; libzfs_handle_t *g_zfs; uint64_t max_inflight = 1000; +static void snprintf_blkptr_compact(char *, size_t, const blkptr_t *); + /* * These libumem hooks provide a reasonable set of defaults for the allocator's * debugging facilities. @@ -418,6 +420,79 @@ dump_zap(objset_t *os, uint64_t object, zap_cursor_fini(&zc); } +static void +dump_bpobj(objset_t *os, uint64_t object, void *data, size_t size) +{ + bpobj_phys_t *bpop = data; + char bytes[32], comp[32], uncomp[32]; + + if (bpop == NULL) + return; + + zdb_nicenum(bpop->bpo_bytes, bytes); + zdb_nicenum(bpop->bpo_comp, comp); + zdb_nicenum(bpop->bpo_uncomp, uncomp); + + (void) printf("\t\tnum_blkptrs = %llu\n", + (u_longlong_t)bpop->bpo_num_blkptrs); + (void) printf("\t\tbytes = %s\n", bytes); + if (size >= BPOBJ_SIZE_V1) { + (void) printf("\t\tcomp = %s\n", comp); + (void) printf("\t\tuncomp = %s\n", uncomp); + } + if (size >= sizeof (*bpop)) { + (void) printf("\t\tsubobjs = %llu\n", + (u_longlong_t)bpop->bpo_subobjs); + (void) printf("\t\tnum_subobjs = %llu\n", + (u_longlong_t)bpop->bpo_num_subobjs); + } + + if (dump_opt['d'] < 5) + return; + + for (uint64_t i = 0; i < bpop->bpo_num_blkptrs; i++) { + char blkbuf[BP_SPRINTF_LEN]; + blkptr_t bp; + + int err = dmu_read(os, object, + i * sizeof (bp), sizeof (bp), &bp, 0); + if (err != 0) { + (void) printf("got error %u from dmu_read\n", err); + break; + } + snprintf_blkptr_compact(blkbuf, sizeof (blkbuf), &bp); + (void) printf("\t%s\n", blkbuf); + } +} + +/* ARGSUSED */ +static void +dump_bpobj_subobjs(objset_t *os, uint64_t object, void *data, size_t size) +{ + dmu_object_info_t doi; + + VERIFY0(dmu_object_info(os, object, &doi)); + uint64_t *subobjs = kmem_alloc(doi.doi_max_offset, KM_SLEEP); + + int err = dmu_read(os, object, 0, doi.doi_max_offset, subobjs, 0); + if (err != 0) { + (void) printf("got error %u from dmu_read\n", err); + kmem_free(subobjs, doi.doi_max_offset); + return; + } + + int64_t last_nonzero = -1; + for (uint64_t i = 0; i < doi.doi_max_offset / 8; i++) { + if (subobjs[i] != 0) + last_nonzero = i; + } + + for (int64_t i = 0; i <= last_nonzero; i++) { + (void) printf("\t%llu\n", (longlong_t)subobjs[i]); + } + kmem_free(subobjs, doi.doi_max_offset); +} + /*ARGSUSED*/ static void dump_ddt_zap(objset_t *os, uint64_t object, void *data, size_t size) @@ -1397,7 +1472,7 @@ dump_bpobj_cb(void *arg, const blkptr_t } static void -dump_bpobj(bpobj_t *bpo, char *name, int indent) +dump_full_bpobj(bpobj_t *bpo, char *name, int indent) { char bytes[32]; char comp[32]; @@ -1411,11 +1486,12 @@ dump_bpobj(bpobj_t *bpo, char *name, int zdb_nicenum(bpo->bpo_phys->bpo_comp, comp); zdb_nicenum(bpo->bpo_phys->bpo_uncomp, uncomp); (void) printf(" %*s: object %llu, %llu local blkptrs, " - "%llu subobjs, %s (%s/%s comp)\n", + "%llu subobjs in object %llu, %s (%s/%s comp)\n", indent * 8, name, (u_longlong_t)bpo->bpo_object, (u_longlong_t)bpo->bpo_phys->bpo_num_blkptrs, (u_longlong_t)bpo->bpo_phys->bpo_num_subobjs, + (u_longlong_t)bpo->bpo_phys->bpo_subobjs, bytes, comp, uncomp); for (uint64_t i = 0; i < bpo->bpo_phys->bpo_num_subobjs; i++) { @@ -1432,7 +1508,7 @@ dump_bpobj(bpobj_t *bpo, char *name, int error, (u_longlong_t)subobj); continue; } - dump_bpobj(&subbpo, "subobj", indent + 1); + dump_full_bpobj(&subbpo, "subobj", indent + 1); bpobj_close(&subbpo); } } else { @@ -1466,7 +1542,7 @@ dump_deadlist(dsl_deadlist_t *dl) return; if (dl->dl_oldfmt) { - dump_bpobj(&dl->dl_bpobj, "old-format deadlist", 0); + dump_full_bpobj(&dl->dl_bpobj, "old-format deadlist", 0); return; } @@ -1491,8 +1567,7 @@ dump_deadlist(dsl_deadlist_t *dl) (void) snprintf(buf, sizeof (buf), "mintxg %llu -> ", (longlong_t)dle->dle_mintxg, (longlong_t)dle->dle_bpobj.bpo_object); - - dump_bpobj(&dle->dle_bpobj, buf, 0); + dump_full_bpobj(&dle->dle_bpobj, buf, 0); } else { (void) printf("mintxg %llu -> obj %llu\n", (longlong_t)dle->dle_mintxg, @@ -1684,8 +1759,8 @@ static object_viewer_t *object_viewer[DM dump_uint64, /* object array */ dump_none, /* packed nvlist */ dump_packed_nvlist, /* packed nvlist size */ - dump_none, /* bplist */ - dump_none, /* bplist header */ + dump_none, /* bpobj */ + dump_bpobj, /* bpobj header */ dump_none, /* SPA space map header */ dump_none, /* SPA space map */ dump_none, /* ZIL intent log */ @@ -1732,7 +1807,7 @@ static object_viewer_t *object_viewer[DM dump_zap, /* deadlist */ dump_none, /* deadlist hdr */ dump_zap, /* dsl clones */ - dump_none, /* bpobj subobjs */ + dump_bpobj_subobjs, /* bpobj subobjs */ dump_unknown, /* Unknown type, must be last */ }; @@ -2961,10 +3036,11 @@ dump_zpool(spa_t *spa) uint64_t refcount; dump_dir(dp->dp_meta_objset); if (dump_opt['d'] >= 3) { - dump_bpobj(&spa->spa_deferred_bpobj, + dump_full_bpobj(&spa->spa_deferred_bpobj, "Deferred frees", 0); if (spa_version(spa) >= SPA_VERSION_DEADLISTS) { - dump_bpobj(&spa->spa_dsl_pool->dp_free_bpobj, + dump_full_bpobj( + &spa->spa_dsl_pool->dp_free_bpobj, "Pool snapshot frees", 0); } Modified: stable/10/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/bpobj.c ============================================================================== --- stable/10/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/bpobj.c Sat Oct 3 07:46:03 2015 (r288558) +++ stable/10/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/bpobj.c Sat Oct 3 07:47:33 2015 (r288559) @@ -20,7 +20,7 @@ */ /* * Copyright (c) 2005, 2010, Oracle and/or its affiliates. All rights reserved. - * Copyright (c) 2013 by Delphix. All rights reserved. + * Copyright (c) 2011, 2014 by Delphix. All rights reserved. */ #include @@ -256,9 +256,8 @@ bpobj_iterate_impl(bpobj_t *bpo, bpobj_i dbuf = NULL; } if (free) { - i++; VERIFY3U(0, ==, dmu_free_range(bpo->bpo_os, bpo->bpo_object, - i * sizeof (blkptr_t), -1ULL, tx)); + (i + 1) * sizeof (blkptr_t), -1ULL, tx)); } if (err || !bpo->bpo_havesubobj || bpo->bpo_phys->bpo_subobjs == 0) goto out; Modified: stable/10/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/sys/bpobj.h ============================================================================== --- stable/10/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/sys/bpobj.h Sat Oct 3 07:46:03 2015 (r288558) +++ stable/10/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/sys/bpobj.h Sat Oct 3 07:47:33 2015 (r288559) @@ -20,7 +20,7 @@ */ /* * Copyright (c) 2010, Oracle and/or its affiliates. All rights reserved. - * Copyright (c) 2012 by Delphix. All rights reserved. + * Copyright (c) 2012, 2014 by Delphix. All rights reserved. */ #ifndef _SYS_BPOBJ_H @@ -77,7 +77,6 @@ void bpobj_close(bpobj_t *bpo); int bpobj_iterate(bpobj_t *bpo, bpobj_itor_t func, void *arg, dmu_tx_t *tx); int bpobj_iterate_nofree(bpobj_t *bpo, bpobj_itor_t func, void *, dmu_tx_t *); -int bpobj_iterate_dbg(bpobj_t *bpo, uint64_t *itorp, blkptr_t *bp); void bpobj_enqueue_subobj(bpobj_t *bpo, uint64_t subobj, dmu_tx_t *tx); void bpobj_enqueue(bpobj_t *bpo, const blkptr_t *bp, dmu_tx_t *tx); From owner-svn-src-stable@freebsd.org Sat Oct 3 07:48:26 2015 Return-Path: Delivered-To: svn-src-stable@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 9ECDCA0F755; Sat, 3 Oct 2015 07:48:26 +0000 (UTC) (envelope-from mav@FreeBSD.org) Received: from repo.freebsd.org (repo.freebsd.org [IPv6:2001:1900:2254:2068::e6a:0]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 8F0071D80; Sat, 3 Oct 2015 07:48:26 +0000 (UTC) (envelope-from mav@FreeBSD.org) Received: from repo.freebsd.org ([127.0.1.70]) by repo.freebsd.org (8.15.2/8.15.2) with ESMTP id t937mQ5M073718; Sat, 3 Oct 2015 07:48:26 GMT (envelope-from mav@FreeBSD.org) Received: (from mav@localhost) by repo.freebsd.org (8.15.2/8.15.2/Submit) id t937mQqv073717; Sat, 3 Oct 2015 07:48:26 GMT (envelope-from mav@FreeBSD.org) Message-Id: <201510030748.t937mQqv073717@repo.freebsd.org> X-Authentication-Warning: repo.freebsd.org: mav set sender to mav@FreeBSD.org using -f From: Alexander Motin Date: Sat, 3 Oct 2015 07:48:26 +0000 (UTC) To: src-committers@freebsd.org, svn-src-all@freebsd.org, svn-src-stable@freebsd.org, svn-src-stable-10@freebsd.org Subject: svn commit: r288560 - stable/10/sys/cddl/contrib/opensolaris/uts/common/fs/zfs X-SVN-Group: stable-10 MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit X-BeenThere: svn-src-stable@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: SVN commit messages for all the -stable branches of the src tree List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 03 Oct 2015 07:48:26 -0000 Author: mav Date: Sat Oct 3 07:48:25 2015 New Revision: 288560 URL: https://svnweb.freebsd.org/changeset/base/288560 Log: MFC r286605: 5812 assertion failed in zrl_tryenter(): zr_owner==NULL Reviewed by: George Wilson Reviewed by: Alex Reece Reviewed by: Will Andrews Approved by: Gordon Ross Author: Matthew Ahrens illumos/illumos-gate@8df173054ca442cd8845a7364c3edad9d6822351 Modified: stable/10/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zrlock.c Directory Properties: stable/10/ (props changed) Modified: stable/10/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zrlock.c ============================================================================== --- stable/10/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zrlock.c Sat Oct 3 07:47:33 2015 (r288559) +++ stable/10/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zrlock.c Sat Oct 3 07:48:25 2015 (r288560) @@ -20,6 +20,7 @@ */ /* * Copyright (c) 2010, Oracle and/or its affiliates. All rights reserved. + * Copyright (c) 2014 by Delphix. All rights reserved. */ /* @@ -42,7 +43,7 @@ * A ZRL can be locked only while there are zero references, so ZRL_LOCKED is * treated as zero references. */ -#define ZRL_LOCKED ((uint32_t)-1) +#define ZRL_LOCKED -1 #define ZRL_DESTROYED -2 void @@ -60,7 +61,7 @@ zrl_init(zrlock_t *zrl) void zrl_destroy(zrlock_t *zrl) { - ASSERT(zrl->zr_refcount == 0); + ASSERT0(zrl->zr_refcount); mutex_destroy(&zrl->zr_mtx); zrl->zr_refcount = ZRL_DESTROYED; @@ -80,7 +81,7 @@ zrl_add(zrlock_t *zrl) uint32_t cas = atomic_cas_32( (uint32_t *)&zrl->zr_refcount, n, n + 1); if (cas == n) { - ASSERT((int32_t)n >= 0); + ASSERT3S((int32_t)n, >=, 0); #ifdef ZFS_DEBUG if (zrl->zr_owner == curthread) { DTRACE_PROBE2(zrlock__reentry, @@ -98,7 +99,7 @@ zrl_add(zrlock_t *zrl) while (zrl->zr_refcount == ZRL_LOCKED) { cv_wait(&zrl->zr_cv, &zrl->zr_mtx); } - ASSERT(zrl->zr_refcount >= 0); + ASSERT3S(zrl->zr_refcount, >=, 0); zrl->zr_refcount++; #ifdef ZFS_DEBUG zrl->zr_owner = curthread; @@ -112,14 +113,14 @@ zrl_remove(zrlock_t *zrl) { uint32_t n; - n = atomic_dec_32_nv((uint32_t *)&zrl->zr_refcount); - ASSERT((int32_t)n >= 0); #ifdef ZFS_DEBUG if (zrl->zr_owner == curthread) { zrl->zr_owner = NULL; zrl->zr_caller = NULL; } #endif + n = atomic_dec_32_nv((uint32_t *)&zrl->zr_refcount); + ASSERT3S((int32_t)n, >=, 0); } int @@ -132,14 +133,14 @@ zrl_tryenter(zrlock_t *zrl) (uint32_t *)&zrl->zr_refcount, 0, ZRL_LOCKED); if (cas == 0) { #ifdef ZFS_DEBUG - ASSERT(zrl->zr_owner == NULL); + ASSERT3P(zrl->zr_owner, ==, NULL); zrl->zr_owner = curthread; #endif return (1); } } - ASSERT((int32_t)n > ZRL_DESTROYED); + ASSERT3S((int32_t)n, >, ZRL_DESTROYED); return (0); } @@ -147,11 +148,11 @@ zrl_tryenter(zrlock_t *zrl) void zrl_exit(zrlock_t *zrl) { - ASSERT(zrl->zr_refcount == ZRL_LOCKED); + ASSERT3S(zrl->zr_refcount, ==, ZRL_LOCKED); mutex_enter(&zrl->zr_mtx); #ifdef ZFS_DEBUG - ASSERT(zrl->zr_owner == curthread); + ASSERT3P(zrl->zr_owner, ==, curthread); zrl->zr_owner = NULL; membar_producer(); /* make sure the owner store happens first */ #endif @@ -163,7 +164,7 @@ zrl_exit(zrlock_t *zrl) int zrl_refcount(zrlock_t *zrl) { - ASSERT(zrl->zr_refcount > ZRL_DESTROYED); + ASSERT3S(zrl->zr_refcount, >, ZRL_DESTROYED); int n = (int)zrl->zr_refcount; return (n <= 0 ? 0 : n); @@ -172,7 +173,7 @@ zrl_refcount(zrlock_t *zrl) int zrl_is_zero(zrlock_t *zrl) { - ASSERT(zrl->zr_refcount > ZRL_DESTROYED); + ASSERT3S(zrl->zr_refcount, >, ZRL_DESTROYED); return (zrl->zr_refcount <= 0); } @@ -180,7 +181,7 @@ zrl_is_zero(zrlock_t *zrl) int zrl_is_locked(zrlock_t *zrl) { - ASSERT(zrl->zr_refcount > ZRL_DESTROYED); + ASSERT3S(zrl->zr_refcount, >, ZRL_DESTROYED); return (zrl->zr_refcount == ZRL_LOCKED); } From owner-svn-src-stable@freebsd.org Sat Oct 3 07:49:17 2015 Return-Path: Delivered-To: svn-src-stable@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 3D768A0F7F2; Sat, 3 Oct 2015 07:49:17 +0000 (UTC) (envelope-from mav@FreeBSD.org) Received: from repo.freebsd.org (repo.freebsd.org [IPv6:2001:1900:2254:2068::e6a:0]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 13E421ECD; Sat, 3 Oct 2015 07:49:17 +0000 (UTC) (envelope-from mav@FreeBSD.org) Received: from repo.freebsd.org ([127.0.1.70]) by repo.freebsd.org (8.15.2/8.15.2) with ESMTP id t937nGtU073868; Sat, 3 Oct 2015 07:49:16 GMT (envelope-from mav@FreeBSD.org) Received: (from mav@localhost) by repo.freebsd.org (8.15.2/8.15.2/Submit) id t937nGqT073867; Sat, 3 Oct 2015 07:49:16 GMT (envelope-from mav@FreeBSD.org) Message-Id: <201510030749.t937nGqT073867@repo.freebsd.org> X-Authentication-Warning: repo.freebsd.org: mav set sender to mav@FreeBSD.org using -f From: Alexander Motin Date: Sat, 3 Oct 2015 07:49:16 +0000 (UTC) To: src-committers@freebsd.org, svn-src-all@freebsd.org, svn-src-stable@freebsd.org, svn-src-stable-10@freebsd.org Subject: svn commit: r288561 - stable/10/sys/cddl/contrib/opensolaris/uts/common/fs/zfs X-SVN-Group: stable-10 MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit X-BeenThere: svn-src-stable@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: SVN commit messages for all the -stable branches of the src tree List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 03 Oct 2015 07:49:17 -0000 Author: mav Date: Sat Oct 3 07:49:16 2015 New Revision: 288561 URL: https://svnweb.freebsd.org/changeset/base/288561 Log: MFC r286623: Remove extra lock, that IMO only creates potential problems now. Modified: stable/10/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/arc.c Directory Properties: stable/10/ (props changed) Modified: stable/10/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/arc.c ============================================================================== --- stable/10/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/arc.c Sat Oct 3 07:48:25 2015 (r288560) +++ stable/10/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/arc.c Sat Oct 3 07:49:16 2015 (r288561) @@ -4776,7 +4776,6 @@ arc_tempreserve_space(uint64_t reserve, return (0); } -static kmutex_t arc_lowmem_lock; #ifdef _KERNEL static eventhandler_tag arc_event_lowmem = NULL; @@ -4784,8 +4783,6 @@ static void arc_lowmem(void *arg __unused, int howto __unused) { - /* Serialize access via arc_lowmem_lock. */ - mutex_enter(&arc_lowmem_lock); mutex_enter(&arc_reclaim_thr_lock); needfree = 1; DTRACE_PROBE(arc__needfree); @@ -4796,12 +4793,9 @@ arc_lowmem(void *arg __unused, int howto * here from ARC itself and may hold ARC locks and thus risk a deadlock * with ARC reclaim thread. */ - if (curproc == pageproc) { - while (needfree) - msleep(&needfree, &arc_reclaim_thr_lock, 0, "zfs:lowmem", 0); - } + if (curproc == pageproc) + msleep(&needfree, &arc_reclaim_thr_lock, 0, "zfs:lowmem", 0); mutex_exit(&arc_reclaim_thr_lock); - mutex_exit(&arc_lowmem_lock); } #endif @@ -4812,7 +4806,6 @@ arc_init(void) mutex_init(&arc_reclaim_thr_lock, NULL, MUTEX_DEFAULT, NULL); cv_init(&arc_reclaim_thr_cv, NULL, CV_DEFAULT, NULL); - mutex_init(&arc_lowmem_lock, NULL, MUTEX_DEFAULT, NULL); /* Convert seconds to clock ticks */ arc_min_prefetch_lifespan = 1 * hz; @@ -5052,7 +5045,6 @@ arc_fini(void) ASSERT0(arc_loaned_bytes); - mutex_destroy(&arc_lowmem_lock); #ifdef _KERNEL if (arc_event_lowmem != NULL) EVENTHANDLER_DEREGISTER(vm_lowmem, arc_event_lowmem); From owner-svn-src-stable@freebsd.org Sat Oct 3 07:50:16 2015 Return-Path: Delivered-To: svn-src-stable@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 2A51DA0F8F9; Sat, 3 Oct 2015 07:50:16 +0000 (UTC) (envelope-from mav@FreeBSD.org) Received: from repo.freebsd.org (repo.freebsd.org [IPv6:2001:1900:2254:2068::e6a:0]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 1A31610F1; Sat, 3 Oct 2015 07:50:16 +0000 (UTC) (envelope-from mav@FreeBSD.org) Received: from repo.freebsd.org ([127.0.1.70]) by repo.freebsd.org (8.15.2/8.15.2) with ESMTP id t937oFF5074565; Sat, 3 Oct 2015 07:50:15 GMT (envelope-from mav@FreeBSD.org) Received: (from mav@localhost) by repo.freebsd.org (8.15.2/8.15.2/Submit) id t937oFXZ074564; Sat, 3 Oct 2015 07:50:15 GMT (envelope-from mav@FreeBSD.org) Message-Id: <201510030750.t937oFXZ074564@repo.freebsd.org> X-Authentication-Warning: repo.freebsd.org: mav set sender to mav@FreeBSD.org using -f From: Alexander Motin Date: Sat, 3 Oct 2015 07:50:15 +0000 (UTC) To: src-committers@freebsd.org, svn-src-all@freebsd.org, svn-src-stable@freebsd.org, svn-src-stable-10@freebsd.org Subject: svn commit: r288562 - stable/10/sys/cddl/contrib/opensolaris/uts/common/fs/zfs X-SVN-Group: stable-10 MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit X-BeenThere: svn-src-stable@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: SVN commit messages for all the -stable branches of the src tree List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 03 Oct 2015 07:50:16 -0000 Author: mav Date: Sat Oct 3 07:50:15 2015 New Revision: 288562 URL: https://svnweb.freebsd.org/changeset/base/288562 Log: MFC r286625: 5376 arc_kmem_reap_now() should not result in clearing arc_no_grow Reviewed by: Christopher Siden Reviewed by: George Wilson Reviewed by: Steven Hartland Reviewed by: Richard Elling Approved by: Dan McDonald Author: Matthew Ahrens illumos/illumos-gate@2ec99e3e987d8aa273f1e9ba2b983557d058198c Modified: stable/10/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/arc.c Directory Properties: stable/10/ (props changed) Modified: stable/10/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/arc.c ============================================================================== --- stable/10/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/arc.c Sat Oct 3 07:49:16 2015 (r288561) +++ stable/10/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/arc.c Sat Oct 3 07:50:15 2015 (r288562) @@ -153,13 +153,7 @@ static kmutex_t arc_reclaim_thr_lock; static kcondvar_t arc_reclaim_thr_cv; /* used to signal reclaim thr */ static uint8_t arc_thread_exit; -#define ARC_REDUCE_DNLC_PERCENT 3 -uint_t arc_reduce_dnlc_percent = ARC_REDUCE_DNLC_PERCENT; - -typedef enum arc_reclaim_strategy { - ARC_RECLAIM_AGGR, /* Aggressive reclaim strategy */ - ARC_RECLAIM_CONS /* Conservative reclaim strategy */ -} arc_reclaim_strategy_t; +uint_t arc_reduce_dnlc_percent = 3; /* * The number of iterations through arc_evict_*() before we @@ -174,7 +168,19 @@ static int arc_grow_retry = 60; static int arc_p_min_shift = 4; /* log2(fraction of arc to reclaim) */ -static int arc_shrink_shift = 5; +static int arc_shrink_shift = 7; + +/* + * log2(fraction of ARC which must be free to allow growing). + * I.e. If there is less than arc_c >> arc_no_grow_shift free memory, + * when reading a new block into the ARC, we will evict an equal-sized block + * from the ARC. + * + * This must be less than arc_shrink_shift, so that when we shrink the ARC, + * we will still not allow it to grow. + */ +int arc_no_grow_shift = 5; + /* * minimum lifespan of a prefetch block in clock ticks @@ -3058,13 +3064,10 @@ arc_flush(spa_t *spa) } void -arc_shrink(void) +arc_shrink(int64_t to_free) { if (arc_c > arc_c_min) { - uint64_t to_free; - - to_free = arc_c >> arc_shrink_shift; DTRACE_PROBE4(arc__shrink, uint64_t, arc_c, uint64_t, arc_c_min, uint64_t, arc_p, uint64_t, to_free); if (arc_c > arc_c_min + to_free) @@ -3092,44 +3095,76 @@ arc_shrink(void) } } -static int needfree = 0; +static long needfree = 0; -static int -arc_reclaim_needed(void) +typedef enum free_memory_reason_t { + FMR_UNKNOWN, + FMR_NEEDFREE, + FMR_LOTSFREE, + FMR_SWAPFS_MINFREE, + FMR_PAGES_PP_MAXIMUM, + FMR_HEAP_ARENA, + FMR_ZIO_ARENA, + FMR_ZIO_FRAG, +} free_memory_reason_t; + +int64_t last_free_memory; +free_memory_reason_t last_free_reason; + +/* + * Additional reserve of pages for pp_reserve. + */ +int64_t arc_pages_pp_reserve = 64; + +/* + * Additional reserve of pages for swapfs. + */ +int64_t arc_swapfs_reserve = 64; + +/* + * Return the amount of memory that can be consumed before reclaim will be + * needed. Positive if there is sufficient free memory, negative indicates + * the amount of memory that needs to be freed up. + */ +static int64_t +arc_available_memory(void) { + int64_t lowest = INT64_MAX; + int64_t n; + free_memory_reason_t r = FMR_UNKNOWN; #ifdef _KERNEL - - if (needfree) { - DTRACE_PROBE(arc__reclaim_needfree); - return (1); + if (needfree > 0) { + n = PAGESIZE * (-needfree); + if (n < lowest) { + lowest = n; + r = FMR_NEEDFREE; + } } /* * Cooperate with pagedaemon when it's time for it to scan * and reclaim some pages. */ - if (freemem < zfs_arc_free_target) { - DTRACE_PROBE2(arc__reclaim_freemem, uint64_t, - freemem, uint64_t, zfs_arc_free_target); - return (1); + n = PAGESIZE * (int64_t)(freemem - zfs_arc_free_target); + if (n < lowest) { + lowest = n; + r = FMR_LOTSFREE; } #ifdef sun /* - * take 'desfree' extra pages, so we reclaim sooner, rather than later - */ - extra = desfree; - - /* * check that we're out of range of the pageout scanner. It starts to * schedule paging if freemem is less than lotsfree and needfree. * lotsfree is the high-water mark for pageout, and needfree is the * number of needed free pages. We add extra pages here to make sure * the scanner doesn't start up while we're freeing memory. */ - if (freemem < lotsfree + needfree + extra) - return (1); + n = PAGESIZE * (freemem - lotsfree - needfree - desfree); + if (n < lowest) { + lowest = n; + r = FMR_LOTSFREE; + } /* * check to make sure that swapfs has enough space so that anon @@ -3138,8 +3173,13 @@ arc_reclaim_needed(void) * swap pages. We also add a bit of extra here just to prevent * circumstances from getting really dire. */ - if (availrmem < swapfs_minfree + swapfs_reserve + extra) - return (1); + n = PAGESIZE * (availrmem - swapfs_minfree - swapfs_reserve - + desfree - arc_swapfs_reserve); + if (n < lowest) { + lowest = n; + r = FMR_SWAPFS_MINFREE; + } + /* * Check that we have enough availrmem that memory locking (e.g., via @@ -3148,8 +3188,12 @@ arc_reclaim_needed(void) * drops below pages_pp_maximum, page locking mechanisms such as * page_pp_lock() will fail.) */ - if (availrmem <= pages_pp_maximum) - return (1); + n = PAGESIZE * (availrmem - pages_pp_maximum - + arc_pages_pp_reserve); + if (n < lowest) { + lowest = n; + r = FMR_PAGES_PP_MAXIMUM; + } #endif /* sun */ #if defined(__i386) || !defined(UMA_MD_SMALL_ALLOC) @@ -3164,12 +3208,11 @@ arc_reclaim_needed(void) * heap is allocated. (Or, in the calculation, if less than 1/4th is * free) */ - if (vmem_size(heap_arena, VMEM_FREE) < - (vmem_size(heap_arena, VMEM_FREE | VMEM_ALLOC) >> 2)) { - DTRACE_PROBE2(arc__reclaim_used, uint64_t, - vmem_size(heap_arena, VMEM_FREE), uint64_t, - (vmem_size(heap_arena, VMEM_FREE | VMEM_ALLOC)) >> 2); - return (1); + n = vmem_size(heap_arena, VMEM_FREE) - + (vmem_size(heap_arena, VMEM_FREE | VMEM_ALLOC) >> 2) + if (n < lowest) { + lowest = n; + r = FMR_HEAP_ARENA; } #define zio_arena NULL #else @@ -3185,29 +3228,50 @@ arc_reclaim_needed(void) * to aggressively evict memory from the arc in order to avoid * memory fragmentation issues. */ - if (zio_arena != NULL && - vmem_size(zio_arena, VMEM_FREE) < - (vmem_size(zio_arena, VMEM_ALLOC) >> 4)) - return (1); + if (zio_arena != NULL) { + n = vmem_size(zio_arena, VMEM_FREE) - + (vmem_size(zio_arena, VMEM_ALLOC) >> 4); + if (n < lowest) { + lowest = n; + r = FMR_ZIO_ARENA; + } + } /* * Above limits know nothing about real level of KVA fragmentation. * Start aggressive reclamation if too little sequential KVA left. */ - if (vmem_size(heap_arena, VMEM_MAXFREE) < zfs_max_recordsize) { - DTRACE_PROBE2(arc__reclaim_maxfree, uint64_t, - vmem_size(heap_arena, VMEM_MAXFREE), - uint64_t, zfs_max_recordsize); - return (1); + if (lowest > 0) { + n = (vmem_size(heap_arena, VMEM_MAXFREE) < zfs_max_recordsize) ? + -(vmem_size(heap_arena, VMEM_ALLOC) >> 4) : INT64_MAX; + if (n < lowest) { + lowest = n; + r = FMR_ZIO_FRAG; + } } #else /* _KERNEL */ + /* Every 100 calls, free a small amount */ if (spa_get_random(100) == 0) - return (1); + lowest = -1024; #endif /* _KERNEL */ - DTRACE_PROBE(arc__reclaim_no); - return (0); + last_free_memory = lowest; + last_free_reason = r; + DTRACE_PROBE2(arc__available_memory, int64_t, lowest, int, r); + return (lowest); +} + + +/* + * Determine if the system is under memory pressure and is asking + * to reclaim memory. A return value of TRUE indicates that the system + * is under memory pressure and that the arc should adjust accordingly. + */ +static boolean_t +arc_reclaim_needed(void) +{ + return (arc_available_memory() < 0); } extern kmem_cache_t *zio_buf_cache[]; @@ -3215,7 +3279,7 @@ extern kmem_cache_t *zio_data_buf_cache[ extern kmem_cache_t *range_seg_cache; static __noinline void -arc_kmem_reap_now(arc_reclaim_strategy_t strat) +arc_kmem_reap_now(void) { size_t i; kmem_cache_t *prev_cache = NULL; @@ -3238,13 +3302,6 @@ arc_kmem_reap_now(arc_reclaim_strategy_t #endif #endif - /* - * An aggressive reclamation will shrink the cache size as well as - * reap free buffers from the arc kmem caches. - */ - if (strat == ARC_RECLAIM_AGGR) - arc_shrink(); - for (i = 0; i < SPA_MAXBLOCKSIZE >> SPA_MINBLOCKSHIFT; i++) { if (zio_buf_cache[i] != prev_cache) { prev_cache = zio_buf_cache[i]; @@ -3261,12 +3318,13 @@ arc_kmem_reap_now(arc_reclaim_strategy_t kmem_cache_reap_now(range_seg_cache); #ifdef sun - /* - * Ask the vmem arena to reclaim unused memory from its - * quantum caches. - */ - if (zio_arena != NULL && strat == ARC_RECLAIM_AGGR) + if (zio_arena != NULL) { + /* + * Ask the vmem arena to reclaim unused memory from its + * quantum caches. + */ vmem_qcache_reap(zio_arena); + } #endif DTRACE_PROBE(arc__kmem_reap_end); } @@ -3275,46 +3333,44 @@ static void arc_reclaim_thread(void *dummy __unused) { clock_t growtime = 0; - arc_reclaim_strategy_t last_reclaim = ARC_RECLAIM_CONS; callb_cpr_t cpr; CALLB_CPR_INIT(&cpr, &arc_reclaim_thr_lock, callb_generic_cpr, FTAG); mutex_enter(&arc_reclaim_thr_lock); while (arc_thread_exit == 0) { - if (arc_reclaim_needed()) { + int64_t free_memory = arc_available_memory(); + if (free_memory < 0) { - if (arc_no_grow) { - if (last_reclaim == ARC_RECLAIM_CONS) { - DTRACE_PROBE(arc__reclaim_aggr_no_grow); - last_reclaim = ARC_RECLAIM_AGGR; - } else { - last_reclaim = ARC_RECLAIM_CONS; - } - } else { - arc_no_grow = TRUE; - last_reclaim = ARC_RECLAIM_AGGR; - DTRACE_PROBE(arc__reclaim_aggr); - membar_producer(); - } + arc_no_grow = B_TRUE; + arc_warm = B_TRUE; - /* reset the growth delay for every reclaim */ + /* + * Wait at least zfs_grow_retry (default 60) seconds + * before considering growing. + */ growtime = ddi_get_lbolt() + (arc_grow_retry * hz); - if (needfree && last_reclaim == ARC_RECLAIM_CONS) { - /* - * If needfree is TRUE our vm_lowmem hook - * was called and in that case we must free some - * memory, so switch to aggressive mode. - */ - arc_no_grow = TRUE; - last_reclaim = ARC_RECLAIM_AGGR; - } - arc_kmem_reap_now(last_reclaim); - arc_warm = B_TRUE; + arc_kmem_reap_now(); - } else if (arc_no_grow && ddi_get_lbolt() >= growtime) { - arc_no_grow = FALSE; + /* + * If we are still low on memory, shrink the ARC + * so that we have arc_shrink_min free space. + */ + free_memory = arc_available_memory(); + + int64_t to_free = + (arc_c >> arc_shrink_shift) - free_memory; + if (to_free > 0) { +#ifdef _KERNEL + to_free = MAX(to_free, ptob(needfree)); +#endif + arc_shrink(to_free); + } + } else if (free_memory < arc_c >> arc_no_grow_shift) { + arc_no_grow = B_TRUE; + } else if (ddi_get_lbolt() >= growtime) { + arc_no_grow = B_FALSE; } arc_adjust(); @@ -4784,7 +4840,8 @@ arc_lowmem(void *arg __unused, int howto { mutex_enter(&arc_reclaim_thr_lock); - needfree = 1; + /* XXX: Memory deficit should be passed as argument. */ + needfree = btoc(arc_c >> arc_shrink_shift); DTRACE_PROBE(arc__needfree); cv_signal(&arc_reclaim_thr_cv); @@ -4868,6 +4925,12 @@ arc_init(void) if (zfs_arc_shrink_shift > 0) arc_shrink_shift = zfs_arc_shrink_shift; + /* + * Ensure that arc_no_grow_shift is less than arc_shrink_shift. + */ + if (arc_no_grow_shift >= arc_shrink_shift) + arc_no_grow_shift = arc_shrink_shift - 1; + if (zfs_arc_p_min_shift > 0) arc_p_min_shift = zfs_arc_p_min_shift; From owner-svn-src-stable@freebsd.org Sat Oct 3 07:51:10 2015 Return-Path: Delivered-To: svn-src-stable@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id B53EEA0FA04; Sat, 3 Oct 2015 07:51:10 +0000 (UTC) (envelope-from mav@FreeBSD.org) Received: from repo.freebsd.org (repo.freebsd.org [IPv6:2001:1900:2254:2068::e6a:0]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id A5B6F1315; Sat, 3 Oct 2015 07:51:10 +0000 (UTC) (envelope-from mav@FreeBSD.org) Received: from repo.freebsd.org ([127.0.1.70]) by repo.freebsd.org (8.15.2/8.15.2) with ESMTP id t937pA8k077131; Sat, 3 Oct 2015 07:51:10 GMT (envelope-from mav@FreeBSD.org) Received: (from mav@localhost) by repo.freebsd.org (8.15.2/8.15.2/Submit) id t937pAlU077130; Sat, 3 Oct 2015 07:51:10 GMT (envelope-from mav@FreeBSD.org) Message-Id: <201510030751.t937pAlU077130@repo.freebsd.org> X-Authentication-Warning: repo.freebsd.org: mav set sender to mav@FreeBSD.org using -f From: Alexander Motin Date: Sat, 3 Oct 2015 07:51:10 +0000 (UTC) To: src-committers@freebsd.org, svn-src-all@freebsd.org, svn-src-stable@freebsd.org, svn-src-stable-10@freebsd.org Subject: svn commit: r288563 - stable/10/sys/cddl/contrib/opensolaris/uts/common/fs/zfs X-SVN-Group: stable-10 MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit X-BeenThere: svn-src-stable@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: SVN commit messages for all the -stable branches of the src tree List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 03 Oct 2015 07:51:10 -0000 Author: mav Date: Sat Oct 3 07:51:09 2015 New Revision: 288563 URL: https://svnweb.freebsd.org/changeset/base/288563 Log: MFC r286626: Fix minor mismerge in r286574. Modified: stable/10/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/arc.c Directory Properties: stable/10/ (props changed) Modified: stable/10/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/arc.c ============================================================================== --- stable/10/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/arc.c Sat Oct 3 07:50:15 2015 (r288562) +++ stable/10/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/arc.c Sat Oct 3 07:51:09 2015 (r288563) @@ -4732,48 +4732,6 @@ arc_memory_throttle(uint64_t reserve, ui return (0); } -static void -arc_kstat_update_state(arc_state_t *state, kstat_named_t *size, - kstat_named_t *evict_data, kstat_named_t *evict_metadata) -{ - size->value.ui64 = state->arcs_size; - evict_data->value.ui64 = state->arcs_lsize[ARC_BUFC_DATA]; - evict_metadata->value.ui64 = state->arcs_lsize[ARC_BUFC_METADATA]; -} - -static int -arc_kstat_update(kstat_t *ksp, int rw) -{ - arc_stats_t *as = ksp->ks_data; - - if (rw == KSTAT_WRITE) { - return (EACCES); - } else { - arc_kstat_update_state(arc_anon, - &as->arcstat_anon_size, - &as->arcstat_anon_evictable_data, - &as->arcstat_anon_evictable_metadata); - arc_kstat_update_state(arc_mru, - &as->arcstat_mru_size, - &as->arcstat_mru_evictable_data, - &as->arcstat_mru_evictable_metadata); - arc_kstat_update_state(arc_mru_ghost, - &as->arcstat_mru_ghost_size, - &as->arcstat_mru_ghost_evictable_data, - &as->arcstat_mru_ghost_evictable_metadata); - arc_kstat_update_state(arc_mfu, - &as->arcstat_mfu_size, - &as->arcstat_mfu_evictable_data, - &as->arcstat_mfu_evictable_metadata); - arc_kstat_update_state(arc_mfu_ghost, - &as->arcstat_mfu_ghost_size, - &as->arcstat_mfu_ghost_evictable_data, - &as->arcstat_mfu_ghost_evictable_metadata); - } - - return (0); -} - void arc_tempreserve_clear(uint64_t reserve) { @@ -4832,6 +4790,48 @@ arc_tempreserve_space(uint64_t reserve, return (0); } +static void +arc_kstat_update_state(arc_state_t *state, kstat_named_t *size, + kstat_named_t *evict_data, kstat_named_t *evict_metadata) +{ + size->value.ui64 = state->arcs_size; + evict_data->value.ui64 = state->arcs_lsize[ARC_BUFC_DATA]; + evict_metadata->value.ui64 = state->arcs_lsize[ARC_BUFC_METADATA]; +} + +static int +arc_kstat_update(kstat_t *ksp, int rw) +{ + arc_stats_t *as = ksp->ks_data; + + if (rw == KSTAT_WRITE) { + return (EACCES); + } else { + arc_kstat_update_state(arc_anon, + &as->arcstat_anon_size, + &as->arcstat_anon_evictable_data, + &as->arcstat_anon_evictable_metadata); + arc_kstat_update_state(arc_mru, + &as->arcstat_mru_size, + &as->arcstat_mru_evictable_data, + &as->arcstat_mru_evictable_metadata); + arc_kstat_update_state(arc_mru_ghost, + &as->arcstat_mru_ghost_size, + &as->arcstat_mru_ghost_evictable_data, + &as->arcstat_mru_ghost_evictable_metadata); + arc_kstat_update_state(arc_mfu, + &as->arcstat_mfu_size, + &as->arcstat_mfu_evictable_data, + &as->arcstat_mfu_evictable_metadata); + arc_kstat_update_state(arc_mfu_ghost, + &as->arcstat_mfu_ghost_size, + &as->arcstat_mfu_ghost_evictable_data, + &as->arcstat_mfu_ghost_evictable_metadata); + } + + return (0); +} + #ifdef _KERNEL static eventhandler_tag arc_event_lowmem = NULL; From owner-svn-src-stable@freebsd.org Sat Oct 3 07:52:11 2015 Return-Path: Delivered-To: svn-src-stable@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id DD927A0FB76; Sat, 3 Oct 2015 07:52:11 +0000 (UTC) (envelope-from mav@FreeBSD.org) Received: from repo.freebsd.org (repo.freebsd.org [IPv6:2001:1900:2254:2068::e6a:0]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id CDEAE175F; Sat, 3 Oct 2015 07:52:11 +0000 (UTC) (envelope-from mav@FreeBSD.org) Received: from repo.freebsd.org ([127.0.1.70]) by repo.freebsd.org (8.15.2/8.15.2) with ESMTP id t937qB11077957; Sat, 3 Oct 2015 07:52:11 GMT (envelope-from mav@FreeBSD.org) Received: (from mav@localhost) by repo.freebsd.org (8.15.2/8.15.2/Submit) id t937qBY9077956; Sat, 3 Oct 2015 07:52:11 GMT (envelope-from mav@FreeBSD.org) Message-Id: <201510030752.t937qBY9077956@repo.freebsd.org> X-Authentication-Warning: repo.freebsd.org: mav set sender to mav@FreeBSD.org using -f From: Alexander Motin Date: Sat, 3 Oct 2015 07:52:11 +0000 (UTC) To: src-committers@freebsd.org, svn-src-all@freebsd.org, svn-src-stable@freebsd.org, svn-src-stable-10@freebsd.org Subject: svn commit: r288564 - stable/10/sys/cddl/contrib/opensolaris/uts/common/fs/zfs X-SVN-Group: stable-10 MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit X-BeenThere: svn-src-stable@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: SVN commit messages for all the -stable branches of the src tree List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 03 Oct 2015 07:52:12 -0000 Author: mav Date: Sat Oct 3 07:52:11 2015 New Revision: 288564 URL: https://svnweb.freebsd.org/changeset/base/288564 Log: MFC r286628: Fix r286625 build on i386. Modified: stable/10/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/arc.c Directory Properties: stable/10/ (props changed) Modified: stable/10/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/arc.c ============================================================================== --- stable/10/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/arc.c Sat Oct 3 07:51:09 2015 (r288563) +++ stable/10/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/arc.c Sat Oct 3 07:52:11 2015 (r288564) @@ -3209,7 +3209,7 @@ arc_available_memory(void) * free) */ n = vmem_size(heap_arena, VMEM_FREE) - - (vmem_size(heap_arena, VMEM_FREE | VMEM_ALLOC) >> 2) + (vmem_size(heap_arena, VMEM_FREE | VMEM_ALLOC) >> 2); if (n < lowest) { lowest = n; r = FMR_HEAP_ARENA; From owner-svn-src-stable@freebsd.org Sat Oct 3 07:53:09 2015 Return-Path: Delivered-To: svn-src-stable@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 31CDCA0FC8E; Sat, 3 Oct 2015 07:53:09 +0000 (UTC) (envelope-from mav@FreeBSD.org) Received: from repo.freebsd.org (repo.freebsd.org [IPv6:2001:1900:2254:2068::e6a:0]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 220C418E2; Sat, 3 Oct 2015 07:53:09 +0000 (UTC) (envelope-from mav@FreeBSD.org) Received: from repo.freebsd.org ([127.0.1.70]) by repo.freebsd.org (8.15.2/8.15.2) with ESMTP id t937r9Ku078120; Sat, 3 Oct 2015 07:53:09 GMT (envelope-from mav@FreeBSD.org) Received: (from mav@localhost) by repo.freebsd.org (8.15.2/8.15.2/Submit) id t937r8n9078119; Sat, 3 Oct 2015 07:53:08 GMT (envelope-from mav@FreeBSD.org) Message-Id: <201510030753.t937r8n9078119@repo.freebsd.org> X-Authentication-Warning: repo.freebsd.org: mav set sender to mav@FreeBSD.org using -f From: Alexander Motin Date: Sat, 3 Oct 2015 07:53:08 +0000 (UTC) To: src-committers@freebsd.org, svn-src-all@freebsd.org, svn-src-stable@freebsd.org, svn-src-stable-10@freebsd.org Subject: svn commit: r288565 - stable/10/sys/cddl/contrib/opensolaris/uts/common/fs/zfs X-SVN-Group: stable-10 MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit X-BeenThere: svn-src-stable@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: SVN commit messages for all the -stable branches of the src tree List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 03 Oct 2015 07:53:09 -0000 Author: mav Date: Sat Oct 3 07:53:08 2015 New Revision: 288565 URL: https://svnweb.freebsd.org/changeset/base/288565 Log: MFC r286647: Fix assertion panic caused by combination of r286598 and TRIM. Modified: stable/10/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/arc.c Directory Properties: stable/10/ (props changed) Modified: stable/10/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/arc.c ============================================================================== --- stable/10/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/arc.c Sat Oct 3 07:52:11 2015 (r288564) +++ stable/10/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/arc.c Sat Oct 3 07:53:08 2015 (r288565) @@ -2329,8 +2329,10 @@ arc_hdr_destroy(arc_buf_hdr_t *hdr) * want to re-destroy the header's L2 portion. */ if (HDR_HAS_L2HDR(hdr)) { - trim_map_free(dev->l2ad_vdev, hdr->b_l2hdr.b_daddr, - hdr->b_l2hdr.b_asize, 0); + if (hdr->b_l2hdr.b_daddr != L2ARC_ADDR_UNSET) + trim_map_free(dev->l2ad_vdev, + hdr->b_l2hdr.b_daddr, + hdr->b_l2hdr.b_asize, 0); arc_hdr_l2hdr_destroy(hdr); } @@ -4415,8 +4417,10 @@ arc_release(arc_buf_t *buf, void *tag) * to acquire the l2ad_mtx. */ if (HDR_HAS_L2HDR(hdr)) { - trim_map_free(hdr->b_l2hdr.b_dev->l2ad_vdev, - hdr->b_l2hdr.b_daddr, hdr->b_l2hdr.b_asize, 0); + if (hdr->b_l2hdr.b_daddr != L2ARC_ADDR_UNSET) + trim_map_free(hdr->b_l2hdr.b_dev->l2ad_vdev, + hdr->b_l2hdr.b_daddr, + hdr->b_l2hdr.b_asize, 0); arc_hdr_l2hdr_destroy(hdr); } From owner-svn-src-stable@freebsd.org Sat Oct 3 07:53:57 2015 Return-Path: Delivered-To: svn-src-stable@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 780A5A0FD27; Sat, 3 Oct 2015 07:53:57 +0000 (UTC) (envelope-from mav@FreeBSD.org) Received: from repo.freebsd.org (repo.freebsd.org [IPv6:2001:1900:2254:2068::e6a:0]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 4FB421A38; Sat, 3 Oct 2015 07:53:57 +0000 (UTC) (envelope-from mav@FreeBSD.org) Received: from repo.freebsd.org ([127.0.1.70]) by repo.freebsd.org (8.15.2/8.15.2) with ESMTP id t937rvhQ078255; Sat, 3 Oct 2015 07:53:57 GMT (envelope-from mav@FreeBSD.org) Received: (from mav@localhost) by repo.freebsd.org (8.15.2/8.15.2/Submit) id t937rvjA078254; Sat, 3 Oct 2015 07:53:57 GMT (envelope-from mav@FreeBSD.org) Message-Id: <201510030753.t937rvjA078254@repo.freebsd.org> X-Authentication-Warning: repo.freebsd.org: mav set sender to mav@FreeBSD.org using -f From: Alexander Motin Date: Sat, 3 Oct 2015 07:53:57 +0000 (UTC) To: src-committers@freebsd.org, svn-src-all@freebsd.org, svn-src-stable@freebsd.org, svn-src-stable-10@freebsd.org Subject: svn commit: r288566 - stable/10/sys/cddl/contrib/opensolaris/uts/common/fs/zfs X-SVN-Group: stable-10 MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit X-BeenThere: svn-src-stable@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: SVN commit messages for all the -stable branches of the src tree List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 03 Oct 2015 07:53:57 -0000 Author: mav Date: Sat Oct 3 07:53:56 2015 New Revision: 288566 URL: https://svnweb.freebsd.org/changeset/base/288566 Log: MFC r286655: Fix set of sign extension bugs in r286625. Modified: stable/10/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/arc.c Directory Properties: stable/10/ (props changed) Modified: stable/10/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/arc.c ============================================================================== --- stable/10/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/arc.c Sat Oct 3 07:53:08 2015 (r288565) +++ stable/10/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/arc.c Sat Oct 3 07:53:56 2015 (r288566) @@ -3148,7 +3148,7 @@ arc_available_memory(void) * Cooperate with pagedaemon when it's time for it to scan * and reclaim some pages. */ - n = PAGESIZE * (int64_t)(freemem - zfs_arc_free_target); + n = PAGESIZE * ((int64_t)freemem - zfs_arc_free_target); if (n < lowest) { lowest = n; r = FMR_LOTSFREE; @@ -3210,7 +3210,7 @@ arc_available_memory(void) * heap is allocated. (Or, in the calculation, if less than 1/4th is * free) */ - n = vmem_size(heap_arena, VMEM_FREE) - + n = (int64_t)vmem_size(heap_arena, VMEM_FREE) - (vmem_size(heap_arena, VMEM_FREE | VMEM_ALLOC) >> 2); if (n < lowest) { lowest = n; @@ -3231,7 +3231,7 @@ arc_available_memory(void) * memory fragmentation issues. */ if (zio_arena != NULL) { - n = vmem_size(zio_arena, VMEM_FREE) - + n = (int64_t)vmem_size(zio_arena, VMEM_FREE) - (vmem_size(zio_arena, VMEM_ALLOC) >> 4); if (n < lowest) { lowest = n; @@ -3245,7 +3245,8 @@ arc_available_memory(void) */ if (lowest > 0) { n = (vmem_size(heap_arena, VMEM_MAXFREE) < zfs_max_recordsize) ? - -(vmem_size(heap_arena, VMEM_ALLOC) >> 4) : INT64_MAX; + -((int64_t)vmem_size(heap_arena, VMEM_ALLOC) >> 4) : + INT64_MAX; if (n < lowest) { lowest = n; r = FMR_ZIO_FRAG; From owner-svn-src-stable@freebsd.org Sat Oct 3 07:55:38 2015 Return-Path: Delivered-To: svn-src-stable@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 61A16A0FE7A; Sat, 3 Oct 2015 07:55:38 +0000 (UTC) (envelope-from mav@FreeBSD.org) Received: from repo.freebsd.org (repo.freebsd.org [IPv6:2001:1900:2254:2068::e6a:0]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 47D311BD4; Sat, 3 Oct 2015 07:55:38 +0000 (UTC) (envelope-from mav@FreeBSD.org) Received: from repo.freebsd.org ([127.0.1.70]) by repo.freebsd.org (8.15.2/8.15.2) with ESMTP id t937tcD8078547; Sat, 3 Oct 2015 07:55:38 GMT (envelope-from mav@FreeBSD.org) Received: (from mav@localhost) by repo.freebsd.org (8.15.2/8.15.2/Submit) id t937tbYH078544; Sat, 3 Oct 2015 07:55:37 GMT (envelope-from mav@FreeBSD.org) Message-Id: <201510030755.t937tbYH078544@repo.freebsd.org> X-Authentication-Warning: repo.freebsd.org: mav set sender to mav@FreeBSD.org using -f From: Alexander Motin Date: Sat, 3 Oct 2015 07:55:37 +0000 (UTC) To: src-committers@freebsd.org, svn-src-all@freebsd.org, svn-src-stable@freebsd.org, svn-src-stable-10@freebsd.org Subject: svn commit: r288567 - in stable/10/sys/cddl/contrib/opensolaris/uts/common/fs/zfs: . sys X-SVN-Group: stable-10 MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit X-BeenThere: svn-src-stable@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: SVN commit messages for all the -stable branches of the src tree List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 03 Oct 2015 07:55:38 -0000 Author: mav Date: Sat Oct 3 07:55:37 2015 New Revision: 288567 URL: https://svnweb.freebsd.org/changeset/base/288567 Log: MFC r286677: 5695 dmu_sync'ed holes do not retain birth time illumos/illumos-gate@70163ac57e58ace1c5c94dfbe85dca5a974eff36 https://www.illumos.org/issues/5695 In dmu_sync_ready(), a hole block pointer will have it's logical size explicitly set as it's necessary for replay purposes. To "undo" this, dmu_sync_done() will zero out any hole that it finds. This becomes a problem when using the "hole_birth" feature, as this will also wipe out any birth time that might have happened to be set on the hole. ... As a fix, the logic to zero out a hole is only applied to old style holes with a birth time of zero. Holes created with the "hole_birth" feature enabled will have a non-zero birth time, and will be skipped (thus preserving the ltime, type, and level information as well). In addition, zdb was updated to also print the ltime, type, and level information for these new style holes. Previously, only the logical birth time would be printed. Author: Prakash Surya Reviewed by: Matthew Ahrens Reviewed by: George Wilson Reviewed by: Christopher Siden Reviewed by: Bayard Bell Approved by: Dan McDonald Modified: stable/10/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/dmu.c stable/10/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/sys/spa.h Directory Properties: stable/10/ (props changed) Modified: stable/10/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/dmu.c ============================================================================== --- stable/10/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/dmu.c Sat Oct 3 07:53:56 2015 (r288566) +++ stable/10/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/dmu.c Sat Oct 3 07:55:37 2015 (r288567) @@ -1485,7 +1485,19 @@ dmu_sync_done(zio_t *zio, arc_buf_t *buf dr->dt.dl.dr_overridden_by = *zio->io_bp; dr->dt.dl.dr_override_state = DR_OVERRIDDEN; dr->dt.dl.dr_copies = zio->io_prop.zp_copies; - if (BP_IS_HOLE(&dr->dt.dl.dr_overridden_by)) + + /* + * Old style holes are filled with all zeros, whereas + * new-style holes maintain their lsize, type, level, + * and birth time (see zio_write_compress). While we + * need to reset the BP_SET_LSIZE() call that happened + * in dmu_sync_ready for old style holes, we do *not* + * want to wipe out the information contained in new + * style holes. Thus, only zero out the block pointer if + * it's an old style hole. + */ + if (BP_IS_HOLE(&dr->dt.dl.dr_overridden_by) && + dr->dt.dl.dr_overridden_by.blk_birth == 0) BP_ZERO(&dr->dt.dl.dr_overridden_by); } else { dr->dt.dl.dr_override_state = DR_NOT_OVERRIDDEN; Modified: stable/10/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/sys/spa.h ============================================================================== --- stable/10/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/sys/spa.h Sat Oct 3 07:53:56 2015 (r288566) +++ stable/10/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/sys/spa.h Sat Oct 3 07:55:37 2015 (r288567) @@ -536,12 +536,13 @@ _NOTE(CONSTCOND) } while (0) if (bp == NULL) { \ len += func(buf + len, size - len, ""); \ } else if (BP_IS_HOLE(bp)) { \ - len += func(buf + len, size - len, ""); \ - if (bp->blk_birth > 0) { \ - len += func(buf + len, size - len, \ - " birth=%lluL", \ - (u_longlong_t)bp->blk_birth); \ - } \ + len += func(buf + len, size - len, \ + "HOLE [L%llu %s] " \ + "size=%llxL birth=%lluL", \ + (u_longlong_t)BP_GET_LEVEL(bp), \ + type, \ + (u_longlong_t)BP_GET_LSIZE(bp), \ + (u_longlong_t)bp->blk_birth); \ } else if (BP_IS_EMBEDDED(bp)) { \ len = func(buf + len, size - len, \ "EMBEDDED [L%llu %s] et=%u %s " \ From owner-svn-src-stable@freebsd.org Sat Oct 3 07:57:34 2015 Return-Path: Delivered-To: svn-src-stable@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 84037A0D02E; Sat, 3 Oct 2015 07:57:34 +0000 (UTC) (envelope-from mav@FreeBSD.org) Received: from repo.freebsd.org (repo.freebsd.org [IPv6:2001:1900:2254:2068::e6a:0]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 71F741D6C; Sat, 3 Oct 2015 07:57:34 +0000 (UTC) (envelope-from mav@FreeBSD.org) Received: from repo.freebsd.org ([127.0.1.70]) by repo.freebsd.org (8.15.2/8.15.2) with ESMTP id t937vYmB078818; Sat, 3 Oct 2015 07:57:34 GMT (envelope-from mav@FreeBSD.org) Received: (from mav@localhost) by repo.freebsd.org (8.15.2/8.15.2/Submit) id t937vXwU078812; Sat, 3 Oct 2015 07:57:33 GMT (envelope-from mav@FreeBSD.org) Message-Id: <201510030757.t937vXwU078812@repo.freebsd.org> X-Authentication-Warning: repo.freebsd.org: mav set sender to mav@FreeBSD.org using -f From: Alexander Motin Date: Sat, 3 Oct 2015 07:57:33 +0000 (UTC) To: src-committers@freebsd.org, svn-src-all@freebsd.org, svn-src-stable@freebsd.org, svn-src-stable-10@freebsd.org Subject: svn commit: r288568 - in stable/10: cddl/contrib/opensolaris/lib/libzfs_core/common sys/cddl/contrib/opensolaris/uts/common/fs/zfs sys/cddl/contrib/opensolaris/uts/common/fs/zfs/sys X-SVN-Group: stable-10 MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit X-BeenThere: svn-src-stable@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: SVN commit messages for all the -stable branches of the src tree List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 03 Oct 2015 07:57:34 -0000 Author: mav Date: Sat Oct 3 07:57:32 2015 New Revision: 288568 URL: https://svnweb.freebsd.org/changeset/base/288568 Log: MFC r286683: 5765 add support for estimating send stream size with lzc_send_space when source is a bookmark Reviewed by: Matthew Ahrens Reviewed by: Christopher Siden Reviewed by: Steven Hartland Reviewed by: Bayard Bell Approved by: Albert Lee Author: Max Grossman illumos/illumos-gate@643da460c8ca583e39ce053081754e24087f84c8 Modified: stable/10/cddl/contrib/opensolaris/lib/libzfs_core/common/libzfs_core.c stable/10/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/dmu_send.c stable/10/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/sys/dmu_send.h stable/10/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/sys/dsl_dataset.h stable/10/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zfs_ioctl.c Directory Properties: stable/10/ (props changed) Modified: stable/10/cddl/contrib/opensolaris/lib/libzfs_core/common/libzfs_core.c ============================================================================== --- stable/10/cddl/contrib/opensolaris/lib/libzfs_core/common/libzfs_core.c Sat Oct 3 07:55:37 2015 (r288567) +++ stable/10/cddl/contrib/opensolaris/lib/libzfs_core/common/libzfs_core.c Sat Oct 3 07:57:32 2015 (r288568) @@ -20,7 +20,7 @@ */ /* - * Copyright (c) 2013 by Delphix. All rights reserved. + * Copyright (c) 2012, 2014 by Delphix. All rights reserved. * Copyright (c) 2013 Steven Hartland. All rights reserved. */ @@ -532,18 +532,30 @@ lzc_send(const char *snapname, const cha } /* - * If fromsnap is NULL, a full (non-incremental) stream will be estimated. + * "from" can be NULL, a snapshot, or a bookmark. + * + * If from is NULL, a full (non-incremental) stream will be estimated. This + * is calculated very efficiently. + * + * If from is a snapshot, lzc_send_space uses the deadlists attached to + * each snapshot to efficiently estimate the stream size. + * + * If from is a bookmark, the indirect blocks in the destination snapshot + * are traversed, looking for blocks with a birth time since the creation TXG of + * the snapshot this bookmark was created from. This will result in + * significantly more I/O and be less efficient than a send space estimation on + * an equivalent snapshot. */ int -lzc_send_space(const char *snapname, const char *fromsnap, uint64_t *spacep) +lzc_send_space(const char *snapname, const char *from, uint64_t *spacep) { nvlist_t *args; nvlist_t *result; int err; args = fnvlist_alloc(); - if (fromsnap != NULL) - fnvlist_add_string(args, "fromsnap", fromsnap); + if (from != NULL) + fnvlist_add_string(args, "from", from); err = lzc_ioctl(ZFS_IOC_SEND_SPACE, snapname, args, &result); nvlist_free(args); if (err == 0) Modified: stable/10/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/dmu_send.c ============================================================================== --- stable/10/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/dmu_send.c Sat Oct 3 07:55:37 2015 (r288567) +++ stable/10/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/dmu_send.c Sat Oct 3 07:57:32 2015 (r288568) @@ -856,6 +856,40 @@ dmu_send(const char *tosnap, const char return (err); } +static int +dmu_adjust_send_estimate_for_indirects(dsl_dataset_t *ds, uint64_t size, + uint64_t *sizep) +{ + int err; + /* + * Assume that space (both on-disk and in-stream) is dominated by + * data. We will adjust for indirect blocks and the copies property, + * but ignore per-object space used (eg, dnodes and DRR_OBJECT records). + */ + + /* + * Subtract out approximate space used by indirect blocks. + * Assume most space is used by data blocks (non-indirect, non-dnode). + * Assume all blocks are recordsize. Assume ditto blocks and + * internal fragmentation counter out compression. + * + * Therefore, space used by indirect blocks is sizeof(blkptr_t) per + * block, which we observe in practice. + */ + uint64_t recordsize; + err = dsl_prop_get_int_ds(ds, "recordsize", &recordsize); + if (err != 0) + return (err); + size -= size / recordsize * sizeof (blkptr_t); + + /* Add in the space for the record associated with each block. */ + size += size / recordsize * sizeof (dmu_replay_record_t); + + *sizep = size; + + return (0); +} + int dmu_send_estimate(dsl_dataset_t *ds, dsl_dataset_t *fromds, uint64_t *sizep) { @@ -891,33 +925,61 @@ dmu_send_estimate(dsl_dataset_t *ds, dsl return (err); } - /* - * Assume that space (both on-disk and in-stream) is dominated by - * data. We will adjust for indirect blocks and the copies property, - * but ignore per-object space used (eg, dnodes and DRR_OBJECT records). - */ + err = dmu_adjust_send_estimate_for_indirects(ds, size, sizep); + return (err); +} + +/* + * Simple callback used to traverse the blocks of a snapshot and sum their + * uncompressed size + */ +/* ARGSUSED */ +static int +dmu_calculate_send_traversal(spa_t *spa, zilog_t *zilog, const blkptr_t *bp, + const zbookmark_phys_t *zb, const dnode_phys_t *dnp, void *arg) +{ + uint64_t *spaceptr = arg; + if (bp != NULL && !BP_IS_HOLE(bp)) { + *spaceptr += BP_GET_UCSIZE(bp); + } + return (0); +} + +/* + * Given a desination snapshot and a TXG, calculate the approximate size of a + * send stream sent from that TXG. from_txg may be zero, indicating that the + * whole snapshot will be sent. + */ +int +dmu_send_estimate_from_txg(dsl_dataset_t *ds, uint64_t from_txg, + uint64_t *sizep) +{ + dsl_pool_t *dp = ds->ds_dir->dd_pool; + int err; + uint64_t size = 0; + + ASSERT(dsl_pool_config_held(dp)); + + /* tosnap must be a snapshot */ + if (!dsl_dataset_is_snapshot(ds)) + return (SET_ERROR(EINVAL)); + + /* verify that from_txg is before the provided snapshot was taken */ + if (from_txg >= dsl_dataset_phys(ds)->ds_creation_txg) { + return (SET_ERROR(EXDEV)); + } /* - * Subtract out approximate space used by indirect blocks. - * Assume most space is used by data blocks (non-indirect, non-dnode). - * Assume all blocks are recordsize. Assume ditto blocks and - * internal fragmentation counter out compression. - * - * Therefore, space used by indirect blocks is sizeof(blkptr_t) per - * block, which we observe in practice. + * traverse the blocks of the snapshot with birth times after + * from_txg, summing their uncompressed size */ - uint64_t recordsize; - err = dsl_prop_get_int_ds(ds, "recordsize", &recordsize); - if (err != 0) + err = traverse_dataset(ds, from_txg, TRAVERSE_POST, + dmu_calculate_send_traversal, &size); + if (err) return (err); - size -= size / recordsize * sizeof (blkptr_t); - - /* Add in the space for the record associated with each block. */ - size += size / recordsize * sizeof (dmu_replay_record_t); - - *sizep = size; - return (0); + err = dmu_adjust_send_estimate_for_indirects(ds, size, sizep); + return (err); } typedef struct dmu_recv_begin_arg { Modified: stable/10/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/sys/dmu_send.h ============================================================================== --- stable/10/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/sys/dmu_send.h Sat Oct 3 07:55:37 2015 (r288567) +++ stable/10/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/sys/dmu_send.h Sat Oct 3 07:57:32 2015 (r288568) @@ -21,7 +21,7 @@ /* * Copyright (c) 2005, 2010, Oracle and/or its affiliates. All rights reserved. - * Copyright (c) 2013 by Delphix. All rights reserved. + * Copyright (c) 2012, 2014 by Delphix. All rights reserved. * Copyright 2011 Nexenta Systems, Inc. All rights reserved. * Copyright (c) 2013, Joyent, Inc. All rights reserved. */ @@ -45,6 +45,8 @@ int dmu_send(const char *tosnap, const c #endif int dmu_send_estimate(struct dsl_dataset *ds, struct dsl_dataset *fromds, uint64_t *sizep); +int dmu_send_estimate_from_txg(struct dsl_dataset *ds, uint64_t fromtxg, + uint64_t *sizep); int dmu_send_obj(const char *pool, uint64_t tosnap, uint64_t fromsnap, boolean_t embedok, boolean_t large_block_ok, #ifdef illumos Modified: stable/10/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/sys/dsl_dataset.h ============================================================================== --- stable/10/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/sys/dsl_dataset.h Sat Oct 3 07:55:37 2015 (r288567) +++ stable/10/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/sys/dsl_dataset.h Sat Oct 3 07:57:32 2015 (r288568) @@ -201,6 +201,9 @@ dsl_dataset_phys(dsl_dataset_t *ds) */ #define MAX_TAG_PREFIX_LEN 17 +#define dsl_dataset_is_snapshot(ds) \ + (dsl_dataset_phys(ds)->ds_num_children != 0) + #define DS_UNIQUE_IS_ACCURATE(ds) \ ((dsl_dataset_phys(ds)->ds_flags & DS_FLAG_UNIQUE_ACCURATE) != 0) Modified: stable/10/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zfs_ioctl.c ============================================================================== --- stable/10/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zfs_ioctl.c Sat Oct 3 07:55:37 2015 (r288567) +++ stable/10/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zfs_ioctl.c Sat Oct 3 07:57:32 2015 (r288568) @@ -5488,7 +5488,8 @@ zfs_ioc_send_new(const char *snapname, n * of bytes that will be written to the fd supplied to zfs_ioc_send_new(). * * innvl: { - * (optional) "fromsnap" -> full snap name to send an incremental from + * (optional) "from" -> full snap or bookmark name to send an incremental + * from * } * * outnvl: { @@ -5499,7 +5500,6 @@ static int zfs_ioc_send_space(const char *snapname, nvlist_t *innvl, nvlist_t *outnvl) { dsl_pool_t *dp; - dsl_dataset_t *fromsnap = NULL; dsl_dataset_t *tosnap; int error; char *fromname; @@ -5515,27 +5515,55 @@ zfs_ioc_send_space(const char *snapname, return (error); } - error = nvlist_lookup_string(innvl, "fromsnap", &fromname); + error = nvlist_lookup_string(innvl, "from", &fromname); if (error == 0) { - error = dsl_dataset_hold(dp, fromname, FTAG, &fromsnap); - if (error != 0) { - dsl_dataset_rele(tosnap, FTAG); - dsl_pool_rele(dp, FTAG); - return (error); + if (strchr(fromname, '@') != NULL) { + /* + * If from is a snapshot, hold it and use the more + * efficient dmu_send_estimate to estimate send space + * size using deadlists. + */ + dsl_dataset_t *fromsnap; + error = dsl_dataset_hold(dp, fromname, FTAG, &fromsnap); + if (error != 0) + goto out; + error = dmu_send_estimate(tosnap, fromsnap, &space); + dsl_dataset_rele(fromsnap, FTAG); + } else if (strchr(fromname, '#') != NULL) { + /* + * If from is a bookmark, fetch the creation TXG of the + * snapshot it was created from and use that to find + * blocks that were born after it. + */ + zfs_bookmark_phys_t frombm; + + error = dsl_bookmark_lookup(dp, fromname, tosnap, + &frombm); + if (error != 0) + goto out; + error = dmu_send_estimate_from_txg(tosnap, + frombm.zbm_creation_txg, &space); + } else { + /* + * from is not properly formatted as a snapshot or + * bookmark + */ + error = SET_ERROR(EINVAL); + goto out; } + } else { + // If estimating the size of a full send, use dmu_send_estimate + error = dmu_send_estimate(tosnap, NULL, &space); } - error = dmu_send_estimate(tosnap, fromsnap, &space); fnvlist_add_uint64(outnvl, "space", space); - if (fromsnap != NULL) - dsl_dataset_rele(fromsnap, FTAG); +out: dsl_dataset_rele(tosnap, FTAG); dsl_pool_rele(dp, FTAG); return (error); } - static zfs_ioc_vec_t zfs_ioc_vec[ZFS_IOC_LAST - ZFS_IOC_FIRST]; static void From owner-svn-src-stable@freebsd.org Sat Oct 3 07:58:31 2015 Return-Path: Delivered-To: svn-src-stable@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id EC3AEA0D10D; Sat, 3 Oct 2015 07:58:31 +0000 (UTC) (envelope-from mav@FreeBSD.org) Received: from repo.freebsd.org (repo.freebsd.org [IPv6:2001:1900:2254:2068::e6a:0]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id DB3071EC0; Sat, 3 Oct 2015 07:58:31 +0000 (UTC) (envelope-from mav@FreeBSD.org) Received: from repo.freebsd.org ([127.0.1.70]) by repo.freebsd.org (8.15.2/8.15.2) with ESMTP id t937wV4t078990; Sat, 3 Oct 2015 07:58:31 GMT (envelope-from mav@FreeBSD.org) Received: (from mav@localhost) by repo.freebsd.org (8.15.2/8.15.2/Submit) id t937wThd078976; Sat, 3 Oct 2015 07:58:29 GMT (envelope-from mav@FreeBSD.org) Message-Id: <201510030758.t937wThd078976@repo.freebsd.org> X-Authentication-Warning: repo.freebsd.org: mav set sender to mav@FreeBSD.org using -f From: Alexander Motin Date: Sat, 3 Oct 2015 07:58:29 +0000 (UTC) To: src-committers@freebsd.org, svn-src-all@freebsd.org, svn-src-stable@freebsd.org, svn-src-stable-10@freebsd.org Subject: svn commit: r288569 - in stable/10/sys/cddl/contrib/opensolaris/uts/common/fs/zfs: . sys X-SVN-Group: stable-10 MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit X-BeenThere: svn-src-stable@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: SVN commit messages for all the -stable branches of the src tree List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 03 Oct 2015 07:58:32 -0000 Author: mav Date: Sat Oct 3 07:58:28 2015 New Revision: 288569 URL: https://svnweb.freebsd.org/changeset/base/288569 Log: MFC r286686: 5269 zpool import slow illumos/illumos-gate@12380e1e701fda28c9e9f32d01cafb54af279eb5 https://www.illumos.org/issues/5269 When importing a pool (at boot or with zpool import) with many filesystem, the process can take minutes. It doesn't matter whether the pool has been exported cleanly or uncleanly. The problem is that each dataset has its own log chain. On import, all datasets have to be checked if there are logs to replay. The idea is to speed up this process by paralellizing it. Reviewed by: Matthew Ahrens Reviewed by: George Wilson Reviewed by: Dan McDonald Approved by: Dan McDonald Author: Arne Jansen Modified: stable/10/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/dmu_objset.c stable/10/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/dsl_pool.c stable/10/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/spa.c stable/10/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/sys/dmu.h stable/10/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/sys/dmu_objset.h stable/10/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/sys/dsl_pool.h stable/10/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/sys/vdev.h stable/10/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/sys/zil.h stable/10/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev.c stable/10/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zil.c Directory Properties: stable/10/ (props changed) Modified: stable/10/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/dmu_objset.c ============================================================================== --- stable/10/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/dmu_objset.c Sat Oct 3 07:57:32 2015 (r288568) +++ stable/10/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/dmu_objset.c Sat Oct 3 07:58:28 2015 (r288569) @@ -25,6 +25,7 @@ * Copyright (c) 2013, Joyent, Inc. All rights reserved. * Copyright (c) 2014 Spectra Logic Corporation, All rights reserved. * Copyright 2015 Nexenta Systems, Inc. All rights reserved. + * Copyright (c) 2015, STRATO AG, Inc. All rights reserved. */ /* Portions Copyright 2010 Robert Milkowski */ @@ -49,6 +50,7 @@ #include #include #include +#include /* * Needed to close a window in dnode_move() that allows the objset to be freed @@ -56,6 +58,16 @@ */ krwlock_t os_lock; +/* + * Tunable to overwrite the maximum number of threads for the parallization + * of dmu_objset_find_dp, needed to speed up the import of pools with many + * datasets. + * Default is 4 times the number of leaf vdevs. + */ +int dmu_find_threads = 0; + +static void dmu_objset_find_dp_cb(void *arg); + void dmu_objset_init(void) { @@ -504,6 +516,25 @@ dmu_objset_hold(const char *name, void * return (err); } +static int +dmu_objset_own_impl(dsl_dataset_t *ds, dmu_objset_type_t type, + boolean_t readonly, void *tag, objset_t **osp) +{ + int err; + + err = dmu_objset_from_ds(ds, osp); + if (err != 0) { + dsl_dataset_disown(ds, tag); + } else if (type != DMU_OST_ANY && type != (*osp)->os_phys->os_type) { + dsl_dataset_disown(ds, tag); + return (SET_ERROR(EINVAL)); + } else if (!readonly && dsl_dataset_is_snapshot(ds)) { + dsl_dataset_disown(ds, tag); + return (SET_ERROR(EROFS)); + } + return (err); +} + /* * dsl_pool must not be held when this is called. * Upon successful return, there will be a longhold on the dataset, @@ -525,21 +556,26 @@ dmu_objset_own(const char *name, dmu_obj dsl_pool_rele(dp, FTAG); return (err); } - - err = dmu_objset_from_ds(ds, osp); + err = dmu_objset_own_impl(ds, type, readonly, tag, osp); dsl_pool_rele(dp, FTAG); - if (err != 0) { - dsl_dataset_disown(ds, tag); - } else if (type != DMU_OST_ANY && type != (*osp)->os_phys->os_type) { - dsl_dataset_disown(ds, tag); - return (SET_ERROR(EINVAL)); - } else if (!readonly && ds->ds_is_snapshot) { - dsl_dataset_disown(ds, tag); - return (SET_ERROR(EROFS)); - } + return (err); } +int +dmu_objset_own_obj(dsl_pool_t *dp, uint64_t obj, dmu_objset_type_t type, + boolean_t readonly, void *tag, objset_t **osp) +{ + dsl_dataset_t *ds; + int err; + + err = dsl_dataset_own_obj(dp, obj, tag, &ds); + if (err != 0) + return (err); + + return (dmu_objset_own_impl(ds, type, readonly, tag, osp)); +} + void dmu_objset_rele(objset_t *os, void *tag) { @@ -1580,30 +1616,41 @@ dmu_dir_list_next(objset_t *os, int name return (0); } -/* - * Find objsets under and including ddobj, call func(ds) on each. - */ -int -dmu_objset_find_dp(dsl_pool_t *dp, uint64_t ddobj, - int func(dsl_pool_t *, dsl_dataset_t *, void *), void *arg, int flags) +typedef struct dmu_objset_find_ctx { + taskq_t *dc_tq; + dsl_pool_t *dc_dp; + uint64_t dc_ddobj; + int (*dc_func)(dsl_pool_t *, dsl_dataset_t *, void *); + void *dc_arg; + int dc_flags; + kmutex_t *dc_error_lock; + int *dc_error; +} dmu_objset_find_ctx_t; + +static void +dmu_objset_find_dp_impl(dmu_objset_find_ctx_t *dcp) { + dsl_pool_t *dp = dcp->dc_dp; + dmu_objset_find_ctx_t *child_dcp; dsl_dir_t *dd; dsl_dataset_t *ds; zap_cursor_t zc; zap_attribute_t *attr; uint64_t thisobj; - int err; + int err = 0; - ASSERT(dsl_pool_config_held(dp)); + /* don't process if there already was an error */ + if (*dcp->dc_error != 0) + goto out; - err = dsl_dir_hold_obj(dp, ddobj, NULL, FTAG, &dd); + err = dsl_dir_hold_obj(dp, dcp->dc_ddobj, NULL, FTAG, &dd); if (err != 0) - return (err); + goto out; /* Don't visit hidden ($MOS & $ORIGIN) objsets. */ if (dd->dd_myname[0] == '$') { dsl_dir_rele(dd, FTAG); - return (0); + goto out; } thisobj = dsl_dir_phys(dd)->dd_head_dataset_obj; @@ -1612,7 +1659,7 @@ dmu_objset_find_dp(dsl_pool_t *dp, uint6 /* * Iterate over all children. */ - if (flags & DS_FIND_CHILDREN) { + if (dcp->dc_flags & DS_FIND_CHILDREN) { for (zap_cursor_init(&zc, dp->dp_meta_objset, dsl_dir_phys(dd)->dd_child_dir_zapobj); zap_cursor_retrieve(&zc, attr) == 0; @@ -1621,24 +1668,22 @@ dmu_objset_find_dp(dsl_pool_t *dp, uint6 sizeof (uint64_t)); ASSERT3U(attr->za_num_integers, ==, 1); - err = dmu_objset_find_dp(dp, attr->za_first_integer, - func, arg, flags); - if (err != 0) - break; + child_dcp = kmem_alloc(sizeof (*child_dcp), KM_SLEEP); + *child_dcp = *dcp; + child_dcp->dc_ddobj = attr->za_first_integer; + if (dcp->dc_tq != NULL) + (void) taskq_dispatch(dcp->dc_tq, + dmu_objset_find_dp_cb, child_dcp, TQ_SLEEP); + else + dmu_objset_find_dp_impl(child_dcp); } zap_cursor_fini(&zc); - - if (err != 0) { - dsl_dir_rele(dd, FTAG); - kmem_free(attr, sizeof (zap_attribute_t)); - return (err); - } } /* * Iterate over all snapshots. */ - if (flags & DS_FIND_SNAPSHOTS) { + if (dcp->dc_flags & DS_FIND_SNAPSHOTS) { dsl_dataset_t *ds; err = dsl_dataset_hold_obj(dp, thisobj, FTAG, &ds); @@ -1659,7 +1704,7 @@ dmu_objset_find_dp(dsl_pool_t *dp, uint6 attr->za_first_integer, FTAG, &ds); if (err != 0) break; - err = func(dp, ds, arg); + err = dcp->dc_func(dp, ds, dcp->dc_arg); dsl_dataset_rele(ds, FTAG); if (err != 0) break; @@ -1672,17 +1717,115 @@ dmu_objset_find_dp(dsl_pool_t *dp, uint6 kmem_free(attr, sizeof (zap_attribute_t)); if (err != 0) - return (err); + goto out; /* * Apply to self. */ err = dsl_dataset_hold_obj(dp, thisobj, FTAG, &ds); if (err != 0) - return (err); - err = func(dp, ds, arg); + goto out; + err = dcp->dc_func(dp, ds, dcp->dc_arg); dsl_dataset_rele(ds, FTAG); - return (err); + +out: + if (err != 0) { + mutex_enter(dcp->dc_error_lock); + /* only keep first error */ + if (*dcp->dc_error == 0) + *dcp->dc_error = err; + mutex_exit(dcp->dc_error_lock); + } + + kmem_free(dcp, sizeof (*dcp)); +} + +static void +dmu_objset_find_dp_cb(void *arg) +{ + dmu_objset_find_ctx_t *dcp = arg; + dsl_pool_t *dp = dcp->dc_dp; + + dsl_pool_config_enter(dp, FTAG); + + dmu_objset_find_dp_impl(dcp); + + dsl_pool_config_exit(dp, FTAG); +} + +/* + * Find objsets under and including ddobj, call func(ds) on each. + * The order for the enumeration is completely undefined. + * func is called with dsl_pool_config held. + */ +int +dmu_objset_find_dp(dsl_pool_t *dp, uint64_t ddobj, + int func(dsl_pool_t *, dsl_dataset_t *, void *), void *arg, int flags) +{ + int error = 0; + taskq_t *tq = NULL; + int ntasks; + dmu_objset_find_ctx_t *dcp; + kmutex_t err_lock; + + mutex_init(&err_lock, NULL, MUTEX_DEFAULT, NULL); + dcp = kmem_alloc(sizeof (*dcp), KM_SLEEP); + dcp->dc_tq = NULL; + dcp->dc_dp = dp; + dcp->dc_ddobj = ddobj; + dcp->dc_func = func; + dcp->dc_arg = arg; + dcp->dc_flags = flags; + dcp->dc_error_lock = &err_lock; + dcp->dc_error = &error; + + if ((flags & DS_FIND_SERIALIZE) || dsl_pool_config_held_writer(dp)) { + /* + * In case a write lock is held we can't make use of + * parallelism, as down the stack of the worker threads + * the lock is asserted via dsl_pool_config_held. + * In case of a read lock this is solved by getting a read + * lock in each worker thread, which isn't possible in case + * of a writer lock. So we fall back to the synchronous path + * here. + * In the future it might be possible to get some magic into + * dsl_pool_config_held in a way that it returns true for + * the worker threads so that a single lock held from this + * thread suffices. For now, stay single threaded. + */ + dmu_objset_find_dp_impl(dcp); + + return (error); + } + + ntasks = dmu_find_threads; + if (ntasks == 0) + ntasks = vdev_count_leaves(dp->dp_spa) * 4; + tq = taskq_create("dmu_objset_find", ntasks, minclsyspri, ntasks, + INT_MAX, 0); + if (tq == NULL) { + kmem_free(dcp, sizeof (*dcp)); + return (SET_ERROR(ENOMEM)); + } + dcp->dc_tq = tq; + + /* dcp will be freed by task */ + (void) taskq_dispatch(tq, dmu_objset_find_dp_cb, dcp, TQ_SLEEP); + + /* + * PORTING: this code relies on the property of taskq_wait to wait + * until no more tasks are queued and no more tasks are active. As + * we always queue new tasks from within other tasks, task_wait + * reliably waits for the full recursion to finish, even though we + * enqueue new tasks after taskq_wait has been called. + * On platforms other than illumos, taskq_wait may not have this + * property. + */ + taskq_wait(tq); + taskq_destroy(tq); + mutex_destroy(&err_lock); + + return (error); } /* Modified: stable/10/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/dsl_pool.c ============================================================================== --- stable/10/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/dsl_pool.c Sat Oct 3 07:57:32 2015 (r288568) +++ stable/10/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/dsl_pool.c Sat Oct 3 07:58:28 2015 (r288569) @@ -855,7 +855,7 @@ dsl_pool_upgrade_clones(dsl_pool_t *dp, ASSERT(dp->dp_origin_snap != NULL); VERIFY0(dmu_objset_find_dp(dp, dp->dp_root_dir_obj, upgrade_clones_cb, - tx, DS_FIND_CHILDREN)); + tx, DS_FIND_CHILDREN | DS_FIND_SERIALIZE)); } /* ARGSUSED */ @@ -909,7 +909,7 @@ dsl_pool_upgrade_dir_clones(dsl_pool_t * VERIFY0(bpobj_open(&dp->dp_free_bpobj, dp->dp_meta_objset, obj)); VERIFY0(dmu_objset_find_dp(dp, dp->dp_root_dir_obj, - upgrade_dir_clones_cb, tx, DS_FIND_CHILDREN)); + upgrade_dir_clones_cb, tx, DS_FIND_CHILDREN | DS_FIND_SERIALIZE)); } void @@ -1153,3 +1153,9 @@ dsl_pool_config_held(dsl_pool_t *dp) { return (RRW_LOCK_HELD(&dp->dp_config_rwlock)); } + +boolean_t +dsl_pool_config_held_writer(dsl_pool_t *dp) +{ + return (RRW_WRITE_HELD(&dp->dp_config_rwlock)); +} Modified: stable/10/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/spa.c ============================================================================== --- stable/10/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/spa.c Sat Oct 3 07:57:32 2015 (r288568) +++ stable/10/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/spa.c Sat Oct 3 07:58:28 2015 (r288569) @@ -1789,13 +1789,14 @@ static boolean_t spa_check_logs(spa_t *spa) { boolean_t rv = B_FALSE; + dsl_pool_t *dp = spa_get_dsl(spa); switch (spa->spa_log_state) { case SPA_LOG_MISSING: /* need to recheck in case slog has been restored */ case SPA_LOG_UNKNOWN: - rv = (dmu_objset_find(spa->spa_name, zil_check_log_chain, - NULL, DS_FIND_CHILDREN) != 0); + rv = (dmu_objset_find_dp(dp, dp->dp_root_dir_obj, + zil_check_log_chain, NULL, DS_FIND_CHILDREN) != 0); if (rv) spa_set_log_state(spa, SPA_LOG_MISSING); break; @@ -2783,6 +2784,7 @@ spa_load_impl(spa_t *spa, uint64_t pool_ spa->spa_load_max_txg == UINT64_MAX)) { dmu_tx_t *tx; int need_update = B_FALSE; + dsl_pool_t *dp = spa_get_dsl(spa); ASSERT(state != SPA_LOAD_TRYIMPORT); @@ -2795,9 +2797,8 @@ spa_load_impl(spa_t *spa, uint64_t pool_ */ spa->spa_claiming = B_TRUE; - tx = dmu_tx_create_assigned(spa_get_dsl(spa), - spa_first_txg(spa)); - (void) dmu_objset_find(spa_name(spa), + tx = dmu_tx_create_assigned(dp, spa_first_txg(spa)); + (void) dmu_objset_find_dp(dp, dp->dp_root_dir_obj, zil_claim, tx, DS_FIND_CHILDREN); dmu_tx_commit(tx); Modified: stable/10/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/sys/dmu.h ============================================================================== --- stable/10/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/sys/dmu.h Sat Oct 3 07:57:32 2015 (r288568) +++ stable/10/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/sys/dmu.h Sat Oct 3 07:58:28 2015 (r288569) @@ -243,6 +243,7 @@ void zfs_znode_byteswap(void *buf, size_ #define DS_FIND_SNAPSHOTS (1<<0) #define DS_FIND_CHILDREN (1<<1) +#define DS_FIND_SERIALIZE (1<<2) /* * The maximum number of bytes that can be accessed as part of one Modified: stable/10/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/sys/dmu_objset.h ============================================================================== --- stable/10/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/sys/dmu_objset.h Sat Oct 3 07:57:32 2015 (r288568) +++ stable/10/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/sys/dmu_objset.h Sat Oct 3 07:58:28 2015 (r288569) @@ -142,6 +142,8 @@ struct objset { int dmu_objset_hold(const char *name, void *tag, objset_t **osp); int dmu_objset_own(const char *name, dmu_objset_type_t type, boolean_t readonly, void *tag, objset_t **osp); +int dmu_objset_own_obj(struct dsl_pool *dp, uint64_t obj, + dmu_objset_type_t type, boolean_t readonly, void *tag, objset_t **osp); void dmu_objset_refresh_ownership(objset_t *os, void *tag); void dmu_objset_rele(objset_t *os, void *tag); void dmu_objset_disown(objset_t *os, void *tag); Modified: stable/10/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/sys/dsl_pool.h ============================================================================== --- stable/10/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/sys/dsl_pool.h Sat Oct 3 07:57:32 2015 (r288568) +++ stable/10/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/sys/dsl_pool.h Sat Oct 3 07:58:28 2015 (r288569) @@ -154,6 +154,7 @@ void dsl_pool_mos_diduse_space(dsl_pool_ void dsl_pool_config_enter(dsl_pool_t *dp, void *tag); void dsl_pool_config_exit(dsl_pool_t *dp, void *tag); boolean_t dsl_pool_config_held(dsl_pool_t *dp); +boolean_t dsl_pool_config_held_writer(dsl_pool_t *dp); boolean_t dsl_pool_need_dirty_delay(dsl_pool_t *dp); taskq_t *dsl_pool_vnrele_taskq(dsl_pool_t *dp); Modified: stable/10/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/sys/vdev.h ============================================================================== --- stable/10/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/sys/vdev.h Sat Oct 3 07:57:32 2015 (r288568) +++ stable/10/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/sys/vdev.h Sat Oct 3 07:58:28 2015 (r288569) @@ -61,6 +61,7 @@ extern zio_t *vdev_probe(vdev_t *vd, zio extern boolean_t vdev_is_bootable(vdev_t *vd); extern vdev_t *vdev_lookup_top(spa_t *spa, uint64_t vdev); extern vdev_t *vdev_lookup_by_guid(vdev_t *vd, uint64_t guid); +extern int vdev_count_leaves(spa_t *spa); extern void vdev_dtl_dirty(vdev_t *vd, vdev_dtl_type_t d, uint64_t txg, uint64_t size); extern boolean_t vdev_dtl_contains(vdev_t *vd, vdev_dtl_type_t d, Modified: stable/10/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/sys/zil.h ============================================================================== --- stable/10/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/sys/zil.h Sat Oct 3 07:57:32 2015 (r288568) +++ stable/10/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/sys/zil.h Sat Oct 3 07:58:28 2015 (r288569) @@ -37,6 +37,9 @@ extern "C" { #endif +struct dsl_pool; +struct dsl_dataset; + /* * Intent log format: * @@ -404,8 +407,10 @@ extern void zil_itx_assign(zilog_t *zilo extern void zil_commit(zilog_t *zilog, uint64_t oid); extern int zil_vdev_offline(const char *osname, void *txarg); -extern int zil_claim(const char *osname, void *txarg); -extern int zil_check_log_chain(const char *osname, void *txarg); +extern int zil_claim(struct dsl_pool *dp, + struct dsl_dataset *ds, void *txarg); +extern int zil_check_log_chain(struct dsl_pool *dp, + struct dsl_dataset *ds, void *tx); extern void zil_sync(zilog_t *zilog, dmu_tx_t *tx); extern void zil_clean(zilog_t *zilog, uint64_t synced_txg); Modified: stable/10/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev.c ============================================================================== --- stable/10/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev.c Sat Oct 3 07:57:32 2015 (r288568) +++ stable/10/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev.c Sat Oct 3 07:58:28 2015 (r288569) @@ -272,6 +272,26 @@ vdev_lookup_by_guid(vdev_t *vd, uint64_t return (NULL); } +static int +vdev_count_leaves_impl(vdev_t *vd) +{ + int n = 0; + + if (vd->vdev_ops->vdev_op_leaf) + return (1); + + for (int c = 0; c < vd->vdev_children; c++) + n += vdev_count_leaves_impl(vd->vdev_child[c]); + + return (n); +} + +int +vdev_count_leaves(spa_t *spa) +{ + return (vdev_count_leaves_impl(spa->spa_root_vdev)); +} + void vdev_add_child(vdev_t *pvd, vdev_t *cvd) { Modified: stable/10/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zil.c ============================================================================== --- stable/10/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zil.c Sat Oct 3 07:57:32 2015 (r288568) +++ stable/10/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zil.c Sat Oct 3 07:58:28 2015 (r288569) @@ -639,7 +639,7 @@ zil_destroy_sync(zilog_t *zilog, dmu_tx_ } int -zil_claim(const char *osname, void *txarg) +zil_claim(dsl_pool_t *dp, dsl_dataset_t *ds, void *txarg) { dmu_tx_t *tx = txarg; uint64_t first_txg = dmu_tx_get_txg(tx); @@ -648,15 +648,16 @@ zil_claim(const char *osname, void *txar objset_t *os; int error; - error = dmu_objset_own(osname, DMU_OST_ANY, B_FALSE, FTAG, &os); + error = dmu_objset_own_obj(dp, ds->ds_object, + DMU_OST_ANY, B_FALSE, FTAG, &os); if (error != 0) { /* * EBUSY indicates that the objset is inconsistent, in which * case it can not have a ZIL. */ if (error != EBUSY) { - cmn_err(CE_WARN, "can't open objset for %s, error %u", - osname, error); + cmn_err(CE_WARN, "can't open objset for %llu, error %u", + (unsigned long long)ds->ds_object, error); } return (0); } @@ -703,8 +704,9 @@ zil_claim(const char *osname, void *txar * Checksum errors are ok as they indicate the end of the chain. * Any other error (no device or read failure) returns an error. */ +/* ARGSUSED */ int -zil_check_log_chain(const char *osname, void *tx) +zil_check_log_chain(dsl_pool_t *dp, dsl_dataset_t *ds, void *tx) { zilog_t *zilog; objset_t *os; @@ -713,9 +715,10 @@ zil_check_log_chain(const char *osname, ASSERT(tx == NULL); - error = dmu_objset_hold(osname, FTAG, &os); + error = dmu_objset_from_ds(ds, &os); if (error != 0) { - cmn_err(CE_WARN, "can't open objset for %s", osname); + cmn_err(CE_WARN, "can't open objset %llu, error %d", + (unsigned long long)ds->ds_object, error); return (0); } @@ -738,10 +741,8 @@ zil_check_log_chain(const char *osname, valid = vdev_log_state_valid(vd); spa_config_exit(os->os_spa, SCL_STATE, FTAG); - if (!valid) { - dmu_objset_rele(os, FTAG); + if (!valid) return (0); - } } /* @@ -754,8 +755,6 @@ zil_check_log_chain(const char *osname, error = zil_parse(zilog, zil_claim_log_block, zil_claim_log_record, tx, zilog->zl_header->zh_claim_txg ? -1ULL : spa_first_txg(os->os_spa)); - dmu_objset_rele(os, FTAG); - return ((error == ECKSUM || error == ENOENT) ? 0 : error); } From owner-svn-src-stable@freebsd.org Sat Oct 3 07:59:28 2015 Return-Path: Delivered-To: svn-src-stable@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id E843BA0D1A6; Sat, 3 Oct 2015 07:59:27 +0000 (UTC) (envelope-from mav@FreeBSD.org) Received: from repo.freebsd.org (repo.freebsd.org [IPv6:2001:1900:2254:2068::e6a:0]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id D4F311012; Sat, 3 Oct 2015 07:59:27 +0000 (UTC) (envelope-from mav@FreeBSD.org) Received: from repo.freebsd.org ([127.0.1.70]) by repo.freebsd.org (8.15.2/8.15.2) with ESMTP id t937xR8s079153; Sat, 3 Oct 2015 07:59:27 GMT (envelope-from mav@FreeBSD.org) Received: (from mav@localhost) by repo.freebsd.org (8.15.2/8.15.2/Submit) id t937xQoa079143; Sat, 3 Oct 2015 07:59:26 GMT (envelope-from mav@FreeBSD.org) Message-Id: <201510030759.t937xQoa079143@repo.freebsd.org> X-Authentication-Warning: repo.freebsd.org: mav set sender to mav@FreeBSD.org using -f From: Alexander Motin Date: Sat, 3 Oct 2015 07:59:26 +0000 (UTC) To: src-committers@freebsd.org, svn-src-all@freebsd.org, svn-src-stable@freebsd.org, svn-src-stable-10@freebsd.org Subject: svn commit: r288570 - in stable/10/sys/cddl/contrib/opensolaris/uts/common/fs/zfs: . sys X-SVN-Group: stable-10 MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit X-BeenThere: svn-src-stable@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: SVN commit messages for all the -stable branches of the src tree List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 03 Oct 2015 07:59:28 -0000 Author: mav Date: Sat Oct 3 07:59:25 2015 New Revision: 288570 URL: https://svnweb.freebsd.org/changeset/base/288570 Log: MFC r286689: 5981 Deadlock in dmu_objset_find_dp illumos/illumos-gate@1d3f896f5469c69c1339890ec3d68e9feddb0343 https://www.illumos.org/issues/5981 When dmu_objset_find_dp gets called with a read lock held, it fans out the work to the task queue. Each task in turn acquires its own read lock before calling the callback. If during this process anyone tries to a acquire a write lock, it will stall all read lock requests.Thus the tasks will never finish, the read lock of the caller will never get freed and the write lock never acquired. deadlock. Reviewed by: Matthew Ahrens Reviewed by: Dan McDonald Approved by: Robert Mustacchi Author: Arne Jansen Modified: stable/10/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/dmu_objset.c stable/10/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/dsl_pool.c stable/10/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/rrwlock.c stable/10/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/sys/dsl_pool.h stable/10/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/sys/rrwlock.h Directory Properties: stable/10/ (props changed) Modified: stable/10/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/dmu_objset.c ============================================================================== --- stable/10/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/dmu_objset.c Sat Oct 3 07:58:28 2015 (r288569) +++ stable/10/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/dmu_objset.c Sat Oct 3 07:59:25 2015 (r288570) @@ -1746,7 +1746,15 @@ dmu_objset_find_dp_cb(void *arg) dmu_objset_find_ctx_t *dcp = arg; dsl_pool_t *dp = dcp->dc_dp; - dsl_pool_config_enter(dp, FTAG); + /* + * We need to get a pool_config_lock here, as there are several + * asssert(pool_config_held) down the stack. Getting a lock via + * dsl_pool_config_enter is risky, as it might be stalled by a + * pending writer. This would deadlock, as the write lock can + * only be granted when our parent thread gives up the lock. + * The _prio interface gives us priority over a pending writer. + */ + dsl_pool_config_enter_prio(dp, FTAG); dmu_objset_find_dp_impl(dcp); Modified: stable/10/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/dsl_pool.c ============================================================================== --- stable/10/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/dsl_pool.c Sat Oct 3 07:58:28 2015 (r288569) +++ stable/10/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/dsl_pool.c Sat Oct 3 07:59:25 2015 (r288570) @@ -1143,6 +1143,13 @@ dsl_pool_config_enter(dsl_pool_t *dp, vo } void +dsl_pool_config_enter_prio(dsl_pool_t *dp, void *tag) +{ + ASSERT(!rrw_held(&dp->dp_config_rwlock, RW_READER)); + rrw_enter_read_prio(&dp->dp_config_rwlock, tag); +} + +void dsl_pool_config_exit(dsl_pool_t *dp, void *tag) { rrw_exit(&dp->dp_config_rwlock, tag); Modified: stable/10/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/rrwlock.c ============================================================================== --- stable/10/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/rrwlock.c Sat Oct 3 07:58:28 2015 (r288569) +++ stable/10/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/rrwlock.c Sat Oct 3 07:59:25 2015 (r288570) @@ -159,8 +159,8 @@ rrw_destroy(rrwlock_t *rrl) refcount_destroy(&rrl->rr_linked_rcount); } -void -rrw_enter_read(rrwlock_t *rrl, void *tag) +static void +rrw_enter_read_impl(rrwlock_t *rrl, boolean_t prio, void *tag) { mutex_enter(&rrl->rr_lock); #if !defined(DEBUG) && defined(_KERNEL) @@ -176,7 +176,7 @@ rrw_enter_read(rrwlock_t *rrl, void *tag ASSERT(refcount_count(&rrl->rr_anon_rcount) >= 0); while (rrl->rr_writer != NULL || (rrl->rr_writer_wanted && - refcount_is_zero(&rrl->rr_anon_rcount) && + refcount_is_zero(&rrl->rr_anon_rcount) && !prio && rrn_find(rrl) == NULL)) cv_wait(&rrl->rr_cv, &rrl->rr_lock); @@ -192,6 +192,25 @@ rrw_enter_read(rrwlock_t *rrl, void *tag } void +rrw_enter_read(rrwlock_t *rrl, void *tag) +{ + rrw_enter_read_impl(rrl, B_FALSE, tag); +} + +/* + * take a read lock even if there are pending write lock requests. if we want + * to take a lock reentrantly, but from different threads (that have a + * relationship to each other), the normal detection mechanism to overrule + * the pending writer does not work, so we have to give an explicit hint here. + */ +void +rrw_enter_read_prio(rrwlock_t *rrl, void *tag) +{ + rrw_enter_read_impl(rrl, B_TRUE, tag); +} + + +void rrw_enter_write(rrwlock_t *rrl) { mutex_enter(&rrl->rr_lock); Modified: stable/10/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/sys/dsl_pool.h ============================================================================== --- stable/10/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/sys/dsl_pool.h Sat Oct 3 07:58:28 2015 (r288569) +++ stable/10/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/sys/dsl_pool.h Sat Oct 3 07:59:25 2015 (r288570) @@ -152,6 +152,7 @@ void dsl_pool_upgrade_dir_clones(dsl_poo void dsl_pool_mos_diduse_space(dsl_pool_t *dp, int64_t used, int64_t comp, int64_t uncomp); void dsl_pool_config_enter(dsl_pool_t *dp, void *tag); +void dsl_pool_config_enter_prio(dsl_pool_t *dp, void *tag); void dsl_pool_config_exit(dsl_pool_t *dp, void *tag); boolean_t dsl_pool_config_held(dsl_pool_t *dp); boolean_t dsl_pool_config_held_writer(dsl_pool_t *dp); Modified: stable/10/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/sys/rrwlock.h ============================================================================== --- stable/10/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/sys/rrwlock.h Sat Oct 3 07:58:28 2015 (r288569) +++ stable/10/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/sys/rrwlock.h Sat Oct 3 07:59:25 2015 (r288570) @@ -69,6 +69,7 @@ void rrw_init(rrwlock_t *rrl, boolean_t void rrw_destroy(rrwlock_t *rrl); void rrw_enter(rrwlock_t *rrl, krw_t rw, void *tag); void rrw_enter_read(rrwlock_t *rrl, void *tag); +void rrw_enter_read_prio(rrwlock_t *rrl, void *tag); void rrw_enter_write(rrwlock_t *rrl); void rrw_exit(rrwlock_t *rrl, void *tag); boolean_t rrw_held(rrwlock_t *rrl, krw_t rw); From owner-svn-src-stable@freebsd.org Sat Oct 3 08:03:41 2015 Return-Path: Delivered-To: svn-src-stable@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id E5BC4A0D955; Sat, 3 Oct 2015 08:03:41 +0000 (UTC) (envelope-from mav@FreeBSD.org) Received: from repo.freebsd.org (repo.freebsd.org [IPv6:2001:1900:2254:2068::e6a:0]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id D2C291558; Sat, 3 Oct 2015 08:03:41 +0000 (UTC) (envelope-from mav@FreeBSD.org) Received: from repo.freebsd.org ([127.0.1.70]) by repo.freebsd.org (8.15.2/8.15.2) with ESMTP id t9383f33083372; Sat, 3 Oct 2015 08:03:41 GMT (envelope-from mav@FreeBSD.org) Received: (from mav@localhost) by repo.freebsd.org (8.15.2/8.15.2/Submit) id t9383bB2083349; Sat, 3 Oct 2015 08:03:37 GMT (envelope-from mav@FreeBSD.org) Message-Id: <201510030803.t9383bB2083349@repo.freebsd.org> X-Authentication-Warning: repo.freebsd.org: mav set sender to mav@FreeBSD.org using -f From: Alexander Motin Date: Sat, 3 Oct 2015 08:03:37 +0000 (UTC) To: src-committers@freebsd.org, svn-src-all@freebsd.org, svn-src-stable@freebsd.org, svn-src-stable-10@freebsd.org Subject: svn commit: r288571 - in stable/10: cddl/contrib/opensolaris/cmd/zdb cddl/contrib/opensolaris/cmd/zfs cddl/contrib/opensolaris/cmd/ztest cddl/contrib/opensolaris/lib/libzfs/common cddl/contrib/open... X-SVN-Group: stable-10 MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit X-BeenThere: svn-src-stable@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: SVN commit messages for all the -stable branches of the src tree List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 03 Oct 2015 08:03:42 -0000 Author: mav Date: Sat Oct 3 08:03:36 2015 New Revision: 288571 URL: https://svnweb.freebsd.org/changeset/base/288571 Log: MFC r286705: 5960 zfs recv should prefetch indirect blocks 5925 zfs receive -o origin= Reviewed by: Prakash Surya Reviewed by: Matthew Ahrens Author: Paul Dagnelie While running 'zfs recv' we noticed that every 128th 8K block required a read. We were seeing that restore_write() was calling dmu_tx_hold_write() and the indirect block was not cached. We should prefetch upcoming indirect blocks to avoid having to go to disk and blocking the restore_write(). Allow an incremental send stream to be received as a clone, even if the stream does not mark it as a clone. Added: stable/10/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/bqueue.c - copied unchanged from r286705, head/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/bqueue.c stable/10/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/sys/bqueue.h - copied unchanged from r286705, head/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/sys/bqueue.h stable/10/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/sys/zio_priority.h - copied unchanged from r286705, head/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/sys/zio_priority.h Modified: stable/10/cddl/contrib/opensolaris/cmd/zdb/zdb.c stable/10/cddl/contrib/opensolaris/cmd/zfs/zfs.8 stable/10/cddl/contrib/opensolaris/cmd/zfs/zfs_main.c stable/10/cddl/contrib/opensolaris/cmd/ztest/ztest.c stable/10/cddl/contrib/opensolaris/lib/libzfs/common/libzfs.h stable/10/cddl/contrib/opensolaris/lib/libzfs/common/libzfs_pool.c stable/10/cddl/contrib/opensolaris/lib/libzfs/common/libzfs_sendrecv.c stable/10/cddl/contrib/opensolaris/lib/libzpool/common/sys/zfs_context.h stable/10/sys/cddl/contrib/opensolaris/uts/common/Makefile.files stable/10/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/bptree.c stable/10/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/dbuf.c stable/10/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/dmu.c stable/10/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/dmu_diff.c stable/10/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/dmu_object.c stable/10/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/dmu_send.c stable/10/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/dmu_traverse.c stable/10/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/dmu_tx.c stable/10/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/dmu_zfetch.c stable/10/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/dnode.c stable/10/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/dnode_sync.c stable/10/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/dsl_dataset.c stable/10/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/dsl_destroy.c stable/10/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/dsl_scan.c stable/10/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/spa.c stable/10/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/space_map.c stable/10/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/sys/dbuf.h stable/10/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/sys/dmu.h stable/10/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/sys/dsl_dataset.h stable/10/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/sys/zio.h stable/10/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/sys/zio_checksum.h stable/10/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zap.c stable/10/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zfs_vfsops.c stable/10/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zfs_vnops.c stable/10/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c stable/10/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zvol.c Directory Properties: stable/10/ (props changed) Modified: stable/10/cddl/contrib/opensolaris/cmd/zdb/zdb.c ============================================================================== --- stable/10/cddl/contrib/opensolaris/cmd/zdb/zdb.c Sat Oct 3 07:59:25 2015 (r288570) +++ stable/10/cddl/contrib/opensolaris/cmd/zdb/zdb.c Sat Oct 3 08:03:36 2015 (r288571) @@ -2429,6 +2429,9 @@ zdb_blkptr_cb(spa_t *spa, zilog_t *zilog dmu_object_type_t type; boolean_t is_metadata; + if (bp == NULL) + return (0); + if (dump_opt['b'] >= 5 && bp->blk_birth > 0) { char blkbuf[BP_SPRINTF_LEN]; snprintf_blkptr(blkbuf, sizeof (blkbuf), bp); @@ -2918,7 +2921,7 @@ zdb_ddt_add_cb(spa_t *spa, zilog_t *zilo avl_index_t where; zdb_ddt_entry_t *zdde, zdde_search; - if (BP_IS_HOLE(bp) || BP_IS_EMBEDDED(bp)) + if (bp == NULL || BP_IS_HOLE(bp) || BP_IS_EMBEDDED(bp)) return (0); if (dump_opt['S'] > 1 && zb->zb_level == ZB_ROOT_LEVEL) { Modified: stable/10/cddl/contrib/opensolaris/cmd/zfs/zfs.8 ============================================================================== --- stable/10/cddl/contrib/opensolaris/cmd/zfs/zfs.8 Sat Oct 3 07:59:25 2015 (r288570) +++ stable/10/cddl/contrib/opensolaris/cmd/zfs/zfs.8 Sat Oct 3 08:03:36 2015 (r288571) @@ -191,11 +191,13 @@ .Nm .Cm receive Ns | Ns Cm recv .Op Fl vnFu +.Op Fl o Sy origin Ns = Ns Ar snapshot .Ar filesystem Ns | Ns Ar volume Ns | Ns Ar snapshot .Nm .Cm receive Ns | Ns Cm recv .Op Fl vnFu .Op Fl d | e +.Op Fl o Sy origin Ns = Ns Ar snapshot .Ar filesystem .Nm .Cm allow @@ -2711,6 +2713,7 @@ feature. .Nm .Cm receive Ns | Ns Cm recv .Op Fl vnFu +.Op Fl o Sy origin Ns = Ns Ar snapshot .Ar filesystem Ns | Ns Ar volume Ns | Ns Ar snapshot .Xc .It Xo @@ -2718,6 +2721,7 @@ feature. .Cm receive Ns | Ns Cm recv .Op Fl vnFu .Op Fl d | e +.Op Fl o Sy origin Ns = Ns Ar snapshot .Ar filesystem .Xc .Pp @@ -2802,6 +2806,10 @@ receive operation. Do not actually receive the stream. This can be useful in conjunction with the .Fl v option to verify the name the receive operation would use. +.It Fl o Sy origin Ns = Ns Ar snapshot +Forces the stream to be received as a clone of the given snapshot. +This is only valid if the stream is an incremental stream whose source +is the same as the provided origin. .It Fl F Force a rollback of the file system to the most recent snapshot before performing the receive operation. If receiving an incremental replication Modified: stable/10/cddl/contrib/opensolaris/cmd/zfs/zfs_main.c ============================================================================== --- stable/10/cddl/contrib/opensolaris/cmd/zfs/zfs_main.c Sat Oct 3 07:59:25 2015 (r288570) +++ stable/10/cddl/contrib/opensolaris/cmd/zfs/zfs_main.c Sat Oct 3 08:03:36 2015 (r288571) @@ -264,8 +264,9 @@ get_usage(zfs_help_t idx) return (gettext("\tpromote \n")); case HELP_RECEIVE: return (gettext("\treceive|recv [-vnFu] \n" - "\treceive|recv [-vnFu] [-d | -e] \n")); + "snapshot>\n" + "\treceive|recv [-vnFu] [-o origin=] [-d | -e] " + "\n")); case HELP_RENAME: return (gettext("\trename [-f] " "\n" @@ -791,7 +792,7 @@ zfs_do_create(int argc, char **argv) nomem(); break; case 'o': - if (parseprop(props, optarg)) + if (parseprop(props, optarg) != 0) goto error; break; case 's': @@ -3663,7 +3664,7 @@ zfs_do_snapshot(int argc, char **argv) while ((c = getopt(argc, argv, "ro:")) != -1) { switch (c) { case 'o': - if (parseprop(props, optarg)) + if (parseprop(props, optarg) != 0) return (1); break; case 'r': @@ -3922,10 +3923,19 @@ zfs_do_receive(int argc, char **argv) { int c, err; recvflags_t flags = { 0 }; + nvlist_t *props; + nvpair_t *nvp = NULL; + + if (nvlist_alloc(&props, NV_UNIQUE_NAME, 0) != 0) + nomem(); /* check options */ - while ((c = getopt(argc, argv, ":denuvF")) != -1) { + while ((c = getopt(argc, argv, ":o:denuvF")) != -1) { switch (c) { + case 'o': + if (parseprop(props, optarg) != 0) + return (1); + break; case 'd': flags.isprefix = B_TRUE; break; @@ -3970,6 +3980,13 @@ zfs_do_receive(int argc, char **argv) usage(B_FALSE); } + while ((nvp = nvlist_next_nvpair(props, nvp))) { + if (strcmp(nvpair_name(nvp), "origin") != 0) { + (void) fprintf(stderr, gettext("invalid option")); + usage(B_FALSE); + } + } + if (isatty(STDIN_FILENO)) { (void) fprintf(stderr, gettext("Error: Backup stream can not be read " @@ -3978,7 +3995,7 @@ zfs_do_receive(int argc, char **argv) return (1); } - err = zfs_receive(g_zfs, argv[0], &flags, STDIN_FILENO, NULL); + err = zfs_receive(g_zfs, argv[0], props, &flags, STDIN_FILENO, NULL); return (err != 0); } Modified: stable/10/cddl/contrib/opensolaris/cmd/ztest/ztest.c ============================================================================== --- stable/10/cddl/contrib/opensolaris/cmd/ztest/ztest.c Sat Oct 3 07:59:25 2015 (r288570) +++ stable/10/cddl/contrib/opensolaris/cmd/ztest/ztest.c Sat Oct 3 08:03:36 2015 (r288571) @@ -3586,7 +3586,8 @@ ztest_dmu_read_write(ztest_ds_t *zd, uin */ n = ztest_random(regions) * stride + ztest_random(width); s = 1 + ztest_random(2 * width - 1); - dmu_prefetch(os, bigobj, n * chunksize, s * chunksize); + dmu_prefetch(os, bigobj, 0, n * chunksize, s * chunksize, + ZIO_PRIORITY_SYNC_READ); /* * Pick a random index and compute the offsets into packobj and bigobj. @@ -5705,8 +5706,10 @@ ztest_run(ztest_shared_t *zs) * Right before closing the pool, kick off a bunch of async I/O; * spa_close() should wait for it to complete. */ - for (uint64_t object = 1; object < 50; object++) - dmu_prefetch(spa->spa_meta_objset, object, 0, 1ULL << 20); + for (uint64_t object = 1; object < 50; object++) { + dmu_prefetch(spa->spa_meta_objset, object, 0, 0, 1ULL << 20, + ZIO_PRIORITY_SYNC_READ); + } spa_close(spa, FTAG); Modified: stable/10/cddl/contrib/opensolaris/lib/libzfs/common/libzfs.h ============================================================================== --- stable/10/cddl/contrib/opensolaris/lib/libzfs/common/libzfs.h Sat Oct 3 07:59:25 2015 (r288570) +++ stable/10/cddl/contrib/opensolaris/lib/libzfs/common/libzfs.h Sat Oct 3 08:03:36 2015 (r288571) @@ -668,8 +668,8 @@ typedef struct recvflags { boolean_t nomount; } recvflags_t; -extern int zfs_receive(libzfs_handle_t *, const char *, recvflags_t *, - int, avl_tree_t *); +extern int zfs_receive(libzfs_handle_t *, const char *, nvlist_t *, + recvflags_t *, int, avl_tree_t *); typedef enum diff_flags { ZFS_DIFF_PARSEABLE = 0x1, Modified: stable/10/cddl/contrib/opensolaris/lib/libzfs/common/libzfs_pool.c ============================================================================== --- stable/10/cddl/contrib/opensolaris/lib/libzfs/common/libzfs_pool.c Sat Oct 3 07:59:25 2015 (r288570) +++ stable/10/cddl/contrib/opensolaris/lib/libzfs/common/libzfs_pool.c Sat Oct 3 08:03:36 2015 (r288571) @@ -3535,7 +3535,7 @@ zpool_vdev_name(libzfs_handle_t *hdl, zp } static int -zbookmark_compare(const void *a, const void *b) +zbookmark_mem_compare(const void *a, const void *b) { return (memcmp(a, b, sizeof (zbookmark_phys_t))); } @@ -3598,7 +3598,7 @@ zpool_get_errlog(zpool_handle_t *zhp, nv zc.zc_nvlist_dst_size; count -= zc.zc_nvlist_dst_size; - qsort(zb, count, sizeof (zbookmark_phys_t), zbookmark_compare); + qsort(zb, count, sizeof (zbookmark_phys_t), zbookmark_mem_compare); verify(nvlist_alloc(nverrlistp, 0, KM_SLEEP) == 0); Modified: stable/10/cddl/contrib/opensolaris/lib/libzfs/common/libzfs_sendrecv.c ============================================================================== --- stable/10/cddl/contrib/opensolaris/lib/libzfs/common/libzfs_sendrecv.c Sat Oct 3 07:59:25 2015 (r288570) +++ stable/10/cddl/contrib/opensolaris/lib/libzfs/common/libzfs_sendrecv.c Sat Oct 3 08:03:36 2015 (r288571) @@ -64,8 +64,9 @@ extern void zfs_setprop_error(libzfs_han /* We need to use something for ENODATA. */ #define ENODATA EIDRM -static int zfs_receive_impl(libzfs_handle_t *, const char *, recvflags_t *, - int, const char *, nvlist_t *, avl_tree_t *, char **, int, uint64_t *); +static int zfs_receive_impl(libzfs_handle_t *, const char *, const char *, + recvflags_t *, int, const char *, nvlist_t *, avl_tree_t *, char **, int, + uint64_t *); static const zio_cksum_t zero_cksum = { 0 }; @@ -2498,7 +2499,7 @@ zfs_receive_package(libzfs_handle_t *hdl * zfs_receive_one() will take care of it (ie, * recv_skip() and return 0). */ - error = zfs_receive_impl(hdl, destname, flags, fd, + error = zfs_receive_impl(hdl, destname, NULL, flags, fd, sendfs, stream_nv, stream_avl, top_zfs, cleanup_fd, action_handlep); if (error == ENODATA) { @@ -2631,9 +2632,9 @@ recv_skip(libzfs_handle_t *hdl, int fd, */ static int zfs_receive_one(libzfs_handle_t *hdl, int infd, const char *tosnap, - recvflags_t *flags, dmu_replay_record_t *drr, - dmu_replay_record_t *drr_noswap, const char *sendfs, - nvlist_t *stream_nv, avl_tree_t *stream_avl, char **top_zfs, int cleanup_fd, + const char *originsnap, recvflags_t *flags, dmu_replay_record_t *drr, + dmu_replay_record_t *drr_noswap, const char *sendfs, nvlist_t *stream_nv, + avl_tree_t *stream_avl, char **top_zfs, int cleanup_fd, uint64_t *action_handlep) { zfs_cmd_t zc = { 0 }; @@ -2798,10 +2799,15 @@ zfs_receive_one(libzfs_handle_t *hdl, in } if (flags->verbose) (void) printf("found clone origin %s\n", zc.zc_string); + } else if (originsnap) { + (void) strncpy(zc.zc_string, originsnap, ZFS_MAXNAMELEN); + if (flags->verbose) + (void) printf("using provided clone origin %s\n", + zc.zc_string); } stream_wantsnewfs = (drrb->drr_fromguid == 0 || - (drrb->drr_flags & DRR_FLAG_CLONE)); + (drrb->drr_flags & DRR_FLAG_CLONE) || originsnap); if (stream_wantsnewfs) { /* @@ -3179,9 +3185,10 @@ zfs_receive_one(libzfs_handle_t *hdl, in } static int -zfs_receive_impl(libzfs_handle_t *hdl, const char *tosnap, recvflags_t *flags, - int infd, const char *sendfs, nvlist_t *stream_nv, avl_tree_t *stream_avl, - char **top_zfs, int cleanup_fd, uint64_t *action_handlep) +zfs_receive_impl(libzfs_handle_t *hdl, const char *tosnap, + const char *originsnap, recvflags_t *flags, int infd, const char *sendfs, + nvlist_t *stream_nv, avl_tree_t *stream_avl, char **top_zfs, int cleanup_fd, + uint64_t *action_handlep) { int err; dmu_replay_record_t drr, drr_noswap; @@ -3200,6 +3207,12 @@ zfs_receive_impl(libzfs_handle_t *hdl, c "(%s) does not exist"), tosnap); return (zfs_error(hdl, EZFS_NOENT, errbuf)); } + if (originsnap && + !zfs_dataset_exists(hdl, originsnap, ZFS_TYPE_DATASET)) { + zfs_error_aux(hdl, dgettext(TEXT_DOMAIN, "specified origin fs " + "(%s) does not exist"), originsnap); + return (zfs_error(hdl, EZFS_NOENT, errbuf)); + } /* read in the BEGIN record */ if (0 != (err = recv_read(hdl, infd, &drr, sizeof (drr), B_FALSE, @@ -3272,14 +3285,14 @@ zfs_receive_impl(libzfs_handle_t *hdl, c *cp = '\0'; sendfs = nonpackage_sendfs; } - return (zfs_receive_one(hdl, infd, tosnap, flags, - &drr, &drr_noswap, sendfs, stream_nv, stream_avl, - top_zfs, cleanup_fd, action_handlep)); + return (zfs_receive_one(hdl, infd, tosnap, originsnap, flags, + &drr, &drr_noswap, sendfs, stream_nv, stream_avl, top_zfs, + cleanup_fd, action_handlep)); } else { assert(DMU_GET_STREAM_HDRTYPE(drrb->drr_versioninfo) == DMU_COMPOUNDSTREAM); - return (zfs_receive_package(hdl, infd, tosnap, flags, - &drr, &zcksum, top_zfs, cleanup_fd, action_handlep)); + return (zfs_receive_package(hdl, infd, tosnap, flags, &drr, + &zcksum, top_zfs, cleanup_fd, action_handlep)); } } @@ -3290,18 +3303,24 @@ zfs_receive_impl(libzfs_handle_t *hdl, c * (-1 will override -2). */ int -zfs_receive(libzfs_handle_t *hdl, const char *tosnap, recvflags_t *flags, - int infd, avl_tree_t *stream_avl) +zfs_receive(libzfs_handle_t *hdl, const char *tosnap, nvlist_t *props, + recvflags_t *flags, int infd, avl_tree_t *stream_avl) { char *top_zfs = NULL; int err; int cleanup_fd; uint64_t action_handle = 0; + char *originsnap = NULL; + if (props) { + err = nvlist_lookup_string(props, "origin", &originsnap); + if (err && err != ENOENT) + return (err); + } cleanup_fd = open(ZFS_DEV, O_RDWR|O_EXCL); VERIFY(cleanup_fd >= 0); - err = zfs_receive_impl(hdl, tosnap, flags, infd, NULL, NULL, + err = zfs_receive_impl(hdl, tosnap, originsnap, flags, infd, NULL, NULL, stream_avl, &top_zfs, cleanup_fd, &action_handle); VERIFY(0 == close(cleanup_fd)); Modified: stable/10/cddl/contrib/opensolaris/lib/libzpool/common/sys/zfs_context.h ============================================================================== --- stable/10/cddl/contrib/opensolaris/lib/libzpool/common/sys/zfs_context.h Sat Oct 3 07:59:25 2015 (r288570) +++ stable/10/cddl/contrib/opensolaris/lib/libzpool/common/sys/zfs_context.h Sat Oct 3 08:03:36 2015 (r288571) @@ -135,8 +135,18 @@ extern int aok; /* * DTrace SDT probes have different signatures in userland than they do in - * kernel. If they're being used in kernel code, re-define them out of + * the kernel. If they're being used in kernel code, re-define them out of * existence for their counterparts in libzpool. + * + * Here's an example of how to use the set-error probes in userland: + * zfs$target:::set-error /arg0 == EBUSY/ {stack();} + * + * Here's an example of how to use DTRACE_PROBE probes in userland: + * If there is a probe declared as follows: + * DTRACE_PROBE2(zfs__probe_name, uint64_t, blkid, dnode_t *, dn); + * Then you can use it as follows: + * zfs$target:::probe2 /copyinstr(arg0) == "zfs__probe_name"/ + * {printf("%u %p\n", arg1, arg2);} */ #ifdef DTRACE_PROBE Modified: stable/10/sys/cddl/contrib/opensolaris/uts/common/Makefile.files ============================================================================== --- stable/10/sys/cddl/contrib/opensolaris/uts/common/Makefile.files Sat Oct 3 07:59:25 2015 (r288570) +++ stable/10/sys/cddl/contrib/opensolaris/uts/common/Makefile.files Sat Oct 3 08:03:36 2015 (r288571) @@ -22,7 +22,9 @@ # # Copyright (c) 1991, 2010, Oracle and/or its affiliates. All rights reserved. # Copyright (c) 2012 Nexenta Systems, Inc. All rights reserved. -# Copyright (c) 2013 by Delphix. All rights reserved. +# Copyright (c) 2012 Nexenta Systems, Inc. All rights reserved. +# Copyright (c) 2012 Joyent, Inc. All rights reserved. +# Copyright (c) 2011, 2014 by Delphix. All rights reserved. # Copyright (c) 2013 by Saso Kiselkov. All rights reserved. # # @@ -36,6 +38,7 @@ ZFS_COMMON_OBJS += \ blkptr.o \ bpobj.o \ bptree.o \ + bqueue.o \ dbuf.o \ ddt.o \ ddt_zap.o \ Modified: stable/10/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/bptree.c ============================================================================== --- stable/10/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/bptree.c Sat Oct 3 07:59:25 2015 (r288570) +++ stable/10/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/bptree.c Sat Oct 3 08:03:36 2015 (r288571) @@ -154,7 +154,7 @@ bptree_visit_cb(spa_t *spa, zilog_t *zil int err; struct bptree_args *ba = arg; - if (BP_IS_HOLE(bp)) + if (bp == NULL || BP_IS_HOLE(bp)) return (0); err = ba->ba_func(ba->ba_arg, bp, ba->ba_tx); Copied: stable/10/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/bqueue.c (from r286705, head/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/bqueue.c) ============================================================================== --- /dev/null 00:00:00 1970 (empty, because file is newly added) +++ stable/10/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/bqueue.c Sat Oct 3 08:03:36 2015 (r288571, copy of r286705, head/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/bqueue.c) @@ -0,0 +1,111 @@ +/* + * CDDL HEADER START + * + * This file and its contents are supplied under the terms of the + * Common Development and Distribution License ("CDDL"), version 1.0. + * You may only use this file in accordance with the terms of version + * 1.0 of the CDDL. + * + * A full copy of the text of the CDDL should have accompanied this + * source. A copy of the CDDL is also available via the Internet at + * http://www.illumos.org/license/CDDL. + * + * CDDL HEADER END + */ +/* + * Copyright (c) 2014 by Delphix. All rights reserved. + */ + +#include +#include + +static inline bqueue_node_t * +obj2node(bqueue_t *q, void *data) +{ + return ((bqueue_node_t *)((char *)data + q->bq_node_offset)); +} + +/* + * Initialize a blocking queue The maximum capacity of the queue is set to + * size. Types that want to be stored in a bqueue must contain a bqueue_node_t, + * and offset should give its offset from the start of the struct. Return 0 on + * success, or -1 on failure. + */ +int +bqueue_init(bqueue_t *q, uint64_t size, size_t node_offset) +{ + list_create(&q->bq_list, node_offset + sizeof (bqueue_node_t), + node_offset + offsetof(bqueue_node_t, bqn_node)); + cv_init(&q->bq_add_cv, NULL, CV_DEFAULT, NULL); + cv_init(&q->bq_pop_cv, NULL, CV_DEFAULT, NULL); + mutex_init(&q->bq_lock, NULL, MUTEX_DEFAULT, NULL); + q->bq_node_offset = node_offset; + q->bq_size = 0; + q->bq_maxsize = size; + return (0); +} + +/* + * Destroy a blocking queue. This function asserts that there are no + * elements in the queue, and no one is blocked on the condition + * variables. + */ +void +bqueue_destroy(bqueue_t *q) +{ + ASSERT0(q->bq_size); + cv_destroy(&q->bq_add_cv); + cv_destroy(&q->bq_pop_cv); + mutex_destroy(&q->bq_lock); + list_destroy(&q->bq_list); +} + +/* + * Add data to q, consuming size units of capacity. If there is insufficient + * capacity to consume size units, block until capacity exists. Asserts size is + * > 0. + */ +void +bqueue_enqueue(bqueue_t *q, void *data, uint64_t item_size) +{ + ASSERT3U(item_size, >, 0); + ASSERT3U(item_size, <, q->bq_maxsize); + mutex_enter(&q->bq_lock); + obj2node(q, data)->bqn_size = item_size; + while (q->bq_size + item_size > q->bq_maxsize) { + cv_wait(&q->bq_add_cv, &q->bq_lock); + } + q->bq_size += item_size; + list_insert_tail(&q->bq_list, data); + cv_signal(&q->bq_pop_cv); + mutex_exit(&q->bq_lock); +} +/* + * Take the first element off of q. If there are no elements on the queue, wait + * until one is put there. Return the removed element. + */ +void * +bqueue_dequeue(bqueue_t *q) +{ + void *ret; + uint64_t item_size; + mutex_enter(&q->bq_lock); + while (q->bq_size == 0) { + cv_wait(&q->bq_pop_cv, &q->bq_lock); + } + ret = list_remove_head(&q->bq_list); + item_size = obj2node(q, ret)->bqn_size; + q->bq_size -= item_size; + mutex_exit(&q->bq_lock); + cv_signal(&q->bq_add_cv); + return (ret); +} + +/* + * Returns true if the space used is 0. + */ +boolean_t +bqueue_empty(bqueue_t *q) +{ + return (q->bq_size == 0); +} Modified: stable/10/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/dbuf.c ============================================================================== --- stable/10/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/dbuf.c Sat Oct 3 07:59:25 2015 (r288570) +++ stable/10/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/dbuf.c Sat Oct 3 08:03:36 2015 (r288571) @@ -548,11 +548,35 @@ dbuf_loan_arcbuf(dmu_buf_impl_t *db) return (abuf); } +/* + * Calculate which level n block references the data at the level 0 offset + * provided. + */ uint64_t -dbuf_whichblock(dnode_t *dn, uint64_t offset) +dbuf_whichblock(dnode_t *dn, int64_t level, uint64_t offset) { - if (dn->dn_datablkshift) { - return (offset >> dn->dn_datablkshift); + if (dn->dn_datablkshift != 0 && dn->dn_indblkshift != 0) { + /* + * The level n blkid is equal to the level 0 blkid divided by + * the number of level 0s in a level n block. + * + * The level 0 blkid is offset >> datablkshift = + * offset / 2^datablkshift. + * + * The number of level 0s in a level n is the number of block + * pointers in an indirect block, raised to the power of level. + * This is 2^(indblkshift - SPA_BLKPTRSHIFT)^level = + * 2^(level*(indblkshift - SPA_BLKPTRSHIFT)). + * + * Thus, the level n blkid is: offset / + * ((2^datablkshift)*(2^(level*(indblkshift - SPA_BLKPTRSHIFT))) + * = offset / 2^(datablkshift + level * + * (indblkshift - SPA_BLKPTRSHIFT)) + * = offset >> (datablkshift + level * + * (indblkshift - SPA_BLKPTRSHIFT)) + */ + return (offset >> (dn->dn_datablkshift + level * + (dn->dn_indblkshift - SPA_BLKPTRSHIFT))); } else { ASSERT3U(offset, <, dn->dn_datablksz); return (0); @@ -1715,6 +1739,12 @@ dbuf_clear(dmu_buf_impl_t *db) dbuf_rele(parent, db); } +/* + * Note: While bpp will always be updated if the function returns success, + * parentp will not be updated if the dnode does not have dn_dbuf filled in; + * this happens when the dnode is the meta-dnode, or a userused or groupused + * object. + */ static int dbuf_findbp(dnode_t *dn, int level, uint64_t blkid, int fail_sparse, dmu_buf_impl_t **parentp, blkptr_t **bpp) @@ -1755,7 +1785,7 @@ dbuf_findbp(dnode_t *dn, int level, uint } else if (level < nlevels-1) { /* this block is referenced from an indirect block */ int err = dbuf_hold_impl(dn, level+1, - blkid >> epbs, fail_sparse, NULL, parentp); + blkid >> epbs, fail_sparse, FALSE, NULL, parentp); if (err) return (err); err = dbuf_read(*parentp, NULL, @@ -1930,11 +1960,96 @@ dbuf_destroy(dmu_buf_impl_t *db) arc_space_return(sizeof (dmu_buf_impl_t), ARC_SPACE_OTHER); } +typedef struct dbuf_prefetch_arg { + spa_t *dpa_spa; /* The spa to issue the prefetch in. */ + zbookmark_phys_t dpa_zb; /* The target block to prefetch. */ + int dpa_epbs; /* Entries (blkptr_t's) Per Block Shift. */ + int dpa_curlevel; /* The current level that we're reading */ + zio_priority_t dpa_prio; /* The priority I/Os should be issued at. */ + zio_t *dpa_zio; /* The parent zio_t for all prefetches. */ + arc_flags_t dpa_aflags; /* Flags to pass to the final prefetch. */ +} dbuf_prefetch_arg_t; + +/* + * Actually issue the prefetch read for the block given. + */ +static void +dbuf_issue_final_prefetch(dbuf_prefetch_arg_t *dpa, blkptr_t *bp) +{ + if (BP_IS_HOLE(bp) || BP_IS_EMBEDDED(bp)) + return; + + arc_flags_t aflags = + dpa->dpa_aflags | ARC_FLAG_NOWAIT | ARC_FLAG_PREFETCH; + + ASSERT3U(dpa->dpa_curlevel, ==, BP_GET_LEVEL(bp)); + ASSERT3U(dpa->dpa_curlevel, ==, dpa->dpa_zb.zb_level); + ASSERT(dpa->dpa_zio != NULL); + (void) arc_read(dpa->dpa_zio, dpa->dpa_spa, bp, NULL, NULL, + dpa->dpa_prio, ZIO_FLAG_CANFAIL | ZIO_FLAG_SPECULATIVE, + &aflags, &dpa->dpa_zb); +} + +/* + * Called when an indirect block above our prefetch target is read in. This + * will either read in the next indirect block down the tree or issue the actual + * prefetch if the next block down is our target. + */ +static void +dbuf_prefetch_indirect_done(zio_t *zio, arc_buf_t *abuf, void *private) +{ + dbuf_prefetch_arg_t *dpa = private; + + ASSERT3S(dpa->dpa_zb.zb_level, <, dpa->dpa_curlevel); + ASSERT3S(dpa->dpa_curlevel, >, 0); + if (zio != NULL) { + ASSERT3S(BP_GET_LEVEL(zio->io_bp), ==, dpa->dpa_curlevel); + ASSERT3U(BP_GET_LSIZE(zio->io_bp), ==, zio->io_size); + ASSERT3P(zio->io_spa, ==, dpa->dpa_spa); + } + + dpa->dpa_curlevel--; + + uint64_t nextblkid = dpa->dpa_zb.zb_blkid >> + (dpa->dpa_epbs * (dpa->dpa_curlevel - dpa->dpa_zb.zb_level)); + blkptr_t *bp = ((blkptr_t *)abuf->b_data) + + P2PHASE(nextblkid, 1ULL << dpa->dpa_epbs); + if (BP_IS_HOLE(bp) || (zio != NULL && zio->io_error != 0)) { + kmem_free(dpa, sizeof (*dpa)); + } else if (dpa->dpa_curlevel == dpa->dpa_zb.zb_level) { + ASSERT3U(nextblkid, ==, dpa->dpa_zb.zb_blkid); + dbuf_issue_final_prefetch(dpa, bp); + kmem_free(dpa, sizeof (*dpa)); + } else { + arc_flags_t iter_aflags = ARC_FLAG_NOWAIT; + zbookmark_phys_t zb; + + ASSERT3U(dpa->dpa_curlevel, ==, BP_GET_LEVEL(bp)); + + SET_BOOKMARK(&zb, dpa->dpa_zb.zb_objset, + dpa->dpa_zb.zb_object, dpa->dpa_curlevel, nextblkid); + + (void) arc_read(dpa->dpa_zio, dpa->dpa_spa, + bp, dbuf_prefetch_indirect_done, dpa, dpa->dpa_prio, + ZIO_FLAG_CANFAIL | ZIO_FLAG_SPECULATIVE, + &iter_aflags, &zb); + } + (void) arc_buf_remove_ref(abuf, private); +} + +/* + * Issue prefetch reads for the given block on the given level. If the indirect + * blocks above that block are not in memory, we will read them in + * asynchronously. As a result, this call never blocks waiting for a read to + * complete. + */ void -dbuf_prefetch(dnode_t *dn, uint64_t blkid, zio_priority_t prio) +dbuf_prefetch(dnode_t *dn, int64_t level, uint64_t blkid, zio_priority_t prio, + arc_flags_t aflags) { - dmu_buf_impl_t *db = NULL; - blkptr_t *bp = NULL; + blkptr_t bp; + int epbs, nlevels, curlevel; + uint64_t curblkid; ASSERT(blkid != DMU_BONUS_BLKID); ASSERT(RW_LOCK_HELD(&dn->dn_struct_rwlock)); @@ -1942,35 +2057,104 @@ dbuf_prefetch(dnode_t *dn, uint64_t blki if (dnode_block_freed(dn, blkid)) return; - /* dbuf_find() returns with db_mtx held */ - if (db = dbuf_find(dn->dn_objset, dn->dn_object, 0, blkid)) { + /* + * This dnode hasn't been written to disk yet, so there's nothing to + * prefetch. + */ + nlevels = dn->dn_phys->dn_nlevels; + if (level >= nlevels || dn->dn_phys->dn_nblkptr == 0) + return; + + epbs = dn->dn_phys->dn_indblkshift - SPA_BLKPTRSHIFT; + if (dn->dn_phys->dn_maxblkid < blkid << (epbs * level)) + return; + + dmu_buf_impl_t *db = dbuf_find(dn->dn_objset, dn->dn_object, + level, blkid); + if (db != NULL) { + mutex_exit(&db->db_mtx); /* - * This dbuf is already in the cache. We assume that - * it is already CACHED, or else about to be either - * read or filled. + * This dbuf already exists. It is either CACHED, or + * (we assume) about to be read or filled. */ - mutex_exit(&db->db_mtx); return; } - if (dbuf_findbp(dn, 0, blkid, TRUE, &db, &bp) == 0) { - if (bp && !BP_IS_HOLE(bp) && !BP_IS_EMBEDDED(bp)) { - dsl_dataset_t *ds = dn->dn_objset->os_dsl_dataset; - arc_flags_t aflags = - ARC_FLAG_NOWAIT | ARC_FLAG_PREFETCH; - zbookmark_phys_t zb; - - SET_BOOKMARK(&zb, ds ? ds->ds_object : DMU_META_OBJSET, - dn->dn_object, 0, blkid); - - (void) arc_read(NULL, dn->dn_objset->os_spa, - bp, NULL, NULL, prio, - ZIO_FLAG_CANFAIL | ZIO_FLAG_SPECULATIVE, - &aflags, &zb); + /* + * Find the closest ancestor (indirect block) of the target block + * that is present in the cache. In this indirect block, we will + * find the bp that is at curlevel, curblkid. + */ + curlevel = level; + curblkid = blkid; + while (curlevel < nlevels - 1) { + int parent_level = curlevel + 1; + uint64_t parent_blkid = curblkid >> epbs; + dmu_buf_impl_t *db; + + if (dbuf_hold_impl(dn, parent_level, parent_blkid, + FALSE, TRUE, FTAG, &db) == 0) { + blkptr_t *bpp = db->db_buf->b_data; + bp = bpp[P2PHASE(curblkid, 1 << epbs)]; + dbuf_rele(db, FTAG); + break; } - if (db) - dbuf_rele(db, NULL); + + curlevel = parent_level; + curblkid = parent_blkid; + } + + if (curlevel == nlevels - 1) { + /* No cached indirect blocks found. */ + ASSERT3U(curblkid, <, dn->dn_phys->dn_nblkptr); + bp = dn->dn_phys->dn_blkptr[curblkid]; } + if (BP_IS_HOLE(&bp)) + return; + + ASSERT3U(curlevel, ==, BP_GET_LEVEL(&bp)); + + zio_t *pio = zio_root(dmu_objset_spa(dn->dn_objset), NULL, NULL, + ZIO_FLAG_CANFAIL); + + dbuf_prefetch_arg_t *dpa = kmem_zalloc(sizeof (*dpa), KM_SLEEP); + dsl_dataset_t *ds = dn->dn_objset->os_dsl_dataset; + SET_BOOKMARK(&dpa->dpa_zb, ds != NULL ? ds->ds_object : DMU_META_OBJSET, + dn->dn_object, level, blkid); + dpa->dpa_curlevel = curlevel; + dpa->dpa_prio = prio; + dpa->dpa_aflags = aflags; + dpa->dpa_spa = dn->dn_objset->os_spa; + dpa->dpa_epbs = epbs; + dpa->dpa_zio = pio; + + /* + * If we have the indirect just above us, no need to do the asynchronous + * prefetch chain; we'll just run the last step ourselves. If we're at + * a higher level, though, we want to issue the prefetches for all the + * indirect blocks asynchronously, so we can go on with whatever we were + * doing. + */ + if (curlevel == level) { + ASSERT3U(curblkid, ==, blkid); + dbuf_issue_final_prefetch(dpa, &bp); + kmem_free(dpa, sizeof (*dpa)); + } else { + arc_flags_t iter_aflags = ARC_FLAG_NOWAIT; + zbookmark_phys_t zb; + + SET_BOOKMARK(&zb, ds != NULL ? ds->ds_object : DMU_META_OBJSET, + dn->dn_object, curlevel, curblkid); + (void) arc_read(dpa->dpa_zio, dpa->dpa_spa, + &bp, dbuf_prefetch_indirect_done, dpa, prio, + ZIO_FLAG_CANFAIL | ZIO_FLAG_SPECULATIVE, + &iter_aflags, &zb); + } + /* + * We use pio here instead of dpa_zio since it's possible that + * dpa may have already been freed. + */ + zio_nowait(pio); } /* @@ -1978,7 +2162,8 @@ dbuf_prefetch(dnode_t *dn, uint64_t blki * Note: dn_struct_rwlock must be held. */ int -dbuf_hold_impl(dnode_t *dn, uint8_t level, uint64_t blkid, int fail_sparse, +dbuf_hold_impl(dnode_t *dn, uint8_t level, uint64_t blkid, + boolean_t fail_sparse, boolean_t fail_uncached, void *tag, dmu_buf_impl_t **dbp) { dmu_buf_impl_t *db, *parent = NULL; @@ -1996,6 +2181,9 @@ top: blkptr_t *bp = NULL; int err; + if (fail_uncached) + return (SET_ERROR(ENOENT)); + ASSERT3P(parent, ==, NULL); err = dbuf_findbp(dn, level, blkid, fail_sparse, &parent, &bp); if (fail_sparse) { @@ -2012,6 +2200,11 @@ top: db = dbuf_create(dn, level, blkid, parent, bp); } + if (fail_uncached && db->db_state != DB_CACHED) { + mutex_exit(&db->db_mtx); + return (SET_ERROR(ENOENT)); + } + if (db->db_buf && refcount_is_zero(&db->db_holds)) { arc_buf_add_ref(db->db_buf, db); if (db->db_buf->b_data == NULL) { @@ -2067,16 +2260,14 @@ top: dmu_buf_impl_t * dbuf_hold(dnode_t *dn, uint64_t blkid, void *tag) { - dmu_buf_impl_t *db; - int err = dbuf_hold_impl(dn, 0, blkid, FALSE, tag, &db); - return (err ? NULL : db); + return (dbuf_hold_level(dn, 0, blkid, tag)); } dmu_buf_impl_t * dbuf_hold_level(dnode_t *dn, int level, uint64_t blkid, void *tag) { dmu_buf_impl_t *db; - int err = dbuf_hold_impl(dn, level, blkid, FALSE, tag, &db); + int err = dbuf_hold_impl(dn, level, blkid, FALSE, FALSE, tag, &db); return (err ? NULL : db); } @@ -2429,8 +2620,8 @@ dbuf_check_blkptr(dnode_t *dn, dmu_buf_i if (parent == NULL) { mutex_exit(&db->db_mtx); rw_enter(&dn->dn_struct_rwlock, RW_READER); - (void) dbuf_hold_impl(dn, db->db_level+1, - db->db_blkid >> epbs, FALSE, db, &parent); + parent = dbuf_hold_level(dn, db->db_level + 1, + db->db_blkid >> epbs, db); rw_exit(&dn->dn_struct_rwlock); mutex_enter(&db->db_mtx); db->db_parent = parent; Modified: stable/10/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/dmu.c ============================================================================== --- stable/10/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/dmu.c Sat Oct 3 07:59:25 2015 (r288570) +++ stable/10/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/dmu.c Sat Oct 3 08:03:36 2015 (r288571) @@ -142,7 +142,7 @@ dmu_buf_hold_noread(objset_t *os, uint64 err = dnode_hold(os, object, FTAG, &dn); if (err) return (err); - blkid = dbuf_whichblock(dn, offset); + blkid = dbuf_whichblock(dn, 0, offset); rw_enter(&dn->dn_struct_rwlock, RW_READER); db = dbuf_hold(dn, blkid, tag); rw_exit(&dn->dn_struct_rwlock); @@ -425,7 +425,7 @@ dmu_buf_hold_array_by_dnode(dnode_t *dn, dbp = kmem_zalloc(sizeof (dmu_buf_t *) * nblks, KM_SLEEP); zio = zio_root(dn->dn_objset->os_spa, NULL, NULL, ZIO_FLAG_CANFAIL); - blkid = dbuf_whichblock(dn, offset); + blkid = dbuf_whichblock(dn, 0, offset); for (i = 0; i < nblks; i++) { dmu_buf_impl_t *db = dbuf_hold(dn, blkid+i, tag); if (db == NULL) { @@ -529,17 +529,16 @@ dmu_buf_rele_array(dmu_buf_t **dbp_fake, } /* - * Issue prefetch i/os for the given blocks. + * Issue prefetch i/os for the given blocks. If level is greater than 0, the + * indirect blocks prefeteched will be those that point to the blocks containing + * the data starting at offset, and continuing to offset + len. * - * Note: The assumption is that we *know* these blocks will be needed - * almost immediately. Therefore, the prefetch i/os will be issued at - * ZIO_PRIORITY_SYNC_READ - * - * Note: indirect blocks and other metadata will be read synchronously, - * causing this function to block if they are not already cached. + * Note that if the indirect blocks above the blocks being prefetched are not in + * cache, they will be asychronously read in. */ void -dmu_prefetch(objset_t *os, uint64_t object, uint64_t offset, uint64_t len) +dmu_prefetch(objset_t *os, uint64_t object, int64_t level, uint64_t offset, + uint64_t len, zio_priority_t pri) { dnode_t *dn; uint64_t blkid; @@ -555,8 +554,9 @@ dmu_prefetch(objset_t *os, uint64_t obje return; rw_enter(&dn->dn_struct_rwlock, RW_READER); - blkid = dbuf_whichblock(dn, object * sizeof (dnode_phys_t)); - dbuf_prefetch(dn, blkid, ZIO_PRIORITY_SYNC_READ); + blkid = dbuf_whichblock(dn, level, + object * sizeof (dnode_phys_t)); + dbuf_prefetch(dn, level, blkid, pri, 0); rw_exit(&dn->dn_struct_rwlock); return; } @@ -571,18 +571,24 @@ dmu_prefetch(objset_t *os, uint64_t obje return; rw_enter(&dn->dn_struct_rwlock, RW_READER); - if (dn->dn_datablkshift) { - int blkshift = dn->dn_datablkshift; - nblks = (P2ROUNDUP(offset + len, 1 << blkshift) - - P2ALIGN(offset, 1 << blkshift)) >> blkshift; + /* + * offset + len - 1 is the last byte we want to prefetch for, and offset + * is the first. Then dbuf_whichblk(dn, level, off + len - 1) is the + * last block we want to prefetch, and dbuf_whichblock(dn, level, + * offset) is the first. Then the number we need to prefetch is the + * last - first + 1. + */ + if (level > 0 || dn->dn_datablkshift != 0) { + nblks = dbuf_whichblock(dn, level, offset + len - 1) - + dbuf_whichblock(dn, level, offset) + 1; } else { nblks = (offset < dn->dn_datablksz); } if (nblks != 0) { - blkid = dbuf_whichblock(dn, offset); + blkid = dbuf_whichblock(dn, level, offset); for (int i = 0; i < nblks; i++) - dbuf_prefetch(dn, blkid + i, ZIO_PRIORITY_SYNC_READ); + dbuf_prefetch(dn, level, blkid + i, pri, 0); } rw_exit(&dn->dn_struct_rwlock); @@ -1394,7 +1400,7 @@ dmu_assign_arcbuf(dmu_buf_t *handle, uin DB_DNODE_ENTER(dbuf); dn = DB_DNODE(dbuf); rw_enter(&dn->dn_struct_rwlock, RW_READER); - blkid = dbuf_whichblock(dn, offset); + blkid = dbuf_whichblock(dn, 0, offset); VERIFY((db = dbuf_hold(dn, blkid, FTAG)) != NULL); rw_exit(&dn->dn_struct_rwlock); DB_DNODE_EXIT(dbuf); Modified: stable/10/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/dmu_diff.c ============================================================================== --- stable/10/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/dmu_diff.c Sat Oct 3 07:59:25 2015 (r288570) +++ stable/10/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/dmu_diff.c Sat Oct 3 08:03:36 2015 (r288571) @@ -138,7 +138,7 @@ diff_cb(spa_t *spa, zilog_t *zilog, cons if (issig(JUSTLOOKING) && issig(FORREAL)) return (SET_ERROR(EINTR)); - if (zb->zb_object != DMU_META_DNODE_OBJECT) + if (bp == NULL || zb->zb_object != DMU_META_DNODE_OBJECT) return (0); if (BP_IS_HOLE(bp)) { Modified: stable/10/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/dmu_object.c ============================================================================== --- stable/10/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/dmu_object.c Sat Oct 3 07:59:25 2015 (r288570) +++ stable/10/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/dmu_object.c Sat Oct 3 08:03:36 2015 (r288571) @@ -148,6 +148,11 @@ dmu_object_free(objset_t *os, uint64_t o return (0); } +/* + * Return (in *objectp) the next object which is allocated (or a hole) + * after *object, taking into account only objects that may have been modified + * after the specified txg. + */ int dmu_object_next(objset_t *os, uint64_t *objectp, boolean_t hole, uint64_t txg) { Modified: stable/10/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/dmu_send.c ============================================================================== --- stable/10/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/dmu_send.c Sat Oct 3 07:59:25 2015 (r288570) +++ stable/10/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/dmu_send.c Sat Oct 3 08:03:36 2015 (r288571) @@ -53,6 +53,7 @@ #include #include #include +#include #ifdef __FreeBSD__ #undef dump_write @@ -61,10 +62,34 @@ /* Set this tunable to TRUE to replace corrupt data with 0x2f5baddb10c */ int zfs_send_corrupt_data = B_FALSE; +int zfs_send_queue_length = 16 * 1024 * 1024; +int zfs_recv_queue_length = 16 * 1024 * 1024; static char *dmu_recv_tag = "dmu_recv_tag"; static const char *recv_clone_name = "%recv"; +#define BP_SPAN(datablkszsec, indblkshift, level) \ + (((uint64_t)datablkszsec) << (SPA_MINBLOCKSHIFT + \ + (level) * (indblkshift - SPA_BLKPTRSHIFT))) + +struct send_thread_arg { + bqueue_t q; + dsl_dataset_t *ds; /* Dataset to traverse */ + uint64_t fromtxg; /* Traverse from this txg */ + int flags; /* flags to pass to traverse_dataset */ + int error_code; + boolean_t cancel; +}; *** DIFF OUTPUT TRUNCATED AT 1000 LINES *** From owner-svn-src-stable@freebsd.org Sat Oct 3 08:05:37 2015 Return-Path: Delivered-To: svn-src-stable@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 7F67AA0DAFD; Sat, 3 Oct 2015 08:05:37 +0000 (UTC) (envelope-from mav@FreeBSD.org) Received: from repo.freebsd.org (repo.freebsd.org [IPv6:2001:1900:2254:2068::e6a:0]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 6A4B517F0; Sat, 3 Oct 2015 08:05:37 +0000 (UTC) (envelope-from mav@FreeBSD.org) Received: from repo.freebsd.org ([127.0.1.70]) by repo.freebsd.org (8.15.2/8.15.2) with ESMTP id t9385bs5083579; Sat, 3 Oct 2015 08:05:37 GMT (envelope-from mav@FreeBSD.org) Received: (from mav@localhost) by repo.freebsd.org (8.15.2/8.15.2/Submit) id t9385YJ9083565; Sat, 3 Oct 2015 08:05:34 GMT (envelope-from mav@FreeBSD.org) Message-Id: <201510030805.t9385YJ9083565@repo.freebsd.org> X-Authentication-Warning: repo.freebsd.org: mav set sender to mav@FreeBSD.org using -f From: Alexander Motin Date: Sat, 3 Oct 2015 08:05:34 +0000 (UTC) To: src-committers@freebsd.org, svn-src-all@freebsd.org, svn-src-stable@freebsd.org, svn-src-stable-10@freebsd.org Subject: svn commit: r288572 - in stable/10: cddl/contrib/opensolaris/cmd/zdb cddl/contrib/opensolaris/cmd/zhack cddl/contrib/opensolaris/cmd/zpool sys/cddl/contrib/opensolaris/common/zfs sys/cddl/contrib/o... X-SVN-Group: stable-10 MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit X-BeenThere: svn-src-stable@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: SVN commit messages for all the -stable branches of the src tree List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 03 Oct 2015 08:05:37 -0000 Author: mav Date: Sat Oct 3 08:05:33 2015 New Revision: 288572 URL: https://svnweb.freebsd.org/changeset/base/288572 Log: MFC r286708: 5959 clean up per-dataset feature count code Reviewed by: Toomas Soome Reviewed by: George Wilson Reviewed by: Alex Reece Approved by: Richard Lowe Author: Matthew Ahrens illumos/illumos-gate@ca0cc3918a1789fa839194af2a9245f801a06b1a A ZFS feature flags (large blocks) tracks its refcounts as the number of datasets that have ever used the feature. Several features of this type are planned to be added (new checksum functions). This code should be made common infrastructure rather than duplicating the code for each feature. Modified: stable/10/cddl/contrib/opensolaris/cmd/zdb/zdb.c stable/10/cddl/contrib/opensolaris/cmd/zhack/zhack.c stable/10/cddl/contrib/opensolaris/cmd/zpool/zpool_main.c stable/10/sys/cddl/contrib/opensolaris/common/zfs/zfeature_common.c stable/10/sys/cddl/contrib/opensolaris/common/zfs/zfeature_common.h stable/10/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/dbuf.c stable/10/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/dmu_send.c stable/10/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/dsl_dataset.c stable/10/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/dsl_destroy.c stable/10/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/sys/dsl_dataset.h stable/10/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zfeature.c Directory Properties: stable/10/ (props changed) Modified: stable/10/cddl/contrib/opensolaris/cmd/zdb/zdb.c ============================================================================== --- stable/10/cddl/contrib/opensolaris/cmd/zdb/zdb.c Sat Oct 3 08:03:36 2015 (r288571) +++ stable/10/cddl/contrib/opensolaris/cmd/zdb/zdb.c Sat Oct 3 08:05:33 2015 (r288572) @@ -21,7 +21,7 @@ /* * Copyright (c) 2005, 2010, Oracle and/or its affiliates. All rights reserved. - * Copyright (c) 2011, 2014 by Delphix. All rights reserved. + * Copyright (c) 2011, 2015 by Delphix. All rights reserved. */ #include @@ -2222,7 +2222,7 @@ dump_label(const char *dev) (void) close(fd); } -static uint64_t num_large_blocks; +static uint64_t dataset_feature_count[SPA_FEATURES]; /*ARGSUSED*/ static int @@ -2236,8 +2236,15 @@ dump_one_dir(const char *dsname, void *a (void) printf("Could not open %s, error %d\n", dsname, error); return (0); } - if (dmu_objset_ds(os)->ds_large_blocks) - num_large_blocks++; + + for (spa_feature_t f = 0; f < SPA_FEATURES; f++) { + if (!dmu_objset_ds(os)->ds_feature_inuse[f]) + continue; + ASSERT(spa_feature_table[f].fi_flags & + ZFEATURE_FLAG_PER_DATASET); + dataset_feature_count[f]++; + } + dump_dir(os); dmu_objset_disown(os, FTAG); fuid_table_destroy(); @@ -3036,7 +3043,6 @@ dump_zpool(spa_t *spa) dump_metaslab_groups(spa); if (dump_opt['d'] || dump_opt['i']) { - uint64_t refcount; dump_dir(dp->dp_meta_objset); if (dump_opt['d'] >= 3) { dump_full_bpobj(&spa->spa_deferred_bpobj, @@ -3058,17 +3064,29 @@ dump_zpool(spa_t *spa) (void) dmu_objset_find(spa_name(spa), dump_one_dir, NULL, DS_FIND_SNAPSHOTS | DS_FIND_CHILDREN); - (void) feature_get_refcount(spa, - &spa_feature_table[SPA_FEATURE_LARGE_BLOCKS], &refcount); - if (num_large_blocks != refcount) { - (void) printf("large_blocks feature refcount mismatch: " - "expected %lld != actual %lld\n", - (longlong_t)num_large_blocks, - (longlong_t)refcount); - rc = 2; - } else { - (void) printf("Verified large_blocks feature refcount " - "is correct (%llu)\n", (longlong_t)refcount); + for (spa_feature_t f = 0; f < SPA_FEATURES; f++) { + uint64_t refcount; + + if (!(spa_feature_table[f].fi_flags & + ZFEATURE_FLAG_PER_DATASET)) { + ASSERT0(dataset_feature_count[f]); + continue; + } + (void) feature_get_refcount(spa, + &spa_feature_table[f], &refcount); + if (dataset_feature_count[f] != refcount) { + (void) printf("%s feature refcount mismatch: " + "%lld datasets != %lld refcount\n", + spa_feature_table[f].fi_uname, + (longlong_t)dataset_feature_count[f], + (longlong_t)refcount); + rc = 2; + } else { + (void) printf("Verified %s feature refcount " + "of %llu is correct\n", + spa_feature_table[f].fi_uname, + (longlong_t)refcount); + } } } if (rc == 0 && (dump_opt['b'] || dump_opt['c'])) Modified: stable/10/cddl/contrib/opensolaris/cmd/zhack/zhack.c ============================================================================== --- stable/10/cddl/contrib/opensolaris/cmd/zhack/zhack.c Sat Oct 3 08:03:36 2015 (r288571) +++ stable/10/cddl/contrib/opensolaris/cmd/zhack/zhack.c Sat Oct 3 08:05:33 2015 (r288572) @@ -20,7 +20,7 @@ */ /* - * Copyright (c) 2011, 2014 by Delphix. All rights reserved. + * Copyright (c) 2011, 2015 by Delphix. All rights reserved. * Copyright (c) 2013 Steven Hartland. All rights reserved. */ @@ -294,8 +294,8 @@ zhack_feature_enable_sync(void *arg, dmu feature_enable_sync(spa, feature, tx); spa_history_log_internal(spa, "zhack enable feature", tx, - "name=%s can_readonly=%u", - feature->fi_guid, feature->fi_can_readonly); + "guid=%s flags=%x", + feature->fi_guid, feature->fi_flags); } static void @@ -314,9 +314,7 @@ zhack_do_feature_enable(int argc, char * */ desc = NULL; feature.fi_uname = "zhack"; - feature.fi_mos = B_FALSE; - feature.fi_can_readonly = B_FALSE; - feature.fi_activate_on_enable = B_FALSE; + feature.fi_flags = 0; feature.fi_depends = nodeps; feature.fi_feature = SPA_FEATURE_NONE; @@ -324,7 +322,7 @@ zhack_do_feature_enable(int argc, char * while ((c = getopt(argc, argv, "rmd:")) != -1) { switch (c) { case 'r': - feature.fi_can_readonly = B_TRUE; + feature.fi_flags |= ZFEATURE_FLAG_READONLY_COMPAT; break; case 'd': desc = strdup(optarg); @@ -413,7 +411,7 @@ zhack_do_feature_ref(int argc, char **ar * disk later. */ feature.fi_uname = "zhack"; - feature.fi_mos = B_FALSE; + feature.fi_flags = 0; feature.fi_desc = NULL; feature.fi_depends = nodeps; feature.fi_feature = SPA_FEATURE_NONE; @@ -422,7 +420,7 @@ zhack_do_feature_ref(int argc, char **ar while ((c = getopt(argc, argv, "md")) != -1) { switch (c) { case 'm': - feature.fi_mos = B_TRUE; + feature.fi_flags |= ZFEATURE_FLAG_MOS; break; case 'd': decr = B_TRUE; @@ -455,10 +453,10 @@ zhack_do_feature_ref(int argc, char **ar if (0 == zap_contains(mos, spa->spa_feat_for_read_obj, feature.fi_guid)) { - feature.fi_can_readonly = B_FALSE; + feature.fi_flags &= ~ZFEATURE_FLAG_READONLY_COMPAT; } else if (0 == zap_contains(mos, spa->spa_feat_for_write_obj, feature.fi_guid)) { - feature.fi_can_readonly = B_TRUE; + feature.fi_flags |= ZFEATURE_FLAG_READONLY_COMPAT; } else { fatal(spa, FTAG, "feature is not enabled: %s", feature.fi_guid); } Modified: stable/10/cddl/contrib/opensolaris/cmd/zpool/zpool_main.c ============================================================================== --- stable/10/cddl/contrib/opensolaris/cmd/zpool/zpool_main.c Sat Oct 3 08:03:36 2015 (r288571) +++ stable/10/cddl/contrib/opensolaris/cmd/zpool/zpool_main.c Sat Oct 3 08:05:33 2015 (r288572) @@ -22,7 +22,7 @@ /* * Copyright (c) 2005, 2010, Oracle and/or its affiliates. All rights reserved. * Copyright 2011 Nexenta Systems, Inc. All rights reserved. - * Copyright (c) 2011, 2014 by Delphix. All rights reserved. + * Copyright (c) 2011, 2015 by Delphix. All rights reserved. * Copyright (c) 2012 by Frederik Wessels. All rights reserved. * Copyright (c) 2012 Martin Matuska . All rights reserved. * Copyright (c) 2013 by Prasad Joshi (sTec). All rights reserved. @@ -4986,7 +4986,8 @@ zpool_do_upgrade(int argc, char **argv) "---------------\n"); for (i = 0; i < SPA_FEATURES; i++) { zfeature_info_t *fi = &spa_feature_table[i]; - const char *ro = fi->fi_can_readonly ? + const char *ro = + (fi->fi_flags & ZFEATURE_FLAG_READONLY_COMPAT) ? " (read-only compatible)" : ""; (void) printf("%-37s%s\n", fi->fi_uname, ro); Modified: stable/10/sys/cddl/contrib/opensolaris/common/zfs/zfeature_common.c ============================================================================== --- stable/10/sys/cddl/contrib/opensolaris/common/zfs/zfeature_common.c Sat Oct 3 08:03:36 2015 (r288571) +++ stable/10/sys/cddl/contrib/opensolaris/common/zfs/zfeature_common.c Sat Oct 3 08:05:33 2015 (r288572) @@ -20,7 +20,7 @@ */ /* - * Copyright (c) 2013 by Delphix. All rights reserved. + * Copyright (c) 2011, 2015 by Delphix. All rights reserved. * Copyright (c) 2013 by Saso Kiselkov. All rights reserved. * Copyright (c) 2013, Joyent, Inc. All rights reserved. * Copyright (c) 2014, Nexenta Systems, Inc. All rights reserved. @@ -129,15 +129,15 @@ zfeature_depends_on(spa_feature_t fid, s static void zfeature_register(spa_feature_t fid, const char *guid, const char *name, - const char *desc, boolean_t readonly, boolean_t mos, - boolean_t activate_on_enable, const spa_feature_t *deps) + const char *desc, zfeature_flags_t flags, const spa_feature_t *deps) { zfeature_info_t *feature = &spa_feature_table[fid]; static spa_feature_t nodeps[] = { SPA_FEATURE_NONE }; ASSERT(name != NULL); ASSERT(desc != NULL); - ASSERT(!readonly || !mos); + ASSERT((flags & ZFEATURE_FLAG_READONLY_COMPAT) == 0 || + (flags & ZFEATURE_FLAG_MOS) == 0); ASSERT3U(fid, <, SPA_FEATURES); ASSERT(zfeature_is_valid_guid(guid)); @@ -148,9 +148,7 @@ zfeature_register(spa_feature_t fid, con feature->fi_guid = guid; feature->fi_uname = name; feature->fi_desc = desc; - feature->fi_can_readonly = readonly; - feature->fi_mos = mos; - feature->fi_activate_on_enable = activate_on_enable; + feature->fi_flags = flags; feature->fi_depends = deps; } @@ -159,45 +157,46 @@ zpool_feature_init(void) { zfeature_register(SPA_FEATURE_ASYNC_DESTROY, "com.delphix:async_destroy", "async_destroy", - "Destroy filesystems asynchronously.", B_TRUE, B_FALSE, - B_FALSE, NULL); + "Destroy filesystems asynchronously.", + ZFEATURE_FLAG_READONLY_COMPAT, NULL); zfeature_register(SPA_FEATURE_EMPTY_BPOBJ, "com.delphix:empty_bpobj", "empty_bpobj", - "Snapshots use less space.", B_TRUE, B_FALSE, - B_FALSE, NULL); + "Snapshots use less space.", + ZFEATURE_FLAG_READONLY_COMPAT, NULL); zfeature_register(SPA_FEATURE_LZ4_COMPRESS, "org.illumos:lz4_compress", "lz4_compress", - "LZ4 compression algorithm support.", B_FALSE, B_FALSE, - B_TRUE, NULL); + "LZ4 compression algorithm support.", + ZFEATURE_FLAG_ACTIVATE_ON_ENABLE, NULL); zfeature_register(SPA_FEATURE_MULTI_VDEV_CRASH_DUMP, "com.joyent:multi_vdev_crash_dump", "multi_vdev_crash_dump", - "Crash dumps to multiple vdev pools.", B_FALSE, B_FALSE, - B_FALSE, NULL); + "Crash dumps to multiple vdev pools.", + 0, NULL); zfeature_register(SPA_FEATURE_SPACEMAP_HISTOGRAM, "com.delphix:spacemap_histogram", "spacemap_histogram", - "Spacemaps maintain space histograms.", B_TRUE, B_FALSE, - B_FALSE, NULL); + "Spacemaps maintain space histograms.", + ZFEATURE_FLAG_READONLY_COMPAT, NULL); zfeature_register(SPA_FEATURE_ENABLED_TXG, "com.delphix:enabled_txg", "enabled_txg", - "Record txg at which a feature is enabled", B_TRUE, B_FALSE, - B_FALSE, NULL); + "Record txg at which a feature is enabled", + ZFEATURE_FLAG_READONLY_COMPAT, NULL); static spa_feature_t hole_birth_deps[] = { SPA_FEATURE_ENABLED_TXG, SPA_FEATURE_NONE }; zfeature_register(SPA_FEATURE_HOLE_BIRTH, "com.delphix:hole_birth", "hole_birth", "Retain hole birth txg for more precise zfs send", - B_FALSE, B_TRUE, B_TRUE, hole_birth_deps); + ZFEATURE_FLAG_MOS | ZFEATURE_FLAG_ACTIVATE_ON_ENABLE, + hole_birth_deps); zfeature_register(SPA_FEATURE_EXTENSIBLE_DATASET, "com.delphix:extensible_dataset", "extensible_dataset", "Enhanced dataset functionality, used by other features.", - B_FALSE, B_FALSE, B_FALSE, NULL); + 0, NULL); static const spa_feature_t bookmarks_deps[] = { SPA_FEATURE_EXTENSIBLE_DATASET, @@ -206,7 +205,7 @@ zpool_feature_init(void) zfeature_register(SPA_FEATURE_BOOKMARKS, "com.delphix:bookmarks", "bookmarks", "\"zfs bookmark\" command", - B_TRUE, B_FALSE, B_FALSE, bookmarks_deps); + ZFEATURE_FLAG_READONLY_COMPAT, bookmarks_deps); static const spa_feature_t filesystem_limits_deps[] = { SPA_FEATURE_EXTENSIBLE_DATASET, @@ -214,13 +213,14 @@ zpool_feature_init(void) }; zfeature_register(SPA_FEATURE_FS_SS_LIMIT, "com.joyent:filesystem_limits", "filesystem_limits", - "Filesystem and snapshot limits.", B_TRUE, B_FALSE, B_FALSE, - filesystem_limits_deps); + "Filesystem and snapshot limits.", + ZFEATURE_FLAG_READONLY_COMPAT, filesystem_limits_deps); zfeature_register(SPA_FEATURE_EMBEDDED_DATA, "com.delphix:embedded_data", "embedded_data", "Blocks which compress very well use even less space.", - B_FALSE, B_TRUE, B_TRUE, NULL); + ZFEATURE_FLAG_MOS | ZFEATURE_FLAG_ACTIVATE_ON_ENABLE, + NULL); static const spa_feature_t large_blocks_deps[] = { SPA_FEATURE_EXTENSIBLE_DATASET, @@ -228,6 +228,6 @@ zpool_feature_init(void) }; zfeature_register(SPA_FEATURE_LARGE_BLOCKS, "org.open-zfs:large_blocks", "large_blocks", - "Support for blocks larger than 128KB.", B_FALSE, B_FALSE, B_FALSE, - large_blocks_deps); + "Support for blocks larger than 128KB.", + ZFEATURE_FLAG_PER_DATASET, large_blocks_deps); } Modified: stable/10/sys/cddl/contrib/opensolaris/common/zfs/zfeature_common.h ============================================================================== --- stable/10/sys/cddl/contrib/opensolaris/common/zfs/zfeature_common.h Sat Oct 3 08:03:36 2015 (r288571) +++ stable/10/sys/cddl/contrib/opensolaris/common/zfs/zfeature_common.h Sat Oct 3 08:05:33 2015 (r288572) @@ -20,7 +20,7 @@ */ /* - * Copyright (c) 2013 by Delphix. All rights reserved. + * Copyright (c) 2011, 2015 by Delphix. All rights reserved. * Copyright (c) 2013 by Saso Kiselkov. All rights reserved. * Copyright (c) 2013, Joyent, Inc. All rights reserved. */ @@ -56,15 +56,23 @@ typedef enum spa_feature { #define SPA_FEATURE_DISABLED (-1ULL) +typedef enum zfeature_flags { + /* Can open pool readonly even if this feature is not supported. */ + ZFEATURE_FLAG_READONLY_COMPAT = (1 << 0), + /* Is this feature necessary to read the MOS? */ + ZFEATURE_FLAG_MOS = (1 << 1), + /* Activate this feature at the same time it is enabled. */ + ZFEATURE_FLAG_ACTIVATE_ON_ENABLE = (1 << 2), + /* Each dataset has a field set if it has ever used this feature. */ + ZFEATURE_FLAG_PER_DATASET = (1 << 3) +} zfeature_flags_t; + typedef struct zfeature_info { spa_feature_t fi_feature; const char *fi_uname; /* User-facing feature name */ const char *fi_guid; /* On-disk feature identifier */ const char *fi_desc; /* Feature description */ - boolean_t fi_can_readonly; /* Can open pool readonly w/o support? */ - boolean_t fi_mos; /* Is the feature necessary to read the MOS? */ - /* Activate this feature at the same time it is enabled */ - boolean_t fi_activate_on_enable; + zfeature_flags_t fi_flags; /* array of dependencies, terminated by SPA_FEATURE_NONE */ const spa_feature_t *fi_depends; } zfeature_info_t; Modified: stable/10/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/dbuf.c ============================================================================== --- stable/10/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/dbuf.c Sat Oct 3 08:03:36 2015 (r288571) +++ stable/10/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/dbuf.c Sat Oct 3 08:05:33 2015 (r288572) @@ -1573,6 +1573,11 @@ dmu_buf_write_embedded(dmu_buf_t *dbuf, struct dirty_leaf *dl; dmu_object_type_t type; + if (etype == BP_EMBEDDED_TYPE_DATA) { + ASSERT(spa_feature_is_active(dmu_objset_spa(db->db_objset), + SPA_FEATURE_EMBEDDED_DATA)); + } + DB_DNODE_ENTER(db); type = DB_DNODE(db)->dn_type; DB_DNODE_EXIT(db); Modified: stable/10/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/dmu_send.c ============================================================================== --- stable/10/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/dmu_send.c Sat Oct 3 08:03:36 2015 (r288571) +++ stable/10/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/dmu_send.c Sat Oct 3 08:05:33 2015 (r288572) @@ -21,7 +21,7 @@ /* * Copyright (c) 2005, 2010, Oracle and/or its affiliates. All rights reserved. * Copyright 2011 Nexenta Systems, Inc. All rights reserved. - * Copyright (c) 2011, 2014 by Delphix. All rights reserved. + * Copyright (c) 2011, 2015 by Delphix. All rights reserved. * Copyright (c) 2014, Joyent, Inc. All rights reserved. * Copyright (c) 2012, Martin Matuska . All rights reserved. * Copyright 2014 HybridCluster. All rights reserved. @@ -721,7 +721,7 @@ dmu_send_impl(void *tag, dsl_pool_t *dp, } #endif - if (large_block_ok && to_ds->ds_large_blocks) + if (large_block_ok && to_ds->ds_feature_inuse[SPA_FEATURE_LARGE_BLOCKS]) featureflags |= DMU_BACKUP_FEATURE_LARGE_BLOCKS; if (embedok && spa_feature_is_active(dp->dp_spa, SPA_FEATURE_EMBEDDED_DATA)) { @@ -1379,13 +1379,6 @@ dmu_recv_begin_sync(void *arg, dmu_tx_t } VERIFY0(dsl_dataset_own_obj(dp, dsobj, dmu_recv_tag, &newds)); - if ((DMU_GET_FEATUREFLAGS(drrb->drr_versioninfo) & - DMU_BACKUP_FEATURE_LARGE_BLOCKS) && - !newds->ds_large_blocks) { - dsl_dataset_activate_large_blocks_sync_impl(dsobj, tx); - newds->ds_large_blocks = B_TRUE; - } - dmu_buf_will_dirty(newds->ds_dbuf, tx); dsl_dataset_phys(newds)->ds_flags |= DS_FLAG_INCONSISTENT; Modified: stable/10/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/dsl_dataset.c ============================================================================== --- stable/10/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/dsl_dataset.c Sat Oct 3 08:03:36 2015 (r288571) +++ stable/10/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/dsl_dataset.c Sat Oct 3 08:05:33 2015 (r288572) @@ -21,7 +21,7 @@ /* * Copyright (c) 2005, 2010, Oracle and/or its affiliates. All rights reserved. * Portions Copyright (c) 2011 Martin Matuska - * Copyright (c) 2011, 2014 by Delphix. All rights reserved. + * Copyright (c) 2011, 2015 by Delphix. All rights reserved. * Copyright (c) 2014, Joyent, Inc. All rights reserved. * Copyright (c) 2014 RackTop Systems. * Copyright (c) 2014 Spectra Logic Corporation, All rights reserved. @@ -130,8 +130,10 @@ dsl_dataset_block_born(dsl_dataset_t *ds dsl_dataset_phys(ds)->ds_compressed_bytes += compressed; dsl_dataset_phys(ds)->ds_uncompressed_bytes += uncompressed; dsl_dataset_phys(ds)->ds_unique_bytes += used; - if (BP_GET_LSIZE(bp) > SPA_OLD_MAXBLOCKSIZE) - ds->ds_need_large_blocks = B_TRUE; + if (BP_GET_LSIZE(bp) > SPA_OLD_MAXBLOCKSIZE) { + ds->ds_feature_activation_needed[SPA_FEATURE_LARGE_BLOCKS] = + B_TRUE; + } mutex_exit(&ds->ds_lock); dsl_dir_diduse_space(ds->ds_dir, DD_USED_HEAD, delta, compressed, uncompressed, tx); @@ -433,19 +435,23 @@ dsl_dataset_hold_obj(dsl_pool_t *dp, uin offsetof(dmu_sendarg_t, dsa_link)); if (doi.doi_type == DMU_OTN_ZAP_METADATA) { - int zaperr = zap_contains(mos, dsobj, - DS_FIELD_LARGE_BLOCKS); - if (zaperr != ENOENT) { - VERIFY0(zaperr); - ds->ds_large_blocks = B_TRUE; + for (spa_feature_t f = 0; f < SPA_FEATURES; f++) { + if (!(spa_feature_table[f].fi_flags & + ZFEATURE_FLAG_PER_DATASET)) + continue; + err = zap_contains(mos, dsobj, + spa_feature_table[f].fi_guid); + if (err == 0) { + ds->ds_feature_inuse[f] = B_TRUE; + } else { + ASSERT3U(err, ==, ENOENT); + err = 0; + } } } - if (err == 0) { - err = dsl_dir_hold_obj(dp, - dsl_dataset_phys(ds)->ds_dir_obj, NULL, ds, - &ds->ds_dir); - } + err = dsl_dir_hold_obj(dp, + dsl_dataset_phys(ds)->ds_dir_obj, NULL, ds, &ds->ds_dir); if (err != 0) { mutex_destroy(&ds->ds_lock); mutex_destroy(&ds->ds_opening_lock); @@ -701,6 +707,34 @@ dsl_dataset_tryown(dsl_dataset_t *ds, vo return (gotit); } +static void +dsl_dataset_activate_feature(uint64_t dsobj, spa_feature_t f, dmu_tx_t *tx) +{ + spa_t *spa = dmu_tx_pool(tx)->dp_spa; + objset_t *mos = dmu_tx_pool(tx)->dp_meta_objset; + uint64_t zero = 0; + + VERIFY(spa_feature_table[f].fi_flags & ZFEATURE_FLAG_PER_DATASET); + + spa_feature_incr(spa, f, tx); + dmu_object_zapify(mos, dsobj, DMU_OT_DSL_DATASET, tx); + + VERIFY0(zap_add(mos, dsobj, spa_feature_table[f].fi_guid, + sizeof (zero), 1, &zero, tx)); +} + +void +dsl_dataset_deactivate_feature(uint64_t dsobj, spa_feature_t f, dmu_tx_t *tx) +{ + spa_t *spa = dmu_tx_pool(tx)->dp_spa; + objset_t *mos = dmu_tx_pool(tx)->dp_meta_objset; + + VERIFY(spa_feature_table[f].fi_flags & ZFEATURE_FLAG_PER_DATASET); + + VERIFY0(zap_remove(mos, dsobj, spa_feature_table[f].fi_guid, tx)); + spa_feature_decr(spa, f, tx); +} + uint64_t dsl_dataset_create_sync_dd(dsl_dir_t *dd, dsl_dataset_t *origin, uint64_t flags, dmu_tx_t *tx) @@ -761,8 +795,10 @@ dsl_dataset_create_sync_dd(dsl_dir_t *dd dsphys->ds_flags |= dsl_dataset_phys(origin)->ds_flags & (DS_FLAG_INCONSISTENT | DS_FLAG_CI_DATASET); - if (origin->ds_large_blocks) - dsl_dataset_activate_large_blocks_sync_impl(dsobj, tx); + for (spa_feature_t f = 0; f < SPA_FEATURES; f++) { + if (origin->ds_feature_inuse[f]) + dsl_dataset_activate_feature(dsobj, f, tx); + } dmu_buf_will_dirty(origin->ds_dbuf, tx); dsl_dataset_phys(origin)->ds_num_children++; @@ -1324,8 +1360,10 @@ dsl_dataset_snapshot_sync_impl(dsl_datas dsphys->ds_bp = dsl_dataset_phys(ds)->ds_bp; dmu_buf_rele(dbuf, FTAG); - if (ds->ds_large_blocks) - dsl_dataset_activate_large_blocks_sync_impl(dsobj, tx); + for (spa_feature_t f = 0; f < SPA_FEATURES; f++) { + if (ds->ds_feature_inuse[f]) + dsl_dataset_activate_feature(dsobj, f, tx); + } ASSERT3U(ds->ds_prev != 0, ==, dsl_dataset_phys(ds)->ds_prev_snap_obj != 0); @@ -1617,9 +1655,13 @@ dsl_dataset_sync(dsl_dataset_t *ds, zio_ dmu_objset_sync(ds->ds_objset, zio, tx); - if (ds->ds_need_large_blocks && !ds->ds_large_blocks) { - dsl_dataset_activate_large_blocks_sync_impl(ds->ds_object, tx); - ds->ds_large_blocks = B_TRUE; + for (spa_feature_t f = 0; f < SPA_FEATURES; f++) { + if (ds->ds_feature_activation_needed[f]) { + if (ds->ds_feature_inuse[f]) + continue; + dsl_dataset_activate_feature(ds->ds_object, f, tx); + ds->ds_feature_inuse[f] = B_TRUE; + } } } @@ -2783,6 +2825,40 @@ dsl_dataset_clone_swap_sync_impl(dsl_dat dsl_dataset_phys(clone)->ds_unique_bytes <= origin_head->ds_quota); ASSERT3P(clone->ds_prev, ==, origin_head->ds_prev); + /* + * Swap per-dataset feature flags. + */ + for (spa_feature_t f = 0; f < SPA_FEATURES; f++) { + if (!(spa_feature_table[f].fi_flags & + ZFEATURE_FLAG_PER_DATASET)) { + ASSERT(!clone->ds_feature_inuse[f]); + ASSERT(!origin_head->ds_feature_inuse[f]); + continue; + } + + boolean_t clone_inuse = clone->ds_feature_inuse[f]; + boolean_t origin_head_inuse = origin_head->ds_feature_inuse[f]; + + if (clone_inuse) { + dsl_dataset_deactivate_feature(clone->ds_object, f, tx); + clone->ds_feature_inuse[f] = B_FALSE; + } + if (origin_head_inuse) { + dsl_dataset_deactivate_feature(origin_head->ds_object, + f, tx); + origin_head->ds_feature_inuse[f] = B_FALSE; + } + if (clone_inuse) { + dsl_dataset_activate_feature(origin_head->ds_object, + f, tx); + origin_head->ds_feature_inuse[f] = B_TRUE; + } + if (origin_head_inuse) { + dsl_dataset_activate_feature(clone->ds_object, f, tx); + clone->ds_feature_inuse[f] = B_TRUE; + } + } + dmu_buf_will_dirty(clone->ds_dbuf, tx); dmu_buf_will_dirty(origin_head->ds_dbuf, tx); @@ -3337,77 +3413,6 @@ dsl_dataset_space_wouldfree(dsl_dataset_ return (err); } -static int -dsl_dataset_activate_large_blocks_check(void *arg, dmu_tx_t *tx) -{ - const char *dsname = arg; - dsl_dataset_t *ds; - dsl_pool_t *dp = dmu_tx_pool(tx); - int error = 0; - - if (!spa_feature_is_enabled(dp->dp_spa, SPA_FEATURE_LARGE_BLOCKS)) - return (SET_ERROR(ENOTSUP)); - - ASSERT(spa_feature_is_enabled(dp->dp_spa, - SPA_FEATURE_EXTENSIBLE_DATASET)); - - error = dsl_dataset_hold(dp, dsname, FTAG, &ds); - if (error != 0) - return (error); - - if (ds->ds_large_blocks) - error = EALREADY; - dsl_dataset_rele(ds, FTAG); - - return (error); -} - -void -dsl_dataset_activate_large_blocks_sync_impl(uint64_t dsobj, dmu_tx_t *tx) -{ - spa_t *spa = dmu_tx_pool(tx)->dp_spa; - objset_t *mos = dmu_tx_pool(tx)->dp_meta_objset; - uint64_t zero = 0; - - spa_feature_incr(spa, SPA_FEATURE_LARGE_BLOCKS, tx); - dmu_object_zapify(mos, dsobj, DMU_OT_DSL_DATASET, tx); - - VERIFY0(zap_add(mos, dsobj, DS_FIELD_LARGE_BLOCKS, - sizeof (zero), 1, &zero, tx)); -} - -static void -dsl_dataset_activate_large_blocks_sync(void *arg, dmu_tx_t *tx) -{ - const char *dsname = arg; - dsl_dataset_t *ds; - - VERIFY0(dsl_dataset_hold(dmu_tx_pool(tx), dsname, FTAG, &ds)); - - dsl_dataset_activate_large_blocks_sync_impl(ds->ds_object, tx); - ASSERT(!ds->ds_large_blocks); - ds->ds_large_blocks = B_TRUE; - dsl_dataset_rele(ds, FTAG); -} - -int -dsl_dataset_activate_large_blocks(const char *dsname) -{ - int error; - - error = dsl_sync_task(dsname, - dsl_dataset_activate_large_blocks_check, - dsl_dataset_activate_large_blocks_sync, (void *)dsname, - 1, ZFS_SPACE_CHECK_RESERVED); - - /* - * EALREADY indicates that this dataset already supports large blocks. - */ - if (error == EALREADY) - error = 0; - return (error); -} - /* * Return TRUE if 'earlier' is an earlier snapshot in 'later's timeline. * For example, they could both be snapshots of the same filesystem, and @@ -3452,7 +3457,6 @@ dsl_dataset_is_before(dsl_dataset_t *lat return (ret); } - void dsl_dataset_zapify(dsl_dataset_t *ds, dmu_tx_t *tx) { Modified: stable/10/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/dsl_destroy.c ============================================================================== --- stable/10/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/dsl_destroy.c Sat Oct 3 08:03:36 2015 (r288571) +++ stable/10/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/dsl_destroy.c Sat Oct 3 08:05:33 2015 (r288572) @@ -20,7 +20,7 @@ */ /* * Copyright (c) 2005, 2010, Oracle and/or its affiliates. All rights reserved. - * Copyright (c) 2012, 2014 by Delphix. All rights reserved. + * Copyright (c) 2012, 2015 by Delphix. All rights reserved. * Copyright (c) 2013 Steven Hartland. All rights reserved. * Copyright (c) 2013 by Joyent, Inc. All rights reserved. */ @@ -267,9 +267,11 @@ dsl_destroy_snapshot_sync_impl(dsl_datas obj = ds->ds_object; - if (ds->ds_large_blocks) { - ASSERT0(zap_contains(mos, obj, DS_FIELD_LARGE_BLOCKS)); - spa_feature_decr(dp->dp_spa, SPA_FEATURE_LARGE_BLOCKS, tx); + for (spa_feature_t f = 0; f < SPA_FEATURES; f++) { + if (ds->ds_feature_inuse[f]) { + dsl_dataset_deactivate_feature(obj, f, tx); + ds->ds_feature_inuse[f] = B_FALSE; + } } if (dsl_dataset_phys(ds)->ds_prev_snap_obj != 0) { ASSERT3P(ds->ds_prev, ==, NULL); @@ -736,12 +738,16 @@ dsl_destroy_head_sync_impl(dsl_dataset_t ASSERT0(ds->ds_reserved); } - if (ds->ds_large_blocks) - spa_feature_decr(dp->dp_spa, SPA_FEATURE_LARGE_BLOCKS, tx); + obj = ds->ds_object; - dsl_scan_ds_destroyed(ds, tx); + for (spa_feature_t f = 0; f < SPA_FEATURES; f++) { + if (ds->ds_feature_inuse[f]) { + dsl_dataset_deactivate_feature(obj, f, tx); + ds->ds_feature_inuse[f] = B_FALSE; + } + } - obj = ds->ds_object; + dsl_scan_ds_destroyed(ds, tx); if (dsl_dataset_phys(ds)->ds_prev_snap_obj != 0) { /* This is a clone */ Modified: stable/10/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/sys/dsl_dataset.h ============================================================================== --- stable/10/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/sys/dsl_dataset.h Sat Oct 3 08:03:36 2015 (r288571) +++ stable/10/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/sys/dsl_dataset.h Sat Oct 3 08:05:33 2015 (r288572) @@ -20,7 +20,7 @@ */ /* * Copyright (c) 2005, 2010, Oracle and/or its affiliates. All rights reserved. - * Copyright (c) 2011, 2014 by Delphix. All rights reserved. + * Copyright (c) 2011, 2015 by Delphix. All rights reserved. * Copyright (c) 2013, Joyent, Inc. All rights reserved. * Copyright (c) 2013 Steven Hartland. All rights reserved. * Copyright (c) 2014 Spectra Logic Corporation, All rights reserved. @@ -38,6 +38,7 @@ #include #include #include +#include #ifdef __cplusplus extern "C" { @@ -145,8 +146,6 @@ typedef struct dsl_dataset { /* only used in syncing context, only valid for non-snapshots: */ struct dsl_dataset *ds_prev; uint64_t ds_bookmarks; /* DMU_OTN_ZAP_METADATA */ - boolean_t ds_large_blocks; - boolean_t ds_need_large_blocks; /* has internal locking: */ dsl_deadlist_t ds_deadlist; @@ -185,6 +184,18 @@ typedef struct dsl_dataset { kmutex_t ds_sendstream_lock; list_t ds_sendstreams; + /* + * For ZFEATURE_FLAG_PER_DATASET features, set if this dataset + * uses this feature. + */ + uint8_t ds_feature_inuse[SPA_FEATURES]; + + /* + * Set if we need to activate the feature on this dataset this txg + * (used only in syncing context). + */ + uint8_t ds_feature_activation_needed[SPA_FEATURES]; + /* Protected by ds_lock; keep at end of struct for better locality */ char ds_snapname[MAXNAMELEN]; } dsl_dataset_t; @@ -264,8 +275,6 @@ int dsl_dataset_space_written(dsl_datase int dsl_dataset_space_wouldfree(dsl_dataset_t *firstsnap, dsl_dataset_t *last, uint64_t *usedp, uint64_t *compp, uint64_t *uncompp); boolean_t dsl_dataset_is_dirty(dsl_dataset_t *ds); -int dsl_dataset_activate_large_blocks(const char *dsname); -void dsl_dataset_activate_large_blocks_sync_impl(uint64_t dsobj, dmu_tx_t *tx); int dsl_dsobj_to_dsname(char *pname, uint64_t obj, char *buf); @@ -305,6 +314,9 @@ void dsl_dataset_set_refreservation_sync void dsl_dataset_zapify(dsl_dataset_t *ds, dmu_tx_t *tx); int dsl_dataset_rollback(const char *fsname, void *owner, nvlist_t *result); +void dsl_dataset_deactivate_feature(uint64_t dsobj, + spa_feature_t f, dmu_tx_t *tx); + #ifdef ZFS_DEBUG #define dprintf_ds(ds, fmt, ...) do { \ if (zfs_flags & ZFS_DEBUG_DPRINTF) { \ Modified: stable/10/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zfeature.c ============================================================================== --- stable/10/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zfeature.c Sat Oct 3 08:03:36 2015 (r288571) +++ stable/10/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zfeature.c Sat Oct 3 08:05:33 2015 (r288572) @@ -20,7 +20,7 @@ */ /* - * Copyright (c) 2013 by Delphix. All rights reserved. + * Copyright (c) 2011, 2015 by Delphix. All rights reserved. */ #include @@ -245,7 +245,7 @@ feature_get_refcount_from_disk(spa_t *sp { int err; uint64_t refcount; - uint64_t zapobj = feature->fi_can_readonly ? + uint64_t zapobj = (feature->fi_flags & ZFEATURE_FLAG_READONLY_COMPAT) ? spa->spa_feat_for_write_obj : spa->spa_feat_for_read_obj; /* @@ -296,7 +296,7 @@ feature_sync(spa_t *spa, zfeature_info_t dmu_tx_t *tx) { ASSERT(VALID_FEATURE_OR_NONE(feature->fi_feature)); - uint64_t zapobj = feature->fi_can_readonly ? + uint64_t zapobj = (feature->fi_flags & ZFEATURE_FLAG_READONLY_COMPAT) ? spa->spa_feat_for_write_obj : spa->spa_feat_for_read_obj; VERIFY0(zap_update(spa->spa_meta_objset, zapobj, feature->fi_guid, @@ -322,7 +322,7 @@ feature_sync(spa_t *spa, zfeature_info_t if (refcount == 0) spa_deactivate_mos_feature(spa, feature->fi_guid); - else if (feature->fi_mos) + else if (feature->fi_flags & ZFEATURE_FLAG_MOS) spa_activate_mos_feature(spa, feature->fi_guid, tx); } @@ -333,8 +333,9 @@ feature_sync(spa_t *spa, zfeature_info_t void feature_enable_sync(spa_t *spa, zfeature_info_t *feature, dmu_tx_t *tx) { - uint64_t initial_refcount = feature->fi_activate_on_enable ? 1 : 0; - uint64_t zapobj = feature->fi_can_readonly ? + uint64_t initial_refcount = + (feature->fi_flags & ZFEATURE_FLAG_ACTIVATE_ON_ENABLE) ? 1 : 0; + uint64_t zapobj = (feature->fi_flags & ZFEATURE_FLAG_READONLY_COMPAT) ? spa->spa_feat_for_write_obj : spa->spa_feat_for_read_obj; ASSERT(0 != zapobj); @@ -379,7 +380,7 @@ feature_do_action(spa_t *spa, spa_featur { uint64_t refcount; zfeature_info_t *feature = &spa_feature_table[fid]; - uint64_t zapobj = feature->fi_can_readonly ? + uint64_t zapobj = (feature->fi_flags & ZFEATURE_FLAG_READONLY_COMPAT) ? spa->spa_feat_for_write_obj : spa->spa_feat_for_read_obj; ASSERT(VALID_FEATURE_FID(fid)); From owner-svn-src-stable@freebsd.org Sat Oct 3 08:06:30 2015 Return-Path: Delivered-To: svn-src-stable@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id B814EA0DC3A; Sat, 3 Oct 2015 08:06:30 +0000 (UTC) (envelope-from mav@FreeBSD.org) Received: from repo.freebsd.org (repo.freebsd.org [IPv6:2001:1900:2254:2068::e6a:0]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id A86FC195F; Sat, 3 Oct 2015 08:06:30 +0000 (UTC) (envelope-from mav@FreeBSD.org) Received: from repo.freebsd.org ([127.0.1.70]) by repo.freebsd.org (8.15.2/8.15.2) with ESMTP id t9386UR7083679; Sat, 3 Oct 2015 08:06:30 GMT (envelope-from mav@FreeBSD.org) Received: (from mav@localhost) by repo.freebsd.org (8.15.2/8.15.2/Submit) id t9386UJ5083678; Sat, 3 Oct 2015 08:06:30 GMT (envelope-from mav@FreeBSD.org) Message-Id: <201510030806.t9386UJ5083678@repo.freebsd.org> X-Authentication-Warning: repo.freebsd.org: mav set sender to mav@FreeBSD.org using -f From: Alexander Motin Date: Sat, 3 Oct 2015 08:06:30 +0000 (UTC) To: src-committers@freebsd.org, svn-src-all@freebsd.org, svn-src-stable@freebsd.org, svn-src-stable-10@freebsd.org Subject: svn commit: r288573 - stable/10/sys/cddl/contrib/opensolaris/uts/common/fs/zfs X-SVN-Group: stable-10 MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit X-BeenThere: svn-src-stable@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: SVN commit messages for all the -stable branches of the src tree List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 03 Oct 2015 08:06:30 -0000 Author: mav Date: Sat Oct 3 08:06:29 2015 New Revision: 288573 URL: https://svnweb.freebsd.org/changeset/base/288573 Log: MFC r286710: 6093 zfsctl_shares_lookup should only VN_RELE() on zfs_zget() success Reviewed by: Gordon Ross Reviewed by: Matthew Ahrens Reviewed by: George Wilson Approved by: Robert Mustacchi Author: Dan McDonald illumos/illumos-gate@0f92170f1ec2737ee5a0e51b5f74093904811452 Modified: stable/10/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zfs_ctldir.c Directory Properties: stable/10/ (props changed) Modified: stable/10/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zfs_ctldir.c ============================================================================== --- stable/10/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zfs_ctldir.c Sat Oct 3 08:05:33 2015 (r288572) +++ stable/10/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zfs_ctldir.c Sat Oct 3 08:06:29 2015 (r288573) @@ -21,6 +21,7 @@ /* * Copyright (c) 2005, 2010, Oracle and/or its affiliates. All rights reserved. * Copyright (c) 2013 by Delphix. All rights reserved. + * Copyright 2015, OmniTI Computer Consulting, Inc. All rights reserved. */ /* @@ -1149,10 +1150,11 @@ zfsctl_shares_lookup(ap) ZFS_EXIT(zfsvfs); return (SET_ERROR(ENOTSUP)); } - if ((error = zfs_zget(zfsvfs, zfsvfs->z_shares_dir, &dzp)) == 0) + if ((error = zfs_zget(zfsvfs, zfsvfs->z_shares_dir, &dzp)) == 0) { error = VOP_LOOKUP(ZTOV(dzp), vpp, cnp); + VN_RELE(ZTOV(dzp)); + } - VN_RELE(ZTOV(dzp)); ZFS_EXIT(zfsvfs); return (error); From owner-svn-src-stable@freebsd.org Sat Oct 3 08:07:22 2015 Return-Path: Delivered-To: svn-src-stable@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 71F22A0DCF7; Sat, 3 Oct 2015 08:07:22 +0000 (UTC) (envelope-from mav@FreeBSD.org) Received: from repo.freebsd.org (repo.freebsd.org [IPv6:2001:1900:2254:2068::e6a:0]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 628451AB8; Sat, 3 Oct 2015 08:07:22 +0000 (UTC) (envelope-from mav@FreeBSD.org) Received: from repo.freebsd.org ([127.0.1.70]) by repo.freebsd.org (8.15.2/8.15.2) with ESMTP id t9387MBX083787; Sat, 3 Oct 2015 08:07:22 GMT (envelope-from mav@FreeBSD.org) Received: (from mav@localhost) by repo.freebsd.org (8.15.2/8.15.2/Submit) id t9387MYN083786; Sat, 3 Oct 2015 08:07:22 GMT (envelope-from mav@FreeBSD.org) Message-Id: <201510030807.t9387MYN083786@repo.freebsd.org> X-Authentication-Warning: repo.freebsd.org: mav set sender to mav@FreeBSD.org using -f From: Alexander Motin Date: Sat, 3 Oct 2015 08:07:22 +0000 (UTC) To: src-committers@freebsd.org, svn-src-all@freebsd.org, svn-src-stable@freebsd.org, svn-src-stable-10@freebsd.org Subject: svn commit: r288574 - stable/10/sys/cddl/contrib/opensolaris/uts/common/fs/zfs X-SVN-Group: stable-10 MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit X-BeenThere: svn-src-stable@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: SVN commit messages for all the -stable branches of the src tree List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 03 Oct 2015 08:07:22 -0000 Author: mav Date: Sat Oct 3 08:07:21 2015 New Revision: 288574 URL: https://svnweb.freebsd.org/changeset/base/288574 Log: MFC r286712: 6096 ZFS_SMB_ACL_RENAME needs to cleanup better Reviewed by: Matthew Ahrens Reviewed by: Gordon Ross Reviewed by: George Wilson Approved by: Robert Mustacchi illumos/illumos-gate@8f5190a540d64d2debee6664bbc740e4c38f5b7f Modified: stable/10/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zfs_ioctl.c Directory Properties: stable/10/ (props changed) Modified: stable/10/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zfs_ioctl.c ============================================================================== --- stable/10/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zfs_ioctl.c Sat Oct 3 08:06:29 2015 (r288573) +++ stable/10/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zfs_ioctl.c Sat Oct 3 08:07:21 2015 (r288574) @@ -5189,6 +5189,7 @@ zfs_ioc_smb_acl(zfs_cmd_t *zc) if ((error = get_nvlist(zc->zc_nvlist_src, zc->zc_nvlist_src_size, zc->zc_iflags, &nvlist)) != 0) { VN_RELE(vp); + VN_RELE(ZTOV(sharedir)); ZFS_EXIT(zfsvfs); return (error); } From owner-svn-src-stable@freebsd.org Sat Oct 3 09:22:22 2015 Return-Path: Delivered-To: svn-src-stable@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id C1E15A0F522; Sat, 3 Oct 2015 09:22:22 +0000 (UTC) (envelope-from mav@FreeBSD.org) Received: from repo.freebsd.org (repo.freebsd.org [IPv6:2001:1900:2254:2068::e6a:0]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 988E0115B; Sat, 3 Oct 2015 09:22:22 +0000 (UTC) (envelope-from mav@FreeBSD.org) Received: from repo.freebsd.org ([127.0.1.70]) by repo.freebsd.org (8.15.2/8.15.2) with ESMTP id t939MMRb016106; Sat, 3 Oct 2015 09:22:22 GMT (envelope-from mav@FreeBSD.org) Received: (from mav@localhost) by repo.freebsd.org (8.15.2/8.15.2/Submit) id t939MMkH016104; Sat, 3 Oct 2015 09:22:22 GMT (envelope-from mav@FreeBSD.org) Message-Id: <201510030922.t939MMkH016104@repo.freebsd.org> X-Authentication-Warning: repo.freebsd.org: mav set sender to mav@FreeBSD.org using -f From: Alexander Motin Date: Sat, 3 Oct 2015 09:22:22 +0000 (UTC) To: src-committers@freebsd.org, svn-src-all@freebsd.org, svn-src-stable@freebsd.org, svn-src-stable-10@freebsd.org Subject: svn commit: r288576 - in stable/10/sys/cddl/contrib/opensolaris/uts/common/fs/zfs: . sys X-SVN-Group: stable-10 MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit X-BeenThere: svn-src-stable@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: SVN commit messages for all the -stable branches of the src tree List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 03 Oct 2015 09:22:22 -0000 Author: mav Date: Sat Oct 3 09:22:21 2015 New Revision: 288576 URL: https://svnweb.freebsd.org/changeset/base/288576 Log: Fix r288549 build on stable. For some reason this (odd) code builds on head, but not on stable. Modified: stable/10/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/dbuf.c stable/10/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/sys/dmu.h Modified: stable/10/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/dbuf.c ============================================================================== --- stable/10/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/dbuf.c Sat Oct 3 09:15:23 2015 (r288575) +++ stable/10/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/dbuf.c Sat Oct 3 09:22:21 2015 (r288576) @@ -55,11 +55,6 @@ static void dbuf_destroy(dmu_buf_impl_t static boolean_t dbuf_undirty(dmu_buf_impl_t *db, dmu_tx_t *tx); static void dbuf_write(dbuf_dirty_record_t *dr, arc_buf_t *data, dmu_tx_t *tx); -#ifndef __lint -extern inline void dmu_buf_init_user(dmu_buf_user_t *dbu, - dmu_buf_evict_func_t *evict_func, dmu_buf_t **clear_on_evict_dbufp); -#endif /* ! __lint */ - /* * Global data structures and functions for the dbuf cache. */ Modified: stable/10/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/sys/dmu.h ============================================================================== --- stable/10/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/sys/dmu.h Sat Oct 3 09:15:23 2015 (r288575) +++ stable/10/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/sys/dmu.h Sat Oct 3 09:22:21 2015 (r288576) @@ -558,7 +558,7 @@ extern void dmu_buf_init_user(dmu_buf_user_t *dbu, dmu_buf_evict_func_t *evict_func, dmu_buf_t **clear_on_evict_dbufp); #else /* __lint */ -inline void +static inline void dmu_buf_init_user(dmu_buf_user_t *dbu, dmu_buf_evict_func_t *evict_func, dmu_buf_t **clear_on_evict_dbufp) { From owner-svn-src-stable@freebsd.org Sat Oct 3 09:33:33 2015 Return-Path: Delivered-To: svn-src-stable@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 92728A0FCAD; Sat, 3 Oct 2015 09:33:33 +0000 (UTC) (envelope-from erwin@FreeBSD.org) Received: from repo.freebsd.org (repo.freebsd.org [IPv6:2001:1900:2254:2068::e6a:0]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 7649A184D; Sat, 3 Oct 2015 09:33:33 +0000 (UTC) (envelope-from erwin@FreeBSD.org) Received: from repo.freebsd.org ([127.0.1.70]) by repo.freebsd.org (8.15.2/8.15.2) with ESMTP id t939XXdo020367; Sat, 3 Oct 2015 09:33:33 GMT (envelope-from erwin@FreeBSD.org) Received: (from erwin@localhost) by repo.freebsd.org (8.15.2/8.15.2/Submit) id t939XUR0020352; Sat, 3 Oct 2015 09:33:30 GMT (envelope-from erwin@FreeBSD.org) Message-Id: <201510030933.t939XUR0020352@repo.freebsd.org> X-Authentication-Warning: repo.freebsd.org: erwin set sender to erwin@FreeBSD.org using -f From: Erwin Lansing Date: Sat, 3 Oct 2015 09:33:30 +0000 (UTC) To: src-committers@freebsd.org, svn-src-all@freebsd.org, svn-src-stable@freebsd.org, svn-src-stable-9@freebsd.org Subject: svn commit: r288577 - in stable/9: contrib/bind9 contrib/bind9/bin/check contrib/bind9/bin/confgen contrib/bind9/bin/dig contrib/bind9/bin/dnssec contrib/bind9/bin/named contrib/bind9/bin/named/inc... X-SVN-Group: stable-9 MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit X-BeenThere: svn-src-stable@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: SVN commit messages for all the -stable branches of the src tree List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 03 Oct 2015 09:33:33 -0000 Author: erwin Date: Sat Oct 3 09:33:29 2015 New Revision: 288577 URL: https://svnweb.freebsd.org/changeset/base/288577 Log: Update BIND to 9.9.8 See release notes for notable changes: https://kb.isc.org/article/AA-01305 Note this is a direct commit to stable/9 as BIND is no longer in head. Sponsored by: DK Hostmaster A/S Added: stable/9/contrib/bind9/doc/arm/html-fixup.pl - copied unchanged from r288438, vendor/bind9/dist/doc/arm/html-fixup.pl Modified: stable/9/contrib/bind9/CHANGES stable/9/contrib/bind9/README stable/9/contrib/bind9/bin/check/check-tool.c stable/9/contrib/bind9/bin/check/named-checkconf.c stable/9/contrib/bind9/bin/check/named-checkzone.c stable/9/contrib/bind9/bin/confgen/keygen.c stable/9/contrib/bind9/bin/confgen/util.c stable/9/contrib/bind9/bin/dig/dig.1 stable/9/contrib/bind9/bin/dig/dig.c stable/9/contrib/bind9/bin/dig/dig.docbook stable/9/contrib/bind9/bin/dig/dig.html stable/9/contrib/bind9/bin/dig/dighost.c stable/9/contrib/bind9/bin/dig/nslookup.c stable/9/contrib/bind9/bin/dnssec/dnssec-dsfromkey.8 stable/9/contrib/bind9/bin/dnssec/dnssec-dsfromkey.c stable/9/contrib/bind9/bin/dnssec/dnssec-dsfromkey.docbook stable/9/contrib/bind9/bin/dnssec/dnssec-dsfromkey.html stable/9/contrib/bind9/bin/dnssec/dnssec-keygen.c stable/9/contrib/bind9/bin/dnssec/dnssec-revoke.c stable/9/contrib/bind9/bin/dnssec/dnssec-settime.c stable/9/contrib/bind9/bin/dnssec/dnssec-signzone.c stable/9/contrib/bind9/bin/named/client.c stable/9/contrib/bind9/bin/named/config.c stable/9/contrib/bind9/bin/named/control.c stable/9/contrib/bind9/bin/named/include/named/lwdclient.h stable/9/contrib/bind9/bin/named/include/named/main.h stable/9/contrib/bind9/bin/named/include/named/server.h stable/9/contrib/bind9/bin/named/interfacemgr.c stable/9/contrib/bind9/bin/named/logconf.c stable/9/contrib/bind9/bin/named/lwdclient.c stable/9/contrib/bind9/bin/named/lwresd.c stable/9/contrib/bind9/bin/named/main.c stable/9/contrib/bind9/bin/named/named.8 stable/9/contrib/bind9/bin/named/named.docbook stable/9/contrib/bind9/bin/named/named.html stable/9/contrib/bind9/bin/named/query.c stable/9/contrib/bind9/bin/named/server.c stable/9/contrib/bind9/bin/named/statschannel.c stable/9/contrib/bind9/bin/named/update.c stable/9/contrib/bind9/bin/named/xfrout.c stable/9/contrib/bind9/bin/nsupdate/nsupdate.1 stable/9/contrib/bind9/bin/nsupdate/nsupdate.c stable/9/contrib/bind9/bin/nsupdate/nsupdate.docbook stable/9/contrib/bind9/bin/nsupdate/nsupdate.html stable/9/contrib/bind9/bin/rndc/rndc.8 stable/9/contrib/bind9/bin/rndc/rndc.c stable/9/contrib/bind9/bin/rndc/rndc.docbook stable/9/contrib/bind9/bin/rndc/rndc.html stable/9/contrib/bind9/bin/rndc/util.c stable/9/contrib/bind9/bin/tools/arpaname.c stable/9/contrib/bind9/bin/tools/isc-hmac-fixup.c stable/9/contrib/bind9/bin/tools/named-journalprint.c stable/9/contrib/bind9/config.h.in stable/9/contrib/bind9/configure.in stable/9/contrib/bind9/doc/arm/Bv9ARM-book.xml stable/9/contrib/bind9/doc/arm/Bv9ARM.ch01.html stable/9/contrib/bind9/doc/arm/Bv9ARM.ch02.html stable/9/contrib/bind9/doc/arm/Bv9ARM.ch03.html stable/9/contrib/bind9/doc/arm/Bv9ARM.ch04.html stable/9/contrib/bind9/doc/arm/Bv9ARM.ch05.html stable/9/contrib/bind9/doc/arm/Bv9ARM.ch06.html stable/9/contrib/bind9/doc/arm/Bv9ARM.ch07.html stable/9/contrib/bind9/doc/arm/Bv9ARM.ch08.html stable/9/contrib/bind9/doc/arm/Bv9ARM.ch09.html stable/9/contrib/bind9/doc/arm/Bv9ARM.ch10.html stable/9/contrib/bind9/doc/arm/Bv9ARM.ch11.html stable/9/contrib/bind9/doc/arm/Bv9ARM.ch12.html stable/9/contrib/bind9/doc/arm/Bv9ARM.ch13.html stable/9/contrib/bind9/doc/arm/Bv9ARM.html stable/9/contrib/bind9/doc/arm/Bv9ARM.pdf stable/9/contrib/bind9/doc/arm/Makefile.in stable/9/contrib/bind9/doc/arm/man.arpaname.html stable/9/contrib/bind9/doc/arm/man.ddns-confgen.html stable/9/contrib/bind9/doc/arm/man.dig.html stable/9/contrib/bind9/doc/arm/man.dnssec-checkds.html stable/9/contrib/bind9/doc/arm/man.dnssec-coverage.html stable/9/contrib/bind9/doc/arm/man.dnssec-dsfromkey.html stable/9/contrib/bind9/doc/arm/man.dnssec-keyfromlabel.html stable/9/contrib/bind9/doc/arm/man.dnssec-keygen.html stable/9/contrib/bind9/doc/arm/man.dnssec-revoke.html stable/9/contrib/bind9/doc/arm/man.dnssec-settime.html stable/9/contrib/bind9/doc/arm/man.dnssec-signzone.html stable/9/contrib/bind9/doc/arm/man.dnssec-verify.html stable/9/contrib/bind9/doc/arm/man.genrandom.html stable/9/contrib/bind9/doc/arm/man.host.html stable/9/contrib/bind9/doc/arm/man.isc-hmac-fixup.html stable/9/contrib/bind9/doc/arm/man.named-checkconf.html stable/9/contrib/bind9/doc/arm/man.named-checkzone.html stable/9/contrib/bind9/doc/arm/man.named-journalprint.html stable/9/contrib/bind9/doc/arm/man.named.html stable/9/contrib/bind9/doc/arm/man.nsec3hash.html stable/9/contrib/bind9/doc/arm/man.nsupdate.html stable/9/contrib/bind9/doc/arm/man.rndc-confgen.html stable/9/contrib/bind9/doc/arm/man.rndc.conf.html stable/9/contrib/bind9/doc/arm/man.rndc.html stable/9/contrib/bind9/doc/arm/notes.html stable/9/contrib/bind9/doc/arm/notes.pdf stable/9/contrib/bind9/doc/arm/notes.xml stable/9/contrib/bind9/doc/misc/rfc-compliance stable/9/contrib/bind9/isc-config.sh.in stable/9/contrib/bind9/lib/bind9/api stable/9/contrib/bind9/lib/bind9/check.c stable/9/contrib/bind9/lib/dns/adb.c stable/9/contrib/bind9/lib/dns/api stable/9/contrib/bind9/lib/dns/cache.c stable/9/contrib/bind9/lib/dns/callbacks.c stable/9/contrib/bind9/lib/dns/client.c stable/9/contrib/bind9/lib/dns/diff.c stable/9/contrib/bind9/lib/dns/dispatch.c stable/9/contrib/bind9/lib/dns/dlz.c stable/9/contrib/bind9/lib/dns/dnssec.c stable/9/contrib/bind9/lib/dns/dst_api.c stable/9/contrib/bind9/lib/dns/dst_openssl.h stable/9/contrib/bind9/lib/dns/dst_parse.c stable/9/contrib/bind9/lib/dns/gssapi_link.c stable/9/contrib/bind9/lib/dns/gssapictx.c stable/9/contrib/bind9/lib/dns/hmac_link.c stable/9/contrib/bind9/lib/dns/include/dns/adb.h stable/9/contrib/bind9/lib/dns/include/dns/log.h stable/9/contrib/bind9/lib/dns/include/dns/message.h stable/9/contrib/bind9/lib/dns/include/dns/name.h stable/9/contrib/bind9/lib/dns/include/dns/resolver.h stable/9/contrib/bind9/lib/dns/include/dns/result.h stable/9/contrib/bind9/lib/dns/include/dns/rrl.h stable/9/contrib/bind9/lib/dns/include/dns/stats.h stable/9/contrib/bind9/lib/dns/include/dns/types.h stable/9/contrib/bind9/lib/dns/include/dns/update.h stable/9/contrib/bind9/lib/dns/include/dns/zone.h stable/9/contrib/bind9/lib/dns/include/dst/dst.h stable/9/contrib/bind9/lib/dns/journal.c stable/9/contrib/bind9/lib/dns/keytable.c stable/9/contrib/bind9/lib/dns/log.c stable/9/contrib/bind9/lib/dns/master.c stable/9/contrib/bind9/lib/dns/message.c stable/9/contrib/bind9/lib/dns/name.c stable/9/contrib/bind9/lib/dns/ncache.c stable/9/contrib/bind9/lib/dns/nsec.c stable/9/contrib/bind9/lib/dns/nsec3.c stable/9/contrib/bind9/lib/dns/openssl_link.c stable/9/contrib/bind9/lib/dns/openssldh_link.c stable/9/contrib/bind9/lib/dns/openssldsa_link.c stable/9/contrib/bind9/lib/dns/opensslecdsa_link.c stable/9/contrib/bind9/lib/dns/opensslgost_link.c stable/9/contrib/bind9/lib/dns/opensslrsa_link.c stable/9/contrib/bind9/lib/dns/order.c stable/9/contrib/bind9/lib/dns/private.c stable/9/contrib/bind9/lib/dns/rbt.c stable/9/contrib/bind9/lib/dns/rbtdb.c stable/9/contrib/bind9/lib/dns/rdata.c stable/9/contrib/bind9/lib/dns/rdata/any_255/tsig_250.c stable/9/contrib/bind9/lib/dns/rdata/ch_3/a_1.c stable/9/contrib/bind9/lib/dns/rdata/generic/afsdb_18.c stable/9/contrib/bind9/lib/dns/rdata/generic/caa_257.c stable/9/contrib/bind9/lib/dns/rdata/generic/cdnskey_60.c stable/9/contrib/bind9/lib/dns/rdata/generic/cds_59.c stable/9/contrib/bind9/lib/dns/rdata/generic/cert_37.c stable/9/contrib/bind9/lib/dns/rdata/generic/cname_5.c stable/9/contrib/bind9/lib/dns/rdata/generic/dlv_32769.c stable/9/contrib/bind9/lib/dns/rdata/generic/dname_39.c stable/9/contrib/bind9/lib/dns/rdata/generic/dnskey_48.c stable/9/contrib/bind9/lib/dns/rdata/generic/ds_43.c stable/9/contrib/bind9/lib/dns/rdata/generic/eui48_108.c stable/9/contrib/bind9/lib/dns/rdata/generic/eui64_109.c stable/9/contrib/bind9/lib/dns/rdata/generic/gpos_27.c stable/9/contrib/bind9/lib/dns/rdata/generic/hinfo_13.c stable/9/contrib/bind9/lib/dns/rdata/generic/hip_55.c stable/9/contrib/bind9/lib/dns/rdata/generic/ipseckey_45.c stable/9/contrib/bind9/lib/dns/rdata/generic/isdn_20.c stable/9/contrib/bind9/lib/dns/rdata/generic/key_25.c stable/9/contrib/bind9/lib/dns/rdata/generic/keydata_65533.c stable/9/contrib/bind9/lib/dns/rdata/generic/l32_105.c stable/9/contrib/bind9/lib/dns/rdata/generic/l64_106.c stable/9/contrib/bind9/lib/dns/rdata/generic/loc_29.c stable/9/contrib/bind9/lib/dns/rdata/generic/lp_107.c stable/9/contrib/bind9/lib/dns/rdata/generic/mb_7.c stable/9/contrib/bind9/lib/dns/rdata/generic/md_3.c stable/9/contrib/bind9/lib/dns/rdata/generic/mf_4.c stable/9/contrib/bind9/lib/dns/rdata/generic/mg_8.c stable/9/contrib/bind9/lib/dns/rdata/generic/minfo_14.c stable/9/contrib/bind9/lib/dns/rdata/generic/mr_9.c stable/9/contrib/bind9/lib/dns/rdata/generic/mx_15.c stable/9/contrib/bind9/lib/dns/rdata/generic/naptr_35.c stable/9/contrib/bind9/lib/dns/rdata/generic/nid_104.c stable/9/contrib/bind9/lib/dns/rdata/generic/ns_2.c stable/9/contrib/bind9/lib/dns/rdata/generic/nsec3_50.c stable/9/contrib/bind9/lib/dns/rdata/generic/nsec3param_51.c stable/9/contrib/bind9/lib/dns/rdata/generic/nsec_47.c stable/9/contrib/bind9/lib/dns/rdata/generic/null_10.c stable/9/contrib/bind9/lib/dns/rdata/generic/nxt_30.c stable/9/contrib/bind9/lib/dns/rdata/generic/openpgpkey_61.c stable/9/contrib/bind9/lib/dns/rdata/generic/opt_41.c stable/9/contrib/bind9/lib/dns/rdata/generic/proforma.c stable/9/contrib/bind9/lib/dns/rdata/generic/ptr_12.c stable/9/contrib/bind9/lib/dns/rdata/generic/rp_17.c stable/9/contrib/bind9/lib/dns/rdata/generic/rrsig_46.c stable/9/contrib/bind9/lib/dns/rdata/generic/rt_21.c stable/9/contrib/bind9/lib/dns/rdata/generic/sig_24.c stable/9/contrib/bind9/lib/dns/rdata/generic/soa_6.c stable/9/contrib/bind9/lib/dns/rdata/generic/spf_99.c stable/9/contrib/bind9/lib/dns/rdata/generic/sshfp_44.c stable/9/contrib/bind9/lib/dns/rdata/generic/tkey_249.c stable/9/contrib/bind9/lib/dns/rdata/generic/tlsa_52.c stable/9/contrib/bind9/lib/dns/rdata/generic/txt_16.c stable/9/contrib/bind9/lib/dns/rdata/generic/unspec_103.c stable/9/contrib/bind9/lib/dns/rdata/generic/uri_256.c stable/9/contrib/bind9/lib/dns/rdata/generic/x25_19.c stable/9/contrib/bind9/lib/dns/rdata/hs_4/a_1.c stable/9/contrib/bind9/lib/dns/rdata/in_1/a6_38.c stable/9/contrib/bind9/lib/dns/rdata/in_1/a_1.c stable/9/contrib/bind9/lib/dns/rdata/in_1/aaaa_28.c stable/9/contrib/bind9/lib/dns/rdata/in_1/apl_42.c stable/9/contrib/bind9/lib/dns/rdata/in_1/dhcid_49.c stable/9/contrib/bind9/lib/dns/rdata/in_1/kx_36.c stable/9/contrib/bind9/lib/dns/rdata/in_1/nsap-ptr_23.c stable/9/contrib/bind9/lib/dns/rdata/in_1/nsap_22.c stable/9/contrib/bind9/lib/dns/rdata/in_1/px_26.c stable/9/contrib/bind9/lib/dns/rdata/in_1/srv_33.c stable/9/contrib/bind9/lib/dns/rdata/in_1/wks_11.c stable/9/contrib/bind9/lib/dns/request.c stable/9/contrib/bind9/lib/dns/resolver.c stable/9/contrib/bind9/lib/dns/result.c stable/9/contrib/bind9/lib/dns/rpz.c stable/9/contrib/bind9/lib/dns/rrl.c stable/9/contrib/bind9/lib/dns/sdb.c stable/9/contrib/bind9/lib/dns/sdlz.c stable/9/contrib/bind9/lib/dns/spnego.c stable/9/contrib/bind9/lib/dns/tcpmsg.c stable/9/contrib/bind9/lib/dns/tkey.c stable/9/contrib/bind9/lib/dns/tsig.c stable/9/contrib/bind9/lib/dns/update.c stable/9/contrib/bind9/lib/dns/view.c stable/9/contrib/bind9/lib/dns/xfrin.c stable/9/contrib/bind9/lib/dns/zone.c stable/9/contrib/bind9/lib/export/isc/unix/include/isc/Makefile.in stable/9/contrib/bind9/lib/export/samples/nsprobe.c stable/9/contrib/bind9/lib/export/samples/sample-async.c stable/9/contrib/bind9/lib/export/samples/sample-gai.c stable/9/contrib/bind9/lib/export/samples/sample-request.c stable/9/contrib/bind9/lib/export/samples/sample-update.c stable/9/contrib/bind9/lib/export/samples/sample.c stable/9/contrib/bind9/lib/irs/api stable/9/contrib/bind9/lib/irs/getaddrinfo.c stable/9/contrib/bind9/lib/isc/api stable/9/contrib/bind9/lib/isc/assertions.c stable/9/contrib/bind9/lib/isc/backtrace.c stable/9/contrib/bind9/lib/isc/commandline.c stable/9/contrib/bind9/lib/isc/entropy.c stable/9/contrib/bind9/lib/isc/error.c stable/9/contrib/bind9/lib/isc/heap.c stable/9/contrib/bind9/lib/isc/hmacmd5.c stable/9/contrib/bind9/lib/isc/hmacsha.c stable/9/contrib/bind9/lib/isc/httpd.c stable/9/contrib/bind9/lib/isc/include/isc/app.h stable/9/contrib/bind9/lib/isc/include/isc/mem.h stable/9/contrib/bind9/lib/isc/include/isc/namespace.h stable/9/contrib/bind9/lib/isc/include/isc/platform.h.in stable/9/contrib/bind9/lib/isc/include/isc/print.h stable/9/contrib/bind9/lib/isc/include/isc/safe.h stable/9/contrib/bind9/lib/isc/include/isc/util.h stable/9/contrib/bind9/lib/isc/lex.c stable/9/contrib/bind9/lib/isc/lib.c stable/9/contrib/bind9/lib/isc/mem.c stable/9/contrib/bind9/lib/isc/pool.c stable/9/contrib/bind9/lib/isc/print.c stable/9/contrib/bind9/lib/isc/pthreads/mutex.c stable/9/contrib/bind9/lib/isc/regex.c stable/9/contrib/bind9/lib/isc/rwlock.c stable/9/contrib/bind9/lib/isc/safe.c stable/9/contrib/bind9/lib/isc/socket_api.c stable/9/contrib/bind9/lib/isc/stats.c stable/9/contrib/bind9/lib/isc/task.c stable/9/contrib/bind9/lib/isc/timer.c stable/9/contrib/bind9/lib/isc/unix/app.c stable/9/contrib/bind9/lib/isc/unix/file.c stable/9/contrib/bind9/lib/isc/unix/ifiter_ioctl.c stable/9/contrib/bind9/lib/isc/unix/ifiter_sysctl.c stable/9/contrib/bind9/lib/isc/unix/net.c stable/9/contrib/bind9/lib/isc/unix/socket.c stable/9/contrib/bind9/lib/isccc/Makefile.in stable/9/contrib/bind9/lib/isccc/alist.c stable/9/contrib/bind9/lib/isccc/api stable/9/contrib/bind9/lib/isccc/cc.c stable/9/contrib/bind9/lib/isccc/sexpr.c stable/9/contrib/bind9/lib/isccfg/api stable/9/contrib/bind9/lib/isccfg/include/isccfg/cfg.h stable/9/contrib/bind9/lib/isccfg/include/isccfg/grammar.h stable/9/contrib/bind9/lib/isccfg/namedconf.c stable/9/contrib/bind9/lib/isccfg/parser.c stable/9/contrib/bind9/lib/lwres/api stable/9/contrib/bind9/lib/lwres/herror.c stable/9/contrib/bind9/lib/lwres/man/lwres.html stable/9/contrib/bind9/lib/lwres/man/lwres_buffer.html stable/9/contrib/bind9/lib/lwres/man/lwres_config.html stable/9/contrib/bind9/lib/lwres/man/lwres_context.html stable/9/contrib/bind9/lib/lwres/man/lwres_gabn.html stable/9/contrib/bind9/lib/lwres/man/lwres_gai_strerror.html stable/9/contrib/bind9/lib/lwres/man/lwres_getaddrinfo.html stable/9/contrib/bind9/lib/lwres/man/lwres_gethostent.html stable/9/contrib/bind9/lib/lwres/man/lwres_getipnode.html stable/9/contrib/bind9/lib/lwres/man/lwres_getnameinfo.html stable/9/contrib/bind9/lib/lwres/man/lwres_getrrsetbyname.html stable/9/contrib/bind9/lib/lwres/man/lwres_gnba.html stable/9/contrib/bind9/lib/lwres/man/lwres_hstrerror.html stable/9/contrib/bind9/lib/lwres/man/lwres_inetntop.html stable/9/contrib/bind9/lib/lwres/man/lwres_noop.html stable/9/contrib/bind9/lib/lwres/man/lwres_packet.html stable/9/contrib/bind9/lib/lwres/man/lwres_resutil.html stable/9/contrib/bind9/lib/lwres/print.c stable/9/contrib/bind9/version stable/9/lib/bind/config.h stable/9/lib/bind/isc/isc/platform.h stable/9/lib/bind/lwres/lwres/platform.h Directory Properties: stable/9/contrib/bind9/ (props changed) Modified: stable/9/contrib/bind9/CHANGES ============================================================================== --- stable/9/contrib/bind9/CHANGES Sat Oct 3 09:22:21 2015 (r288576) +++ stable/9/contrib/bind9/CHANGES Sat Oct 3 09:33:29 2015 (r288577) @@ -1,14 +1,318 @@ - --- 9.9.7-P2 released --- + --- 9.9.8 released --- + + --- 9.9.8rc1 released --- + +4193. [bug] Handle broken servers that return BADVERS incorrectly. + [RT #40427] + +4192. [bug] The default rrset-order of random was not always being + applied. [RT #40456] + +4191. [protocol] Accept DNS-SD non LDH PTR records in reverse zones + as per RFC 6763. [RT #37889] + +4190. [protocol] Accept Active Diretory gc._msdcs. name as + valid with check-names. still needs to be + LDH. [RT #40399] + +4189. [cleanup] Don't exit on overly long tokens in named.conf. + [RT #40418] + +4188. [bug] Support HTTP/1.0 client properly on the statistics + channel. [RT #40261] + +4187. [func] When any RR type implementation doesn't + implement totext() for the RDATA's wire + representation and returns ISC_R_NOTIMPLEMENTED, + such RDATA is now printed in unknown + presentation format (RFC 3597). RR types affected + include LOC(29) and APL(42). [RT #40317]. + +4183. [cleanup] Use timing-safe memory comparisons in cryptographic + code. Also, the timing-safe comparison functions have + been renamed to avoid possible confusion with + memcmp(). Thanks to Loganaden Velvindron of + AFRINIC. [RT #40148] + +4182. [cleanup] Use mnemonics for RR class and type comparisons. + [RT #40297] + +4181. [bug] Queued notify messages could be dequeued from the + wrong rate limiter queue. [RT #40350] + +4179. [bug] Fix double frees in getaddrinfo() in libirs. + [RT #40209] + +4178. [bug] Fix assertion failure in parsing UNSPEC(103) RR from + text. [RT #40274] + +4177. [bug] Fix assertion failure in parsing NSAP records from + text. [RT #40285] + +4176. [bug] Address race issues with lwresd. [RT #40284] + +4175. [bug] TKEY with GSS-API keys needed bigger buffers. + [RT #40333] + +4174. [bug] "dnssec-coverage -r" didn't handle time unit + suffixes correctly. [RT #38444] + +4173. [bug] dig +sigchase was not properly matching the trusted + key. [RT #40188] + +4172. [bug] Named / named-checkconf didn't handle a view of CLASS0. + [RT #40265] + +4171. [bug] Fixed incorrect class checks in TSIG RR + implementation. [RT #40287] + +4170. [security] An incorrect boundary check in the OPENPGPKEY + rdatatype could trigger an assertion failure. + (CVE-2015-5986) [RT #40286] + +4169. [test] Added a 'wire_test -d' option to read input as + raw binary data, for use as a fuzzing harness. + [RT #40312] + +4168. [security] A buffer accounting error could trigger an + assertion failure when parsing certain malformed + DNSSEC keys. (CVE-2015-5722) [RT #40212] + + --- 9.9.8b1 released --- 4165. [security] A failure to reset a value to NULL in tkey.c could result in an assertion failure. (CVE-2015-5477) [RT #40046] - --- 9.9.7-P1 released --- +4164. [bug] Don't rename slave files and journals on out of memory. + [RT #40033] + +4163. [bug] Address compiler warnings. [RT #40024] + +4162. [bug] httpdmgr->flags was not being initialized. [RT #40017] + +4159. [cleanup] Alphabetize dig's help output. [RT #39966] + +4158. [protocol] Support the printing of EDNS COOKIE and EXPIRE options. + [RT #39928] + +4154. [bug] A OPT record should be included with the FORMERR + response when there is a malformed EDNS option. + [RT #39647] + +4153. [bug] Check that non significant ECS bits are zero on + receipt. [RT #39647] + +4151. [bug] 'rndc flush' could cause a deadlock. [RT #39835] + +4150. [bug] win32: listen-on-v6 { any; }; was not working. Apply + minimal fix. [RT #39667] + +4149. [bug] Fixed a race condition in the getaddrinfo() + implementation in libirs. [RT #39899] + +4148. [bug] Fix a bug when printing zone names with '/' character + in XML and JSON statistics output. [RT #39873] + +4147. [bug] Filter-aaaa / filter-aaaa-on-v4 / filter-aaaa-on-v6 + was returning referrals rather than nodata responses + when the AAAA records were filtered. [RT #39843] -4138. [bug] An uninitialized value in validator.c could result +4146. [bug] Address reference leak that could prevent a clean + shutdown. [RT #37125] + +4145. [bug] Not all unassociated adb entries where being printed. + [RT #37125] + +4143. [bug] serial-query-rate was not effective for notify. + [RT #39858] + +4142. [bug] rndc addzone with view specified saved NZF config + that could not be read back by named. This has now + been fixed. [RT #39845] + +4138. [security] An uninitialized value in validator.c could result in an assertion failure. (CVE-2015-4620) [RT #39795] +4137. [bug] Make rndc reconfig report configuration errors the + same way rndc reload does. [RT #39635] + +4132. [cleanup] dig: added +rd as a synonym for +recurse, + added +class as an unabbreviated alternative + to +cl. [RT #39686] + +4130. [bug] The compatibility shim for *printf() misprinted some + large numbers. [RT #39586] + +4129. [port] Address API changes in OpenSSL 1.1.0. [RT #39532] + +4128. [bug] Address issues raised by Coverity 7.6. [RT #39537] + +4127. [protocol] CDS and CDNSKEY need to be signed by the key signing + key as per RFC 7344, Section 4.1. [RT #37215] + +4123. [port] Added %z (size_t) format options to the portable + internal printf/sprintf implementation. [RT #39586] + +4118. [bug] Teach isc-config.sh about irs. [RT #39213] + +4117. [protocol] Add EMPTY.AS112.ARPA as per RFC 7534. + +4113. [test] Check for Net::DNS is some system test + prerequisites. [RT #39369] + +4112. [bug] Named failed to load when "root-delegation-only" + was used without a list of domains to exclude. + [RT #39380] + +4111. [doc] Alphabetize rndc man page. [RT #39360] + +4110. [bug] Address memory leaks / null pointer dereferences + on out of memory. [RT #39310] + +4109. [port] linux: support reading the local port range from + net.ipv4.ip_local_port_range. [RT # 39379] + +4107. [bug] Address potential deadlock when updating zone content. + [RT #39269] + +4106. [port] Improve readline support. [RT #38938] + +4105. [port] Misc fixes for Microsoft Visual Studio + 2015 CTP6 in 64 bit mode. [RT #39308] + +4104. [bug] Address uninitialized elements. [RT #39252] + +4102. [bug] Fix a use after free bug introduced in change + #4094. [RT #39281] + +4101. [bug] dig: the +split option didn't work with +short. + [RT #39291] + +4100. [bug] Inherited owernames on the line immediately following + a $INCLUDE were not working. [RT #39268] + +4099. [port] clang: make unknown commandline options hard errors + when determining what options are supported. + [RT #39273] + +4098. [bug] Address use-after-free issue when using a + predecessor key with dnssec-settime. [RT #39272] + +4097. [func] Add additional logging about xfrin transfer status. + [RT #39170] + +4096. [bug] Fix a use after free of query->sendevent. + [RT #39132] + +4094. [bug] A race during shutdown or reconfiguration could + cause an assertion in mem.c. [RT #38979] + +4091. [cleanup] Some cleanups in isc mem code. [RT #38896] + +4090. [bug] Fix a crash while parsing malformed CAA RRs in + presentation format, i.e., from text such as + from master files. Thanks to John Van de + Meulebrouck Brendgard for discovering and + reporting this problem. [RT #39003] + +4089. [bug] Send notifies immediately for slave zones during + startup. [RT #38843] + +4088. [port] Fixed errors when building with libressl. [RT #38899] + +4087. [bug] Fix a crash due to use-after-free due to sequencing + of tasks actions. [RT #38495] + +4085. [bug] ISC_PLATFORM_HAVEXADDQ could be inconsistently set. + [RT #38828] + +4084. [bug] Fix a possible race in updating stats counters. + [RT #38826] + +4082. [bug] Incrementally sign large inline zone deltas. + [RT #37927] + +4081. [cleanup] Use dns_rdatalist_init consistently. [RT #38759] + +4077. [test] Add static-stub regression test for DS NXDOMAIN + return making the static stub disappear. [RT #38564] + +4076. [bug] Named could crash on shutdown with outstanding + reload / reconfig events. [RT #38622] + +4075. [bug] Increase nsupdate's input buffer to accomodate + very large RRs. [RT #38689] + +4074. [cleanup] Cleaned up more warnings from gcc -Wshadow. [RT #38708] + +4073. [cleanup] Add libjson-c version number reporting to + "named -V"; normalize version number formatting. + [RT #38056] + +4072. [func] Add a --enable-querytrace configure switch for + very verbose query trace logging. (This option + has a negative performance impact and should be + used only for debugging.) [RT #37520] + +4070. [bug] Fix a segfault in nslookup in a query such as + "nslookup isc.org AMS.SNS-PB.ISC.ORG -all". + [RT #38548] + +4069. [doc] Reorganize options in the nsupdate man page. + [RT #38515] + +4067. [cleanup] Reduce noise from RRL when query logging is + disabled. [RT #38648] + +4066. [doc] Reorganize options in the dig man page. [RT #38516] + +4064. [contrib] dnssec-keyset.sh: Generates a specified number + of DNSSEC keys with timing set to implement a + pre-publication key rollover strategy. Thanks + to Jeffry A. Spain. [RT #38459] + +4063. [bug] Asynchronous zone loads were not handled + correctly when the zone load was already in + progress; this could trigger a crash in zt.c. + [RT #37573] + +4062. [bug] Fix an out-of-bounds read in RPZ code. If the + read succeeded, it doesn't result in a bug + during operation. If the read failed, named + could segfault. [RT #38559] + +3938. [func] Added quotas to be used in recursive resolvers + that are under high query load for names in zones + whose authoritative servers are nonresponsive or + are experiencing a denial of service attack. + + - "fetches-per-server" limits the number of + simultaneous queries that can be sent to any + single authoritative server. The configured + value is a starting point; it is automatically + adjusted downward if the server is partially or + completely non-responsive. The algorithm used to + adjust the quota can be configured via the + "fetch-quota-params" option. + - "fetches-per-zone" limits the number of + simultaneous queries that can be sent for names + within a single domain. (Note: Unlike + "fetches-per-server", this value is not + self-tuning.) + - New stats counters have been added to count + queries spilled due to these quotas. + + These options are not available by default; + use "configure --enable-fetchlimit" (or + --enable-developer) to include them in the build. + + See the ARM for details of these options. [RT #37125] + +3937. [func] Added some debug logging to better indicate the + conditions causing SERVFAILs when resolving. + [RT #35538] + --- 9.9.7 released --- --- 9.9.7rc2 released --- @@ -16,7 +320,7 @@ 4061. [bug] Handle timeout in legacy system test. [RT #38573] 4060. [bug] dns_rdata_freestruct could be called on a - uninitialised structure when handling a error. + uninitialized structure when handling a error. [RT #38568] 4059. [bug] Addressed valgrind warnings. [RT #38549] Modified: stable/9/contrib/bind9/README ============================================================================== --- stable/9/contrib/bind9/README Sat Oct 3 09:22:21 2015 (r288576) +++ stable/9/contrib/bind9/README Sat Oct 3 09:33:29 2015 (r288577) @@ -51,15 +51,35 @@ BIND 9 For up-to-date release notes and errata, see http://www.isc.org/software/bind9/releasenotes -BIND 9.9.7-P2 +BIND 9.9.8 - BIND 9.9.7-P1 is a security release addressing the flaw - described in CVE-2015-5477. + BIND 9.9.8 is a maintenance release and addresses bugs + found in BIND 9.9.7 and earlier, as well as the security + flaws described in CVE-2015-4620, CVE-2015-5477, + CVE-2015-5722, and CVE-2015-5986. + + It also makes the following new features available via a + compile-time option: + + - New "fetchlimit" quotas are now available for the use of + recursive resolvers that are are under high query load for + domains whose authoritative servers are nonresponsive or are + experiencing a denial of service attack. + + + "fetches-per-server" limits the number of simultaneous queries + that can be sent to any single authoritative server. The + configured value is a starting point; it is automatically + adjusted downward if the server is partially or completely + non-responsive. The algorithm used to adjust the quota can be + configured via the "fetch-quota-params" option. + + "fetches-per-zone" limits the number of simultaneous queries + that can be sent for names within a single domain. (Note: + Unlike "fetches-per-server", this value is not self-tuning.) + + New stats counters have been added to count + queries spilled due to these quotas. -BIND 9.9.7-P1 - - BIND 9.9.7-P1 is a security release addressing the flaw - described in CVE-2015-4620. + NOTE: These options are NOT built in by default; use + "configure --enable-fetchlimit" to enable them. BIND 9.9.7 Modified: stable/9/contrib/bind9/bin/check/check-tool.c ============================================================================== --- stable/9/contrib/bind9/bin/check/check-tool.c Sat Oct 3 09:22:21 2015 (r288576) +++ stable/9/contrib/bind9/bin/check/check-tool.c Sat Oct 3 09:33:29 2015 (r288577) @@ -1,5 +1,5 @@ /* - * Copyright (C) 2004-2012 Internet Systems Consortium, Inc. ("ISC") + * Copyright (C) 2004-2012, 2015 Internet Systems Consortium, Inc. ("ISC") * Copyright (C) 2000-2002 Internet Software Consortium. * * Permission to use, copy, modify, and/or distribute this software for any @@ -33,6 +33,7 @@ #include #include #include +#include #include #include #include Modified: stable/9/contrib/bind9/bin/check/named-checkconf.c ============================================================================== --- stable/9/contrib/bind9/bin/check/named-checkconf.c Sat Oct 3 09:22:21 2015 (r288576) +++ stable/9/contrib/bind9/bin/check/named-checkconf.c Sat Oct 3 09:33:29 2015 (r288577) @@ -1,5 +1,5 @@ /* - * Copyright (C) 2004-2007, 2009-2014 Internet Systems Consortium, Inc. ("ISC") + * Copyright (C) 2004-2007, 2009-2015 Internet Systems Consortium, Inc. ("ISC") * Copyright (C) 1999-2002 Internet Software Consortium. * * Permission to use, copy, modify, and/or distribute this software for any @@ -31,6 +31,7 @@ #include #include #include +#include #include #include #include Modified: stable/9/contrib/bind9/bin/check/named-checkzone.c ============================================================================== --- stable/9/contrib/bind9/bin/check/named-checkzone.c Sat Oct 3 09:22:21 2015 (r288576) +++ stable/9/contrib/bind9/bin/check/named-checkzone.c Sat Oct 3 09:33:29 2015 (r288577) @@ -1,5 +1,5 @@ /* - * Copyright (C) 2004-2013 Internet Systems Consortium, Inc. ("ISC") + * Copyright (C) 2004-2013, 2015 Internet Systems Consortium, Inc. ("ISC") * Copyright (C) 1999-2003 Internet Software Consortium. * * Permission to use, copy, modify, and/or distribute this software for any @@ -30,6 +30,7 @@ #include #include #include +#include #include #include #include Modified: stable/9/contrib/bind9/bin/confgen/keygen.c ============================================================================== --- stable/9/contrib/bind9/bin/confgen/keygen.c Sat Oct 3 09:22:21 2015 (r288576) +++ stable/9/contrib/bind9/bin/confgen/keygen.c Sat Oct 3 09:33:29 2015 (r288577) @@ -1,5 +1,5 @@ /* - * Copyright (C) 2009, 2012, 2013 Internet Systems Consortium, Inc. ("ISC") + * Copyright (C) 2009, 2012, 2013, 2015 Internet Systems Consortium, Inc. ("ISC") * * Permission to use, copy, modify, and/or distribute this software for any * purpose with or without fee is hereby granted, provided that the above @@ -29,6 +29,7 @@ #include #include #include +#include #include #include Modified: stable/9/contrib/bind9/bin/confgen/util.c ============================================================================== --- stable/9/contrib/bind9/bin/confgen/util.c Sat Oct 3 09:22:21 2015 (r288576) +++ stable/9/contrib/bind9/bin/confgen/util.c Sat Oct 3 09:33:29 2015 (r288577) @@ -1,5 +1,5 @@ /* - * Copyright (C) 2009 Internet Systems Consortium, Inc. ("ISC") + * Copyright (C) 2009, 2015 Internet Systems Consortium, Inc. ("ISC") * * Permission to use, copy, modify, and/or distribute this software for any * purpose with or without fee is hereby granted, provided that the above @@ -25,6 +25,7 @@ #include #include +#include #include "util.h" Modified: stable/9/contrib/bind9/bin/dig/dig.1 ============================================================================== --- stable/9/contrib/bind9/bin/dig/dig.1 Sat Oct 3 09:22:21 2015 (r288576) +++ stable/9/contrib/bind9/bin/dig/dig.1 Sat Oct 3 09:33:29 2015 (r288577) @@ -1,4 +1,4 @@ -.\" Copyright (C) 2004-2011, 2013, 2014 Internet Systems Consortium, Inc. ("ISC") +.\" Copyright (C) 2004-2011, 2013-2015 Internet Systems Consortium, Inc. ("ISC") .\" Copyright (C) 2000-2003 Internet Software Consortium. .\" .\" Permission to use, copy, modify, and/or distribute this software for any @@ -130,77 +130,97 @@ will perform a lookup for an A record. .RE .SH "OPTIONS" .PP -The -\fB\-b\fR -option sets the source IP address of the query to -\fIaddress\fR. This must be a valid address on one of the host's network interfaces or "0.0.0.0" or "::". An optional port may be specified by appending "#" +\-4 +.RS 4 +Use IPv4 only. +.RE .PP -The default query class (IN for internet) is overridden by the -\fB\-c\fR -option. +\-6 +.RS 4 +Use IPv6 only. +.RE +.PP +\-b \fIaddress\fR\fI[#port]\fR +.RS 4 +Set the source IP address of the query. The +\fIaddress\fR +must be a valid address on one of the host's network interfaces, or "0.0.0.0" or "::". An optional port may be specified by appending "#" +.RE +.PP +\-c \fIclass\fR +.RS 4 +Set the query class. The default \fIclass\fR -is any valid class, such as HS for Hesiod records or CH for Chaosnet records. +is IN; other classes are HS for Hesiod records or CH for Chaosnet records. +.RE .PP -The -\fB\-f\fR -option makes -\fBdig \fR -operate in batch mode by reading a list of lookup requests to process from the file -\fIfilename\fR. The file contains a number of queries, one per line. Each entry in the file should be organized in the same way they would be presented as queries to +\-f \fIfile\fR +.RS 4 +Batch mode: +\fBdig\fR +reads a list of lookup requests to process from the given +\fIfile\fR. Each line in the file should be organized in the same way they would be presented as queries to \fBdig\fR using the command\-line interface. +.RE .PP -The -\fB\-m\fR -option enables memory usage debugging. -.PP -If a non\-standard port number is to be queried, the -\fB\-p\fR -option is used. -\fIport#\fR -is the port number that -\fBdig\fR -will send its queries instead of the standard DNS port number 53. This option would be used to test a name server that has been configured to listen for queries on a non\-standard port number. +\-i +.RS 4 +Do reverse IPv6 lookups using the obsolete RFC1886 IP6.INT domain, which is no longer in use. Obsolete bit string label queries (RFC2874) are not attempted. +.RE .PP -The -\fB\-4\fR -option forces -\fBdig\fR -to only use IPv4 query transport. The -\fB\-6\fR -option forces -\fBdig\fR -to only use IPv6 query transport. +\-k \fIkeyfile\fR +.RS 4 +Sign queries using TSIG using a key read from the given file. Key files can be generated using +\fBtsig\-keygen\fR(8). When using TSIG authentication with +\fBdig\fR, the name server that is queried needs to know the key and algorithm that is being used. In BIND, this is done by providing appropriate +\fBkey\fR +and +\fBserver\fR +statements in +\fInamed.conf\fR. +.RE .PP -The -\fB\-t\fR -option sets the query type to -\fItype\fR. It can be any valid query type which is supported in BIND 9. The default query type is "A", unless the +\-m +.RS 4 +Enable memory usage debugging. +.RE +.PP +\-p \fIport\fR +.RS 4 +Send the query to a non\-standard port on the server, instead of the defaut port 53. This option would be used to test a name server that has been configured to listen for queries on a non\-standard port number. +.RE +.PP +\-q \fIname\fR +.RS 4 +The domain name to query. This is useful to distinguish the +\fIname\fR +from other arguments. +.RE +.PP +\-t \fItype\fR +.RS 4 +The resource record type to query. It can be any valid query type which is supported in BIND 9. The default query type is "A", unless the \fB\-x\fR -option is supplied to indicate a reverse lookup. A zone transfer can be requested by specifying a type of AXFR. When an incremental zone transfer (IXFR) is required, +option is supplied to indicate a reverse lookup. A zone transfer can be requested by specifying a type of AXFR. When an incremental zone transfer (IXFR) is required, set the \fItype\fR -is set to +to ixfr=N. The incremental zone transfer will contain the changes made to the zone since the serial number in the zone's SOA record was \fIN\fR. +.RE .PP -The -\fB\-q\fR -option sets the query name to -\fIname\fR. This is useful to distinguish the -\fIname\fR -from other arguments. -.PP -The -\fB\-v\fR -causes -\fBdig\fR -to print the version number and exit. +\-v +.RS 4 +Print the version number and exit. +.RE .PP -Reverse lookups \(em mapping addresses to names \(em are simplified by the -\fB\-x\fR -option. +\-x \fIaddr\fR +.RS 4 +Simplified reverse lookups, for mapping addresses to names. The \fIaddr\fR -is an IPv4 address in dotted\-decimal notation, or a colon\-delimited IPv6 address. When this option is used, there is no need to provide the +is an IPv4 address in dotted\-decimal notation, or a colon\-delimited IPv6 address. When the +\fB\-x\fR +is used, there is no need to provide the \fIname\fR, \fIclass\fR and @@ -208,35 +228,41 @@ and arguments. \fBdig\fR automatically performs a lookup for a name like -11.12.13.10.in\-addr.arpa -and sets the query type and class to PTR and IN respectively. By default, IPv6 addresses are looked up using nibble format under the IP6.ARPA domain. To use the older RFC1886 method using the IP6.INT domain specify the +94.2.0.192.in\-addr.arpa +and sets the query type and class to PTR and IN respectively. IPv6 addresses are looked up using nibble format under the IP6.ARPA domain (but see also the \fB\-i\fR -option. Bit string labels (RFC2874) are now experimental and are not attempted. +option). +.RE .PP -To sign the DNS queries sent by -\fBdig\fR -and their responses using transaction signatures (TSIG), specify a TSIG key file using the +\-y \fI[hmac:]\fR\fIkeyname:secret\fR +.RS 4 +Sign queries using TSIG with the given authentication key. +\fIkeyname\fR +is the name of the key, and +\fIsecret\fR +is the base64 encoded shared secret. +\fIhmac\fR +is the name of the key algorithm; valid choices are +hmac\-md5, +hmac\-sha1, +hmac\-sha224, +hmac\-sha256, +hmac\-sha384, or +hmac\-sha512. If +\fIhmac\fR +is not specified, the default is +hmac\-md5. +.sp +NOTE: You should use the \fB\-k\fR -option. You can also specify the TSIG key itself on the command line using the +option and avoid the \fB\-y\fR -option; -\fIhmac\fR -is the type of the TSIG, default HMAC\-MD5, -\fIname\fR -is the name of the TSIG key and -\fIkey\fR -is the actual key. The key is a base\-64 encoded string, typically generated by -\fBdnssec\-keygen\fR(8). Caution should be taken when using the +option, because with \fB\-y\fR -option on multi\-user systems as the key can be visible in the output from +the shared secret is supplied as a command line argument in clear text. This may be visible in the output from \fBps\fR(1) -or in the shell's history file. When using TSIG authentication with -\fBdig\fR, the name server that is queried needs to know the key and algorithm that is being used. In BIND, this is done by providing appropriate -\fBkey\fR -and -\fBserver\fR -statements in -\fInamed.conf\fR. +or in a history file maintained by the user's shell. +.RE .SH "QUERY OPTIONS" .PP \fBdig\fR @@ -245,7 +271,10 @@ provides a number of query options which Each query option is identified by a keyword preceded by a plus sign (+). Some keywords set or reset an option. These may be preceded by the string no to negate the meaning of that keyword. Other keywords assign values to options like the timeout interval. They have the form -\fB+keyword=value\fR. The query options are: +\fB+keyword=value\fR. Keywords may be abbreviated, provided the abbreviation is unambiguous; for example, ++cd +is equivalent to ++cdflag. The query options are: .PP \fB+[no]aaflag\fR .RS 4 @@ -300,7 +329,7 @@ bytes. The maximum and minimum sizes of Set [do not set] the CD (checking disabled) bit in the query. This requests the server to not perform DNSSEC validation of responses. .RE .PP -\fB+[no]cl\fR +\fB+[no]class\fR .RS 4 Display [do not display] the CLASS when printing the record. .RE @@ -421,6 +450,12 @@ Print [do not print] the query as it is Print [do not print] the question section of a query when an answer is returned. The default is to print the question section as a comment. .RE .PP +\fB+[no]rdflag\fR +.RS 4 +A synonym for +\fI+[no]recurse\fR. +.RE +.PP \fB+[no]recurse\fR .RS 4 Toggle the setting of the RD (recursion desired) bit in the query. This bit is set by default, which means @@ -518,6 +553,8 @@ Toggle tracing of the delegation path fr \fBdig\fR makes iterative queries to resolve the name being looked up. It will follow referrals from the root servers, showing the answer from each server that was used to resolve the lookup. .sp +If @server is also specified, it affects only the initial query for the root zone name servers. +.sp \fB+dnssec\fR is also set when +trace is set to better emulate the default queries from a nameserver. .RE @@ -620,7 +657,7 @@ RFC1035. .PP There are probably too many query options. .SH "COPYRIGHT" -Copyright \(co 2004\-2011, 2013, 2014 Internet Systems Consortium, Inc. ("ISC") +Copyright \(co 2004\-2011, 2013\-2015 Internet Systems Consortium, Inc. ("ISC") .br Copyright \(co 2000\-2003 Internet Software Consortium. .br Modified: stable/9/contrib/bind9/bin/dig/dig.c ============================================================================== --- stable/9/contrib/bind9/bin/dig/dig.c Sat Oct 3 09:22:21 2015 (r288576) +++ stable/9/contrib/bind9/bin/dig/dig.c Sat Oct 3 09:33:29 2015 (r288577) @@ -1,5 +1,5 @@ /* - * Copyright (C) 2004-2014 Internet Systems Consortium, Inc. ("ISC") + * Copyright (C) 2004-2015 Internet Systems Consortium, Inc. ("ISC") * Copyright (C) 2000-2003 Internet Software Consortium. * * Permission to use, copy, modify, and/or distribute this software for any @@ -165,71 +165,75 @@ help(void) { " q-type is one of (a,any,mx,ns,soa,hinfo,axfr,txt,...) [default:a]\n" " (Use ixfr=version for type ixfr)\n" " q-opt is one of:\n" -" -x dot-notation (shortcut for reverse lookups)\n" -" -i (use IP6.INT for IPv6 reverse lookups)\n" -" -f filename (batch mode)\n" +" -4 (use IPv4 query transport only)\n" +" -6 (use IPv6 query transport only)\n" " -b address[#port] (bind to source address/port)\n" +" -c class (specify query class)\n" +" -f filename (batch mode)\n" +" -i (use IP6.INT for IPv6 reverse lookups)\n" +" -k keyfile (specify tsig key file)\n" +" -m (enable memory usage debugging)\n" " -p port (specify port number)\n" " -q name (specify query name)\n" " -t type (specify query type)\n" -" -c class (specify query class)\n" -" -k keyfile (specify tsig key file)\n" +" -x dot-notation (shortcut for reverse lookups)\n" " -y [hmac:]name:key (specify named base64 tsig key)\n" -" -4 (use IPv4 query transport only)\n" -" -6 (use IPv6 query transport only)\n" -" -m (enable memory usage debugging)\n" " d-opt is of the form +keyword[=value], where keyword is:\n" -" +[no]vc (TCP mode)\n" -" +[no]tcp (TCP mode, alternate syntax)\n" -" +time=### (Set query timeout) [5]\n" -" +tries=### (Set number of UDP attempts) [3]\n" -" +retry=### (Set number of UDP retries) [2]\n" -" +domain=### (Set default domainname)\n" -" +bufsize=### (Set EDNS0 Max UDP packet size)\n" -" +ndots=### (Set NDOTS value)\n" -" +[no]edns[=###] (Set EDNS version) [0]\n" -" +[no]search (Set whether to use searchlist)\n" -" +[no]showsearch (Search with intermediate results)\n" -" +[no]defname (Ditto)\n" -" +[no]recurse (Recursive mode)\n" -" +[no]ignore (Don't revert to TCP for TC responses.)" -"\n" -" +[no]fail (Don't try next server on SERVFAIL)\n" -" +[no]besteffort (Try to parse even illegal messages)\n" " +[no]aaonly (Set AA flag in query (+[no]aaflag))\n" -" +[no]adflag (Set AD flag in query)\n" -" +[no]cdflag (Set CD flag in query)\n" +" +[no]additional (Control display of additional section)\n" +" +[no]adflag (Set AD flag in query (default on))\n" +" +[no]all (Set or clear all display flags)\n" +" +[no]answer (Control display of answer section)\n" +" +[no]authority (Control display of authority section)\n" +" +[no]besteffort (Try to parse even illegal messages)\n" +" +bufsize=### (Set EDNS0 Max UDP packet size)\n" +" +[no]cdflag (Set checking disabled flag in query)\n" " +[no]cl (Control display of class in records)\n" " +[no]cmd (Control display of command line)\n" " +[no]comments (Control display of comment lines)\n" +" +[no]defname (Use search list (+[no]search))\n" +" +[no]dnssec (Request DNSSEC records)\n" +" +domain=### (Set default domainname)\n" +" +[no]edns[=###] (Set EDNS version) [0]\n" +" +[no]fail (Don't try next server on SERVFAIL)\n" +" +[no]identify (ID responders in short answers)\n" +" +[no]ignore (Don't revert to TCP for TC responses.)" +"\n" +" +[no]keepopen (Keep the TCP socket open between queries)\n" +" +[no]multiline (Print records in an expanded format)\n" +" +ndots=### (Set search NDOTS value)\n" +" +[no]nsid (Request Name Server ID)\n" +" +[no]nssearch (Search all authoritative nameservers)\n" +" +[no]onesoa (AXFR prints only one soa record)\n" +" +[no]qr (Print question before sending)\n" +" +[no]question (Control display of question section)\n" +" +[no]recurse (Recursive mode)\n" +" +retry=### (Set number of UDP retries) [2]\n" " +[no]rrcomments (Control display of per-record " "comments)\n" -" +[no]question (Control display of question)\n" -" +[no]answer (Control display of answer)\n" -" +[no]authority (Control display of authority)\n" -" +[no]additional (Control display of additional)\n" -" +[no]stats (Control display of statistics)\n" -" +[no]short (Disable everything except short\n" +" +[no]search (Set whether to use searchlist)\n" +" +[no]short (Display nothing except short\n" " form of answer)\n" -" +[no]ttlid (Control display of ttls in records)\n" -" +[no]all (Set or clear all display flags)\n" -" +[no]qr (Print question before sending)\n" -" +[no]nssearch (Search all authoritative nameservers)\n" -" +[no]identify (ID responders in short answers)\n" -" +[no]trace (Trace delegation down from root [+dnssec])\n" -" +[no]dnssec (Request DNSSEC records)\n" -" +[no]nsid (Request Name Server ID)\n" +" +[no]showsearch (Search with intermediate results)\n" #ifdef DIG_SIGCHASE " +[no]sigchase (Chase DNSSEC signatures)\n" -" +trusted-key=#### (Trusted Key when chasing DNSSEC sigs)\n" +#endif +" +[no]split=## (Split hex/base64 fields into chunks)\n" +" +[no]stats (Control display of statistics)\n" +" +[no]tcp (TCP mode (+[no]vc))\n" +" +time=### (Set query timeout) [5]\n" +#ifdef DIG_SIGCHASE #if DIG_SIGCHASE_TD " +[no]topdown (Do DNSSEC validation top down mode)\n" #endif #endif -" +[no]split=## (Split hex/base64 fields into chunks)\n" -" +[no]multiline (Print records in an expanded format)\n" -" +[no]onesoa (AXFR prints only one soa record)\n" -" +[no]keepopen (Keep the TCP socket open between queries)\n" +" +[no]trace (Trace delegation down from root [+dnssec])\n" +" +tries=### (Set number of UDP attempts) [3]\n" +#ifdef DIG_SIGCHASE +" +trusted-key=#### (Trusted Key when chasing DNSSEC sigs)\n" +#endif +" +[no]ttlid (Control display of ttls in records)\n" +" +[no]vc (TCP mode (+[no]tcp))\n" " global d-opts and servers (before host name) affect all queries.\n" " local d-opts and servers (after host name) affect only that lookup.\n" " -h (print help and exit)\n" @@ -306,6 +310,7 @@ say_message(dns_rdata_t *rdata, dig_quer isc_result_t result; isc_uint64_t diff; char store[sizeof("12345678901234567890")]; + unsigned int styleflags = 0; if (query->lookup->trace || query->lookup->ns_search_only) { result = dns_rdatatype_totext(rdata->type, buf); @@ -313,7 +318,11 @@ say_message(dns_rdata_t *rdata, dig_quer return (result); ADD_STRING(buf, " "); } - result = dns_rdata_totext(rdata, NULL, buf); + + if (rrcomments) + styleflags |= DNS_STYLEFLAG_RRCOMMENT; + result = dns_rdata_tofmttext(rdata, NULL, styleflags, 0, + splitwidth, " ", buf); if (result == ISC_R_NOSPACE) return (result); check_result(result, "dns_rdata_totext"); @@ -831,8 +840,9 @@ plus_option(char *option, isc_boolean_t goto invalid_option; } break; - case 'l': /* cl */ - FULLCHECK("cl"); + case 'l': /* class */ + /* keep +cl for backwards compatibility */ + FULLCHECK2("cl", "class"); noclass = ISC_TF(!state); break; case 'm': /* cmd */ @@ -984,6 +994,10 @@ plus_option(char *option, isc_boolean_t break; case 'r': switch (cmd[1]) { + case 'd': /* rdflag */ + FULLCHECK("rdflag"); + lookup->recurse = state; + break; case 'e': switch (cmd[2]) { case 'c': /* recurse */ Modified: stable/9/contrib/bind9/bin/dig/dig.docbook ============================================================================== --- stable/9/contrib/bind9/bin/dig/dig.docbook Sat Oct 3 09:22:21 2015 (r288576) +++ stable/9/contrib/bind9/bin/dig/dig.docbook Sat Oct 3 09:33:29 2015 (r288577) @@ -2,7 +2,7 @@ "http://www.oasis-open.org/docbook/xml/4.2/docbookx.dtd" []> - - - - If a non-standard port number is to be queried, the - option is used. port# is - the port number that dig will send its - queries - instead of the standard DNS port number 53. This option would be used - to test a name server that has been configured to listen for queries - on a non-standard port number. - - - - The option forces dig - to only - use IPv4 query transport. The option forces - dig to only use IPv6 query transport. - - - - The option sets the query type to - type. It can be any valid query type - which is - supported in BIND 9. The default query type is "A", unless the - option is supplied to indicate a reverse lookup. - A zone transfer can be requested by specifying a type of AXFR. When - an incremental zone transfer (IXFR) is required, - type is set to ixfr=N. - The incremental zone transfer will contain the changes made to the zone *** DIFF OUTPUT TRUNCATED AT 1000 LINES *** From owner-svn-src-stable@freebsd.org Sat Oct 3 11:10:55 2015 Return-Path: Delivered-To: svn-src-stable@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 7E1E7A0E702; Sat, 3 Oct 2015 11:10:55 +0000 (UTC) (envelope-from mav@FreeBSD.org) Received: from repo.freebsd.org (repo.freebsd.org [IPv6:2001:1900:2254:2068::e6a:0]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 6DB1F18E8; Sat, 3 Oct 2015 11:10:55 +0000 (UTC) (envelope-from mav@FreeBSD.org) Received: from repo.freebsd.org ([127.0.1.70]) by repo.freebsd.org (8.15.2/8.15.2) with ESMTP id t93BAtDI060094; Sat, 3 Oct 2015 11:10:55 GMT (envelope-from mav@FreeBSD.org) Received: (from mav@localhost) by repo.freebsd.org (8.15.2/8.15.2/Submit) id t93BAtPw060093; Sat, 3 Oct 2015 11:10:55 GMT (envelope-from mav@FreeBSD.org) Message-Id: <201510031110.t93BAtPw060093@repo.freebsd.org> X-Authentication-Warning: repo.freebsd.org: mav set sender to mav@FreeBSD.org using -f From: Alexander Motin Date: Sat, 3 Oct 2015 11:10:55 +0000 (UTC) To: src-committers@freebsd.org, svn-src-all@freebsd.org, svn-src-stable@freebsd.org, svn-src-stable-10@freebsd.org Subject: svn commit: r288580 - stable/10/sys/cddl/contrib/opensolaris/uts/common/fs/zfs X-SVN-Group: stable-10 MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit X-BeenThere: svn-src-stable@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: SVN commit messages for all the -stable branches of the src tree List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 03 Oct 2015 11:10:55 -0000 Author: mav Date: Sat Oct 3 11:10:54 2015 New Revision: 288580 URL: https://svnweb.freebsd.org/changeset/base/288580 Log: MFC r286762: Revert part of r205231, introducing multiple ARC state locks. This local implementation will be replaced by one from Illumos to reduce code divergence and make further merges easier. Modified: stable/10/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/arc.c Directory Properties: stable/10/ (props changed) Modified: stable/10/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/arc.c ============================================================================== --- stable/10/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/arc.c Sat Oct 3 11:05:58 2015 (r288579) +++ stable/10/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/arc.c Sat Oct 3 11:10:54 2015 (r288580) @@ -201,6 +201,9 @@ extern int zfs_prefetch_disable; */ static boolean_t arc_warm; +/* + * These tunables are for performance analysis. + */ uint64_t zfs_arc_max; uint64_t zfs_arc_min; uint64_t zfs_arc_meta_limit = 0; @@ -315,31 +318,13 @@ SYSCTL_PROC(_vfs_zfs, OID_AUTO, arc_meta * second level ARC benefit from these fast lookups. */ -#define ARCS_LOCK_PAD CACHE_LINE_SIZE -struct arcs_lock { - kmutex_t arcs_lock; -#ifdef _KERNEL - unsigned char pad[(ARCS_LOCK_PAD - sizeof (kmutex_t))]; -#endif -}; - -/* - * must be power of two for mask use to work - * - */ -#define ARC_BUFC_NUMDATALISTS 16 -#define ARC_BUFC_NUMMETADATALISTS 16 -#define ARC_BUFC_NUMLISTS (ARC_BUFC_NUMMETADATALISTS + ARC_BUFC_NUMDATALISTS) - typedef struct arc_state { + list_t arcs_list[ARC_BUFC_NUMTYPES]; /* list of evictable buffers */ uint64_t arcs_lsize[ARC_BUFC_NUMTYPES]; /* amount of evictable data */ uint64_t arcs_size; /* total amount of data in this state */ - list_t arcs_lists[ARC_BUFC_NUMLISTS]; /* list of evictable buffers */ - struct arcs_lock arcs_locks[ARC_BUFC_NUMLISTS] __aligned(CACHE_LINE_SIZE); + kmutex_t arcs_mtx; } arc_state_t; -#define ARCS_LOCK(s, i) (&((s)->arcs_locks[(i)].arcs_lock)) - /* The 6 states: */ static arc_state_t ARC_anon; static arc_state_t ARC_mru; @@ -365,7 +350,6 @@ typedef struct arc_stats { kstat_named_t arcstat_mfu_ghost_hits; kstat_named_t arcstat_allocated; kstat_named_t arcstat_deleted; - kstat_named_t arcstat_stolen; kstat_named_t arcstat_recycle_miss; /* * Number of buffers that could not be evicted because the hash lock @@ -587,7 +571,6 @@ static arc_stats_t arc_stats = { { "mfu_ghost_hits", KSTAT_DATA_UINT64 }, { "allocated", KSTAT_DATA_UINT64 }, { "deleted", KSTAT_DATA_UINT64 }, - { "stolen", KSTAT_DATA_UINT64 }, { "recycle_miss", KSTAT_DATA_UINT64 }, { "mutex_miss", KSTAT_DATA_UINT64 }, { "evict_skip", KSTAT_DATA_UINT64 }, @@ -1684,23 +1667,6 @@ arc_buf_freeze(arc_buf_t *buf) } static void -get_buf_info(arc_buf_hdr_t *hdr, arc_state_t *state, list_t **list, kmutex_t **lock) -{ - uint64_t buf_hashid = buf_hash(hdr->b_spa, &hdr->b_dva, hdr->b_birth); - - if (arc_buf_type(hdr) == ARC_BUFC_METADATA) - buf_hashid &= (ARC_BUFC_NUMMETADATALISTS - 1); - else { - buf_hashid &= (ARC_BUFC_NUMDATALISTS - 1); - buf_hashid += ARC_BUFC_NUMMETADATALISTS; - } - - *list = &state->arcs_lists[buf_hashid]; - *lock = ARCS_LOCK(state, buf_hashid); -} - - -static void add_reference(arc_buf_hdr_t *hdr, kmutex_t *hash_lock, void *tag) { ASSERT(HDR_HAS_L1HDR(hdr)); @@ -1712,13 +1678,11 @@ add_reference(arc_buf_hdr_t *hdr, kmutex /* We don't use the L2-only state list. */ if (state != arc_l2c_only) { uint64_t delta = hdr->b_size * hdr->b_l1hdr.b_datacnt; + list_t *list = &state->arcs_list[arc_buf_type(hdr)]; uint64_t *size = &state->arcs_lsize[arc_buf_type(hdr)]; - list_t *list; - kmutex_t *lock; - get_buf_info(hdr, state, &list, &lock); - ASSERT(!MUTEX_HELD(lock)); - mutex_enter(lock); + ASSERT(!MUTEX_HELD(&state->arcs_mtx)); + mutex_enter(&state->arcs_mtx); ASSERT(list_link_active(&hdr->b_l1hdr.b_arc_node)); list_remove(list, hdr); if (GHOST_STATE(state)) { @@ -1729,7 +1693,7 @@ add_reference(arc_buf_hdr_t *hdr, kmutex ASSERT(delta > 0); ASSERT3U(*size, >=, delta); atomic_add_64(size, -delta); - mutex_exit(lock); + mutex_exit(&state->arcs_mtx); } /* remove the prefetch flag if we get a reference */ hdr->b_flags &= ~ARC_FLAG_PREFETCH; @@ -1753,18 +1717,15 @@ remove_reference(arc_buf_hdr_t *hdr, kmu if (((cnt = refcount_remove(&hdr->b_l1hdr.b_refcnt, tag)) == 0) && (state != arc_anon)) { uint64_t *size = &state->arcs_lsize[arc_buf_type(hdr)]; - list_t *list; - kmutex_t *lock; - get_buf_info(hdr, state, &list, &lock); - ASSERT(!MUTEX_HELD(lock)); - mutex_enter(lock); + ASSERT(!MUTEX_HELD(&state->arcs_mtx)); + mutex_enter(&state->arcs_mtx); ASSERT(!list_link_active(&hdr->b_l1hdr.b_arc_node)); - list_insert_head(list, hdr); + list_insert_head(&state->arcs_list[arc_buf_type(hdr)], hdr); ASSERT(hdr->b_l1hdr.b_datacnt > 0); atomic_add_64(size, hdr->b_size * hdr->b_l1hdr.b_datacnt); - mutex_exit(lock); + mutex_exit(&state->arcs_mtx); } return (cnt); } @@ -1782,8 +1743,6 @@ arc_change_state(arc_state_t *new_state, uint32_t datacnt; uint64_t from_delta, to_delta; arc_buf_contents_t buftype = arc_buf_type(hdr); - list_t *list; - kmutex_t *lock; /* * We almost always have an L1 hdr here, since we call arc_hdr_realloc() @@ -1816,17 +1775,15 @@ arc_change_state(arc_state_t *new_state, */ if (refcnt == 0) { if (old_state != arc_anon && old_state != arc_l2c_only) { - int use_mutex; + int use_mutex = !MUTEX_HELD(&old_state->arcs_mtx); uint64_t *size = &old_state->arcs_lsize[buftype]; - get_buf_info(hdr, old_state, &list, &lock); - use_mutex = !MUTEX_HELD(lock); if (use_mutex) - mutex_enter(lock); + mutex_enter(&old_state->arcs_mtx); ASSERT(HDR_HAS_L1HDR(hdr)); ASSERT(list_link_active(&hdr->b_l1hdr.b_arc_node)); - list_remove(list, hdr); + list_remove(&old_state->arcs_list[buftype], hdr); /* * If prefetching out of the ghost cache, @@ -1841,10 +1798,10 @@ arc_change_state(arc_state_t *new_state, atomic_add_64(size, -from_delta); if (use_mutex) - mutex_exit(lock); + mutex_exit(&old_state->arcs_mtx); } if (new_state != arc_anon && new_state != arc_l2c_only) { - int use_mutex; + int use_mutex = !MUTEX_HELD(&new_state->arcs_mtx); uint64_t *size = &new_state->arcs_lsize[buftype]; /* @@ -1854,23 +1811,21 @@ arc_change_state(arc_state_t *new_state, * beforehand. */ ASSERT(HDR_HAS_L1HDR(hdr)); - get_buf_info(hdr, new_state, &list, &lock); - use_mutex = !MUTEX_HELD(lock); if (use_mutex) - mutex_enter(lock); + mutex_enter(&new_state->arcs_mtx); - list_insert_head(list, hdr); + list_insert_head(&new_state->arcs_list[buftype], hdr); /* ghost elements have a ghost size */ if (GHOST_STATE(new_state)) { - ASSERT(datacnt == 0); + ASSERT0(datacnt); ASSERT(hdr->b_l1hdr.b_buf == NULL); to_delta = hdr->b_size; } atomic_add_64(size, to_delta); if (use_mutex) - mutex_exit(lock); + mutex_exit(&new_state->arcs_mtx); } } @@ -1892,10 +1847,8 @@ arc_change_state(arc_state_t *new_state, * L2 headers should never be on the L2 state list since they don't * have L1 headers allocated. */ -#ifdef illumos ASSERT(list_is_empty(&arc_l2c_only->arcs_list[ARC_BUFC_DATA]) && list_is_empty(&arc_l2c_only->arcs_list[ARC_BUFC_METADATA])); -#endif } void @@ -2535,41 +2488,55 @@ arc_evict(arc_state_t *state, uint64_t s { arc_state_t *evicted_state; uint64_t bytes_evicted = 0, skipped = 0, missed = 0; - int64_t bytes_remaining; arc_buf_hdr_t *hdr, *hdr_prev = NULL; - list_t *evicted_list, *list, *evicted_list_start, *list_start; - kmutex_t *lock, *evicted_lock; kmutex_t *hash_lock; boolean_t have_lock; void *stolen = NULL; arc_buf_hdr_t marker = { 0 }; int count = 0; - static int evict_metadata_offset, evict_data_offset; - int i, idx, offset, list_count, lists; ASSERT(state == arc_mru || state == arc_mfu); evicted_state = (state == arc_mru) ? arc_mru_ghost : arc_mfu_ghost; /* + * The ghost list lock must be acquired first in order to prevent + * a 3 party deadlock: + * + * - arc_evict_ghost acquires arc_*_ghost->arcs_mtx, followed by + * l2ad_mtx in arc_hdr_realloc + * - l2arc_write_buffers acquires l2ad_mtx, followed by arc_*->arcs_mtx + * - arc_evict acquires arc_*_ghost->arcs_mtx, followed by + * arc_*_ghost->arcs_mtx and forms a deadlock cycle. + * + * This situation is avoided by acquiring the ghost list lock first. + */ + mutex_enter(&evicted_state->arcs_mtx); + mutex_enter(&state->arcs_mtx); + + /* * Decide which "type" (data vs metadata) to recycle from. * * If we are over the metadata limit, recycle from metadata. * If we are under the metadata minimum, recycle from data. * Otherwise, recycle from whichever type has the oldest (least - * recently accessed) header. This is not yet implemented. + * recently accessed) header. */ if (recycle) { + arc_buf_hdr_t *data_hdr = + list_tail(&state->arcs_list[ARC_BUFC_DATA]); + arc_buf_hdr_t *metadata_hdr = + list_tail(&state->arcs_list[ARC_BUFC_METADATA]); arc_buf_contents_t realtype; - if (state->arcs_lsize[ARC_BUFC_DATA] == 0) { + + if (data_hdr == NULL) { realtype = ARC_BUFC_METADATA; - } else if (state->arcs_lsize[ARC_BUFC_METADATA] == 0) { + } else if (metadata_hdr == NULL) { realtype = ARC_BUFC_DATA; } else if (arc_meta_used >= arc_meta_limit) { realtype = ARC_BUFC_METADATA; } else if (arc_meta_used <= arc_meta_min) { realtype = ARC_BUFC_DATA; -#ifdef illumos } else if (HDR_HAS_L1HDR(data_hdr) && HDR_HAS_L1HDR(metadata_hdr) && data_hdr->b_l1hdr.b_arc_access < @@ -2577,11 +2544,6 @@ arc_evict(arc_state_t *state, uint64_t s realtype = ARC_BUFC_DATA; } else { realtype = ARC_BUFC_METADATA; -#else - } else { - /* TODO */ - realtype = type; -#endif } if (realtype != type) { /* @@ -2595,49 +2557,10 @@ arc_evict(arc_state_t *state, uint64_t s } } - if (type == ARC_BUFC_METADATA) { - offset = 0; - list_count = ARC_BUFC_NUMMETADATALISTS; - list_start = &state->arcs_lists[0]; - evicted_list_start = &evicted_state->arcs_lists[0]; - idx = evict_metadata_offset; - } else { - offset = ARC_BUFC_NUMMETADATALISTS; - list_start = &state->arcs_lists[offset]; - evicted_list_start = &evicted_state->arcs_lists[offset]; - list_count = ARC_BUFC_NUMDATALISTS; - idx = evict_data_offset; - } - bytes_remaining = evicted_state->arcs_lsize[type]; - lists = 0; - -evict_start: - list = &list_start[idx]; - evicted_list = &evicted_list_start[idx]; - lock = ARCS_LOCK(state, (offset + idx)); - evicted_lock = ARCS_LOCK(evicted_state, (offset + idx)); - - /* - * The ghost list lock must be acquired first in order to prevent - * a 3 party deadlock: - * - * - arc_evict_ghost acquires arc_*_ghost->arcs_mtx, followed by - * l2ad_mtx in arc_hdr_realloc - * - l2arc_write_buffers acquires l2ad_mtx, followed by arc_*->arcs_mtx - * - arc_evict acquires arc_*_ghost->arcs_mtx, followed by - * arc_*_ghost->arcs_mtx and forms a deadlock cycle. - * - * This situation is avoided by acquiring the ghost list lock first. - */ - mutex_enter(evicted_lock); - mutex_enter(lock); + list_t *list = &state->arcs_list[type]; for (hdr = list_tail(list); hdr; hdr = hdr_prev) { hdr_prev = list_prev(list, hdr); - if (HDR_HAS_L1HDR(hdr)) { - bytes_remaining -= - (hdr->b_size * hdr->b_l1hdr.b_datacnt); - } /* prefetch buffers have a minimum lifespan */ if (HDR_IO_IN_PROGRESS(hdr) || (spa && hdr->b_spa != spa) || @@ -2667,11 +2590,11 @@ evict_start: */ if (!recycle && count++ > arc_evict_iterations) { list_insert_after(list, hdr, &marker); - mutex_exit(lock); - mutex_exit(evicted_lock); + mutex_exit(&state->arcs_mtx); + mutex_exit(&evicted_state->arcs_mtx); kpreempt(KPREEMPT_SYNC); - mutex_enter(evicted_lock); - mutex_enter(lock); + mutex_enter(&evicted_state->arcs_mtx); + mutex_enter(&state->arcs_mtx); hdr_prev = list_prev(list, &marker); list_remove(list, &marker); count = 0; @@ -2741,35 +2664,17 @@ evict_start: mutex_exit(hash_lock); if (bytes >= 0 && bytes_evicted >= bytes) break; - if (bytes_remaining > 0) { - mutex_exit(evicted_lock); - mutex_exit(lock); - idx = ((idx + 1) & (list_count - 1)); - lists++; - goto evict_start; - } } else { missed += 1; } } - mutex_exit(lock); - mutex_exit(evicted_lock); + mutex_exit(&state->arcs_mtx); + mutex_exit(&evicted_state->arcs_mtx); - idx = ((idx + 1) & (list_count - 1)); - lists++; - - if (bytes_evicted < bytes) { - if (lists < list_count) - goto evict_start; - else - dprintf("only evicted %lld bytes from %x", - (longlong_t)bytes_evicted, state); - } - if (type == ARC_BUFC_METADATA) - evict_metadata_offset = idx; - else - evict_data_offset = idx; + if (bytes_evicted < bytes) + dprintf("only evicted %lld bytes from %x", + (longlong_t)bytes_evicted, state); if (skipped) ARCSTAT_INCR(arcstat_evict_skip, skipped); @@ -2784,8 +2689,6 @@ evict_start: * this chore to the arc_reclaim_thread(). */ - if (stolen) - ARCSTAT_BUMP(arcstat_stolen); return (stolen); } @@ -2798,29 +2701,15 @@ arc_evict_ghost(arc_state_t *state, uint { arc_buf_hdr_t *hdr, *hdr_prev; arc_buf_hdr_t marker = { 0 }; - list_t *list, *list_start; - kmutex_t *hash_lock, *lock; + list_t *list = &state->arcs_list[ARC_BUFC_DATA]; + kmutex_t *hash_lock; uint64_t bytes_deleted = 0; uint64_t bufs_skipped = 0; int count = 0; - static int evict_offset; - int list_count, idx = evict_offset; - int offset, lists = 0; ASSERT(GHOST_STATE(state)); - - /* - * data lists come after metadata lists - */ - list_start = &state->arcs_lists[ARC_BUFC_NUMMETADATALISTS]; - list_count = ARC_BUFC_NUMDATALISTS; - offset = ARC_BUFC_NUMMETADATALISTS; - -evict_start: - list = &list_start[idx]; - lock = ARCS_LOCK(state, idx + offset); - - mutex_enter(lock); +top: + mutex_enter(&state->arcs_mtx); for (hdr = list_tail(list); hdr; hdr = hdr_prev) { hdr_prev = list_prev(list, hdr); if (arc_buf_type(hdr) >= ARC_BUFC_NUMTYPES) @@ -2845,9 +2734,9 @@ evict_start: */ if (count++ > arc_evict_iterations) { list_insert_after(list, hdr, &marker); - mutex_exit(lock); + mutex_exit(&state->arcs_mtx); kpreempt(KPREEMPT_SYNC); - mutex_enter(lock); + mutex_enter(&state->arcs_mtx); hdr_prev = list_prev(list, &marker); list_remove(list, &marker); count = 0; @@ -2889,10 +2778,10 @@ evict_start: * available, restart from where we left off. */ list_insert_after(list, hdr, &marker); - mutex_exit(lock); + mutex_exit(&state->arcs_mtx); mutex_enter(hash_lock); mutex_exit(hash_lock); - mutex_enter(lock); + mutex_enter(&state->arcs_mtx); hdr_prev = list_prev(list, &marker); list_remove(list, &marker); } else { @@ -2900,20 +2789,12 @@ evict_start: } } - mutex_exit(lock); - idx = ((idx + 1) & (ARC_BUFC_NUMDATALISTS - 1)); - lists++; - - if (lists < list_count) - goto evict_start; + mutex_exit(&state->arcs_mtx); - evict_offset = idx; - if ((uintptr_t)list > (uintptr_t)&state->arcs_lists[ARC_BUFC_NUMMETADATALISTS] && + if (list == &state->arcs_list[ARC_BUFC_DATA] && (bytes < 0 || bytes_deleted < bytes)) { - list_start = &state->arcs_lists[0]; - list_count = ARC_BUFC_NUMMETADATALISTS; - offset = lists = 0; - goto evict_start; + list = &state->arcs_list[ARC_BUFC_METADATA]; + goto top; } if (bufs_skipped) { @@ -2993,23 +2874,14 @@ arc_adjust(void) static void arc_do_user_evicts(void) { - static arc_buf_t *tmp_arc_eviction_list; - - /* - * Move list over to avoid LOR - */ -restart: mutex_enter(&arc_eviction_mtx); - tmp_arc_eviction_list = arc_eviction_list; - arc_eviction_list = NULL; - mutex_exit(&arc_eviction_mtx); - - while (tmp_arc_eviction_list != NULL) { - arc_buf_t *buf = tmp_arc_eviction_list; - tmp_arc_eviction_list = buf->b_next; + while (arc_eviction_list != NULL) { + arc_buf_t *buf = arc_eviction_list; + arc_eviction_list = buf->b_next; mutex_enter(&buf->b_evict_lock); buf->b_hdr = NULL; mutex_exit(&buf->b_evict_lock); + mutex_exit(&arc_eviction_mtx); if (buf->b_efunc != NULL) VERIFY0(buf->b_efunc(buf->b_private)); @@ -3017,10 +2889,9 @@ restart: buf->b_efunc = NULL; buf->b_private = NULL; kmem_cache_free(buf_cache, buf); + mutex_enter(&arc_eviction_mtx); } - - if (arc_eviction_list != NULL) - goto restart; + mutex_exit(&arc_eviction_mtx); } /* @@ -3068,7 +2939,6 @@ arc_flush(spa_t *spa) void arc_shrink(int64_t to_free) { - if (arc_c > arc_c_min) { DTRACE_PROBE4(arc__shrink, uint64_t, arc_c, uint64_t, arc_c_min, uint64_t, arc_p, uint64_t, to_free); @@ -3906,7 +3776,7 @@ arc_read_done(zio_t *zio) } /* - * "Read" the block block at the specified DVA (in bp) via the + * "Read" the block at the specified DVA (in bp) via the * cache. If the block is found in the cache, invoke the provided * callback immediately and return. Note that the `zio' parameter * in the callback will be NULL in this case, since no IO was @@ -4300,8 +4170,6 @@ arc_clear_callback(arc_buf_t *buf) kmutex_t *hash_lock; arc_evict_func_t *efunc = buf->b_efunc; void *private = buf->b_private; - list_t *list, *evicted_list; - kmutex_t *lock, *evicted_lock; mutex_enter(&buf->b_evict_lock); hdr = buf->b_hdr; @@ -4956,39 +4824,43 @@ arc_init(void) arc_l2c_only = &ARC_l2c_only; arc_size = 0; - for (i = 0; i < ARC_BUFC_NUMLISTS; i++) { - mutex_init(&arc_anon->arcs_locks[i].arcs_lock, - NULL, MUTEX_DEFAULT, NULL); - mutex_init(&arc_mru->arcs_locks[i].arcs_lock, - NULL, MUTEX_DEFAULT, NULL); - mutex_init(&arc_mru_ghost->arcs_locks[i].arcs_lock, - NULL, MUTEX_DEFAULT, NULL); - mutex_init(&arc_mfu->arcs_locks[i].arcs_lock, - NULL, MUTEX_DEFAULT, NULL); - mutex_init(&arc_mfu_ghost->arcs_locks[i].arcs_lock, - NULL, MUTEX_DEFAULT, NULL); - mutex_init(&arc_l2c_only->arcs_locks[i].arcs_lock, - NULL, MUTEX_DEFAULT, NULL); - - list_create(&arc_mru->arcs_lists[i], - sizeof (arc_buf_hdr_t), - offsetof(arc_buf_hdr_t, b_l1hdr.b_arc_node)); - list_create(&arc_mru_ghost->arcs_lists[i], - sizeof (arc_buf_hdr_t), - offsetof(arc_buf_hdr_t, b_l1hdr.b_arc_node)); - list_create(&arc_mfu->arcs_lists[i], - sizeof (arc_buf_hdr_t), - offsetof(arc_buf_hdr_t, b_l1hdr.b_arc_node)); - list_create(&arc_mfu_ghost->arcs_lists[i], - sizeof (arc_buf_hdr_t), - offsetof(arc_buf_hdr_t, b_l1hdr.b_arc_node)); - list_create(&arc_mfu_ghost->arcs_lists[i], - sizeof (arc_buf_hdr_t), - offsetof(arc_buf_hdr_t, b_l1hdr.b_arc_node)); - list_create(&arc_l2c_only->arcs_lists[i], - sizeof (arc_buf_hdr_t), - offsetof(arc_buf_hdr_t, b_l1hdr.b_arc_node)); - } + mutex_init(&arc_anon->arcs_mtx, NULL, MUTEX_DEFAULT, NULL); + mutex_init(&arc_mru->arcs_mtx, NULL, MUTEX_DEFAULT, NULL); + mutex_init(&arc_mru_ghost->arcs_mtx, NULL, MUTEX_DEFAULT, NULL); + mutex_init(&arc_mfu->arcs_mtx, NULL, MUTEX_DEFAULT, NULL); + mutex_init(&arc_mfu_ghost->arcs_mtx, NULL, MUTEX_DEFAULT, NULL); + mutex_init(&arc_l2c_only->arcs_mtx, NULL, MUTEX_DEFAULT, NULL); + + list_create(&arc_mru->arcs_list[ARC_BUFC_METADATA], + sizeof (arc_buf_hdr_t), + offsetof(arc_buf_hdr_t, b_l1hdr.b_arc_node)); + list_create(&arc_mru->arcs_list[ARC_BUFC_DATA], + sizeof (arc_buf_hdr_t), + offsetof(arc_buf_hdr_t, b_l1hdr.b_arc_node)); + list_create(&arc_mru_ghost->arcs_list[ARC_BUFC_METADATA], + sizeof (arc_buf_hdr_t), + offsetof(arc_buf_hdr_t, b_l1hdr.b_arc_node)); + list_create(&arc_mru_ghost->arcs_list[ARC_BUFC_DATA], + sizeof (arc_buf_hdr_t), + offsetof(arc_buf_hdr_t, b_l1hdr.b_arc_node)); + list_create(&arc_mfu->arcs_list[ARC_BUFC_METADATA], + sizeof (arc_buf_hdr_t), + offsetof(arc_buf_hdr_t, b_l1hdr.b_arc_node)); + list_create(&arc_mfu->arcs_list[ARC_BUFC_DATA], + sizeof (arc_buf_hdr_t), + offsetof(arc_buf_hdr_t, b_l1hdr.b_arc_node)); + list_create(&arc_mfu_ghost->arcs_list[ARC_BUFC_METADATA], + sizeof (arc_buf_hdr_t), + offsetof(arc_buf_hdr_t, b_l1hdr.b_arc_node)); + list_create(&arc_mfu_ghost->arcs_list[ARC_BUFC_DATA], + sizeof (arc_buf_hdr_t), + offsetof(arc_buf_hdr_t, b_l1hdr.b_arc_node)); + list_create(&arc_l2c_only->arcs_list[ARC_BUFC_METADATA], + sizeof (arc_buf_hdr_t), + offsetof(arc_buf_hdr_t, b_l1hdr.b_arc_node)); + list_create(&arc_l2c_only->arcs_list[ARC_BUFC_DATA], + sizeof (arc_buf_hdr_t), + offsetof(arc_buf_hdr_t, b_l1hdr.b_arc_node)); buf_init(); @@ -5072,8 +4944,6 @@ arc_init(void) void arc_fini(void) { - int i; - mutex_enter(&arc_reclaim_thr_lock); arc_thread_exit = 1; cv_signal(&arc_reclaim_thr_cv); @@ -5094,20 +4964,21 @@ arc_fini(void) mutex_destroy(&arc_reclaim_thr_lock); cv_destroy(&arc_reclaim_thr_cv); - for (i = 0; i < ARC_BUFC_NUMLISTS; i++) { - list_destroy(&arc_mru->arcs_lists[i]); - list_destroy(&arc_mru_ghost->arcs_lists[i]); - list_destroy(&arc_mfu->arcs_lists[i]); - list_destroy(&arc_mfu_ghost->arcs_lists[i]); - list_destroy(&arc_l2c_only->arcs_lists[i]); - - mutex_destroy(&arc_anon->arcs_locks[i].arcs_lock); - mutex_destroy(&arc_mru->arcs_locks[i].arcs_lock); - mutex_destroy(&arc_mru_ghost->arcs_locks[i].arcs_lock); - mutex_destroy(&arc_mfu->arcs_locks[i].arcs_lock); - mutex_destroy(&arc_mfu_ghost->arcs_locks[i].arcs_lock); - mutex_destroy(&arc_l2c_only->arcs_locks[i].arcs_lock); - } + list_destroy(&arc_mru->arcs_list[ARC_BUFC_METADATA]); + list_destroy(&arc_mru_ghost->arcs_list[ARC_BUFC_METADATA]); + list_destroy(&arc_mfu->arcs_list[ARC_BUFC_METADATA]); + list_destroy(&arc_mfu_ghost->arcs_list[ARC_BUFC_METADATA]); + list_destroy(&arc_mru->arcs_list[ARC_BUFC_DATA]); + list_destroy(&arc_mru_ghost->arcs_list[ARC_BUFC_DATA]); + list_destroy(&arc_mfu->arcs_list[ARC_BUFC_DATA]); + list_destroy(&arc_mfu_ghost->arcs_list[ARC_BUFC_DATA]); + + mutex_destroy(&arc_anon->arcs_mtx); + mutex_destroy(&arc_mru->arcs_mtx); + mutex_destroy(&arc_mru_ghost->arcs_mtx); + mutex_destroy(&arc_mfu->arcs_mtx); + mutex_destroy(&arc_mfu_ghost->arcs_mtx); + mutex_destroy(&arc_l2c_only->arcs_mtx); buf_fini(); @@ -5611,27 +5482,26 @@ static list_t * l2arc_list_locked(int list_num, kmutex_t **lock) { list_t *list = NULL; - int idx; - ASSERT(list_num >= 0 && list_num < 2 * ARC_BUFC_NUMLISTS); + ASSERT(list_num >= 0 && list_num <= 3); - if (list_num < ARC_BUFC_NUMMETADATALISTS) { - idx = list_num; - list = &arc_mfu->arcs_lists[idx]; - *lock = ARCS_LOCK(arc_mfu, idx); - } else if (list_num < ARC_BUFC_NUMMETADATALISTS * 2) { - idx = list_num - ARC_BUFC_NUMMETADATALISTS; - list = &arc_mru->arcs_lists[idx]; - *lock = ARCS_LOCK(arc_mru, idx); - } else if (list_num < (ARC_BUFC_NUMMETADATALISTS * 2 + - ARC_BUFC_NUMDATALISTS)) { - idx = list_num - ARC_BUFC_NUMMETADATALISTS; - list = &arc_mfu->arcs_lists[idx]; - *lock = ARCS_LOCK(arc_mfu, idx); - } else { - idx = list_num - ARC_BUFC_NUMLISTS; - list = &arc_mru->arcs_lists[idx]; - *lock = ARCS_LOCK(arc_mru, idx); + switch (list_num) { + case 0: + list = &arc_mfu->arcs_list[ARC_BUFC_METADATA]; + *lock = &arc_mfu->arcs_mtx; + break; + case 1: + list = &arc_mru->arcs_list[ARC_BUFC_METADATA]; + *lock = &arc_mru->arcs_mtx; + break; + case 2: + list = &arc_mfu->arcs_list[ARC_BUFC_DATA]; + *lock = &arc_mfu->arcs_mtx; + break; + case 3: + list = &arc_mru->arcs_list[ARC_BUFC_DATA]; + *lock = &arc_mru->arcs_mtx; + break; } ASSERT(!(MUTEX_HELD(*lock))); @@ -5793,7 +5663,7 @@ l2arc_write_buffers(spa_t *spa, l2arc_de * Copy buffers for L2ARC writing. */ mutex_enter(&dev->l2ad_mtx); - for (try = 0; try < 2 * ARC_BUFC_NUMLISTS; try++) { + for (try = 0; try <= 3; try++) { uint64_t passed_sz = 0; list = l2arc_list_locked(try, &list_lock); @@ -5812,7 +5682,7 @@ l2arc_write_buffers(spa_t *spa, l2arc_de if (hdr == NULL) ARCSTAT_BUMP(arcstat_l2_write_buffer_list_null_iter); - headroom = target_sz * l2arc_headroom * 2 / ARC_BUFC_NUMLISTS; + headroom = target_sz * l2arc_headroom; if (do_headroom_boost) headroom = (headroom * l2arc_headroom_boost) / 100; From owner-svn-src-stable@freebsd.org Sat Oct 3 11:11:58 2015 Return-Path: Delivered-To: svn-src-stable@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id EA52FA0E9EC; Sat, 3 Oct 2015 11:11:57 +0000 (UTC) (envelope-from mav@FreeBSD.org) Received: from repo.freebsd.org (repo.freebsd.org [IPv6:2001:1900:2254:2068::e6a:0]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id D18771CB5; Sat, 3 Oct 2015 11:11:57 +0000 (UTC) (envelope-from mav@FreeBSD.org) Received: from repo.freebsd.org ([127.0.1.70]) by repo.freebsd.org (8.15.2/8.15.2) with ESMTP id t93BBvr7061227; Sat, 3 Oct 2015 11:11:57 GMT (envelope-from mav@FreeBSD.org) Received: (from mav@localhost) by repo.freebsd.org (8.15.2/8.15.2/Submit) id t93BBvIv061224; Sat, 3 Oct 2015 11:11:57 GMT (envelope-from mav@FreeBSD.org) Message-Id: <201510031111.t93BBvIv061224@repo.freebsd.org> X-Authentication-Warning: repo.freebsd.org: mav set sender to mav@FreeBSD.org using -f From: Alexander Motin Date: Sat, 3 Oct 2015 11:11:57 +0000 (UTC) To: src-committers@freebsd.org, svn-src-all@freebsd.org, svn-src-stable@freebsd.org, svn-src-stable-10@freebsd.org Subject: svn commit: r288581 - in stable/10/sys: cddl/contrib/opensolaris/uts/common cddl/contrib/opensolaris/uts/common/fs/zfs cddl/contrib/opensolaris/uts/common/fs/zfs/sys conf X-SVN-Group: stable-10 MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit X-BeenThere: svn-src-stable@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: SVN commit messages for all the -stable branches of the src tree List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 03 Oct 2015 11:11:58 -0000 Author: mav Date: Sat Oct 3 11:11:56 2015 New Revision: 288581 URL: https://svnweb.freebsd.org/changeset/base/288581 Log: MFC r286763: 5497 lock contention on arcs_mtx Reviewed by: George Wilson Reviewed by: Matthew Ahrens Reviewed by: Richard Elling Approved by: Dan McDonald Author: Prakash Surya illumos/illumos-gate@244781f10dcd82684fd8163c016540667842f203 This patch attempts to reduce lock contention on the current arc_state_t mutexes. These mutexes are used liberally to protect the number of LRU lists within the ARC (e.g. ARC_mru, ARC_mfu, etc). The granularity at which these locks are acquired has been shown to greatly affect the performance of highly concurrent, cached workloads. Added: stable/10/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/multilist.c - copied unchanged from r286763, head/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/multilist.c stable/10/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/sys/multilist.h - copied unchanged from r286763, head/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/sys/multilist.h Modified: stable/10/sys/cddl/contrib/opensolaris/uts/common/Makefile.files stable/10/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/arc.c stable/10/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/dsl_pool.c stable/10/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/sys/arc.h stable/10/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio_inject.c stable/10/sys/conf/files Directory Properties: stable/10/ (props changed) Modified: stable/10/sys/cddl/contrib/opensolaris/uts/common/Makefile.files ============================================================================== --- stable/10/sys/cddl/contrib/opensolaris/uts/common/Makefile.files Sat Oct 3 11:10:54 2015 (r288580) +++ stable/10/sys/cddl/contrib/opensolaris/uts/common/Makefile.files Sat Oct 3 11:11:56 2015 (r288581) @@ -68,6 +68,7 @@ ZFS_COMMON_OBJS += \ lz4.o \ lzjb.o \ metaslab.o \ + multilist.o \ range_tree.o \ refcount.o \ rrwlock.o \ Modified: stable/10/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/arc.c ============================================================================== --- stable/10/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/arc.c Sat Oct 3 11:10:54 2015 (r288580) +++ stable/10/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/arc.c Sat Oct 3 11:11:56 2015 (r288581) @@ -129,6 +129,7 @@ #include #include #include +#include #ifdef _KERNEL #include #endif @@ -149,21 +150,39 @@ int arc_procfd; #endif #endif /* illumos */ -static kmutex_t arc_reclaim_thr_lock; -static kcondvar_t arc_reclaim_thr_cv; /* used to signal reclaim thr */ -static uint8_t arc_thread_exit; +static kmutex_t arc_reclaim_lock; +static kcondvar_t arc_reclaim_thread_cv; +static boolean_t arc_reclaim_thread_exit; +static kcondvar_t arc_reclaim_waiters_cv; + +static kmutex_t arc_user_evicts_lock; +static kcondvar_t arc_user_evicts_cv; +static boolean_t arc_user_evicts_thread_exit; uint_t arc_reduce_dnlc_percent = 3; /* - * The number of iterations through arc_evict_*() before we - * drop & reacquire the lock. + * The number of headers to evict in arc_evict_state_impl() before + * dropping the sublist lock and evicting from another sublist. A lower + * value means we're more likely to evict the "correct" header (i.e. the + * oldest header in the arc state), but comes with higher overhead + * (i.e. more invocations of arc_evict_state_impl()). */ -int arc_evict_iterations = 100; +int zfs_arc_evict_batch_limit = 10; + +/* + * The number of sublists used for each of the arc state lists. If this + * is not set to a suitable value by the user, it will be configured to + * the number of CPUs on the system in arc_init(). + */ +int zfs_arc_num_sublists_per_state = 0; /* number of seconds before growing cache again */ static int arc_grow_retry = 60; +/* shift of arc_c for calculating overflow limit in arc_get_data_buf */ +int zfs_arc_overflow_shift = 8; + /* shift of arc_c for calculating both min and max arc_p */ static int arc_p_min_shift = 4; @@ -319,10 +338,19 @@ SYSCTL_PROC(_vfs_zfs, OID_AUTO, arc_meta */ typedef struct arc_state { - list_t arcs_list[ARC_BUFC_NUMTYPES]; /* list of evictable buffers */ - uint64_t arcs_lsize[ARC_BUFC_NUMTYPES]; /* amount of evictable data */ - uint64_t arcs_size; /* total amount of data in this state */ - kmutex_t arcs_mtx; + /* + * list of evictable buffers + */ + multilist_t arcs_list[ARC_BUFC_NUMTYPES]; + /* + * total amount of evictable data in this state + */ + uint64_t arcs_lsize[ARC_BUFC_NUMTYPES]; + /* + * total amount of data in this state; this includes: evictable, + * non-evictable, ARC_BUFC_DATA, and ARC_BUFC_METADATA. + */ + uint64_t arcs_size; } arc_state_t; /* The 6 states: */ @@ -350,7 +378,6 @@ typedef struct arc_stats { kstat_named_t arcstat_mfu_ghost_hits; kstat_named_t arcstat_allocated; kstat_named_t arcstat_deleted; - kstat_named_t arcstat_recycle_miss; /* * Number of buffers that could not be evicted because the hash lock * was held by another thread. The lock may not necessarily be held @@ -364,9 +391,15 @@ typedef struct arc_stats { * not from the spa we're trying to evict from. */ kstat_named_t arcstat_evict_skip; + /* + * Number of times arc_evict_state() was unable to evict enough + * buffers to reach it's target amount. + */ + kstat_named_t arcstat_evict_not_enough; kstat_named_t arcstat_evict_l2_cached; kstat_named_t arcstat_evict_l2_eligible; kstat_named_t arcstat_evict_l2_ineligible; + kstat_named_t arcstat_evict_l2_skip; kstat_named_t arcstat_hash_elements; kstat_named_t arcstat_hash_elements_max; kstat_named_t arcstat_hash_collisions; @@ -517,7 +550,7 @@ typedef struct arc_stats { kstat_named_t arcstat_l2_writes_sent; kstat_named_t arcstat_l2_writes_done; kstat_named_t arcstat_l2_writes_error; - kstat_named_t arcstat_l2_writes_hdr_miss; + kstat_named_t arcstat_l2_writes_lock_retry; kstat_named_t arcstat_l2_evict_lock_retry; kstat_named_t arcstat_l2_evict_reading; kstat_named_t arcstat_l2_evict_l1cached; @@ -571,12 +604,13 @@ static arc_stats_t arc_stats = { { "mfu_ghost_hits", KSTAT_DATA_UINT64 }, { "allocated", KSTAT_DATA_UINT64 }, { "deleted", KSTAT_DATA_UINT64 }, - { "recycle_miss", KSTAT_DATA_UINT64 }, { "mutex_miss", KSTAT_DATA_UINT64 }, { "evict_skip", KSTAT_DATA_UINT64 }, + { "evict_not_enough", KSTAT_DATA_UINT64 }, { "evict_l2_cached", KSTAT_DATA_UINT64 }, { "evict_l2_eligible", KSTAT_DATA_UINT64 }, { "evict_l2_ineligible", KSTAT_DATA_UINT64 }, + { "evict_l2_skip", KSTAT_DATA_UINT64 }, { "hash_elements", KSTAT_DATA_UINT64 }, { "hash_elements_max", KSTAT_DATA_UINT64 }, { "hash_collisions", KSTAT_DATA_UINT64 }, @@ -615,7 +649,7 @@ static arc_stats_t arc_stats = { { "l2_writes_sent", KSTAT_DATA_UINT64 }, { "l2_writes_done", KSTAT_DATA_UINT64 }, { "l2_writes_error", KSTAT_DATA_UINT64 }, - { "l2_writes_hdr_miss", KSTAT_DATA_UINT64 }, + { "l2_writes_lock_retry", KSTAT_DATA_UINT64 }, { "l2_evict_lock_retry", KSTAT_DATA_UINT64 }, { "l2_evict_reading", KSTAT_DATA_UINT64 }, { "l2_evict_l1cached", KSTAT_DATA_UINT64 }, @@ -792,7 +826,7 @@ typedef struct l1arc_buf_hdr { /* protected by arc state mutex */ arc_state_t *b_state; - list_node_t b_arc_node; + multilist_node_t b_arc_node; /* updated atomically */ clock_t b_arc_access; @@ -863,7 +897,6 @@ sysctl_vfs_zfs_arc_meta_limit(SYSCTL_HAN #endif static arc_buf_t *arc_eviction_list; -static kmutex_t arc_eviction_mtx; static arc_buf_hdr_t arc_eviction_hdr; #define GHOST_STATE(state) \ @@ -1092,8 +1125,7 @@ static uint8_t l2arc_thread_exit; static void arc_get_data_buf(arc_buf_t *); static void arc_access(arc_buf_hdr_t *, kmutex_t *); -static int arc_evict_needed(arc_buf_contents_t); -static void arc_evict_ghost(arc_state_t *, uint64_t, int64_t); +static boolean_t arc_is_overflowing(); static void arc_buf_watch(arc_buf_t *); static arc_buf_contents_t arc_buf_type(arc_buf_hdr_t *); @@ -1274,6 +1306,7 @@ hdr_full_cons(void *vbuf, void *unused, cv_init(&hdr->b_l1hdr.b_cv, NULL, CV_DEFAULT, NULL); refcount_create(&hdr->b_l1hdr.b_refcnt); mutex_init(&hdr->b_l1hdr.b_freeze_lock, NULL, MUTEX_DEFAULT, NULL); + multilist_link_init(&hdr->b_l1hdr.b_arc_node); arc_space_consume(HDR_FULL_SIZE, ARC_SPACE_HDRS); return (0); @@ -1318,6 +1351,7 @@ hdr_full_dest(void *vbuf, void *unused) cv_destroy(&hdr->b_l1hdr.b_cv); refcount_destroy(&hdr->b_l1hdr.b_refcnt); mutex_destroy(&hdr->b_l1hdr.b_freeze_lock); + ASSERT(!multilist_link_active(&hdr->b_l1hdr.b_arc_node)); arc_space_return(HDR_FULL_SIZE, ARC_SPACE_HDRS); } @@ -1354,7 +1388,7 @@ hdr_recl(void *unused) * which is after we do arc_fini(). */ if (!arc_dead) - cv_signal(&arc_reclaim_thr_cv); + cv_signal(&arc_reclaim_thread_cv); } static void @@ -1433,18 +1467,31 @@ arc_hdr_realloc(arc_buf_hdr_t *hdr, kmem * l2c_only even though it's about to change. */ nhdr->b_l1hdr.b_state = arc_l2c_only; + + /* Verify previous threads set to NULL before freeing */ + ASSERT3P(nhdr->b_l1hdr.b_tmp_cdata, ==, NULL); } else { ASSERT(hdr->b_l1hdr.b_buf == NULL); ASSERT0(hdr->b_l1hdr.b_datacnt); - ASSERT(!list_link_active(&hdr->b_l1hdr.b_arc_node)); + /* - * We might be removing the L1hdr of a buffer which was just - * written out to L2ARC. If such a buffer is compressed then we - * need to free its b_tmp_cdata before destroying the header. - */ - if (hdr->b_l1hdr.b_tmp_cdata != NULL && - HDR_GET_COMPRESS(hdr) != ZIO_COMPRESS_OFF) - l2arc_release_cdata_buf(hdr); + * If we've reached here, We must have been called from + * arc_evict_hdr(), as such we should have already been + * removed from any ghost list we were previously on + * (which protects us from racing with arc_evict_state), + * thus no locking is needed during this check. + */ + ASSERT(!multilist_link_active(&hdr->b_l1hdr.b_arc_node)); + + /* + * A buffer must not be moved into the arc_l2c_only + * state if it's not finished being written out to the + * l2arc device. Otherwise, the b_l1hdr.b_tmp_cdata field + * might try to be accessed, even though it was removed. + */ + VERIFY(!HDR_L2_WRITING(hdr)); + VERIFY3P(hdr->b_l1hdr.b_tmp_cdata, ==, NULL); + nhdr->b_flags &= ~ARC_FLAG_HAS_L1HDR; } /* @@ -1677,14 +1724,13 @@ add_reference(arc_buf_hdr_t *hdr, kmutex (state != arc_anon)) { /* We don't use the L2-only state list. */ if (state != arc_l2c_only) { + arc_buf_contents_t type = arc_buf_type(hdr); uint64_t delta = hdr->b_size * hdr->b_l1hdr.b_datacnt; - list_t *list = &state->arcs_list[arc_buf_type(hdr)]; - uint64_t *size = &state->arcs_lsize[arc_buf_type(hdr)]; + multilist_t *list = &state->arcs_list[type]; + uint64_t *size = &state->arcs_lsize[type]; + + multilist_remove(list, hdr); - ASSERT(!MUTEX_HELD(&state->arcs_mtx)); - mutex_enter(&state->arcs_mtx); - ASSERT(list_link_active(&hdr->b_l1hdr.b_arc_node)); - list_remove(list, hdr); if (GHOST_STATE(state)) { ASSERT0(hdr->b_l1hdr.b_datacnt); ASSERT3P(hdr->b_l1hdr.b_buf, ==, NULL); @@ -1693,7 +1739,6 @@ add_reference(arc_buf_hdr_t *hdr, kmutex ASSERT(delta > 0); ASSERT3U(*size, >=, delta); atomic_add_64(size, -delta); - mutex_exit(&state->arcs_mtx); } /* remove the prefetch flag if we get a reference */ hdr->b_flags &= ~ARC_FLAG_PREFETCH; @@ -1716,22 +1761,21 @@ remove_reference(arc_buf_hdr_t *hdr, kmu */ if (((cnt = refcount_remove(&hdr->b_l1hdr.b_refcnt, tag)) == 0) && (state != arc_anon)) { - uint64_t *size = &state->arcs_lsize[arc_buf_type(hdr)]; + arc_buf_contents_t type = arc_buf_type(hdr); + multilist_t *list = &state->arcs_list[type]; + uint64_t *size = &state->arcs_lsize[type]; + + multilist_insert(list, hdr); - ASSERT(!MUTEX_HELD(&state->arcs_mtx)); - mutex_enter(&state->arcs_mtx); - ASSERT(!list_link_active(&hdr->b_l1hdr.b_arc_node)); - list_insert_head(&state->arcs_list[arc_buf_type(hdr)], hdr); ASSERT(hdr->b_l1hdr.b_datacnt > 0); atomic_add_64(size, hdr->b_size * hdr->b_l1hdr.b_datacnt); - mutex_exit(&state->arcs_mtx); } return (cnt); } /* - * Move the supplied buffer to the indicated state. The mutex + * Move the supplied buffer to the indicated state. The hash lock * for the buffer must be held by the caller. */ static void @@ -1775,15 +1819,10 @@ arc_change_state(arc_state_t *new_state, */ if (refcnt == 0) { if (old_state != arc_anon && old_state != arc_l2c_only) { - int use_mutex = !MUTEX_HELD(&old_state->arcs_mtx); uint64_t *size = &old_state->arcs_lsize[buftype]; - if (use_mutex) - mutex_enter(&old_state->arcs_mtx); - ASSERT(HDR_HAS_L1HDR(hdr)); - ASSERT(list_link_active(&hdr->b_l1hdr.b_arc_node)); - list_remove(&old_state->arcs_list[buftype], hdr); + multilist_remove(&old_state->arcs_list[buftype], hdr); /* * If prefetching out of the ghost cache, @@ -1796,12 +1835,8 @@ arc_change_state(arc_state_t *new_state, } ASSERT3U(*size, >=, from_delta); atomic_add_64(size, -from_delta); - - if (use_mutex) - mutex_exit(&old_state->arcs_mtx); } if (new_state != arc_anon && new_state != arc_l2c_only) { - int use_mutex = !MUTEX_HELD(&new_state->arcs_mtx); uint64_t *size = &new_state->arcs_lsize[buftype]; /* @@ -1811,10 +1846,7 @@ arc_change_state(arc_state_t *new_state, * beforehand. */ ASSERT(HDR_HAS_L1HDR(hdr)); - if (use_mutex) - mutex_enter(&new_state->arcs_mtx); - - list_insert_head(&new_state->arcs_list[buftype], hdr); + multilist_insert(&new_state->arcs_list[buftype], hdr); /* ghost elements have a ghost size */ if (GHOST_STATE(new_state)) { @@ -1823,9 +1855,6 @@ arc_change_state(arc_state_t *new_state, to_delta = hdr->b_size; } atomic_add_64(size, to_delta); - - if (use_mutex) - mutex_exit(&new_state->arcs_mtx); } } @@ -1847,8 +1876,8 @@ arc_change_state(arc_state_t *new_state, * L2 headers should never be on the L2 state list since they don't * have L1 headers allocated. */ - ASSERT(list_is_empty(&arc_l2c_only->arcs_list[ARC_BUFC_DATA]) && - list_is_empty(&arc_l2c_only->arcs_list[ARC_BUFC_METADATA])); + ASSERT(multilist_is_empty(&arc_l2c_only->arcs_list[ARC_BUFC_DATA]) && + multilist_is_empty(&arc_l2c_only->arcs_list[ARC_BUFC_METADATA])); } void @@ -1941,6 +1970,7 @@ arc_buf_alloc(spa_t *spa, int32_t size, hdr->b_l1hdr.b_state = arc_anon; hdr->b_l1hdr.b_arc_access = 0; hdr->b_l1hdr.b_datacnt = 1; + hdr->b_l1hdr.b_tmp_cdata = NULL; arc_get_data_buf(buf); ASSERT(refcount_is_zero(&hdr->b_l1hdr.b_refcnt)); @@ -2076,7 +2106,7 @@ arc_buf_free_on_write(void *data, size_t { l2arc_data_free_t *df; - df = kmem_alloc(sizeof (l2arc_data_free_t), KM_SLEEP); + df = kmem_alloc(sizeof (*df), KM_SLEEP); df->l2df_data = data; df->l2df_size = size; df->l2df_func = free_func; @@ -2120,19 +2150,49 @@ arc_buf_l2_cdata_free(arc_buf_hdr_t *hdr if (!HDR_HAS_L1HDR(hdr)) return; - if (hdr->b_l1hdr.b_tmp_cdata == NULL) + /* + * The header isn't being written to the l2arc device, thus it + * shouldn't have a b_tmp_cdata to free. + */ + if (!HDR_L2_WRITING(hdr)) { + ASSERT3P(hdr->b_l1hdr.b_tmp_cdata, ==, NULL); + return; + } + + /* + * The header does not have compression enabled. This can be due + * to the buffer not being compressible, or because we're + * freeing the buffer before the second phase of + * l2arc_write_buffer() has started (which does the compression + * step). In either case, b_tmp_cdata does not point to a + * separately compressed buffer, so there's nothing to free (it + * points to the same buffer as the arc_buf_t's b_data field). + */ + if (HDR_GET_COMPRESS(hdr) == ZIO_COMPRESS_OFF) { + hdr->b_l1hdr.b_tmp_cdata = NULL; + return; + } + + /* + * There's nothing to free since the buffer was all zero's and + * compressed to a zero length buffer. + */ + if (HDR_GET_COMPRESS(hdr) == ZIO_COMPRESS_EMPTY) { + ASSERT3P(hdr->b_l1hdr.b_tmp_cdata, ==, NULL); return; + } + + ASSERT(L2ARC_IS_VALID_COMPRESS(HDR_GET_COMPRESS(hdr))); - ASSERT(HDR_L2_WRITING(hdr)); - arc_buf_free_on_write(hdr->b_l1hdr.b_tmp_cdata, hdr->b_size, - zio_data_buf_free); + arc_buf_free_on_write(hdr->b_l1hdr.b_tmp_cdata, + hdr->b_size, zio_data_buf_free); ARCSTAT_BUMP(arcstat_l2_cdata_free_on_write); hdr->b_l1hdr.b_tmp_cdata = NULL; } static void -arc_buf_destroy(arc_buf_t *buf, boolean_t recycle, boolean_t remove) +arc_buf_destroy(arc_buf_t *buf, boolean_t remove) { arc_buf_t **bufp; @@ -2147,17 +2207,17 @@ arc_buf_destroy(arc_buf_t *buf, boolean_ arc_buf_unwatch(buf); #endif /* illumos */ - if (!recycle) { - if (type == ARC_BUFC_METADATA) { - arc_buf_data_free(buf, zio_buf_free); - arc_space_return(size, ARC_SPACE_META); - } else { - ASSERT(type == ARC_BUFC_DATA); - arc_buf_data_free(buf, zio_data_buf_free); - arc_space_return(size, ARC_SPACE_DATA); - } + if (type == ARC_BUFC_METADATA) { + arc_buf_data_free(buf, zio_buf_free); + arc_space_return(size, ARC_SPACE_META); + } else { + ASSERT(type == ARC_BUFC_DATA); + arc_buf_data_free(buf, zio_data_buf_free); + arc_space_return(size, ARC_SPACE_DATA); } - if (list_link_active(&buf->b_hdr->b_l1hdr.b_arc_node)) { + + /* protected by hash lock, if in the hash table */ + if (multilist_link_active(&buf->b_hdr->b_l1hdr.b_arc_node)) { uint64_t *cnt = &state->arcs_lsize[type]; ASSERT(refcount_is_zero( @@ -2305,20 +2365,19 @@ arc_hdr_destroy(arc_buf_hdr_t *hdr) arc_buf_t *buf = hdr->b_l1hdr.b_buf; if (buf->b_efunc != NULL) { - mutex_enter(&arc_eviction_mtx); + mutex_enter(&arc_user_evicts_lock); mutex_enter(&buf->b_evict_lock); ASSERT(buf->b_hdr != NULL); - arc_buf_destroy(hdr->b_l1hdr.b_buf, FALSE, - FALSE); + arc_buf_destroy(hdr->b_l1hdr.b_buf, FALSE); hdr->b_l1hdr.b_buf = buf->b_next; buf->b_hdr = &arc_eviction_hdr; buf->b_next = arc_eviction_list; arc_eviction_list = buf; mutex_exit(&buf->b_evict_lock); - mutex_exit(&arc_eviction_mtx); + cv_signal(&arc_user_evicts_cv); + mutex_exit(&arc_user_evicts_lock); } else { - arc_buf_destroy(hdr->b_l1hdr.b_buf, FALSE, - TRUE); + arc_buf_destroy(hdr->b_l1hdr.b_buf, TRUE); } } #ifdef ZFS_DEBUG @@ -2331,7 +2390,7 @@ arc_hdr_destroy(arc_buf_hdr_t *hdr) ASSERT3P(hdr->b_hash_next, ==, NULL); if (HDR_HAS_L1HDR(hdr)) { - ASSERT(!list_link_active(&hdr->b_l1hdr.b_arc_node)); + ASSERT(!multilist_link_active(&hdr->b_l1hdr.b_arc_node)); ASSERT3P(hdr->b_l1hdr.b_acb, ==, NULL); kmem_cache_free(hdr_full_cache, hdr); } else { @@ -2357,7 +2416,7 @@ arc_buf_free(arc_buf_t *buf, void *tag) (void) remove_reference(hdr, hash_lock, tag); if (hdr->b_l1hdr.b_datacnt > 1) { - arc_buf_destroy(buf, FALSE, TRUE); + arc_buf_destroy(buf, TRUE); } else { ASSERT(buf == hdr->b_l1hdr.b_buf); ASSERT(buf->b_efunc == NULL); @@ -2371,16 +2430,16 @@ arc_buf_free(arc_buf_t *buf, void *tag) * this buffer unless the write completes before we finish * decrementing the reference count. */ - mutex_enter(&arc_eviction_mtx); + mutex_enter(&arc_user_evicts_lock); (void) remove_reference(hdr, NULL, tag); ASSERT(refcount_is_zero(&hdr->b_l1hdr.b_refcnt)); destroy_hdr = !HDR_IO_IN_PROGRESS(hdr); - mutex_exit(&arc_eviction_mtx); + mutex_exit(&arc_user_evicts_lock); if (destroy_hdr) arc_hdr_destroy(hdr); } else { if (remove_reference(hdr, NULL, tag) > 0) - arc_buf_destroy(buf, FALSE, TRUE); + arc_buf_destroy(buf, TRUE); else arc_hdr_destroy(hdr); } @@ -2409,7 +2468,7 @@ arc_buf_remove_ref(arc_buf_t *buf, void* (void) remove_reference(hdr, hash_lock, tag); if (hdr->b_l1hdr.b_datacnt > 1) { if (no_callback) - arc_buf_destroy(buf, FALSE, TRUE); + arc_buf_destroy(buf, TRUE); } else if (no_callback) { ASSERT(hdr->b_l1hdr.b_buf == buf && buf->b_next == NULL); ASSERT(buf->b_efunc == NULL); @@ -2470,418 +2529,675 @@ arc_buf_eviction_needed(arc_buf_t *buf) } /* - * Evict buffers from list until we've removed the specified number of - * bytes. Move the removed buffers to the appropriate evict state. - * If the recycle flag is set, then attempt to "recycle" a buffer: - * - look for a buffer to evict that is `bytes' long. - * - return the data block from this buffer rather than freeing it. - * This flag is used by callers that are trying to make space for a - * new buffer in a full arc cache. + * Evict the arc_buf_hdr that is provided as a parameter. The resultant + * state of the header is dependent on it's state prior to entering this + * function. The following transitions are possible: * - * This function makes a "best effort". It skips over any buffers - * it can't get a hash_lock on, and so may not catch all candidates. - * It may also return without evicting as much space as requested. + * - arc_mru -> arc_mru_ghost + * - arc_mfu -> arc_mfu_ghost + * - arc_mru_ghost -> arc_l2c_only + * - arc_mru_ghost -> deleted + * - arc_mfu_ghost -> arc_l2c_only + * - arc_mfu_ghost -> deleted */ -static void * -arc_evict(arc_state_t *state, uint64_t spa, int64_t bytes, boolean_t recycle, - arc_buf_contents_t type) +static int64_t +arc_evict_hdr(arc_buf_hdr_t *hdr, kmutex_t *hash_lock) { - arc_state_t *evicted_state; - uint64_t bytes_evicted = 0, skipped = 0, missed = 0; - arc_buf_hdr_t *hdr, *hdr_prev = NULL; - kmutex_t *hash_lock; - boolean_t have_lock; - void *stolen = NULL; - arc_buf_hdr_t marker = { 0 }; - int count = 0; + arc_state_t *evicted_state, *state; + int64_t bytes_evicted = 0; - ASSERT(state == arc_mru || state == arc_mfu); - - evicted_state = (state == arc_mru) ? arc_mru_ghost : arc_mfu_ghost; + ASSERT(MUTEX_HELD(hash_lock)); + ASSERT(HDR_HAS_L1HDR(hdr)); - /* - * The ghost list lock must be acquired first in order to prevent - * a 3 party deadlock: - * - * - arc_evict_ghost acquires arc_*_ghost->arcs_mtx, followed by - * l2ad_mtx in arc_hdr_realloc - * - l2arc_write_buffers acquires l2ad_mtx, followed by arc_*->arcs_mtx - * - arc_evict acquires arc_*_ghost->arcs_mtx, followed by - * arc_*_ghost->arcs_mtx and forms a deadlock cycle. - * - * This situation is avoided by acquiring the ghost list lock first. - */ - mutex_enter(&evicted_state->arcs_mtx); - mutex_enter(&state->arcs_mtx); + state = hdr->b_l1hdr.b_state; + if (GHOST_STATE(state)) { + ASSERT(!HDR_IO_IN_PROGRESS(hdr)); + ASSERT(hdr->b_l1hdr.b_buf == NULL); - /* - * Decide which "type" (data vs metadata) to recycle from. - * - * If we are over the metadata limit, recycle from metadata. - * If we are under the metadata minimum, recycle from data. - * Otherwise, recycle from whichever type has the oldest (least - * recently accessed) header. - */ - if (recycle) { - arc_buf_hdr_t *data_hdr = - list_tail(&state->arcs_list[ARC_BUFC_DATA]); - arc_buf_hdr_t *metadata_hdr = - list_tail(&state->arcs_list[ARC_BUFC_METADATA]); - arc_buf_contents_t realtype; - - if (data_hdr == NULL) { - realtype = ARC_BUFC_METADATA; - } else if (metadata_hdr == NULL) { - realtype = ARC_BUFC_DATA; - } else if (arc_meta_used >= arc_meta_limit) { - realtype = ARC_BUFC_METADATA; - } else if (arc_meta_used <= arc_meta_min) { - realtype = ARC_BUFC_DATA; - } else if (HDR_HAS_L1HDR(data_hdr) && - HDR_HAS_L1HDR(metadata_hdr) && - data_hdr->b_l1hdr.b_arc_access < - metadata_hdr->b_l1hdr.b_arc_access) { - realtype = ARC_BUFC_DATA; - } else { - realtype = ARC_BUFC_METADATA; + /* + * l2arc_write_buffers() relies on a header's L1 portion + * (i.e. it's b_tmp_cdata field) during it's write phase. + * Thus, we cannot push a header onto the arc_l2c_only + * state (removing it's L1 piece) until the header is + * done being written to the l2arc. + */ + if (HDR_HAS_L2HDR(hdr) && HDR_L2_WRITING(hdr)) { + ARCSTAT_BUMP(arcstat_evict_l2_skip); + return (bytes_evicted); } - if (realtype != type) { + + ARCSTAT_BUMP(arcstat_deleted); + bytes_evicted += hdr->b_size; + + DTRACE_PROBE1(arc__delete, arc_buf_hdr_t *, hdr); + + if (HDR_HAS_L2HDR(hdr)) { /* - * If we want to evict from a different list, - * we can not recycle, because DATA vs METADATA - * buffers are segregated into different kmem - * caches (and vmem arenas). + * This buffer is cached on the 2nd Level ARC; + * don't destroy the header. */ - type = realtype; - recycle = B_FALSE; + arc_change_state(arc_l2c_only, hdr, hash_lock); + /* + * dropping from L1+L2 cached to L2-only, + * realloc to remove the L1 header. + */ + hdr = arc_hdr_realloc(hdr, hdr_full_cache, + hdr_l2only_cache); + } else { + arc_change_state(arc_anon, hdr, hash_lock); + arc_hdr_destroy(hdr); } + return (bytes_evicted); } - list_t *list = &state->arcs_list[type]; + ASSERT(state == arc_mru || state == arc_mfu); + evicted_state = (state == arc_mru) ? arc_mru_ghost : arc_mfu_ghost; - for (hdr = list_tail(list); hdr; hdr = hdr_prev) { - hdr_prev = list_prev(list, hdr); - /* prefetch buffers have a minimum lifespan */ - if (HDR_IO_IN_PROGRESS(hdr) || - (spa && hdr->b_spa != spa) || - ((hdr->b_flags & (ARC_FLAG_PREFETCH | ARC_FLAG_INDIRECT)) && - ddi_get_lbolt() - hdr->b_l1hdr.b_arc_access < - arc_min_prefetch_lifespan)) { - skipped++; - continue; + /* prefetch buffers have a minimum lifespan */ + if (HDR_IO_IN_PROGRESS(hdr) || + ((hdr->b_flags & (ARC_FLAG_PREFETCH | ARC_FLAG_INDIRECT)) && + ddi_get_lbolt() - hdr->b_l1hdr.b_arc_access < + arc_min_prefetch_lifespan)) { + ARCSTAT_BUMP(arcstat_evict_skip); + return (bytes_evicted); + } + + ASSERT0(refcount_count(&hdr->b_l1hdr.b_refcnt)); + ASSERT3U(hdr->b_l1hdr.b_datacnt, >, 0); + while (hdr->b_l1hdr.b_buf) { + arc_buf_t *buf = hdr->b_l1hdr.b_buf; + if (!mutex_tryenter(&buf->b_evict_lock)) { + ARCSTAT_BUMP(arcstat_mutex_miss); + break; } - /* "lookahead" for better eviction candidate */ - if (recycle && hdr->b_size != bytes && - hdr_prev && hdr_prev->b_size == bytes) - continue; + if (buf->b_data != NULL) + bytes_evicted += hdr->b_size; + if (buf->b_efunc != NULL) { + mutex_enter(&arc_user_evicts_lock); + arc_buf_destroy(buf, FALSE); + hdr->b_l1hdr.b_buf = buf->b_next; + buf->b_hdr = &arc_eviction_hdr; + buf->b_next = arc_eviction_list; + arc_eviction_list = buf; + cv_signal(&arc_user_evicts_cv); + mutex_exit(&arc_user_evicts_lock); + mutex_exit(&buf->b_evict_lock); + } else { + mutex_exit(&buf->b_evict_lock); + arc_buf_destroy(buf, TRUE); + } + } - /* ignore markers */ - if (hdr->b_spa == 0) - continue; + if (HDR_HAS_L2HDR(hdr)) { + ARCSTAT_INCR(arcstat_evict_l2_cached, hdr->b_size); + } else { + if (l2arc_write_eligible(hdr->b_spa, hdr)) + ARCSTAT_INCR(arcstat_evict_l2_eligible, hdr->b_size); + else + ARCSTAT_INCR(arcstat_evict_l2_ineligible, hdr->b_size); + } + + if (hdr->b_l1hdr.b_datacnt == 0) { + arc_change_state(evicted_state, hdr, hash_lock); + ASSERT(HDR_IN_HASH_TABLE(hdr)); + hdr->b_flags |= ARC_FLAG_IN_HASH_TABLE; + hdr->b_flags &= ~ARC_FLAG_BUF_AVAILABLE; + DTRACE_PROBE1(arc__evict, arc_buf_hdr_t *, hdr); + } + + return (bytes_evicted); +} + +static uint64_t +arc_evict_state_impl(multilist_t *ml, int idx, arc_buf_hdr_t *marker, + uint64_t spa, int64_t bytes) +{ + multilist_sublist_t *mls; + uint64_t bytes_evicted = 0; + arc_buf_hdr_t *hdr; + kmutex_t *hash_lock; + int evict_count = 0; + + ASSERT3P(marker, !=, NULL); + IMPLY(bytes < 0, bytes == ARC_EVICT_ALL); + + mls = multilist_sublist_lock(ml, idx); + + for (hdr = multilist_sublist_prev(mls, marker); hdr != NULL; + hdr = multilist_sublist_prev(mls, marker)) { + if ((bytes != ARC_EVICT_ALL && bytes_evicted >= bytes) || + (evict_count >= zfs_arc_evict_batch_limit)) + break; /* - * It may take a long time to evict all the bufs requested. - * To avoid blocking all arc activity, periodically drop - * the arcs_mtx and give other threads a chance to run - * before reacquiring the lock. - * - * If we are looking for a buffer to recycle, we are in - * the hot code path, so don't sleep. + * To keep our iteration location, move the marker + * forward. Since we're not holding hdr's hash lock, we + * must be very careful and not remove 'hdr' from the + * sublist. Otherwise, other consumers might mistake the + * 'hdr' as not being on a sublist when they call the + * multilist_link_active() function (they all rely on + * the hash lock protecting concurrent insertions and + * removals). multilist_sublist_move_forward() was + * specifically implemented to ensure this is the case + * (only 'marker' will be removed and re-inserted). + */ + multilist_sublist_move_forward(mls, marker); + + /* + * The only case where the b_spa field should ever be + * zero, is the marker headers inserted by + * arc_evict_state(). It's possible for multiple threads + * to be calling arc_evict_state() concurrently (e.g. + * dsl_pool_close() and zio_inject_fault()), so we must + * skip any markers we see from these other threads. */ - if (!recycle && count++ > arc_evict_iterations) { - list_insert_after(list, hdr, &marker); - mutex_exit(&state->arcs_mtx); - mutex_exit(&evicted_state->arcs_mtx); - kpreempt(KPREEMPT_SYNC); - mutex_enter(&evicted_state->arcs_mtx); - mutex_enter(&state->arcs_mtx); - hdr_prev = list_prev(list, &marker); - list_remove(list, &marker); - count = 0; + if (hdr->b_spa == 0) + continue; + + /* we're only interested in evicting buffers of a certain spa */ + if (spa != 0 && hdr->b_spa != spa) { + ARCSTAT_BUMP(arcstat_evict_skip); continue; } hash_lock = HDR_LOCK(hdr); - have_lock = MUTEX_HELD(hash_lock); - if (have_lock || mutex_tryenter(hash_lock)) { - ASSERT0(refcount_count(&hdr->b_l1hdr.b_refcnt)); - ASSERT3U(hdr->b_l1hdr.b_datacnt, >, 0); - while (hdr->b_l1hdr.b_buf) { - arc_buf_t *buf = hdr->b_l1hdr.b_buf; - if (!mutex_tryenter(&buf->b_evict_lock)) { - missed += 1; - break; - } - if (buf->b_data != NULL) { - bytes_evicted += hdr->b_size; - if (recycle && - arc_buf_type(hdr) == type && - hdr->b_size == bytes && - !HDR_L2_WRITING(hdr)) { - stolen = buf->b_data; - recycle = FALSE; - } - } - if (buf->b_efunc != NULL) { - mutex_enter(&arc_eviction_mtx); - arc_buf_destroy(buf, - buf->b_data == stolen, FALSE); - hdr->b_l1hdr.b_buf = buf->b_next; - buf->b_hdr = &arc_eviction_hdr; - buf->b_next = arc_eviction_list; - arc_eviction_list = buf; - mutex_exit(&arc_eviction_mtx); - mutex_exit(&buf->b_evict_lock); - } else { - mutex_exit(&buf->b_evict_lock); - arc_buf_destroy(buf, - buf->b_data == stolen, TRUE); - } - } - if (HDR_HAS_L2HDR(hdr)) { - ARCSTAT_INCR(arcstat_evict_l2_cached, - hdr->b_size); - } else { - if (l2arc_write_eligible(hdr->b_spa, hdr)) { - ARCSTAT_INCR(arcstat_evict_l2_eligible, - hdr->b_size); - } else { - ARCSTAT_INCR( - arcstat_evict_l2_ineligible, - hdr->b_size); - } - } + /* + * We aren't calling this function from any code path + * that would already be holding a hash lock, so we're + * asserting on this assumption to be defensive in case + * this ever changes. Without this check, it would be + * possible to incorrectly increment arcstat_mutex_miss + * below (e.g. if the code changed such that we called + * this function with a hash lock held). + */ + ASSERT(!MUTEX_HELD(hash_lock)); - if (hdr->b_l1hdr.b_datacnt == 0) { - arc_change_state(evicted_state, hdr, hash_lock); - ASSERT(HDR_IN_HASH_TABLE(hdr)); - hdr->b_flags |= ARC_FLAG_IN_HASH_TABLE; - hdr->b_flags &= ~ARC_FLAG_BUF_AVAILABLE; - DTRACE_PROBE1(arc__evict, arc_buf_hdr_t *, hdr); - } - if (!have_lock) - mutex_exit(hash_lock); - if (bytes >= 0 && bytes_evicted >= bytes) - break; + if (mutex_tryenter(hash_lock)) { + uint64_t evicted = arc_evict_hdr(hdr, hash_lock); + mutex_exit(hash_lock); + + bytes_evicted += evicted; + + /* + * If evicted is zero, arc_evict_hdr() must have + * decided to skip this header, don't increment + * evict_count in this case. + */ + if (evicted != 0) + evict_count++; + + /* + * If arc_size isn't overflowing, signal any + * threads that might happen to be waiting. + * + * For each header evicted, we wake up a single + * thread. If we used cv_broadcast, we could + * wake up "too many" threads causing arc_size + * to significantly overflow arc_c; since + * arc_get_data_buf() doesn't check for overflow + * when it's woken up (it doesn't because it's + * possible for the ARC to be overflowing while + * full of un-evictable buffers, and the + * function should proceed in this case). + * + * If threads are left sleeping, due to not + * using cv_broadcast, they will be woken up + * just before arc_reclaim_thread() sleeps. + */ + mutex_enter(&arc_reclaim_lock); + if (!arc_is_overflowing()) + cv_signal(&arc_reclaim_waiters_cv); + mutex_exit(&arc_reclaim_lock); } else { - missed += 1; + ARCSTAT_BUMP(arcstat_mutex_miss); } } - mutex_exit(&state->arcs_mtx); - mutex_exit(&evicted_state->arcs_mtx); + multilist_sublist_unlock(mls); - if (bytes_evicted < bytes) - dprintf("only evicted %lld bytes from %x", - (longlong_t)bytes_evicted, state); + return (bytes_evicted); +} + +/* + * Evict buffers from the given arc state, until we've removed the + * specified number of bytes. Move the removed buffers to the + * appropriate evict state. + * + * This function makes a "best effort". It skips over any buffers + * it can't get a hash_lock on, and so, may not catch all candidates. + * It may also return without evicting as much space as requested. + * + * If bytes is specified using the special value ARC_EVICT_ALL, this + * will evict all available (i.e. unlocked and evictable) buffers from + * the given arc state; which is used by arc_flush(). + */ +static uint64_t +arc_evict_state(arc_state_t *state, uint64_t spa, int64_t bytes, + arc_buf_contents_t type) +{ + uint64_t total_evicted = 0; + multilist_t *ml = &state->arcs_list[type]; + int num_sublists; + arc_buf_hdr_t **markers; - if (skipped) - ARCSTAT_INCR(arcstat_evict_skip, skipped); + IMPLY(bytes < 0, bytes == ARC_EVICT_ALL); - if (missed) - ARCSTAT_INCR(arcstat_mutex_miss, missed); + num_sublists = multilist_get_num_sublists(ml); /* - * Note: we have just evicted some data into the ghost state, - * potentially putting the ghost size over the desired size. Rather - * that evicting from the ghost list in this hot code path, leave - * this chore to the arc_reclaim_thread(). + * If we've tried to evict from each sublist, made some + * progress, but still have not hit the target number of bytes + * to evict, we want to keep trying. The markers allow us to + * pick up where we left off for each individual sublist, rather + * than starting from the tail each time. */ + markers = kmem_zalloc(sizeof (*markers) * num_sublists, KM_SLEEP); + for (int i = 0; i < num_sublists; i++) { + markers[i] = kmem_cache_alloc(hdr_full_cache, KM_SLEEP); - return (stolen); -} + /* + * A b_spa of 0 is used to indicate that this header is + * a marker. This fact is used in arc_adjust_type() and + * arc_evict_state_impl(). + */ + markers[i]->b_spa = 0; -/* - * Remove buffers from list until we've removed the specified number of - * bytes. Destroy the buffers that are removed. - */ *** DIFF OUTPUT TRUNCATED AT 1000 LINES *** From owner-svn-src-stable@freebsd.org Sat Oct 3 11:14:06 2015 Return-Path: Delivered-To: svn-src-stable@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 051CCA0EBF5; Sat, 3 Oct 2015 11:14:06 +0000 (UTC) (envelope-from mav@FreeBSD.org) Received: from repo.freebsd.org (repo.freebsd.org [IPv6:2001:1900:2254:2068::e6a:0]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id DDC241FBF; Sat, 3 Oct 2015 11:14:05 +0000 (UTC) (envelope-from mav@FreeBSD.org) Received: from repo.freebsd.org ([127.0.1.70]) by repo.freebsd.org (8.15.2/8.15.2) with ESMTP id t93BE5eP061425; Sat, 3 Oct 2015 11:14:05 GMT (envelope-from mav@FreeBSD.org) Received: (from mav@localhost) by repo.freebsd.org (8.15.2/8.15.2/Submit) id t93BE5cf061424; Sat, 3 Oct 2015 11:14:05 GMT (envelope-from mav@FreeBSD.org) Message-Id: <201510031114.t93BE5cf061424@repo.freebsd.org> X-Authentication-Warning: repo.freebsd.org: mav set sender to mav@FreeBSD.org using -f From: Alexander Motin Date: Sat, 3 Oct 2015 11:14:05 +0000 (UTC) To: src-committers@freebsd.org, svn-src-all@freebsd.org, svn-src-stable@freebsd.org, svn-src-stable-10@freebsd.org Subject: svn commit: r288583 - stable/10/sys/cddl/contrib/opensolaris/uts/common/fs/zfs X-SVN-Group: stable-10 MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit X-BeenThere: svn-src-stable@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: SVN commit messages for all the -stable branches of the src tree List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 03 Oct 2015 11:14:06 -0000 Author: mav Date: Sat Oct 3 11:14:05 2015 New Revision: 288583 URL: https://svnweb.freebsd.org/changeset/base/288583 Log: MFC r286766: 5817 change type of arcs_size from uint64_t to refcount_t Reviewed by: Matthew Ahrens Reviewed by: Paul Dagnelie Reviewed by: Adam Leventhal Reviewed by: Alex Reece Reviewed by: Richard Elling Approved by: Garrett D'Amore Author: Prakash Surya illumos/illumos-gate@2fd872a734cf486007a8dba532cec52bfb4d40e5 As a way to make it more difficult to introduce bugs into the ARC, and to make it easier to diagnose issues when bugs do creep in, it would be beneficial to change the type of the arc_state_t's arcs_size field to be a refcount_t instead of a uint64_t. This would allow us to make stricter checks when incrementing and decrementing the value with debugging enabled, but still fallback to simple, fast atomic operations when debugging is disabled. Modified: stable/10/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/arc.c Directory Properties: stable/10/ (props changed) Modified: stable/10/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/arc.c ============================================================================== --- stable/10/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/arc.c Sat Oct 3 11:13:01 2015 (r288582) +++ stable/10/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/arc.c Sat Oct 3 11:14:05 2015 (r288583) @@ -21,7 +21,7 @@ /* * Copyright (c) 2005, 2010, Oracle and/or its affiliates. All rights reserved. * Copyright (c) 2012, Joyent, Inc. All rights reserved. - * Copyright (c) 2011, 2014 by Delphix. All rights reserved. + * Copyright (c) 2011, 2015 by Delphix. All rights reserved. * Copyright (c) 2014 by Saso Kiselkov. All rights reserved. * Copyright 2015 Nexenta Systems, Inc. All rights reserved. */ @@ -350,7 +350,7 @@ typedef struct arc_state { * total amount of data in this state; this includes: evictable, * non-evictable, ARC_BUFC_DATA, and ARC_BUFC_METADATA. */ - uint64_t arcs_size; + refcount_t arcs_size; } arc_state_t; /* The 6 states: */ @@ -1863,12 +1863,73 @@ arc_change_state(arc_state_t *new_state, buf_hash_remove(hdr); /* adjust state sizes (ignore arc_l2c_only) */ - if (to_delta && new_state != arc_l2c_only) - atomic_add_64(&new_state->arcs_size, to_delta); + + if (to_delta && new_state != arc_l2c_only) { + ASSERT(HDR_HAS_L1HDR(hdr)); + if (GHOST_STATE(new_state)) { + ASSERT0(datacnt); + + /* + * We moving a header to a ghost state, we first + * remove all arc buffers. Thus, we'll have a + * datacnt of zero, and no arc buffer to use for + * the reference. As a result, we use the arc + * header pointer for the reference. + */ + (void) refcount_add_many(&new_state->arcs_size, + hdr->b_size, hdr); + } else { + ASSERT3U(datacnt, !=, 0); + + /* + * Each individual buffer holds a unique reference, + * thus we must remove each of these references one + * at a time. + */ + for (arc_buf_t *buf = hdr->b_l1hdr.b_buf; buf != NULL; + buf = buf->b_next) { + (void) refcount_add_many(&new_state->arcs_size, + hdr->b_size, buf); + } + } + } + if (from_delta && old_state != arc_l2c_only) { - ASSERT3U(old_state->arcs_size, >=, from_delta); - atomic_add_64(&old_state->arcs_size, -from_delta); + ASSERT(HDR_HAS_L1HDR(hdr)); + if (GHOST_STATE(old_state)) { + /* + * When moving a header off of a ghost state, + * there's the possibility for datacnt to be + * non-zero. This is because we first add the + * arc buffer to the header prior to changing + * the header's state. Since we used the header + * for the reference when putting the header on + * the ghost state, we must balance that and use + * the header when removing off the ghost state + * (even though datacnt is non zero). + */ + + IMPLY(datacnt == 0, new_state == arc_anon || + new_state == arc_l2c_only); + + (void) refcount_remove_many(&old_state->arcs_size, + hdr->b_size, hdr); + } else { + ASSERT3P(datacnt, !=, 0); + + /* + * Each individual buffer holds a unique reference, + * thus we must remove each of these references one + * at a time. + */ + for (arc_buf_t *buf = hdr->b_l1hdr.b_buf; buf != NULL; + buf = buf->b_next) { + (void) refcount_remove_many( + &old_state->arcs_size, hdr->b_size, buf); + } + } } + if (HDR_HAS_L1HDR(hdr)) hdr->b_l1hdr.b_state = new_state; @@ -2227,8 +2288,8 @@ arc_buf_destroy(arc_buf_t *buf, boolean_ ASSERT3U(*cnt, >=, size); atomic_add_64(cnt, -size); } - ASSERT3U(state->arcs_size, >=, size); - atomic_add_64(&state->arcs_size, -size); + + (void) refcount_remove_many(&state->arcs_size, size, buf); buf->b_data = NULL; /* @@ -2952,7 +3013,8 @@ arc_adjust_meta(void) * evict some from the MRU here, and some from the MFU below. */ target = MIN((int64_t)(arc_meta_used - arc_meta_limit), - (int64_t)(arc_anon->arcs_size + arc_mru->arcs_size - arc_p)); + (int64_t)(refcount_count(&arc_anon->arcs_size) + + refcount_count(&arc_mru->arcs_size) - arc_p)); total_evicted += arc_adjust_impl(arc_mru, 0, target, ARC_BUFC_METADATA); @@ -2962,7 +3024,7 @@ arc_adjust_meta(void) * space alloted to the MFU (which is defined as arc_c - arc_p). */ target = MIN((int64_t)(arc_meta_used - arc_meta_limit), - (int64_t)(arc_mfu->arcs_size - (arc_c - arc_p))); + (int64_t)(refcount_count(&arc_mfu->arcs_size) - (arc_c - arc_p))); total_evicted += arc_adjust_impl(arc_mfu, 0, target, ARC_BUFC_METADATA); @@ -3071,8 +3133,8 @@ arc_adjust(void) * arc_p here, and then evict more from the MFU below. */ target = MIN((int64_t)(arc_size - arc_c), - (int64_t)(arc_anon->arcs_size + arc_mru->arcs_size + arc_meta_used - - arc_p)); + (int64_t)(refcount_count(&arc_anon->arcs_size) + + refcount_count(&arc_mru->arcs_size) + arc_meta_used - arc_p)); /* * If we're below arc_meta_min, always prefer to evict data. @@ -3156,7 +3218,8 @@ arc_adjust(void) * cache. The following logic enforces these limits on the ghost * caches, and evicts from them as needed. */ - target = arc_mru->arcs_size + arc_mru_ghost->arcs_size - arc_c; + target = refcount_count(&arc_mru->arcs_size) + + refcount_count(&arc_mru_ghost->arcs_size) - arc_c; bytes = arc_adjust_impl(arc_mru_ghost, 0, target, ARC_BUFC_DATA); total_evicted += bytes; @@ -3174,7 +3237,8 @@ arc_adjust(void) * mru + mfu + mru ghost + mfu ghost <= 2 * arc_c * mru ghost + mfu ghost <= arc_c */ - target = arc_mru_ghost->arcs_size + arc_mfu_ghost->arcs_size - arc_c; + target = refcount_count(&arc_mru_ghost->arcs_size) + + refcount_count(&arc_mfu_ghost->arcs_size) - arc_c; bytes = arc_adjust_impl(arc_mfu_ghost, 0, target, ARC_BUFC_DATA); total_evicted += bytes; @@ -3668,6 +3732,8 @@ arc_adapt(int bytes, arc_state_t *state) { int mult; uint64_t arc_p_min = (arc_c >> arc_p_min_shift); + int64_t mrug_size = refcount_count(&arc_mru_ghost->arcs_size); + int64_t mfug_size = refcount_count(&arc_mfu_ghost->arcs_size); if (state == arc_l2c_only) return; @@ -3682,16 +3748,14 @@ arc_adapt(int bytes, arc_state_t *state) * target size of the MRU list. */ if (state == arc_mru_ghost) { - mult = ((arc_mru_ghost->arcs_size >= arc_mfu_ghost->arcs_size) ? - 1 : (arc_mfu_ghost->arcs_size/arc_mru_ghost->arcs_size)); + mult = (mrug_size >= mfug_size) ? 1 : (mfug_size / mrug_size); mult = MIN(mult, 10); /* avoid wild arc_p adjustment */ arc_p = MIN(arc_c - arc_p_min, arc_p + bytes * mult); } else if (state == arc_mfu_ghost) { uint64_t delta; - mult = ((arc_mfu_ghost->arcs_size >= arc_mru_ghost->arcs_size) ? - 1 : (arc_mru_ghost->arcs_size/arc_mfu_ghost->arcs_size)); + mult = (mfug_size >= mrug_size) ? 1 : (mrug_size / mfug_size); mult = MIN(mult, 10); delta = MIN(bytes * mult, arc_p); @@ -3808,8 +3872,9 @@ arc_get_data_buf(arc_buf_t *buf) */ if (!GHOST_STATE(buf->b_hdr->b_l1hdr.b_state)) { arc_buf_hdr_t *hdr = buf->b_hdr; + arc_state_t *state = hdr->b_l1hdr.b_state; - atomic_add_64(&hdr->b_l1hdr.b_state->arcs_size, size); + (void) refcount_add_many(&state->arcs_size, size, buf); /* * If this is reached via arc_read, the link is @@ -3830,7 +3895,8 @@ arc_get_data_buf(arc_buf_t *buf) * data, and we have outgrown arc_p, update arc_p */ if (arc_size < arc_c && hdr->b_l1hdr.b_state == arc_anon && - arc_anon->arcs_size + arc_mru->arcs_size > arc_p) + (refcount_count(&arc_anon->arcs_size) + + refcount_count(&arc_mru->arcs_size) > arc_p)) arc_p = MIN(arc_c, arc_p + size); } ARCSTAT_BUMP(arcstat_allocated); @@ -4686,8 +4752,10 @@ arc_release(arc_buf_t *buf, void *tag) buf->b_next = NULL; ASSERT3P(state, !=, arc_l2c_only); - ASSERT3U(state->arcs_size, >=, hdr->b_size); - atomic_add_64(&state->arcs_size, -hdr->b_size); + + (void) refcount_remove_many( + &state->arcs_size, hdr->b_size, buf); + if (refcount_is_zero(&hdr->b_l1hdr.b_refcnt)) { ASSERT3P(state, !=, arc_l2c_only); uint64_t *size = &state->arcs_lsize[type]; @@ -4730,7 +4798,7 @@ arc_release(arc_buf_t *buf, void *tag) (void) refcount_add(&nhdr->b_l1hdr.b_refcnt, tag); buf->b_hdr = nhdr; mutex_exit(&buf->b_evict_lock); - atomic_add_64(&arc_anon->arcs_size, blksz); + (void) refcount_add_many(&arc_anon->arcs_size, blksz, buf); } else { mutex_exit(&buf->b_evict_lock); ASSERT(refcount_count(&hdr->b_l1hdr.b_refcnt) == 1); @@ -4997,7 +5065,8 @@ arc_tempreserve_space(uint64_t reserve, * network delays from blocking transactions that are ready to be * assigned to a txg. */ - anon_size = MAX((int64_t)(arc_anon->arcs_size - arc_loaned_bytes), 0); + anon_size = MAX((int64_t)(refcount_count(&arc_anon->arcs_size) - + arc_loaned_bytes), 0); /* * Writes will, almost always, require additional memory allocations @@ -5034,7 +5103,7 @@ static void arc_kstat_update_state(arc_state_t *state, kstat_named_t *size, kstat_named_t *evict_data, kstat_named_t *evict_metadata) { - size->value.ui64 = state->arcs_size; + size->value.ui64 = refcount_count(&state->arcs_size); evict_data->value.ui64 = state->arcs_lsize[ARC_BUFC_DATA]; evict_metadata->value.ui64 = state->arcs_lsize[ARC_BUFC_METADATA]; } @@ -5274,6 +5343,13 @@ arc_init(void) offsetof(arc_buf_hdr_t, b_l1hdr.b_arc_node), zfs_arc_num_sublists_per_state, arc_state_multilist_index_func); + refcount_create(&arc_anon->arcs_size); + refcount_create(&arc_mru->arcs_size); + refcount_create(&arc_mru_ghost->arcs_size); + refcount_create(&arc_mfu->arcs_size); + refcount_create(&arc_mfu_ghost->arcs_size); + refcount_create(&arc_l2c_only->arcs_size); + buf_init(); arc_reclaim_thread_exit = FALSE; @@ -5400,6 +5476,13 @@ arc_fini(void) mutex_destroy(&arc_user_evicts_lock); cv_destroy(&arc_user_evicts_cv); + refcount_destroy(&arc_anon->arcs_size); + refcount_destroy(&arc_mru->arcs_size); + refcount_destroy(&arc_mru_ghost->arcs_size); + refcount_destroy(&arc_mfu->arcs_size); + refcount_destroy(&arc_mfu_ghost->arcs_size); + refcount_destroy(&arc_l2c_only->arcs_size); + multilist_destroy(&arc_mru->arcs_list[ARC_BUFC_METADATA]); multilist_destroy(&arc_mru_ghost->arcs_list[ARC_BUFC_METADATA]); multilist_destroy(&arc_mfu->arcs_list[ARC_BUFC_METADATA]); From owner-svn-src-stable@freebsd.org Sat Oct 3 11:13:02 2015 Return-Path: Delivered-To: svn-src-stable@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id C5B35A0EB2C; Sat, 3 Oct 2015 11:13:02 +0000 (UTC) (envelope-from mav@FreeBSD.org) Received: from repo.freebsd.org (repo.freebsd.org [IPv6:2001:1900:2254:2068::e6a:0]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 9D6CE1E51; Sat, 3 Oct 2015 11:13:02 +0000 (UTC) (envelope-from mav@FreeBSD.org) Received: from repo.freebsd.org ([127.0.1.70]) by repo.freebsd.org (8.15.2/8.15.2) with ESMTP id t93BD2Iu061329; Sat, 3 Oct 2015 11:13:02 GMT (envelope-from mav@FreeBSD.org) Received: (from mav@localhost) by repo.freebsd.org (8.15.2/8.15.2/Submit) id t93BD2JT061328; Sat, 3 Oct 2015 11:13:02 GMT (envelope-from mav@FreeBSD.org) Message-Id: <201510031113.t93BD2JT061328@repo.freebsd.org> X-Authentication-Warning: repo.freebsd.org: mav set sender to mav@FreeBSD.org using -f From: Alexander Motin Date: Sat, 3 Oct 2015 11:13:02 +0000 (UTC) To: src-committers@freebsd.org, svn-src-all@freebsd.org, svn-src-stable@freebsd.org, svn-src-stable-10@freebsd.org Subject: svn commit: r288582 - stable/10/sys/cddl/contrib/opensolaris/uts/common/fs/zfs X-SVN-Group: stable-10 MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit X-BeenThere: svn-src-stable@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: SVN commit messages for all the -stable branches of the src tree List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 03 Oct 2015 11:13:02 -0000 Author: mav Date: Sat Oct 3 11:13:01 2015 New Revision: 288582 URL: https://svnweb.freebsd.org/changeset/base/288582 Log: MFC r286764: 6033 arc_adjust() should search MFU lists for oldest buffer when adjusting MFU size. illumos/illumos-gate@31c46cf23cd1cf4d66390a983dc5072d7d299ba2 https://www.illumos.org/issues/6033 When we're looking for the list containing oldest buffer we never actually look at the MFU lists even when we try to evict from MFU. looks like a copy paste error, the fix is here: Reviewed by: Saso Kiselkov Reviewed by: Xin Li Reviewed by: Prakash Surya Approved by: Matthew Ahrens Author: Alek Pinchuk Obtained from: illumos Modified: stable/10/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/arc.c Directory Properties: stable/10/ (props changed) Modified: stable/10/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/arc.c ============================================================================== --- stable/10/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/arc.c Sat Oct 3 11:11:56 2015 (r288581) +++ stable/10/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/arc.c Sat Oct 3 11:13:01 2015 (r288582) @@ -23,7 +23,7 @@ * Copyright (c) 2012, Joyent, Inc. All rights reserved. * Copyright (c) 2011, 2014 by Delphix. All rights reserved. * Copyright (c) 2014 by Saso Kiselkov. All rights reserved. - * Copyright 2014 Nexenta Systems, Inc. All rights reserved. + * Copyright 2015 Nexenta Systems, Inc. All rights reserved. */ /* @@ -3118,7 +3118,7 @@ arc_adjust(void) */ target = arc_size - arc_c; - if (arc_adjust_type(arc_mru) == ARC_BUFC_METADATA && + if (arc_adjust_type(arc_mfu) == ARC_BUFC_METADATA && arc_meta_used > arc_meta_min) { bytes = arc_adjust_impl(arc_mfu, 0, target, ARC_BUFC_METADATA); total_evicted += bytes; From owner-svn-src-stable@freebsd.org Sat Oct 3 11:15:02 2015 Return-Path: Delivered-To: svn-src-stable@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 0A91EA0ECED; Sat, 3 Oct 2015 11:15:02 +0000 (UTC) (envelope-from mav@FreeBSD.org) Received: from repo.freebsd.org (repo.freebsd.org [IPv6:2001:1900:2254:2068::e6a:0]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id EF1021120; Sat, 3 Oct 2015 11:15:01 +0000 (UTC) (envelope-from mav@FreeBSD.org) Received: from repo.freebsd.org ([127.0.1.70]) by repo.freebsd.org (8.15.2/8.15.2) with ESMTP id t93BF1oX061529; Sat, 3 Oct 2015 11:15:01 GMT (envelope-from mav@FreeBSD.org) Received: (from mav@localhost) by repo.freebsd.org (8.15.2/8.15.2/Submit) id t93BF1dT061528; Sat, 3 Oct 2015 11:15:01 GMT (envelope-from mav@FreeBSD.org) Message-Id: <201510031115.t93BF1dT061528@repo.freebsd.org> X-Authentication-Warning: repo.freebsd.org: mav set sender to mav@FreeBSD.org using -f From: Alexander Motin Date: Sat, 3 Oct 2015 11:15:01 +0000 (UTC) To: src-committers@freebsd.org, svn-src-all@freebsd.org, svn-src-stable@freebsd.org, svn-src-stable-10@freebsd.org Subject: svn commit: r288584 - stable/10/sys/cddl/contrib/opensolaris/uts/common/fs/zfs X-SVN-Group: stable-10 MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit X-BeenThere: svn-src-stable@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: SVN commit messages for all the -stable branches of the src tree List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 03 Oct 2015 11:15:02 -0000 Author: mav Date: Sat Oct 3 11:15:01 2015 New Revision: 288584 URL: https://svnweb.freebsd.org/changeset/base/288584 Log: MFC r286767: Fix minor mismerge sometimes earlier. Modified: stable/10/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/arc.c Directory Properties: stable/10/ (props changed) Modified: stable/10/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/arc.c ============================================================================== --- stable/10/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/arc.c Sat Oct 3 11:14:05 2015 (r288583) +++ stable/10/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/arc.c Sat Oct 3 11:15:01 2015 (r288584) @@ -2193,10 +2193,6 @@ arc_buf_data_free(arc_buf_t *buf, void ( } } -/* - * Free up buf->b_data and if 'remove' is set, then pull the - * arc_buf_t off of the the arc_buf_hdr_t's list and free it. - */ static void arc_buf_l2_cdata_free(arc_buf_hdr_t *hdr) { @@ -2252,6 +2248,10 @@ arc_buf_l2_cdata_free(arc_buf_hdr_t *hdr hdr->b_l1hdr.b_tmp_cdata = NULL; } +/* + * Free up buf->b_data and if 'remove' is set, then pull the + * arc_buf_t off of the the arc_buf_hdr_t's list and free it. + */ static void arc_buf_destroy(arc_buf_t *buf, boolean_t remove) { From owner-svn-src-stable@freebsd.org Sat Oct 3 11:15:50 2015 Return-Path: Delivered-To: svn-src-stable@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 8A493A0EDAE; Sat, 3 Oct 2015 11:15:50 +0000 (UTC) (envelope-from mav@FreeBSD.org) Received: from repo.freebsd.org (repo.freebsd.org [IPv6:2001:1900:2254:2068::e6a:0]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 7A5031276; Sat, 3 Oct 2015 11:15:50 +0000 (UTC) (envelope-from mav@FreeBSD.org) Received: from repo.freebsd.org ([127.0.1.70]) by repo.freebsd.org (8.15.2/8.15.2) with ESMTP id t93BFovj061612; Sat, 3 Oct 2015 11:15:50 GMT (envelope-from mav@FreeBSD.org) Received: (from mav@localhost) by repo.freebsd.org (8.15.2/8.15.2/Submit) id t93BFoSk061611; Sat, 3 Oct 2015 11:15:50 GMT (envelope-from mav@FreeBSD.org) Message-Id: <201510031115.t93BFoSk061611@repo.freebsd.org> X-Authentication-Warning: repo.freebsd.org: mav set sender to mav@FreeBSD.org using -f From: Alexander Motin Date: Sat, 3 Oct 2015 11:15:50 +0000 (UTC) To: src-committers@freebsd.org, svn-src-all@freebsd.org, svn-src-stable@freebsd.org, svn-src-stable-10@freebsd.org Subject: svn commit: r288585 - stable/10/sys/cddl/contrib/opensolaris/uts/common/fs/zfs X-SVN-Group: stable-10 MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit X-BeenThere: svn-src-stable@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: SVN commit messages for all the -stable branches of the src tree List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 03 Oct 2015 11:15:50 -0000 Author: mav Date: Sat Oct 3 11:15:49 2015 New Revision: 288585 URL: https://svnweb.freebsd.org/changeset/base/288585 Log: MFC r286770: Fix r286766 build with debug. Modified: stable/10/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/arc.c Directory Properties: stable/10/ (props changed) Modified: stable/10/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/arc.c ============================================================================== --- stable/10/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/arc.c Sat Oct 3 11:15:01 2015 (r288584) +++ stable/10/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/arc.c Sat Oct 3 11:15:49 2015 (r288585) @@ -1030,21 +1030,21 @@ SYSCTL_INT(_vfs_zfs, OID_AUTO, l2arc_nor &l2arc_norw, 0, "no reads during writes"); SYSCTL_UQUAD(_vfs_zfs, OID_AUTO, anon_size, CTLFLAG_RD, - &ARC_anon.arcs_size, 0, "size of anonymous state"); + &ARC_anon.arcs_size.rc_count, 0, "size of anonymous state"); SYSCTL_UQUAD(_vfs_zfs, OID_AUTO, anon_metadata_lsize, CTLFLAG_RD, &ARC_anon.arcs_lsize[ARC_BUFC_METADATA], 0, "size of anonymous state"); SYSCTL_UQUAD(_vfs_zfs, OID_AUTO, anon_data_lsize, CTLFLAG_RD, &ARC_anon.arcs_lsize[ARC_BUFC_DATA], 0, "size of anonymous state"); SYSCTL_UQUAD(_vfs_zfs, OID_AUTO, mru_size, CTLFLAG_RD, - &ARC_mru.arcs_size, 0, "size of mru state"); + &ARC_mru.arcs_size.rc_count, 0, "size of mru state"); SYSCTL_UQUAD(_vfs_zfs, OID_AUTO, mru_metadata_lsize, CTLFLAG_RD, &ARC_mru.arcs_lsize[ARC_BUFC_METADATA], 0, "size of metadata in mru state"); SYSCTL_UQUAD(_vfs_zfs, OID_AUTO, mru_data_lsize, CTLFLAG_RD, &ARC_mru.arcs_lsize[ARC_BUFC_DATA], 0, "size of data in mru state"); SYSCTL_UQUAD(_vfs_zfs, OID_AUTO, mru_ghost_size, CTLFLAG_RD, - &ARC_mru_ghost.arcs_size, 0, "size of mru ghost state"); + &ARC_mru_ghost.arcs_size.rc_count, 0, "size of mru ghost state"); SYSCTL_UQUAD(_vfs_zfs, OID_AUTO, mru_ghost_metadata_lsize, CTLFLAG_RD, &ARC_mru_ghost.arcs_lsize[ARC_BUFC_METADATA], 0, "size of metadata in mru ghost state"); @@ -1053,14 +1053,14 @@ SYSCTL_UQUAD(_vfs_zfs, OID_AUTO, mru_gho "size of data in mru ghost state"); SYSCTL_UQUAD(_vfs_zfs, OID_AUTO, mfu_size, CTLFLAG_RD, - &ARC_mfu.arcs_size, 0, "size of mfu state"); + &ARC_mfu.arcs_size.rc_count, 0, "size of mfu state"); SYSCTL_UQUAD(_vfs_zfs, OID_AUTO, mfu_metadata_lsize, CTLFLAG_RD, &ARC_mfu.arcs_lsize[ARC_BUFC_METADATA], 0, "size of metadata in mfu state"); SYSCTL_UQUAD(_vfs_zfs, OID_AUTO, mfu_data_lsize, CTLFLAG_RD, &ARC_mfu.arcs_lsize[ARC_BUFC_DATA], 0, "size of data in mfu state"); SYSCTL_UQUAD(_vfs_zfs, OID_AUTO, mfu_ghost_size, CTLFLAG_RD, - &ARC_mfu_ghost.arcs_size, 0, "size of mfu ghost state"); + &ARC_mfu_ghost.arcs_size.rc_count, 0, "size of mfu ghost state"); SYSCTL_UQUAD(_vfs_zfs, OID_AUTO, mfu_ghost_metadata_lsize, CTLFLAG_RD, &ARC_mfu_ghost.arcs_lsize[ARC_BUFC_METADATA], 0, "size of metadata in mfu ghost state"); @@ -1069,7 +1069,7 @@ SYSCTL_UQUAD(_vfs_zfs, OID_AUTO, mfu_gho "size of data in mfu ghost state"); SYSCTL_UQUAD(_vfs_zfs, OID_AUTO, l2c_only_size, CTLFLAG_RD, - &ARC_l2c_only.arcs_size, 0, "size of mru state"); + &ARC_l2c_only.arcs_size.rc_count, 0, "size of mru state"); /* * L2ARC Internals From owner-svn-src-stable@freebsd.org Sat Oct 3 11:16:43 2015 Return-Path: Delivered-To: svn-src-stable@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 4D0C2A0EE81; Sat, 3 Oct 2015 11:16:43 +0000 (UTC) (envelope-from mav@FreeBSD.org) Received: from repo.freebsd.org (repo.freebsd.org [IPv6:2001:1900:2254:2068::e6a:0]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 3D7E213C2; Sat, 3 Oct 2015 11:16:43 +0000 (UTC) (envelope-from mav@FreeBSD.org) Received: from repo.freebsd.org ([127.0.1.70]) by repo.freebsd.org (8.15.2/8.15.2) with ESMTP id t93BGhwA061706; Sat, 3 Oct 2015 11:16:43 GMT (envelope-from mav@FreeBSD.org) Received: (from mav@localhost) by repo.freebsd.org (8.15.2/8.15.2/Submit) id t93BGhFI061705; Sat, 3 Oct 2015 11:16:43 GMT (envelope-from mav@FreeBSD.org) Message-Id: <201510031116.t93BGhFI061705@repo.freebsd.org> X-Authentication-Warning: repo.freebsd.org: mav set sender to mav@FreeBSD.org using -f From: Alexander Motin Date: Sat, 3 Oct 2015 11:16:43 +0000 (UTC) To: src-committers@freebsd.org, svn-src-all@freebsd.org, svn-src-stable@freebsd.org, svn-src-stable-10@freebsd.org Subject: svn commit: r288586 - stable/10/sys/cddl/contrib/opensolaris/uts/common/fs/zfs X-SVN-Group: stable-10 MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit X-BeenThere: svn-src-stable@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: SVN commit messages for all the -stable branches of the src tree List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 03 Oct 2015 11:16:43 -0000 Author: mav Date: Sat Oct 3 11:16:42 2015 New Revision: 288586 URL: https://svnweb.freebsd.org/changeset/base/288586 Log: MFC r286774: 2618 arc.c mistypes in the comments Reviewed by: Jason King Reviewed by: Josef Sipek Approved by: Richard Lowe Author: Bart Coddens illumos/illumos-gate@fc98fea58e89224f6f13d7fae246d6cb5dfa35ea Modified: stable/10/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/arc.c Directory Properties: stable/10/ (props changed) Modified: stable/10/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/arc.c ============================================================================== --- stable/10/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/arc.c Sat Oct 3 11:15:49 2015 (r288585) +++ stable/10/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/arc.c Sat Oct 3 11:16:42 2015 (r288586) @@ -82,9 +82,9 @@ * types of locks: 1) the hash table lock array, and 2) the * arc list locks. * - * Buffers do not have their own mutexs, rather they rely on the - * hash table mutexs for the bulk of their protection (i.e. most - * fields in the arc_buf_hdr_t are protected by these mutexs). + * Buffers do not have their own mutexes, rather they rely on the + * hash table mutexes for the bulk of their protection (i.e. most + * fields in the arc_buf_hdr_t are protected by these mutexes). * * buf_hash_find() returns the appropriate mutex (held) when it * locates the requested buffer in the hash table. It returns From owner-svn-src-stable@freebsd.org Sat Oct 3 11:17:32 2015 Return-Path: Delivered-To: svn-src-stable@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id AA3F4A0EF4F; Sat, 3 Oct 2015 11:17:32 +0000 (UTC) (envelope-from mav@FreeBSD.org) Received: from repo.freebsd.org (repo.freebsd.org [IPv6:2001:1900:2254:2068::e6a:0]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 9A8AF15AC; Sat, 3 Oct 2015 11:17:32 +0000 (UTC) (envelope-from mav@FreeBSD.org) Received: from repo.freebsd.org ([127.0.1.70]) by repo.freebsd.org (8.15.2/8.15.2) with ESMTP id t93BHWKo061796; Sat, 3 Oct 2015 11:17:32 GMT (envelope-from mav@FreeBSD.org) Received: (from mav@localhost) by repo.freebsd.org (8.15.2/8.15.2/Submit) id t93BHW6K061795; Sat, 3 Oct 2015 11:17:32 GMT (envelope-from mav@FreeBSD.org) Message-Id: <201510031117.t93BHW6K061795@repo.freebsd.org> X-Authentication-Warning: repo.freebsd.org: mav set sender to mav@FreeBSD.org using -f From: Alexander Motin Date: Sat, 3 Oct 2015 11:17:32 +0000 (UTC) To: src-committers@freebsd.org, svn-src-all@freebsd.org, svn-src-stable@freebsd.org, svn-src-stable-10@freebsd.org Subject: svn commit: r288587 - stable/10/sys/cddl/contrib/opensolaris/uts/common/fs/zfs X-SVN-Group: stable-10 MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit X-BeenThere: svn-src-stable@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: SVN commit messages for all the -stable branches of the src tree List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 03 Oct 2015 11:17:32 -0000 Author: mav Date: Sat Oct 3 11:17:31 2015 New Revision: 288587 URL: https://svnweb.freebsd.org/changeset/base/288587 Log: MFC r286776: Remove some random accumulated diff from Illumos. Modified: stable/10/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/arc.c Directory Properties: stable/10/ (props changed) Modified: stable/10/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/arc.c ============================================================================== --- stable/10/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/arc.c Sat Oct 3 11:16:42 2015 (r288586) +++ stable/10/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/arc.c Sat Oct 3 11:17:31 2015 (r288587) @@ -2416,6 +2416,7 @@ arc_hdr_destroy(arc_buf_hdr_t *hdr) if (!BUF_EMPTY(hdr)) buf_discard_identity(hdr); + if (hdr->b_freeze_cksum != NULL) { kmem_free(hdr->b_freeze_cksum, sizeof (zio_cksum_t)); hdr->b_freeze_cksum = NULL; @@ -4658,8 +4659,6 @@ arc_release(arc_buf_t *buf, void *tag) { arc_buf_hdr_t *hdr = buf->b_hdr; - ASSERT(HDR_HAS_L1HDR(hdr)); - /* * It would be nice to assert that if it's DMU metadata (level > * 0 || it's the dnode file), then it must be syncing context. @@ -4667,6 +4666,9 @@ arc_release(arc_buf_t *buf, void *tag) */ mutex_enter(&buf->b_evict_lock); + + ASSERT(HDR_HAS_L1HDR(hdr)); + /* * We don't grab the hash lock prior to this check, because if * the buffer's header is in the arc_anon state, it won't be @@ -5897,6 +5899,7 @@ top: /* * Error - drop L2ARC entry. */ + list_remove(buflist, hdr); trim_map_free(hdr->b_l2hdr.b_dev->l2ad_vdev, hdr->b_l2hdr.b_daddr, hdr->b_l2hdr.b_asize, 0); hdr->b_flags &= ~ARC_FLAG_HAS_L2HDR; @@ -6437,14 +6440,6 @@ l2arc_write_buffers(spa_t *spa, l2arc_de buf_sz = hdr->b_l2hdr.b_asize; /* - * If the data has not been compressed, then clear b_tmp_cdata - * to make sure that it points only to a temporary compression - * buffer. - */ - if (!L2ARC_IS_VALID_COMPRESS(HDR_GET_COMPRESS(hdr))) - hdr->b_l1hdr.b_tmp_cdata = NULL; - - /* * We need to do this regardless if buf_sz is zero or * not, otherwise, when this l2hdr is evicted we'll * remove a reference that was never added. @@ -6537,6 +6532,12 @@ l2arc_compress_buf(arc_buf_hdr_t *hdr) csize = zio_compress_data(ZIO_COMPRESS_LZ4, hdr->b_l1hdr.b_tmp_cdata, cdata, l2hdr->b_asize); + rounded = P2ROUNDUP(csize, (size_t)SPA_MINBLOCKSIZE); + if (rounded > csize) { + bzero((char *)cdata + csize, rounded - csize); + csize = rounded; + } + if (csize == 0) { /* zero block, indicate that there's nothing to write */ zio_data_buf_free(cdata, len); @@ -6545,19 +6546,11 @@ l2arc_compress_buf(arc_buf_hdr_t *hdr) hdr->b_l1hdr.b_tmp_cdata = NULL; ARCSTAT_BUMP(arcstat_l2_compress_zeros); return (B_TRUE); - } - - rounded = P2ROUNDUP(csize, - (size_t)1 << l2hdr->b_dev->l2ad_vdev->vdev_ashift); - if (rounded < len) { + } else if (csize > 0 && csize < len) { /* * Compression succeeded, we'll keep the cdata around for * writing and release it afterwards. */ - if (rounded > csize) { - bzero((char *)cdata + csize, rounded - csize); - csize = rounded; - } HDR_SET_COMPRESS(hdr, ZIO_COMPRESS_LZ4); l2hdr->b_asize = csize; hdr->b_l1hdr.b_tmp_cdata = cdata; @@ -6674,6 +6667,7 @@ l2arc_release_cdata_buf(arc_buf_hdr_t *h hdr->b_size); hdr->b_l1hdr.b_tmp_cdata = NULL; } + } /* From owner-svn-src-stable@freebsd.org Sat Oct 3 11:18:31 2015 Return-Path: Delivered-To: svn-src-stable@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 5B5C8A0F037; Sat, 3 Oct 2015 11:18:31 +0000 (UTC) (envelope-from mav@FreeBSD.org) Received: from repo.freebsd.org (repo.freebsd.org [IPv6:2001:1900:2254:2068::e6a:0]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 4B9F01812; Sat, 3 Oct 2015 11:18:31 +0000 (UTC) (envelope-from mav@FreeBSD.org) Received: from repo.freebsd.org ([127.0.1.70]) by repo.freebsd.org (8.15.2/8.15.2) with ESMTP id t93BIVj6061892; Sat, 3 Oct 2015 11:18:31 GMT (envelope-from mav@FreeBSD.org) Received: (from mav@localhost) by repo.freebsd.org (8.15.2/8.15.2/Submit) id t93BIV3C061891; Sat, 3 Oct 2015 11:18:31 GMT (envelope-from mav@FreeBSD.org) Message-Id: <201510031118.t93BIV3C061891@repo.freebsd.org> X-Authentication-Warning: repo.freebsd.org: mav set sender to mav@FreeBSD.org using -f From: Alexander Motin Date: Sat, 3 Oct 2015 11:18:31 +0000 (UTC) To: src-committers@freebsd.org, svn-src-all@freebsd.org, svn-src-stable@freebsd.org, svn-src-stable-10@freebsd.org Subject: svn commit: r288588 - stable/10/sys/cddl/contrib/opensolaris/uts/common/fs/zfs X-SVN-Group: stable-10 MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit X-BeenThere: svn-src-stable@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: SVN commit messages for all the -stable branches of the src tree List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 03 Oct 2015 11:18:31 -0000 Author: mav Date: Sat Oct 3 11:18:30 2015 New Revision: 288588 URL: https://svnweb.freebsd.org/changeset/base/288588 Log: MFC r286951: Restore part of r274628, reverted at r286776. Modified: stable/10/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/arc.c Directory Properties: stable/10/ (props changed) Modified: stable/10/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/arc.c ============================================================================== --- stable/10/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/arc.c Sat Oct 3 11:17:31 2015 (r288587) +++ stable/10/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/arc.c Sat Oct 3 11:18:30 2015 (r288588) @@ -6532,7 +6532,8 @@ l2arc_compress_buf(arc_buf_hdr_t *hdr) csize = zio_compress_data(ZIO_COMPRESS_LZ4, hdr->b_l1hdr.b_tmp_cdata, cdata, l2hdr->b_asize); - rounded = P2ROUNDUP(csize, (size_t)SPA_MINBLOCKSIZE); + rounded = P2ROUNDUP(csize, + (size_t)1 << l2hdr->b_dev->l2ad_vdev->vdev_ashift); if (rounded > csize) { bzero((char *)cdata + csize, rounded - csize); csize = rounded; From owner-svn-src-stable@freebsd.org Sat Oct 3 11:19:33 2015 Return-Path: Delivered-To: svn-src-stable@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 2FC24A0F0CD; Sat, 3 Oct 2015 11:19:33 +0000 (UTC) (envelope-from mav@FreeBSD.org) Received: from repo.freebsd.org (repo.freebsd.org [IPv6:2001:1900:2254:2068::e6a:0]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 2011E1979; Sat, 3 Oct 2015 11:19:33 +0000 (UTC) (envelope-from mav@FreeBSD.org) Received: from repo.freebsd.org ([127.0.1.70]) by repo.freebsd.org (8.15.2/8.15.2) with ESMTP id t93BJXSo061988; Sat, 3 Oct 2015 11:19:33 GMT (envelope-from mav@FreeBSD.org) Received: (from mav@localhost) by repo.freebsd.org (8.15.2/8.15.2/Submit) id t93BJWTf061987; Sat, 3 Oct 2015 11:19:32 GMT (envelope-from mav@FreeBSD.org) Message-Id: <201510031119.t93BJWTf061987@repo.freebsd.org> X-Authentication-Warning: repo.freebsd.org: mav set sender to mav@FreeBSD.org using -f From: Alexander Motin Date: Sat, 3 Oct 2015 11:19:32 +0000 (UTC) To: src-committers@freebsd.org, svn-src-all@freebsd.org, svn-src-stable@freebsd.org, svn-src-stable-10@freebsd.org Subject: svn commit: r288589 - stable/10/sys/cddl/contrib/opensolaris/uts/common/fs/zfs X-SVN-Group: stable-10 MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit X-BeenThere: svn-src-stable@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: SVN commit messages for all the -stable branches of the src tree List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 03 Oct 2015 11:19:33 -0000 Author: mav Date: Sat Oct 3 11:19:32 2015 New Revision: 288589 URL: https://svnweb.freebsd.org/changeset/base/288589 Log: MFC r286983 (by avg): fix a mismerge in r286539 (MFV 286538: 5562 ZFS sa_handle's violate...) Modified: stable/10/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/sa.c Directory Properties: stable/10/ (props changed) Modified: stable/10/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/sa.c ============================================================================== --- stable/10/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/sa.c Sat Oct 3 11:18:30 2015 (r288588) +++ stable/10/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/sa.c Sat Oct 3 11:19:32 2015 (r288589) @@ -220,7 +220,6 @@ static void sa_cache_destructor(void *buf, void *unused) { sa_handle_t *hdl = buf; - hdl->sa_dbu.dbu_evict_func = NULL; mutex_destroy(&hdl->sa_lock); } @@ -1385,6 +1384,7 @@ sa_handle_get_from_db(objset_t *os, dmu_ sa_handle_t *winner = NULL; handle = kmem_cache_alloc(sa_cache, KM_SLEEP); + handle->sa_dbu.dbu_evict_func = NULL; handle->sa_userp = userp; handle->sa_bonus = db; handle->sa_os = os; From owner-svn-src-stable@freebsd.org Sat Oct 3 11:21:52 2015 Return-Path: Delivered-To: svn-src-stable@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 613ADA0F355; Sat, 3 Oct 2015 11:21:52 +0000 (UTC) (envelope-from mav@FreeBSD.org) Received: from repo.freebsd.org (repo.freebsd.org [IPv6:2001:1900:2254:2068::e6a:0]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 510851D11; Sat, 3 Oct 2015 11:21:52 +0000 (UTC) (envelope-from mav@FreeBSD.org) Received: from repo.freebsd.org ([127.0.1.70]) by repo.freebsd.org (8.15.2/8.15.2) with ESMTP id t93BLqPX065737; Sat, 3 Oct 2015 11:21:52 GMT (envelope-from mav@FreeBSD.org) Received: (from mav@localhost) by repo.freebsd.org (8.15.2/8.15.2/Submit) id t93BLpXV065734; Sat, 3 Oct 2015 11:21:51 GMT (envelope-from mav@FreeBSD.org) Message-Id: <201510031121.t93BLpXV065734@repo.freebsd.org> X-Authentication-Warning: repo.freebsd.org: mav set sender to mav@FreeBSD.org using -f From: Alexander Motin Date: Sat, 3 Oct 2015 11:21:51 +0000 (UTC) To: src-committers@freebsd.org, svn-src-all@freebsd.org, svn-src-stable@freebsd.org, svn-src-stable-10@freebsd.org Subject: svn commit: r288590 - in stable/10/sys/cddl/contrib/opensolaris/uts/common/fs/zfs: . sys X-SVN-Group: stable-10 MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit X-BeenThere: svn-src-stable@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: SVN commit messages for all the -stable branches of the src tree List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 03 Oct 2015 11:21:52 -0000 Author: mav Date: Sat Oct 3 11:21:50 2015 New Revision: 288590 URL: https://svnweb.freebsd.org/changeset/base/288590 Log: MFC r287103 (by avg): 5692 expose the number of hole blocks in a file FreeBSD porting notes: - only kernel-side changes are merged - the new ioctl is not actually implemented yet - thus, the goal is to synchronize DMU code illumos/illumos-gate@2bcf0248e992f292c7b814458bcdce2f004925d6 https://www.illumos.org/issues/5692 we would like to expose the number of hole (sparse) blocks in a file. this can be useful to for example if you want to fill in the holes with some data; knowing the number of holes in advances allows you to report progress on hole filling. We could use SEEK_HOLE to do that but it would be O(n) where n is the number of holes present in the file. Author: Max Grossman Reviewed by: Adam Leventhal Reviewed by: Matthew Ahrens Reviewed by: Boris Protopopov Approved by: Richard Lowe Modified: stable/10/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/dmu.c stable/10/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/sys/dmu.h stable/10/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zfs_vnops.c Directory Properties: stable/10/ (props changed) Modified: stable/10/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/dmu.c ============================================================================== --- stable/10/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/dmu.c Sat Oct 3 11:19:32 2015 (r288589) +++ stable/10/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/dmu.c Sat Oct 3 11:21:50 2015 (r288590) @@ -1902,25 +1902,20 @@ int dmu_offset_next(objset_t *os, uint64_t object, boolean_t hole, uint64_t *off) { dnode_t *dn; - int i, err; + int err; - err = dnode_hold(os, object, FTAG, &dn); - if (err) - return (err); /* * Sync any current changes before * we go trundling through the block pointers. */ - for (i = 0; i < TXG_SIZE; i++) { - if (list_link_active(&dn->dn_dirty_link[i])) - break; + err = dmu_object_wait_synced(os, object); + if (err) { + return (err); } - if (i != TXG_SIZE) { - dnode_rele(dn, FTAG); - txg_wait_synced(dmu_objset_pool(os), 0); - err = dnode_hold(os, object, FTAG, &dn); - if (err) - return (err); + + err = dnode_hold(os, object, FTAG, &dn); + if (err) { + return (err); } err = dnode_next_offset(dn, (hole ? DNODE_FIND_HOLE : 0), off, 1, 1, 0); @@ -1929,6 +1924,36 @@ dmu_offset_next(objset_t *os, uint64_t o return (err); } +/* + * Given the ZFS object, if it contains any dirty nodes + * this function flushes all dirty blocks to disk. This + * ensures the DMU object info is updated. A more efficient + * future version might just find the TXG with the maximum + * ID and wait for that to be synced. + */ +int +dmu_object_wait_synced(objset_t *os, uint64_t object) { + dnode_t *dn; + int error, i; + + error = dnode_hold(os, object, FTAG, &dn); + if (error) { + return (error); + } + + for (i = 0; i < TXG_SIZE; i++) { + if (list_link_active(&dn->dn_dirty_link[i])) { + break; + } + } + dnode_rele(dn, FTAG); + if (i != TXG_SIZE) { + txg_wait_synced(dmu_objset_pool(os), 0); + } + + return (0); +} + void dmu_object_info_from_dnode(dnode_t *dn, dmu_object_info_t *doi) { Modified: stable/10/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/sys/dmu.h ============================================================================== --- stable/10/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/sys/dmu.h Sat Oct 3 11:19:32 2015 (r288589) +++ stable/10/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/sys/dmu.h Sat Oct 3 11:21:50 2015 (r288590) @@ -915,6 +915,15 @@ int dmu_offset_next(objset_t *os, uint64 uint64_t *off); /* + * Check if a DMU object has any dirty blocks. If so, sync out + * all pending transaction groups. Otherwise, this function + * does not alter DMU state. This could be improved to only sync + * out the necessary transaction groups for this particular + * object. + */ +int dmu_object_wait_synced(objset_t *os, uint64_t object); + +/* * Initial setup and final teardown. */ extern void dmu_init(void); Modified: stable/10/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zfs_vnops.c ============================================================================== --- stable/10/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zfs_vnops.c Sat Oct 3 11:19:32 2015 (r288589) +++ stable/10/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zfs_vnops.c Sat Oct 3 11:21:50 2015 (r288590) @@ -293,25 +293,32 @@ zfs_ioctl(vnode_t *vp, u_long com, intpt int *rvalp, caller_context_t *ct) { offset_t off; + offset_t ndata; + dmu_object_info_t doi; int error; zfsvfs_t *zfsvfs; znode_t *zp; switch (com) { case _FIOFFS: + { return (0); /* * The following two ioctls are used by bfu. Faking out, * necessary to avoid bfu errors. */ + } case _FIOGDIO: case _FIOSDIO: + { return (0); + } case _FIO_SEEK_DATA: case _FIO_SEEK_HOLE: -#ifdef sun + { +#ifdef illumos if (ddi_copyin((void *)data, &off, sizeof (off), flag)) return (SET_ERROR(EFAULT)); #else @@ -335,6 +342,48 @@ zfs_ioctl(vnode_t *vp, u_long com, intpt #endif return (0); } +#ifdef illumos + case _FIO_COUNT_FILLED: + { + /* + * _FIO_COUNT_FILLED adds a new ioctl command which + * exposes the number of filled blocks in a + * ZFS object. + */ + zp = VTOZ(vp); + zfsvfs = zp->z_zfsvfs; + ZFS_ENTER(zfsvfs); + ZFS_VERIFY_ZP(zp); + + /* + * Wait for all dirty blocks for this object + * to get synced out to disk, and the DMU info + * updated. + */ + error = dmu_object_wait_synced(zfsvfs->z_os, zp->z_id); + if (error) { + ZFS_EXIT(zfsvfs); + return (error); + } + + /* + * Retrieve fill count from DMU object. + */ + error = dmu_object_info(zfsvfs->z_os, zp->z_id, &doi); + if (error) { + ZFS_EXIT(zfsvfs); + return (error); + } + + ndata = doi.doi_fill_count; + + ZFS_EXIT(zfsvfs); + if (ddi_copyout(&ndata, (void *)data, sizeof (ndata), flag)) + return (SET_ERROR(EFAULT)); + return (0); + } +#endif + } return (SET_ERROR(ENOTTY)); } From owner-svn-src-stable@freebsd.org Sat Oct 3 11:23:09 2015 Return-Path: Delivered-To: svn-src-stable@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id BDDA9A0F455; Sat, 3 Oct 2015 11:23:09 +0000 (UTC) (envelope-from mav@FreeBSD.org) Received: from repo.freebsd.org (repo.freebsd.org [IPv6:2001:1900:2254:2068::e6a:0]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id AE2A51E9E; Sat, 3 Oct 2015 11:23:09 +0000 (UTC) (envelope-from mav@FreeBSD.org) Received: from repo.freebsd.org ([127.0.1.70]) by repo.freebsd.org (8.15.2/8.15.2) with ESMTP id t93BN9OU065866; Sat, 3 Oct 2015 11:23:09 GMT (envelope-from mav@FreeBSD.org) Received: (from mav@localhost) by repo.freebsd.org (8.15.2/8.15.2/Submit) id t93BN9F7065865; Sat, 3 Oct 2015 11:23:09 GMT (envelope-from mav@FreeBSD.org) Message-Id: <201510031123.t93BN9F7065865@repo.freebsd.org> X-Authentication-Warning: repo.freebsd.org: mav set sender to mav@FreeBSD.org using -f From: Alexander Motin Date: Sat, 3 Oct 2015 11:23:09 +0000 (UTC) To: src-committers@freebsd.org, svn-src-all@freebsd.org, svn-src-stable@freebsd.org, svn-src-stable-10@freebsd.org Subject: svn commit: r288591 - stable/10/sys/cddl/contrib/opensolaris/uts/common/fs/zfs X-SVN-Group: stable-10 MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit X-BeenThere: svn-src-stable@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: SVN commit messages for all the -stable branches of the src tree List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 03 Oct 2015 11:23:09 -0000 Author: mav Date: Sat Oct 3 11:23:08 2015 New Revision: 288591 URL: https://svnweb.freebsd.org/changeset/base/288591 Log: MFC r287280 (by delphij): In r286705 (Illumos 5960/a2cdcdd), a separate thread is created with curproc as parent. In the case of a send or receive, the curproc would be the userland application that issues the ioctl. This would trigger an assertion failure introduced in Solaris compatibility shims in r196458 when kernel is compiled with INVARIANTS. Fix this by using p0 (proc0 or kernel) as the parent thread when creating the kernel threads. Modified: stable/10/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/dmu_send.c Directory Properties: stable/10/ (props changed) Modified: stable/10/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/dmu_send.c ============================================================================== --- stable/10/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/dmu_send.c Sat Oct 3 11:21:50 2015 (r288590) +++ stable/10/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/dmu_send.c Sat Oct 3 11:23:08 2015 (r288591) @@ -786,7 +786,7 @@ dmu_send_impl(void *tag, dsl_pool_t *dp, to_arg.ds = to_ds; to_arg.fromtxg = fromtxg; to_arg.flags = TRAVERSE_PRE | TRAVERSE_PREFETCH; - (void) thread_create(NULL, 0, send_traverse_thread, &to_arg, 0, curproc, + (void) thread_create(NULL, 0, send_traverse_thread, &to_arg, 0, &p0, TS_RUN, minclsyspri); struct send_block_record *to_data; @@ -2446,7 +2446,7 @@ dmu_recv_stream(dmu_recv_cookie_t *drc, rwa.os = ra.os; rwa.byteswap = drc->drc_byteswap; - (void) thread_create(NULL, 0, receive_writer_thread, &rwa, 0, curproc, + (void) thread_create(NULL, 0, receive_writer_thread, &rwa, 0, &p0, TS_RUN, minclsyspri); /* * We're reading rwa.err without locks, which is safe since we are the From owner-svn-src-stable@freebsd.org Sat Oct 3 11:24:48 2015 Return-Path: Delivered-To: svn-src-stable@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 95E4DA0F535; Sat, 3 Oct 2015 11:24:48 +0000 (UTC) (envelope-from mav@FreeBSD.org) Received: from repo.freebsd.org (repo.freebsd.org [IPv6:2001:1900:2254:2068::e6a:0]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 7A90B1FFF; Sat, 3 Oct 2015 11:24:48 +0000 (UTC) (envelope-from mav@FreeBSD.org) Received: from repo.freebsd.org ([127.0.1.70]) by repo.freebsd.org (8.15.2/8.15.2) with ESMTP id t93BOmto065980; Sat, 3 Oct 2015 11:24:48 GMT (envelope-from mav@FreeBSD.org) Received: (from mav@localhost) by repo.freebsd.org (8.15.2/8.15.2/Submit) id t93BOmgU065979; Sat, 3 Oct 2015 11:24:48 GMT (envelope-from mav@FreeBSD.org) Message-Id: <201510031124.t93BOmgU065979@repo.freebsd.org> X-Authentication-Warning: repo.freebsd.org: mav set sender to mav@FreeBSD.org using -f From: Alexander Motin Date: Sat, 3 Oct 2015 11:24:48 +0000 (UTC) To: src-committers@freebsd.org, svn-src-all@freebsd.org, svn-src-stable@freebsd.org, svn-src-stable-10@freebsd.org Subject: svn commit: r288592 - stable/10/sys/cddl/contrib/opensolaris/uts/common/fs/zfs X-SVN-Group: stable-10 MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit X-BeenThere: svn-src-stable@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: SVN commit messages for all the -stable branches of the src tree List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 03 Oct 2015 11:24:48 -0000 Author: mav Date: Sat Oct 3 11:24:47 2015 New Revision: 288592 URL: https://svnweb.freebsd.org/changeset/base/288592 Log: MFC r287283 (by delphij): Fix a buffer overrun which may lead to data corruption, introduced in r286951 by reinstating changes in r274628. In l2arc_compress_buf(), we allocate a buffer to stash away the compressed data in 'cdata', allocated of l2hdr->b_asize bytes. We then ask zio_compress_data() to compress the buffer, b_l1hdr.b_tmp_cdata, which is of l2hdr->b_asize bytes, and have the compressed size (or original size, if compress didn't gain enough) stored in csize. To pad the buffer to fit the optimal write size, we round up the compressed size to L2 device's vdev_ashift. Illumos code rounds up the size by at most SPA_MINBLOCKSIZE. Because we know csize <= b_asize, and b_asize is integer multiple of SPA_MINBLOCKSIZE, we are guaranteed that the rounded up csize would be <= b_asize. However, this is not necessarily true when we round up to 1 << vdev_ashift, because it could be larger than SPA_MINBLOCKSIZE. So, in the worst case scenario, we are overwriting at most (1 << vdev_ashift - SPA_MINBLOCKSIZE) bytes of memory next to the compressed data buffer. Andriy's original change in r274628 reorganized the code a little bit, by moving the padding to after we determined that the compression was beneficial. At which point, we would check rounded size against the allocated buffer size, and the buffer overrun would not be possible. Modified: stable/10/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/arc.c Directory Properties: stable/10/ (props changed) Modified: stable/10/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/arc.c ============================================================================== --- stable/10/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/arc.c Sat Oct 3 11:23:08 2015 (r288591) +++ stable/10/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/arc.c Sat Oct 3 11:24:47 2015 (r288592) @@ -6532,13 +6532,6 @@ l2arc_compress_buf(arc_buf_hdr_t *hdr) csize = zio_compress_data(ZIO_COMPRESS_LZ4, hdr->b_l1hdr.b_tmp_cdata, cdata, l2hdr->b_asize); - rounded = P2ROUNDUP(csize, - (size_t)1 << l2hdr->b_dev->l2ad_vdev->vdev_ashift); - if (rounded > csize) { - bzero((char *)cdata + csize, rounded - csize); - csize = rounded; - } - if (csize == 0) { /* zero block, indicate that there's nothing to write */ zio_data_buf_free(cdata, len); @@ -6547,11 +6540,19 @@ l2arc_compress_buf(arc_buf_hdr_t *hdr) hdr->b_l1hdr.b_tmp_cdata = NULL; ARCSTAT_BUMP(arcstat_l2_compress_zeros); return (B_TRUE); - } else if (csize > 0 && csize < len) { + } + + rounded = P2ROUNDUP(csize, + (size_t)1 << l2hdr->b_dev->l2ad_vdev->vdev_ashift); + if (rounded < len) { /* * Compression succeeded, we'll keep the cdata around for * writing and release it afterwards. */ + if (rounded > csize) { + bzero((char *)cdata + csize, rounded - csize); + csize = rounded; + } HDR_SET_COMPRESS(hdr, ZIO_COMPRESS_LZ4); l2hdr->b_asize = csize; hdr->b_l1hdr.b_tmp_cdata = cdata; From owner-svn-src-stable@freebsd.org Sat Oct 3 11:27:24 2015 Return-Path: Delivered-To: svn-src-stable@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 372DBA0F754; Sat, 3 Oct 2015 11:27:24 +0000 (UTC) (envelope-from mav@FreeBSD.org) Received: from repo.freebsd.org (repo.freebsd.org [IPv6:2001:1900:2254:2068::e6a:0]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 27A9911FD; Sat, 3 Oct 2015 11:27:24 +0000 (UTC) (envelope-from mav@FreeBSD.org) Received: from repo.freebsd.org ([127.0.1.70]) by repo.freebsd.org (8.15.2/8.15.2) with ESMTP id t93BROd2066138; Sat, 3 Oct 2015 11:27:24 GMT (envelope-from mav@FreeBSD.org) Received: (from mav@localhost) by repo.freebsd.org (8.15.2/8.15.2/Submit) id t93BROI7066137; Sat, 3 Oct 2015 11:27:24 GMT (envelope-from mav@FreeBSD.org) Message-Id: <201510031127.t93BROI7066137@repo.freebsd.org> X-Authentication-Warning: repo.freebsd.org: mav set sender to mav@FreeBSD.org using -f From: Alexander Motin Date: Sat, 3 Oct 2015 11:27:24 +0000 (UTC) To: src-committers@freebsd.org, svn-src-all@freebsd.org, svn-src-stable@freebsd.org, svn-src-stable-10@freebsd.org Subject: svn commit: r288593 - stable/10/cddl/contrib/opensolaris/lib/libzpool/common/sys X-SVN-Group: stable-10 MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit X-BeenThere: svn-src-stable@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: SVN commit messages for all the -stable branches of the src tree List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 03 Oct 2015 11:27:24 -0000 Author: mav Date: Sat Oct 3 11:27:23 2015 New Revision: 288593 URL: https://svnweb.freebsd.org/changeset/base/288593 Log: MFC r287335 (by allanjude): Remove duplicate defines introduced in initial ZFS import (r168404) This change reduces compiler warnings by removing duplicate defines Line numbers are from r168404 (and r284648) #define lbolt: lines 384 and 459 (531 and 648) (original was renamed later) #define lbolt64: lines 385 and 460 (532 and 649) (original was renamed later) #define gethrestime_sec: lines 390 and 465 (540 and 653) uint64_t physmem: lines 402 and 463 (561 and 651) Modified: stable/10/cddl/contrib/opensolaris/lib/libzpool/common/sys/zfs_context.h Directory Properties: stable/10/ (props changed) Modified: stable/10/cddl/contrib/opensolaris/lib/libzpool/common/sys/zfs_context.h ============================================================================== --- stable/10/cddl/contrib/opensolaris/lib/libzpool/common/sys/zfs_context.h Sat Oct 3 11:24:47 2015 (r288592) +++ stable/10/cddl/contrib/opensolaris/lib/libzpool/common/sys/zfs_context.h Sat Oct 3 11:27:23 2015 (r288593) @@ -655,13 +655,6 @@ extern int zfs_secpolicy_rename_perms(co extern int zfs_secpolicy_destroy_perms(const char *name, cred_t *cr); extern zoneid_t getzoneid(void); /* Random compatibility stuff. */ -#define lbolt (gethrtime() >> 23) -#define lbolt64 (gethrtime() >> 23) - -extern uint64_t physmem; - -#define gethrestime_sec() time(NULL) - #define pwrite64(d, p, n, o) pwrite(d, p, n, o) #define readdir64(d) readdir(d) #define SIGPENDING(td) (0) From owner-svn-src-stable@freebsd.org Sat Oct 3 11:35:20 2015 Return-Path: Delivered-To: svn-src-stable@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 9A5C1A0FCD3; Sat, 3 Oct 2015 11:35:20 +0000 (UTC) (envelope-from mav@FreeBSD.org) Received: from repo.freebsd.org (repo.freebsd.org [IPv6:2001:1900:2254:2068::e6a:0]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 88ECA1830; Sat, 3 Oct 2015 11:35:20 +0000 (UTC) (envelope-from mav@FreeBSD.org) Received: from repo.freebsd.org ([127.0.1.70]) by repo.freebsd.org (8.15.2/8.15.2) with ESMTP id t93BZKuR070135; Sat, 3 Oct 2015 11:35:20 GMT (envelope-from mav@FreeBSD.org) Received: (from mav@localhost) by repo.freebsd.org (8.15.2/8.15.2/Submit) id t93BZJJI070130; Sat, 3 Oct 2015 11:35:19 GMT (envelope-from mav@FreeBSD.org) Message-Id: <201510031135.t93BZJJI070130@repo.freebsd.org> X-Authentication-Warning: repo.freebsd.org: mav set sender to mav@FreeBSD.org using -f From: Alexander Motin Date: Sat, 3 Oct 2015 11:35:19 +0000 (UTC) To: src-committers@freebsd.org, svn-src-all@freebsd.org, svn-src-stable@freebsd.org, svn-src-stable-10@freebsd.org Subject: svn commit: r288594 - in stable/10/sys/cddl/contrib/opensolaris/uts/common/fs/zfs: . sys X-SVN-Group: stable-10 MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit X-BeenThere: svn-src-stable@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: SVN commit messages for all the -stable branches of the src tree List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 03 Oct 2015 11:35:20 -0000 Author: mav Date: Sat Oct 3 11:35:18 2015 New Revision: 288594 URL: https://svnweb.freebsd.org/changeset/base/288594 Log: MFC r287702: 5987 zfs prefetch code needs work Rewrite the ZFS prefetch code to detect only forward, sequential streams. The following kstats have been added: kstat.zfs.misc.arcstats.sync_wait_for_async How many sync reads have waited for async read to complete. (less is better) kstat.zfs.misc.arcstats.demand_hit_predictive_prefetch How many demand read didn't have to wait for I/O because of predictive prefetch. (more is better) zfetch kstats have been similified to hits, misses, and max_streams, with max_streams representing times when we were not able to create new stream because we already have the maximum number of sequences for a file. The sysctl variable/loader tunable vfs.zfs.zfetch.block_cap have been replaced by vfs.zfs.zfetch.max_distance, which controls maximum bytes to prefetch per stream. illumos/illumos-gate@cf6106c8a0d6598b045811f9650d66e07eb332af Illumos ZFS issues: 5987 zfs prefetch code needs work https://www.illumos.org/issues/5987 Modified: stable/10/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/arc.c stable/10/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/dbuf.c stable/10/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/dmu.c stable/10/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/dmu_zfetch.c stable/10/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/dnode.c stable/10/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/sys/arc.h stable/10/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/sys/dmu.h stable/10/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/sys/dmu_zfetch.h Directory Properties: stable/10/ (props changed) Modified: stable/10/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/arc.c ============================================================================== --- stable/10/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/arc.c Sat Oct 3 11:27:23 2015 (r288593) +++ stable/10/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/arc.c Sat Oct 3 11:35:18 2015 (r288594) @@ -213,7 +213,7 @@ static int arc_min_prefetch_lifespan; int arc_lotsfree_percent = 10; static int arc_dead; -extern int zfs_prefetch_disable; +extern boolean_t zfs_prefetch_disable; /* * The arc has filled available memory and has now warmed up. @@ -585,6 +585,8 @@ typedef struct arc_stats { kstat_named_t arcstat_meta_limit; kstat_named_t arcstat_meta_max; kstat_named_t arcstat_meta_min; + kstat_named_t arcstat_sync_wait_for_async; + kstat_named_t arcstat_demand_hit_predictive_prefetch; } arc_stats_t; static arc_stats_t arc_stats = { @@ -683,7 +685,9 @@ static arc_stats_t arc_stats = { { "arc_meta_used", KSTAT_DATA_UINT64 }, { "arc_meta_limit", KSTAT_DATA_UINT64 }, { "arc_meta_max", KSTAT_DATA_UINT64 }, - { "arc_meta_min", KSTAT_DATA_UINT64 } + { "arc_meta_min", KSTAT_DATA_UINT64 }, + { "sync_wait_for_async", KSTAT_DATA_UINT64 }, + { "demand_hit_predictive_prefetch", KSTAT_DATA_UINT64 }, }; #define ARCSTAT(stat) (arc_stats.stat.value.ui64) @@ -4253,6 +4257,36 @@ top: if (HDR_IO_IN_PROGRESS(hdr)) { + if ((hdr->b_flags & ARC_FLAG_PRIO_ASYNC_READ) && + priority == ZIO_PRIORITY_SYNC_READ) { + /* + * This sync read must wait for an + * in-progress async read (e.g. a predictive + * prefetch). Async reads are queued + * separately at the vdev_queue layer, so + * this is a form of priority inversion. + * Ideally, we would "inherit" the demand + * i/o's priority by moving the i/o from + * the async queue to the synchronous queue, + * but there is currently no mechanism to do + * so. Track this so that we can evaluate + * the magnitude of this potential performance + * problem. + * + * Note that if the prefetch i/o is already + * active (has been issued to the device), + * the prefetch improved performance, because + * we issued it sooner than we would have + * without the prefetch. + */ + DTRACE_PROBE1(arc__sync__wait__for__async, + arc_buf_hdr_t *, hdr); + ARCSTAT_BUMP(arcstat_sync_wait_for_async); + } + if (hdr->b_flags & ARC_FLAG_PREDICTIVE_PREFETCH) { + hdr->b_flags &= ~ARC_FLAG_PREDICTIVE_PREFETCH; + } + if (*arc_flags & ARC_FLAG_WAIT) { cv_wait(&hdr->b_l1hdr.b_cv, hash_lock); mutex_exit(hash_lock); @@ -4261,7 +4295,7 @@ top: ASSERT(*arc_flags & ARC_FLAG_NOWAIT); if (done) { - arc_callback_t *acb = NULL; + arc_callback_t *acb = NULL; acb = kmem_zalloc(sizeof (arc_callback_t), KM_SLEEP); @@ -4286,6 +4320,19 @@ top: hdr->b_l1hdr.b_state == arc_mfu); if (done) { + if (hdr->b_flags & ARC_FLAG_PREDICTIVE_PREFETCH) { + /* + * This is a demand read which does not have to + * wait for i/o because we did a predictive + * prefetch i/o for it, which has completed. + */ + DTRACE_PROBE1( + arc__demand__hit__predictive__prefetch, + arc_buf_hdr_t *, hdr); + ARCSTAT_BUMP( + arcstat_demand_hit_predictive_prefetch); + hdr->b_flags &= ~ARC_FLAG_PREDICTIVE_PREFETCH; + } add_reference(hdr, hash_lock, private); /* * If this block is already in use, create a new @@ -4348,12 +4395,16 @@ top: goto top; /* restart the IO request */ } - /* if this is a prefetch, we don't have a reference */ - if (*arc_flags & ARC_FLAG_PREFETCH) { + /* + * If there is a callback, we pass our reference to + * it; otherwise we remove our reference. + */ + if (done == NULL) { (void) remove_reference(hdr, hash_lock, private); - hdr->b_flags |= ARC_FLAG_PREFETCH; } + if (*arc_flags & ARC_FLAG_PREFETCH) + hdr->b_flags |= ARC_FLAG_PREFETCH; if (*arc_flags & ARC_FLAG_L2CACHE) hdr->b_flags |= ARC_FLAG_L2CACHE; if (*arc_flags & ARC_FLAG_L2COMPRESS) @@ -4376,11 +4427,13 @@ top: ASSERT(refcount_is_zero(&hdr->b_l1hdr.b_refcnt)); ASSERT3P(hdr->b_l1hdr.b_buf, ==, NULL); - /* if this is a prefetch, we don't have a reference */ + /* + * If there is a callback, we pass a reference to it. + */ + if (done != NULL) + add_reference(hdr, hash_lock, private); if (*arc_flags & ARC_FLAG_PREFETCH) hdr->b_flags |= ARC_FLAG_PREFETCH; - else - add_reference(hdr, hash_lock, private); if (*arc_flags & ARC_FLAG_L2CACHE) hdr->b_flags |= ARC_FLAG_L2CACHE; if (*arc_flags & ARC_FLAG_L2COMPRESS) @@ -4398,6 +4451,8 @@ top: arc_access(hdr, hash_lock); } + if (*arc_flags & ARC_FLAG_PREDICTIVE_PREFETCH) + hdr->b_flags |= ARC_FLAG_PREDICTIVE_PREFETCH; ASSERT(!GHOST_STATE(hdr->b_l1hdr.b_state)); acb = kmem_zalloc(sizeof (arc_callback_t), KM_SLEEP); @@ -4440,6 +4495,11 @@ top: curthread->td_ru.ru_inblock++; #endif + if (priority == ZIO_PRIORITY_ASYNC_READ) + hdr->b_flags |= ARC_FLAG_PRIO_ASYNC_READ; + else + hdr->b_flags &= ~ARC_FLAG_PRIO_ASYNC_READ; + if (vd != NULL && l2arc_ndev != 0 && !(l2arc_norw && devw)) { /* * Read from the L2ARC if the following are true: Modified: stable/10/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/dbuf.c ============================================================================== --- stable/10/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/dbuf.c Sat Oct 3 11:27:23 2015 (r288593) +++ stable/10/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/dbuf.c Sat Oct 3 11:35:18 2015 (r288594) @@ -613,7 +613,7 @@ dbuf_read_done(zio_t *zio, arc_buf_t *bu } static void -dbuf_read_impl(dmu_buf_impl_t *db, zio_t *zio, uint32_t *flags) +dbuf_read_impl(dmu_buf_impl_t *db, zio_t *zio, uint32_t flags) { dnode_t *dn; zbookmark_phys_t zb; @@ -659,7 +659,6 @@ dbuf_read_impl(dmu_buf_impl_t *db, zio_t db->db.db_size, db, type)); bzero(db->db.db_data, db->db.db_size); db->db_state = DB_CACHED; - *flags |= DB_RF_CACHED; mutex_exit(&db->db_mtx); return; } @@ -682,10 +681,8 @@ dbuf_read_impl(dmu_buf_impl_t *db, zio_t (void) arc_read(zio, db->db_objset->os_spa, db->db_blkptr, dbuf_read_done, db, ZIO_PRIORITY_SYNC_READ, - (*flags & DB_RF_CANFAIL) ? ZIO_FLAG_CANFAIL : ZIO_FLAG_MUSTSUCCEED, + (flags & DB_RF_CANFAIL) ? ZIO_FLAG_CANFAIL : ZIO_FLAG_MUSTSUCCEED, &aflags, &zb); - if (aflags & ARC_FLAG_CACHED) - *flags |= DB_RF_CACHED; } int @@ -718,8 +715,7 @@ dbuf_read(dmu_buf_impl_t *db, zio_t *zio if (db->db_state == DB_CACHED) { mutex_exit(&db->db_mtx); if (prefetch) - dmu_zfetch(&dn->dn_zfetch, db->db.db_offset, - db->db.db_size, TRUE); + dmu_zfetch(&dn->dn_zfetch, db->db_blkid, 1); if ((flags & DB_RF_HAVESTRUCT) == 0) rw_exit(&dn->dn_struct_rwlock); DB_DNODE_EXIT(db); @@ -728,13 +724,12 @@ dbuf_read(dmu_buf_impl_t *db, zio_t *zio if (zio == NULL) zio = zio_root(spa, NULL, NULL, ZIO_FLAG_CANFAIL); - dbuf_read_impl(db, zio, &flags); + dbuf_read_impl(db, zio, flags); /* dbuf_read_impl has dropped db_mtx for us */ if (prefetch) - dmu_zfetch(&dn->dn_zfetch, db->db.db_offset, - db->db.db_size, flags & DB_RF_CACHED); + dmu_zfetch(&dn->dn_zfetch, db->db_blkid, 1); if ((flags & DB_RF_HAVESTRUCT) == 0) rw_exit(&dn->dn_struct_rwlock); @@ -753,8 +748,7 @@ dbuf_read(dmu_buf_impl_t *db, zio_t *zio */ mutex_exit(&db->db_mtx); if (prefetch) - dmu_zfetch(&dn->dn_zfetch, db->db.db_offset, - db->db.db_size, TRUE); + dmu_zfetch(&dn->dn_zfetch, db->db_blkid, 1); if ((flags & DB_RF_HAVESTRUCT) == 0) rw_exit(&dn->dn_struct_rwlock); DB_DNODE_EXIT(db); @@ -2054,6 +2048,9 @@ dbuf_prefetch(dnode_t *dn, int64_t level ASSERT(blkid != DMU_BONUS_BLKID); ASSERT(RW_LOCK_HELD(&dn->dn_struct_rwlock)); + if (blkid > dn->dn_maxblkid) + return; + if (dnode_block_freed(dn, blkid)) return; Modified: stable/10/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/dmu.c ============================================================================== --- stable/10/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/dmu.c Sat Oct 3 11:27:23 2015 (r288593) +++ stable/10/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/dmu.c Sat Oct 3 11:35:18 2015 (r288594) @@ -20,7 +20,7 @@ */ /* * Copyright (c) 2005, 2010, Oracle and/or its affiliates. All rights reserved. - * Copyright (c) 2011, 2014 by Delphix. All rights reserved. + * Copyright (c) 2011, 2015 by Delphix. All rights reserved. */ /* Copyright (c) 2013 by Saso Kiselkov. All rights reserved. */ /* Copyright (c) 2013, Joyent, Inc. All rights reserved. */ @@ -390,7 +390,7 @@ dmu_spill_hold_by_bonus(dmu_buf_t *bonus */ static int dmu_buf_hold_array_by_dnode(dnode_t *dn, uint64_t offset, uint64_t length, - int read, void *tag, int *numbufsp, dmu_buf_t ***dbpp, uint32_t flags) + boolean_t read, void *tag, int *numbufsp, dmu_buf_t ***dbpp, uint32_t flags) { dmu_buf_t **dbp; uint64_t blkid, nblks, i; @@ -400,15 +400,19 @@ dmu_buf_hold_array_by_dnode(dnode_t *dn, ASSERT(length <= DMU_MAX_ACCESS); - dbuf_flags = DB_RF_CANFAIL | DB_RF_NEVERWAIT | DB_RF_HAVESTRUCT; - if (flags & DMU_READ_NO_PREFETCH || length > zfetch_array_rd_sz) - dbuf_flags |= DB_RF_NOPREFETCH; + /* + * Note: We directly notify the prefetch code of this read, so that + * we can tell it about the multi-block read. dbuf_read() only knows + * about the one block it is accessing. + */ + dbuf_flags = DB_RF_CANFAIL | DB_RF_NEVERWAIT | DB_RF_HAVESTRUCT | + DB_RF_NOPREFETCH; rw_enter(&dn->dn_struct_rwlock, RW_READER); if (dn->dn_datablkshift) { int blkshift = dn->dn_datablkshift; - nblks = (P2ROUNDUP(offset+length, 1ULL<> blkshift; + nblks = (P2ROUNDUP(offset + length, 1ULL << blkshift) - + P2ALIGN(offset, 1ULL << blkshift)) >> blkshift; } else { if (offset + length > dn->dn_datablksz) { zfs_panic_recover("zfs: accessing past end of object " @@ -427,13 +431,14 @@ dmu_buf_hold_array_by_dnode(dnode_t *dn, zio = zio_root(dn->dn_objset->os_spa, NULL, NULL, ZIO_FLAG_CANFAIL); blkid = dbuf_whichblock(dn, 0, offset); for (i = 0; i < nblks; i++) { - dmu_buf_impl_t *db = dbuf_hold(dn, blkid+i, tag); + dmu_buf_impl_t *db = dbuf_hold(dn, blkid + i, tag); if (db == NULL) { rw_exit(&dn->dn_struct_rwlock); dmu_buf_rele_array(dbp, nblks, tag); zio_nowait(zio); return (SET_ERROR(EIO)); } + /* initiate async i/o */ if (read) (void) dbuf_read(db, zio, dbuf_flags); @@ -443,6 +448,11 @@ dmu_buf_hold_array_by_dnode(dnode_t *dn, #endif dbp[i] = &db->db; } + + if ((flags & DMU_READ_NO_PREFETCH) == 0 && read && + length < zfetch_array_rd_sz) { + dmu_zfetch(&dn->dn_zfetch, blkid, nblks); + } rw_exit(&dn->dn_struct_rwlock); /* wait for async i/o */ @@ -496,7 +506,8 @@ dmu_buf_hold_array(objset_t *os, uint64_ int dmu_buf_hold_array_by_bonus(dmu_buf_t *db_fake, uint64_t offset, - uint64_t length, int read, void *tag, int *numbufsp, dmu_buf_t ***dbpp) + uint64_t length, boolean_t read, void *tag, int *numbufsp, + dmu_buf_t ***dbpp) { dmu_buf_impl_t *db = (dmu_buf_impl_t *)db_fake; dnode_t *dn; @@ -544,9 +555,6 @@ dmu_prefetch(objset_t *os, uint64_t obje uint64_t blkid; int nblks, err; - if (zfs_prefetch_disable) - return; - if (len == 0) { /* they're interested in the bonus buffer */ dn = DMU_META_DNODE(os); Modified: stable/10/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/dmu_zfetch.c ============================================================================== --- stable/10/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/dmu_zfetch.c Sat Oct 3 11:27:23 2015 (r288593) +++ stable/10/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/dmu_zfetch.c Sat Oct 3 11:35:18 2015 (r288594) @@ -24,7 +24,7 @@ */ /* - * Copyright (c) 2013 by Delphix. All rights reserved. + * Copyright (c) 2013, 2014 by Delphix. All rights reserved. */ #include @@ -36,19 +36,20 @@ #include /* - * I'm against tune-ables, but these should probably exist as tweakable globals - * until we can get this working the way we want it to. + * This tunable disables predictive prefetch. Note that it leaves "prescient" + * prefetch (e.g. prefetch for zfs send) intact. Unlike predictive prefetch, + * prescient prefetch never issues i/os that end up not being needed, + * so it can't hurt performance. */ - -int zfs_prefetch_disable = 0; +boolean_t zfs_prefetch_disable = B_FALSE; /* max # of streams per zfetch */ uint32_t zfetch_max_streams = 8; /* min time before stream reclaim */ uint32_t zfetch_min_sec_reap = 2; -/* max number of blocks to fetch at a time */ -uint32_t zfetch_block_cap = 256; -/* number of bytes in a array_read at which we stop prefetching (1Mb) */ +/* max bytes to prefetch per stream (default 8MB) */ +uint32_t zfetch_max_distance = 8 * 1024 * 1024; +/* number of bytes in a array_read at which we stop prefetching (1MB) */ uint64_t zfetch_array_rd_sz = 1024 * 1024; SYSCTL_DECL(_vfs_zfs); @@ -61,200 +62,34 @@ SYSCTL_UINT(_vfs_zfs_zfetch, OID_AUTO, m TUNABLE_INT("vfs.zfs.zfetch.min_sec_reap", &zfetch_min_sec_reap); SYSCTL_UINT(_vfs_zfs_zfetch, OID_AUTO, min_sec_reap, CTLFLAG_RWTUN, &zfetch_min_sec_reap, 0, "Min time before stream reclaim"); -TUNABLE_INT("vfs.zfs.zfetch.block_cap", &zfetch_block_cap); -SYSCTL_UINT(_vfs_zfs_zfetch, OID_AUTO, block_cap, CTLFLAG_RWTUN, - &zfetch_block_cap, 0, "Max number of blocks to fetch at a time"); +TUNABLE_INT("vfs.zfs.zfetch.max_distance", &zfetch_max_distance); +SYSCTL_UINT(_vfs_zfs_zfetch, OID_AUTO, max_distance, CTLFLAG_RWTUN, + &zfetch_max_distance, 0, "Max bytes to prefetch per stream"); TUNABLE_QUAD("vfs.zfs.zfetch.array_rd_sz", &zfetch_array_rd_sz); SYSCTL_UQUAD(_vfs_zfs_zfetch, OID_AUTO, array_rd_sz, CTLFLAG_RWTUN, &zfetch_array_rd_sz, 0, "Number of bytes in a array_read at which we stop prefetching"); -/* forward decls for static routines */ -static boolean_t dmu_zfetch_colinear(zfetch_t *, zstream_t *); -static void dmu_zfetch_dofetch(zfetch_t *, zstream_t *); -static uint64_t dmu_zfetch_fetch(dnode_t *, uint64_t, uint64_t); -static uint64_t dmu_zfetch_fetchsz(dnode_t *, uint64_t, uint64_t); -static boolean_t dmu_zfetch_find(zfetch_t *, zstream_t *, int); -static int dmu_zfetch_stream_insert(zfetch_t *, zstream_t *); -static zstream_t *dmu_zfetch_stream_reclaim(zfetch_t *); -static void dmu_zfetch_stream_remove(zfetch_t *, zstream_t *); -static int dmu_zfetch_streams_equal(zstream_t *, zstream_t *); - typedef struct zfetch_stats { kstat_named_t zfetchstat_hits; kstat_named_t zfetchstat_misses; - kstat_named_t zfetchstat_colinear_hits; - kstat_named_t zfetchstat_colinear_misses; - kstat_named_t zfetchstat_stride_hits; - kstat_named_t zfetchstat_stride_misses; - kstat_named_t zfetchstat_reclaim_successes; - kstat_named_t zfetchstat_reclaim_failures; - kstat_named_t zfetchstat_stream_resets; - kstat_named_t zfetchstat_stream_noresets; - kstat_named_t zfetchstat_bogus_streams; + kstat_named_t zfetchstat_max_streams; } zfetch_stats_t; static zfetch_stats_t zfetch_stats = { { "hits", KSTAT_DATA_UINT64 }, { "misses", KSTAT_DATA_UINT64 }, - { "colinear_hits", KSTAT_DATA_UINT64 }, - { "colinear_misses", KSTAT_DATA_UINT64 }, - { "stride_hits", KSTAT_DATA_UINT64 }, - { "stride_misses", KSTAT_DATA_UINT64 }, - { "reclaim_successes", KSTAT_DATA_UINT64 }, - { "reclaim_failures", KSTAT_DATA_UINT64 }, - { "streams_resets", KSTAT_DATA_UINT64 }, - { "streams_noresets", KSTAT_DATA_UINT64 }, - { "bogus_streams", KSTAT_DATA_UINT64 }, + { "max_streams", KSTAT_DATA_UINT64 }, }; -#define ZFETCHSTAT_INCR(stat, val) \ - atomic_add_64(&zfetch_stats.stat.value.ui64, (val)); - -#define ZFETCHSTAT_BUMP(stat) ZFETCHSTAT_INCR(stat, 1); +#define ZFETCHSTAT_BUMP(stat) \ + atomic_inc_64(&zfetch_stats.stat.value.ui64); kstat_t *zfetch_ksp; -/* - * Given a zfetch structure and a zstream structure, determine whether the - * blocks to be read are part of a co-linear pair of existing prefetch - * streams. If a set is found, coalesce the streams, removing one, and - * configure the prefetch so it looks for a strided access pattern. - * - * In other words: if we find two sequential access streams that are - * the same length and distance N appart, and this read is N from the - * last stream, then we are probably in a strided access pattern. So - * combine the two sequential streams into a single strided stream. - * - * Returns whether co-linear streams were found. - */ -static boolean_t -dmu_zfetch_colinear(zfetch_t *zf, zstream_t *zh) -{ - zstream_t *z_walk; - zstream_t *z_comp; - - if (! rw_tryenter(&zf->zf_rwlock, RW_WRITER)) - return (0); - - if (zh == NULL) { - rw_exit(&zf->zf_rwlock); - return (0); - } - - for (z_walk = list_head(&zf->zf_stream); z_walk; - z_walk = list_next(&zf->zf_stream, z_walk)) { - for (z_comp = list_next(&zf->zf_stream, z_walk); z_comp; - z_comp = list_next(&zf->zf_stream, z_comp)) { - int64_t diff; - - if (z_walk->zst_len != z_walk->zst_stride || - z_comp->zst_len != z_comp->zst_stride) { - continue; - } - - diff = z_comp->zst_offset - z_walk->zst_offset; - if (z_comp->zst_offset + diff == zh->zst_offset) { - z_walk->zst_offset = zh->zst_offset; - z_walk->zst_direction = diff < 0 ? -1 : 1; - z_walk->zst_stride = - diff * z_walk->zst_direction; - z_walk->zst_ph_offset = - zh->zst_offset + z_walk->zst_stride; - dmu_zfetch_stream_remove(zf, z_comp); - mutex_destroy(&z_comp->zst_lock); - kmem_free(z_comp, sizeof (zstream_t)); - - dmu_zfetch_dofetch(zf, z_walk); - - rw_exit(&zf->zf_rwlock); - return (1); - } - - diff = z_walk->zst_offset - z_comp->zst_offset; - if (z_walk->zst_offset + diff == zh->zst_offset) { - z_walk->zst_offset = zh->zst_offset; - z_walk->zst_direction = diff < 0 ? -1 : 1; - z_walk->zst_stride = - diff * z_walk->zst_direction; - z_walk->zst_ph_offset = - zh->zst_offset + z_walk->zst_stride; - dmu_zfetch_stream_remove(zf, z_comp); - mutex_destroy(&z_comp->zst_lock); - kmem_free(z_comp, sizeof (zstream_t)); - - dmu_zfetch_dofetch(zf, z_walk); - - rw_exit(&zf->zf_rwlock); - return (1); - } - } - } - - rw_exit(&zf->zf_rwlock); - return (0); -} - -/* - * Given a zstream_t, determine the bounds of the prefetch. Then call the - * routine that actually prefetches the individual blocks. - */ -static void -dmu_zfetch_dofetch(zfetch_t *zf, zstream_t *zs) -{ - uint64_t prefetch_tail; - uint64_t prefetch_limit; - uint64_t prefetch_ofst; - uint64_t prefetch_len; - uint64_t blocks_fetched; - - zs->zst_stride = MAX((int64_t)zs->zst_stride, zs->zst_len); - zs->zst_cap = MIN(zfetch_block_cap, 2 * zs->zst_cap); - - prefetch_tail = MAX((int64_t)zs->zst_ph_offset, - (int64_t)(zs->zst_offset + zs->zst_stride)); - /* - * XXX: use a faster division method? - */ - prefetch_limit = zs->zst_offset + zs->zst_len + - (zs->zst_cap * zs->zst_stride) / zs->zst_len; - - while (prefetch_tail < prefetch_limit) { - prefetch_ofst = zs->zst_offset + zs->zst_direction * - (prefetch_tail - zs->zst_offset); - - prefetch_len = zs->zst_len; - - /* - * Don't prefetch beyond the end of the file, if working - * backwards. - */ - if ((zs->zst_direction == ZFETCH_BACKWARD) && - (prefetch_ofst > prefetch_tail)) { - prefetch_len += prefetch_ofst; - prefetch_ofst = 0; - } - - /* don't prefetch more than we're supposed to */ - if (prefetch_len > zs->zst_len) - break; - - blocks_fetched = dmu_zfetch_fetch(zf->zf_dnode, - prefetch_ofst, zs->zst_len); - - prefetch_tail += zs->zst_stride; - /* stop if we've run out of stuff to prefetch */ - if (blocks_fetched < zs->zst_len) - break; - } - zs->zst_ph_offset = prefetch_tail; - zs->zst_last = ddi_get_lbolt(); -} - void zfetch_init(void) { - zfetch_ksp = kstat_create("zfs", 0, "zfetchstats", "misc", KSTAT_TYPE_NAMED, sizeof (zfetch_stats) / sizeof (kstat_named_t), KSTAT_FLAG_VIRTUAL); @@ -282,285 +117,41 @@ zfetch_fini(void) void dmu_zfetch_init(zfetch_t *zf, dnode_t *dno) { - if (zf == NULL) { + if (zf == NULL) return; - } zf->zf_dnode = dno; - zf->zf_stream_cnt = 0; - zf->zf_alloc_fail = 0; list_create(&zf->zf_stream, sizeof (zstream_t), - offsetof(zstream_t, zst_node)); + offsetof(zstream_t, zs_node)); rw_init(&zf->zf_rwlock, NULL, RW_DEFAULT, NULL); } -/* - * This function computes the actual size, in blocks, that can be prefetched, - * and fetches it. - */ -static uint64_t -dmu_zfetch_fetch(dnode_t *dn, uint64_t blkid, uint64_t nblks) -{ - uint64_t fetchsz; - uint64_t i; - - fetchsz = dmu_zfetch_fetchsz(dn, blkid, nblks); - - for (i = 0; i < fetchsz; i++) { - dbuf_prefetch(dn, 0, blkid + i, ZIO_PRIORITY_ASYNC_READ, - ARC_FLAG_PREFETCH); - } - - return (fetchsz); -} - -/* - * this function returns the number of blocks that would be prefetched, based - * upon the supplied dnode, blockid, and nblks. This is used so that we can - * update streams in place, and then prefetch with their old value after the - * fact. This way, we can delay the prefetch, but subsequent accesses to the - * stream won't result in the same data being prefetched multiple times. - */ -static uint64_t -dmu_zfetch_fetchsz(dnode_t *dn, uint64_t blkid, uint64_t nblks) -{ - uint64_t fetchsz; - - if (blkid > dn->dn_maxblkid) { - return (0); - } - - /* compute fetch size */ - if (blkid + nblks + 1 > dn->dn_maxblkid) { - fetchsz = (dn->dn_maxblkid - blkid) + 1; - ASSERT(blkid + fetchsz - 1 <= dn->dn_maxblkid); - } else { - fetchsz = nblks; - } - - - return (fetchsz); -} - -/* - * given a zfetch and a zstream structure, see if there is an associated zstream - * for this block read. If so, it starts a prefetch for the stream it - * located and returns true, otherwise it returns false - */ -static boolean_t -dmu_zfetch_find(zfetch_t *zf, zstream_t *zh, int prefetched) +static void +dmu_zfetch_stream_remove(zfetch_t *zf, zstream_t *zs) { - zstream_t *zs; - int64_t diff; - int reset = !prefetched; - int rc = 0; - - if (zh == NULL) - return (0); - - /* - * XXX: This locking strategy is a bit coarse; however, it's impact has - * yet to be tested. If this turns out to be an issue, it can be - * modified in a number of different ways. - */ - - rw_enter(&zf->zf_rwlock, RW_READER); -top: - - for (zs = list_head(&zf->zf_stream); zs; - zs = list_next(&zf->zf_stream, zs)) { - - /* - * XXX - should this be an assert? - */ - if (zs->zst_len == 0) { - /* bogus stream */ - ZFETCHSTAT_BUMP(zfetchstat_bogus_streams); - continue; - } - - /* - * We hit this case when we are in a strided prefetch stream: - * we will read "len" blocks before "striding". - */ - if (zh->zst_offset >= zs->zst_offset && - zh->zst_offset < zs->zst_offset + zs->zst_len) { - if (prefetched) { - /* already fetched */ - ZFETCHSTAT_BUMP(zfetchstat_stride_hits); - rc = 1; - goto out; - } else { - ZFETCHSTAT_BUMP(zfetchstat_stride_misses); - } - } - - /* - * This is the forward sequential read case: we increment - * len by one each time we hit here, so we will enter this - * case on every read. - */ - if (zh->zst_offset == zs->zst_offset + zs->zst_len) { - - reset = !prefetched && zs->zst_len > 1; - - if (mutex_tryenter(&zs->zst_lock) == 0) { - rc = 1; - goto out; - } - - if (zh->zst_offset != zs->zst_offset + zs->zst_len) { - mutex_exit(&zs->zst_lock); - goto top; - } - zs->zst_len += zh->zst_len; - diff = zs->zst_len - zfetch_block_cap; - if (diff > 0) { - zs->zst_offset += diff; - zs->zst_len = zs->zst_len > diff ? - zs->zst_len - diff : 0; - } - zs->zst_direction = ZFETCH_FORWARD; - - break; - - /* - * Same as above, but reading backwards through the file. - */ - } else if (zh->zst_offset == zs->zst_offset - zh->zst_len) { - /* backwards sequential access */ - - reset = !prefetched && zs->zst_len > 1; - - if (mutex_tryenter(&zs->zst_lock) == 0) { - rc = 1; - goto out; - } - - if (zh->zst_offset != zs->zst_offset - zh->zst_len) { - mutex_exit(&zs->zst_lock); - goto top; - } - - zs->zst_offset = zs->zst_offset > zh->zst_len ? - zs->zst_offset - zh->zst_len : 0; - zs->zst_ph_offset = zs->zst_ph_offset > zh->zst_len ? - zs->zst_ph_offset - zh->zst_len : 0; - zs->zst_len += zh->zst_len; - - diff = zs->zst_len - zfetch_block_cap; - if (diff > 0) { - zs->zst_ph_offset = zs->zst_ph_offset > diff ? - zs->zst_ph_offset - diff : 0; - zs->zst_len = zs->zst_len > diff ? - zs->zst_len - diff : zs->zst_len; - } - zs->zst_direction = ZFETCH_BACKWARD; - - break; - - } else if ((zh->zst_offset - zs->zst_offset - zs->zst_stride < - zs->zst_len) && (zs->zst_len != zs->zst_stride)) { - /* strided forward access */ - - if (mutex_tryenter(&zs->zst_lock) == 0) { - rc = 1; - goto out; - } - - if ((zh->zst_offset - zs->zst_offset - zs->zst_stride >= - zs->zst_len) || (zs->zst_len == zs->zst_stride)) { - mutex_exit(&zs->zst_lock); - goto top; - } - - zs->zst_offset += zs->zst_stride; - zs->zst_direction = ZFETCH_FORWARD; - - break; - - } else if ((zh->zst_offset - zs->zst_offset + zs->zst_stride < - zs->zst_len) && (zs->zst_len != zs->zst_stride)) { - /* strided reverse access */ - - if (mutex_tryenter(&zs->zst_lock) == 0) { - rc = 1; - goto out; - } - - if ((zh->zst_offset - zs->zst_offset + zs->zst_stride >= - zs->zst_len) || (zs->zst_len == zs->zst_stride)) { - mutex_exit(&zs->zst_lock); - goto top; - } - - zs->zst_offset = zs->zst_offset > zs->zst_stride ? - zs->zst_offset - zs->zst_stride : 0; - zs->zst_ph_offset = (zs->zst_ph_offset > - (2 * zs->zst_stride)) ? - (zs->zst_ph_offset - (2 * zs->zst_stride)) : 0; - zs->zst_direction = ZFETCH_BACKWARD; - - break; - } - } - - if (zs) { - if (reset) { - zstream_t *remove = zs; - - ZFETCHSTAT_BUMP(zfetchstat_stream_resets); - rc = 0; - mutex_exit(&zs->zst_lock); - rw_exit(&zf->zf_rwlock); - rw_enter(&zf->zf_rwlock, RW_WRITER); - /* - * Relocate the stream, in case someone removes - * it while we were acquiring the WRITER lock. - */ - for (zs = list_head(&zf->zf_stream); zs; - zs = list_next(&zf->zf_stream, zs)) { - if (zs == remove) { - dmu_zfetch_stream_remove(zf, zs); - mutex_destroy(&zs->zst_lock); - kmem_free(zs, sizeof (zstream_t)); - break; - } - } - } else { - ZFETCHSTAT_BUMP(zfetchstat_stream_noresets); - rc = 1; - dmu_zfetch_dofetch(zf, zs); - mutex_exit(&zs->zst_lock); - } - } -out: - rw_exit(&zf->zf_rwlock); - return (rc); + ASSERT(RW_WRITE_HELD(&zf->zf_rwlock)); + list_remove(&zf->zf_stream, zs); + mutex_destroy(&zs->zs_lock); + kmem_free(zs, sizeof (*zs)); } /* - * Clean-up state associated with a zfetch structure. This frees allocated - * structure members, empties the zf_stream tree, and generally makes things - * nice. This doesn't free the zfetch_t itself, that's left to the caller. + * Clean-up state associated with a zfetch structure (e.g. destroy the + * streams). This doesn't free the zfetch_t itself, that's left to the caller. */ void -dmu_zfetch_rele(zfetch_t *zf) +dmu_zfetch_fini(zfetch_t *zf) { - zstream_t *zs; - zstream_t *zs_next; + zstream_t *zs; ASSERT(!RW_LOCK_HELD(&zf->zf_rwlock)); - for (zs = list_head(&zf->zf_stream); zs; zs = zs_next) { - zs_next = list_next(&zf->zf_stream, zs); - - list_remove(&zf->zf_stream, zs); - mutex_destroy(&zs->zst_lock); - kmem_free(zs, sizeof (zstream_t)); - } + rw_enter(&zf->zf_rwlock, RW_WRITER); + while ((zs = list_head(&zf->zf_stream)) != NULL) + dmu_zfetch_stream_remove(zf, zs); + rw_exit(&zf->zf_rwlock); list_destroy(&zf->zf_stream); rw_destroy(&zf->zf_rwlock); @@ -568,103 +159,55 @@ dmu_zfetch_rele(zfetch_t *zf) } /* - * Given a zfetch and zstream structure, insert the zstream structure into the - * AVL tree contained within the zfetch structure. Peform the appropriate - * book-keeping. It is possible that another thread has inserted a stream which - * matches one that we are about to insert, so we must be sure to check for this - * case. If one is found, return failure, and let the caller cleanup the - * duplicates. + * If there aren't too many streams already, create a new stream. + * The "blkid" argument is the next block that we expect this stream to access. + * While we're here, clean up old streams (which haven't been + * accessed for at least zfetch_min_sec_reap seconds). */ -static int -dmu_zfetch_stream_insert(zfetch_t *zf, zstream_t *zs) +static void +dmu_zfetch_stream_create(zfetch_t *zf, uint64_t blkid) { - zstream_t *zs_walk; - zstream_t *zs_next; + zstream_t *zs_next; + int numstreams = 0; ASSERT(RW_WRITE_HELD(&zf->zf_rwlock)); - for (zs_walk = list_head(&zf->zf_stream); zs_walk; zs_walk = zs_next) { - zs_next = list_next(&zf->zf_stream, zs_walk); - - if (dmu_zfetch_streams_equal(zs_walk, zs)) { - return (0); - } - } - - list_insert_head(&zf->zf_stream, zs); - zf->zf_stream_cnt++; - return (1); -} - - -/* - * Walk the list of zstreams in the given zfetch, find an old one (by time), and - * reclaim it for use by the caller. - */ -static zstream_t * -dmu_zfetch_stream_reclaim(zfetch_t *zf) -{ - zstream_t *zs; - clock_t ticks; - - ticks = zfetch_min_sec_reap * hz; - if (! rw_tryenter(&zf->zf_rwlock, RW_WRITER)) - return (0); - - for (zs = list_head(&zf->zf_stream); zs; - zs = list_next(&zf->zf_stream, zs)) { - - if (ddi_get_lbolt() - zs->zst_last > ticks) - break; + /* + * Clean up old streams. + */ + for (zstream_t *zs = list_head(&zf->zf_stream); + zs != NULL; zs = zs_next) { + zs_next = list_next(&zf->zf_stream, zs); + if (((gethrtime() - zs->zs_atime) / NANOSEC) > + zfetch_min_sec_reap) + dmu_zfetch_stream_remove(zf, zs); + else + numstreams++; } - if (zs) { - dmu_zfetch_stream_remove(zf, zs); - mutex_destroy(&zs->zst_lock); - bzero(zs, sizeof (zstream_t)); - } else { - zf->zf_alloc_fail++; + /* + * The maximum number of streams is normally zfetch_max_streams, + * but for small files we lower it such that it's at least possible + * for all the streams to be non-overlapping. + * + * If we are already at the maximum number of streams for this file, + * even after removing old streams, then don't create this stream. + */ + uint32_t max_streams = MAX(1, MIN(zfetch_max_streams, + zf->zf_dnode->dn_maxblkid * zf->zf_dnode->dn_datablksz / + zfetch_max_distance)); + if (numstreams >= max_streams) { + ZFETCHSTAT_BUMP(zfetchstat_max_streams); + return; } - rw_exit(&zf->zf_rwlock); - - return (zs); -} - -/* - * Given a zfetch and zstream structure, remove the zstream structure from its - * container in the zfetch structure. Perform the appropriate book-keeping. - */ -static void -dmu_zfetch_stream_remove(zfetch_t *zf, zstream_t *zs) -{ - ASSERT(RW_WRITE_HELD(&zf->zf_rwlock)); - - list_remove(&zf->zf_stream, zs); - zf->zf_stream_cnt--; -} - -static int -dmu_zfetch_streams_equal(zstream_t *zs1, zstream_t *zs2) -{ - if (zs1->zst_offset != zs2->zst_offset) - return (0); - - if (zs1->zst_len != zs2->zst_len) - return (0); - if (zs1->zst_stride != zs2->zst_stride) - return (0); + zstream_t *zs = kmem_zalloc(sizeof (*zs), KM_SLEEP); + zs->zs_blkid = blkid; + zs->zs_pf_blkid = blkid; *** DIFF OUTPUT TRUNCATED AT 1000 LINES *** From owner-svn-src-stable@freebsd.org Sat Oct 3 11:37:01 2015 Return-Path: Delivered-To: svn-src-stable@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id C906BA0FE97; Sat, 3 Oct 2015 11:37:01 +0000 (UTC) (envelope-from mav@FreeBSD.org) Received: from repo.freebsd.org (repo.freebsd.org [IPv6:2001:1900:2254:2068::e6a:0]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 9ECF219D6; Sat, 3 Oct 2015 11:37:01 +0000 (UTC) (envelope-from mav@FreeBSD.org) Received: from repo.freebsd.org ([127.0.1.70]) by repo.freebsd.org (8.15.2/8.15.2) with ESMTP id t93Bb1Iq070266; Sat, 3 Oct 2015 11:37:01 GMT (envelope-from mav@FreeBSD.org) Received: (from mav@localhost) by repo.freebsd.org (8.15.2/8.15.2/Submit) id t93Bb1fB070264; Sat, 3 Oct 2015 11:37:01 GMT (envelope-from mav@FreeBSD.org) Message-Id: <201510031137.t93Bb1fB070264@repo.freebsd.org> X-Authentication-Warning: repo.freebsd.org: mav set sender to mav@FreeBSD.org using -f From: Alexander Motin Date: Sat, 3 Oct 2015 11:37:01 +0000 (UTC) To: src-committers@freebsd.org, svn-src-all@freebsd.org, svn-src-stable@freebsd.org, svn-src-stable-10@freebsd.org Subject: svn commit: r288595 - in stable/10: cddl/contrib/opensolaris/cmd/sgs/tools/common sys/cddl/contrib/opensolaris/common/avl X-SVN-Group: stable-10 MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit X-BeenThere: svn-src-stable@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: SVN commit messages for all the -stable branches of the src tree List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 03 Oct 2015 11:37:01 -0000 Author: mav Date: Sat Oct 3 11:37:00 2015 New Revision: 288595 URL: https://svnweb.freebsd.org/changeset/base/288595 Log: MFV r287703, r287705 (by delphij): 6091 avl_add doesn't assert on non-debug builds Use assfail() from libuutil instead of ASSERT() in userland AVL avl_add. illumos/illumos-gate@faa2b6be2fc102adf9ed584fc1a667b4ddf50d78 Illumos issues: 6091 avl_add doesn't assert on non-debug builds https://www.illumos.org/issues/6091 Modified: stable/10/cddl/contrib/opensolaris/cmd/sgs/tools/common/sgsmsg.c stable/10/sys/cddl/contrib/opensolaris/common/avl/avl.c Directory Properties: stable/10/ (props changed) Modified: stable/10/cddl/contrib/opensolaris/cmd/sgs/tools/common/sgsmsg.c ============================================================================== --- stable/10/cddl/contrib/opensolaris/cmd/sgs/tools/common/sgsmsg.c Sat Oct 3 11:35:18 2015 (r288594) +++ stable/10/cddl/contrib/opensolaris/cmd/sgs/tools/common/sgsmsg.c Sat Oct 3 11:37:00 2015 (r288595) @@ -132,6 +132,8 @@ typedef struct msg_string { static msg_string *msg_head; static msg_string *msg_tail; +int aok; + /* * message_append() is responsible for both inserting strings into * the master Str_tbl as well as maintaining a list of the Modified: stable/10/sys/cddl/contrib/opensolaris/common/avl/avl.c ============================================================================== --- stable/10/sys/cddl/contrib/opensolaris/common/avl/avl.c Sat Oct 3 11:35:18 2015 (r288594) +++ stable/10/sys/cddl/contrib/opensolaris/common/avl/avl.c Sat Oct 3 11:37:00 2015 (r288595) @@ -25,6 +25,7 @@ /* * Copyright (c) 2014 by Delphix. All rights reserved. + * Copyright 2015 Nexenta Systems, Inc. All rights reserved. */ /* @@ -635,14 +636,17 @@ avl_add(avl_tree_t *tree, void *new_node /* * This is unfortunate. We want to call panic() here, even for * non-DEBUG kernels. In userland, however, we can't depend on anything - * in libc or else the rtld build process gets confused. So, all we can - * do in userland is resort to a normal ASSERT(). + * in libc or else the rtld build process gets confused. + * Thankfully, rtld provides us with its own assfail() so we can use + * that here. We use assfail() directly to get a nice error message + * in the core - much like what panic() does for crashdumps. */ if (avl_find(tree, new_node, &where) != NULL) #ifdef _KERNEL panic("avl_find() succeeded inside avl_add()"); #else - ASSERT(0); + (void) assfail("avl_find() succeeded inside avl_add()", + __FILE__, __LINE__); #endif avl_insert(tree, new_node, where); } From owner-svn-src-stable@freebsd.org Sat Oct 3 11:38:11 2015 Return-Path: Delivered-To: svn-src-stable@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 85AC5A0FF88; Sat, 3 Oct 2015 11:38:11 +0000 (UTC) (envelope-from mav@FreeBSD.org) Received: from repo.freebsd.org (repo.freebsd.org [IPv6:2001:1900:2254:2068::e6a:0]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 756201B5D; Sat, 3 Oct 2015 11:38:11 +0000 (UTC) (envelope-from mav@FreeBSD.org) Received: from repo.freebsd.org ([127.0.1.70]) by repo.freebsd.org (8.15.2/8.15.2) with ESMTP id t93BcB0L070366; Sat, 3 Oct 2015 11:38:11 GMT (envelope-from mav@FreeBSD.org) Received: (from mav@localhost) by repo.freebsd.org (8.15.2/8.15.2/Submit) id t93BcBEi070364; Sat, 3 Oct 2015 11:38:11 GMT (envelope-from mav@FreeBSD.org) Message-Id: <201510031138.t93BcBEi070364@repo.freebsd.org> X-Authentication-Warning: repo.freebsd.org: mav set sender to mav@FreeBSD.org using -f From: Alexander Motin Date: Sat, 3 Oct 2015 11:38:11 +0000 (UTC) To: src-committers@freebsd.org, svn-src-all@freebsd.org, svn-src-stable@freebsd.org, svn-src-stable-10@freebsd.org Subject: svn commit: r288596 - in stable/10/sys/cddl/contrib/opensolaris/uts/common/fs/zfs: . sys X-SVN-Group: stable-10 MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit X-BeenThere: svn-src-stable@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: SVN commit messages for all the -stable branches of the src tree List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 03 Oct 2015 11:38:11 -0000 Author: mav Date: Sat Oct 3 11:38:10 2015 New Revision: 288596 URL: https://svnweb.freebsd.org/changeset/base/288596 Log: MFC r287706 (by delphij): 6214 zpools going south In r286570 (MFV of r277426) an unprotected write to b_flags to set the compression mode was introduced. This would open a race window where data is partially decompressed, modified, checksummed and written to the pool, resulting in pool corruption due to the partial decompression. Prevent this by reintroducing b_compress illumos/illumos-gate@d4cd038c92c36fd0ae35945831a8fc2975b5272c Illumos issues: 6214 zpools going south https://www.illumos.org/issues/6214 Modified: stable/10/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/arc.c stable/10/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/sys/arc.h Directory Properties: stable/10/ (props changed) Modified: stable/10/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/arc.c ============================================================================== --- stable/10/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/arc.c Sat Oct 3 11:37:00 2015 (r288595) +++ stable/10/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/arc.c Sat Oct 3 11:38:10 2015 (r288596) @@ -851,6 +851,7 @@ typedef struct l2arc_buf_hdr { uint64_t b_daddr; /* disk address, offset byte */ /* real alloc'd buffer size depending on b_compress applied */ int32_t b_asize; + uint8_t b_compress; list_node_t b_l2node; } l2arc_buf_hdr_t; @@ -930,15 +931,6 @@ static arc_buf_hdr_t arc_eviction_hdr; #define HDR_HAS_L1HDR(hdr) ((hdr)->b_flags & ARC_FLAG_HAS_L1HDR) #define HDR_HAS_L2HDR(hdr) ((hdr)->b_flags & ARC_FLAG_HAS_L2HDR) -/* For storing compression mode in b_flags */ -#define HDR_COMPRESS_OFFSET 24 -#define HDR_COMPRESS_NBITS 7 - -#define HDR_GET_COMPRESS(hdr) ((enum zio_compress)BF32_GET(hdr->b_flags, \ - HDR_COMPRESS_OFFSET, HDR_COMPRESS_NBITS)) -#define HDR_SET_COMPRESS(hdr, cmp) BF32_SET(hdr->b_flags, \ - HDR_COMPRESS_OFFSET, HDR_COMPRESS_NBITS, (cmp)) - /* * Other sizes */ @@ -2229,7 +2221,7 @@ arc_buf_l2_cdata_free(arc_buf_hdr_t *hdr * separately compressed buffer, so there's nothing to free (it * points to the same buffer as the arc_buf_t's b_data field). */ - if (HDR_GET_COMPRESS(hdr) == ZIO_COMPRESS_OFF) { + if (hdr->b_l2hdr.b_compress == ZIO_COMPRESS_OFF) { hdr->b_l1hdr.b_tmp_cdata = NULL; return; } @@ -2238,12 +2230,12 @@ arc_buf_l2_cdata_free(arc_buf_hdr_t *hdr * There's nothing to free since the buffer was all zero's and * compressed to a zero length buffer. */ - if (HDR_GET_COMPRESS(hdr) == ZIO_COMPRESS_EMPTY) { + if (hdr->b_l2hdr.b_compress == ZIO_COMPRESS_EMPTY) { ASSERT3P(hdr->b_l1hdr.b_tmp_cdata, ==, NULL); return; } - ASSERT(L2ARC_IS_VALID_COMPRESS(HDR_GET_COMPRESS(hdr))); + ASSERT(L2ARC_IS_VALID_COMPRESS(hdr->b_l2hdr.b_compress)); arc_buf_free_on_write(hdr->b_l1hdr.b_tmp_cdata, hdr->b_size, zio_data_buf_free); @@ -4467,7 +4459,7 @@ top: (vd = hdr->b_l2hdr.b_dev->l2ad_vdev) != NULL) { devw = hdr->b_l2hdr.b_dev->l2ad_writing; addr = hdr->b_l2hdr.b_daddr; - b_compress = HDR_GET_COMPRESS(hdr); + b_compress = hdr->b_l2hdr.b_compress; b_asize = hdr->b_l2hdr.b_asize; /* * Lock out device removal. @@ -6028,6 +6020,8 @@ l2arc_read_done(zio_t *zio) if (cb->l2rcb_compress != ZIO_COMPRESS_OFF) l2arc_decompress_zio(zio, hdr, cb->l2rcb_compress); ASSERT(zio->io_data != NULL); + ASSERT3U(zio->io_size, ==, hdr->b_size); + ASSERT3U(BP_GET_LSIZE(&cb->l2rcb_bp), ==, hdr->b_size); /* * Check this survived the L2ARC journey. @@ -6064,7 +6058,7 @@ l2arc_read_done(zio_t *zio) ASSERT(!pio || pio->io_child_type == ZIO_CHILD_LOGICAL); zio_nowait(zio_read(pio, cb->l2rcb_spa, &cb->l2rcb_bp, - buf->b_data, zio->io_size, arc_read_done, buf, + buf->b_data, hdr->b_size, arc_read_done, buf, zio->io_priority, cb->l2rcb_flags, &cb->l2rcb_zb)); } } @@ -6381,7 +6375,7 @@ l2arc_write_buffers(spa_t *spa, l2arc_de * can't access without holding the ARC list locks * (which we want to avoid during compression/writing). */ - HDR_SET_COMPRESS(hdr, ZIO_COMPRESS_OFF); + hdr->b_l2hdr.b_compress = ZIO_COMPRESS_OFF; hdr->b_l2hdr.b_asize = hdr->b_size; hdr->b_l1hdr.b_tmp_cdata = hdr->b_l1hdr.b_buf->b_data; @@ -6583,7 +6577,7 @@ l2arc_compress_buf(arc_buf_hdr_t *hdr) l2arc_buf_hdr_t *l2hdr = &hdr->b_l2hdr; ASSERT(HDR_HAS_L1HDR(hdr)); - ASSERT(HDR_GET_COMPRESS(hdr) == ZIO_COMPRESS_OFF); + ASSERT3S(l2hdr->b_compress, ==, ZIO_COMPRESS_OFF); ASSERT(hdr->b_l1hdr.b_tmp_cdata != NULL); len = l2hdr->b_asize; @@ -6595,7 +6589,7 @@ l2arc_compress_buf(arc_buf_hdr_t *hdr) if (csize == 0) { /* zero block, indicate that there's nothing to write */ zio_data_buf_free(cdata, len); - HDR_SET_COMPRESS(hdr, ZIO_COMPRESS_EMPTY); + l2hdr->b_compress = ZIO_COMPRESS_EMPTY; l2hdr->b_asize = 0; hdr->b_l1hdr.b_tmp_cdata = NULL; ARCSTAT_BUMP(arcstat_l2_compress_zeros); @@ -6613,7 +6607,7 @@ l2arc_compress_buf(arc_buf_hdr_t *hdr) bzero((char *)cdata + csize, rounded - csize); csize = rounded; } - HDR_SET_COMPRESS(hdr, ZIO_COMPRESS_LZ4); + l2hdr->b_compress = ZIO_COMPRESS_LZ4; l2hdr->b_asize = csize; hdr->b_l1hdr.b_tmp_cdata = cdata; ARCSTAT_BUMP(arcstat_l2_compress_successes); @@ -6700,7 +6694,8 @@ l2arc_decompress_zio(zio_t *zio, arc_buf static void l2arc_release_cdata_buf(arc_buf_hdr_t *hdr) { - enum zio_compress comp = HDR_GET_COMPRESS(hdr); + ASSERT(HDR_HAS_L2HDR(hdr)); + enum zio_compress comp = hdr->b_l2hdr.b_compress; ASSERT(HDR_HAS_L1HDR(hdr)); ASSERT(comp == ZIO_COMPRESS_OFF || L2ARC_IS_VALID_COMPRESS(comp)); Modified: stable/10/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/sys/arc.h ============================================================================== --- stable/10/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/sys/arc.h Sat Oct 3 11:37:00 2015 (r288595) +++ stable/10/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/sys/arc.h Sat Oct 3 11:38:10 2015 (r288596) @@ -88,21 +88,6 @@ typedef enum arc_flags /* Flags specifying whether optional hdr struct fields are defined */ ARC_FLAG_HAS_L1HDR = 1 << 19, ARC_FLAG_HAS_L2HDR = 1 << 20, - - - /* - * The arc buffer's compression mode is stored in the top 7 bits of the - * flags field, so these dummy flags are included so that MDB can - * interpret the enum properly. - */ - ARC_FLAG_COMPRESS_0 = 1 << 24, - ARC_FLAG_COMPRESS_1 = 1 << 25, - ARC_FLAG_COMPRESS_2 = 1 << 26, - ARC_FLAG_COMPRESS_3 = 1 << 27, - ARC_FLAG_COMPRESS_4 = 1 << 28, - ARC_FLAG_COMPRESS_5 = 1 << 29, - ARC_FLAG_COMPRESS_6 = 1 << 30 - } arc_flags_t; struct arc_buf { From owner-svn-src-stable@freebsd.org Sat Oct 3 11:39:39 2015 Return-Path: Delivered-To: svn-src-stable@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id E401AA0E07D; Sat, 3 Oct 2015 11:39:39 +0000 (UTC) (envelope-from mav@FreeBSD.org) Received: from repo.freebsd.org (repo.freebsd.org [IPv6:2001:1900:2254:2068::e6a:0]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id D44241CD5; Sat, 3 Oct 2015 11:39:39 +0000 (UTC) (envelope-from mav@FreeBSD.org) Received: from repo.freebsd.org ([127.0.1.70]) by repo.freebsd.org (8.15.2/8.15.2) with ESMTP id t93Bdd2Q070474; Sat, 3 Oct 2015 11:39:39 GMT (envelope-from mav@FreeBSD.org) Received: (from mav@localhost) by repo.freebsd.org (8.15.2/8.15.2/Submit) id t93Bddbu070473; Sat, 3 Oct 2015 11:39:39 GMT (envelope-from mav@FreeBSD.org) Message-Id: <201510031139.t93Bddbu070473@repo.freebsd.org> X-Authentication-Warning: repo.freebsd.org: mav set sender to mav@FreeBSD.org using -f From: Alexander Motin Date: Sat, 3 Oct 2015 11:39:39 +0000 (UTC) To: src-committers@freebsd.org, svn-src-all@freebsd.org, svn-src-stable@freebsd.org, svn-src-stable-10@freebsd.org Subject: svn commit: r288597 - stable/10/sys/cddl/contrib/opensolaris/uts/common/fs/zfs X-SVN-Group: stable-10 MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit X-BeenThere: svn-src-stable@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: SVN commit messages for all the -stable branches of the src tree List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 03 Oct 2015 11:39:40 -0000 Author: mav Date: Sat Oct 3 11:39:39 2015 New Revision: 288597 URL: https://svnweb.freebsd.org/changeset/base/288597 Log: MFC r287744 (by delphij): Reduce diff against upstream. Modified: stable/10/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/spa.c Directory Properties: stable/10/ (props changed) Modified: stable/10/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/spa.c ============================================================================== --- stable/10/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/spa.c Sat Oct 3 11:38:10 2015 (r288596) +++ stable/10/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/spa.c Sat Oct 3 11:39:39 2015 (r288597) @@ -1630,6 +1630,7 @@ load_nvlist(spa_t *spa, uint64_t obj, nv error = dmu_bonus_hold(spa->spa_meta_objset, obj, FTAG, &db); if (error != 0) return (error); + nvsize = *(uint64_t *)db->db_data; dmu_buf_rele(db, FTAG); From owner-svn-src-stable@freebsd.org Sat Oct 3 11:41:28 2015 Return-Path: Delivered-To: svn-src-stable@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id C4F40A0E226; Sat, 3 Oct 2015 11:41:28 +0000 (UTC) (envelope-from mav@FreeBSD.org) Received: from repo.freebsd.org (repo.freebsd.org [IPv6:2001:1900:2254:2068::e6a:0]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id B540E1F3B; Sat, 3 Oct 2015 11:41:28 +0000 (UTC) (envelope-from mav@FreeBSD.org) Received: from repo.freebsd.org ([127.0.1.70]) by repo.freebsd.org (8.15.2/8.15.2) with ESMTP id t93BfSGG074200; Sat, 3 Oct 2015 11:41:28 GMT (envelope-from mav@FreeBSD.org) Received: (from mav@localhost) by repo.freebsd.org (8.15.2/8.15.2/Submit) id t93BfSnF074199; Sat, 3 Oct 2015 11:41:28 GMT (envelope-from mav@FreeBSD.org) Message-Id: <201510031141.t93BfSnF074199@repo.freebsd.org> X-Authentication-Warning: repo.freebsd.org: mav set sender to mav@FreeBSD.org using -f From: Alexander Motin Date: Sat, 3 Oct 2015 11:41:28 +0000 (UTC) To: src-committers@freebsd.org, svn-src-all@freebsd.org, svn-src-stable@freebsd.org, svn-src-stable-10@freebsd.org Subject: svn commit: r288598 - stable/10/cddl/contrib/opensolaris/cmd/zdb X-SVN-Group: stable-10 MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit X-BeenThere: svn-src-stable@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: SVN commit messages for all the -stable branches of the src tree List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 03 Oct 2015 11:41:28 -0000 Author: mav Date: Sat Oct 3 11:41:27 2015 New Revision: 288598 URL: https://svnweb.freebsd.org/changeset/base/288598 Log: MFC r287771: 5695 dmu_sync'ed holes do not retain birth time (userland portion that was not merged in r286677) Update zdb to also print ltime, type, and level information for these new style holes. Previously, only the logical birth time would be printed. Modified: stable/10/cddl/contrib/opensolaris/cmd/zdb/zdb.c Directory Properties: stable/10/ (props changed) Modified: stable/10/cddl/contrib/opensolaris/cmd/zdb/zdb.c ============================================================================== --- stable/10/cddl/contrib/opensolaris/cmd/zdb/zdb.c Sat Oct 3 11:39:39 2015 (r288597) +++ stable/10/cddl/contrib/opensolaris/cmd/zdb/zdb.c Sat Oct 3 11:41:27 2015 (r288598) @@ -1205,7 +1205,9 @@ snprintf_blkptr_compact(char *blkbuf, si if (BP_IS_HOLE(bp)) { (void) snprintf(blkbuf + strlen(blkbuf), - buflen - strlen(blkbuf), "B=%llu", + buflen - strlen(blkbuf), + "%llxL B=%llu", + (u_longlong_t)BP_GET_LSIZE(bp), (u_longlong_t)bp->blk_birth); } else { (void) snprintf(blkbuf + strlen(blkbuf), From owner-svn-src-stable@freebsd.org Sat Oct 3 11:43:55 2015 Return-Path: Delivered-To: svn-src-stable@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 2E344A0E4F8; Sat, 3 Oct 2015 11:43:55 +0000 (UTC) (envelope-from mav@FreeBSD.org) Received: from repo.freebsd.org (repo.freebsd.org [IPv6:2001:1900:2254:2068::e6a:0]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 1E82811F7; Sat, 3 Oct 2015 11:43:55 +0000 (UTC) (envelope-from mav@FreeBSD.org) Received: from repo.freebsd.org ([127.0.1.70]) by repo.freebsd.org (8.15.2/8.15.2) with ESMTP id t93Bhs64074348; Sat, 3 Oct 2015 11:43:54 GMT (envelope-from mav@FreeBSD.org) Received: (from mav@localhost) by repo.freebsd.org (8.15.2/8.15.2/Submit) id t93Bhs3E074347; Sat, 3 Oct 2015 11:43:54 GMT (envelope-from mav@FreeBSD.org) Message-Id: <201510031143.t93Bhs3E074347@repo.freebsd.org> X-Authentication-Warning: repo.freebsd.org: mav set sender to mav@FreeBSD.org using -f From: Alexander Motin Date: Sat, 3 Oct 2015 11:43:54 +0000 (UTC) To: src-committers@freebsd.org, svn-src-all@freebsd.org, svn-src-stable@freebsd.org, svn-src-stable-10@freebsd.org Subject: svn commit: r288599 - stable/10/sys/cddl/contrib/opensolaris/uts/common/fs/zfs X-SVN-Group: stable-10 MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit X-BeenThere: svn-src-stable@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: SVN commit messages for all the -stable branches of the src tree List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 03 Oct 2015 11:43:55 -0000 Author: mav Date: Sat Oct 3 11:43:54 2015 New Revision: 288599 URL: https://svnweb.freebsd.org/changeset/base/288599 Log: MFC r288064 (by avg): 6220 memleak in l2arc on debug build illumos/illumos-gate/commit/c546f36aa898d913ff77674fb5ff97f15b2e08b4 https://www.illumos.org/issues/6220 5408 introduced a memleak in l2arc, namely the member b_thawed gets leaked when an arc_hdr is realloced from full to l2only. Reviewed by: Saso Kiselkov Reviewed by: Simon Klinkert Reviewed by: George Wilson Approved by: Robert Mustacchi Author: Arne Jansen Modified: stable/10/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/arc.c Directory Properties: stable/10/ (props changed) Modified: stable/10/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/arc.c ============================================================================== --- stable/10/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/arc.c Sat Oct 3 11:41:27 2015 (r288598) +++ stable/10/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/arc.c Sat Oct 3 11:43:54 2015 (r288599) @@ -1488,6 +1488,13 @@ arc_hdr_realloc(arc_buf_hdr_t *hdr, kmem VERIFY(!HDR_L2_WRITING(hdr)); VERIFY3P(hdr->b_l1hdr.b_tmp_cdata, ==, NULL); +#ifdef ZFS_DEBUG + if (hdr->b_l1hdr.b_thawed != NULL) { + kmem_free(hdr->b_l1hdr.b_thawed, 1); + hdr->b_l1hdr.b_thawed = NULL; + } +#endif + nhdr->b_flags &= ~ARC_FLAG_HAS_L1HDR; } /* From owner-svn-src-stable@freebsd.org Sat Oct 3 16:01:20 2015 Return-Path: Delivered-To: svn-src-stable@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 04DF0A0D745; Sat, 3 Oct 2015 16:01:20 +0000 (UTC) (envelope-from bdrewery@FreeBSD.org) Received: from repo.freebsd.org (repo.freebsd.org [IPv6:2001:1900:2254:2068::e6a:0]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id E69F113DA; Sat, 3 Oct 2015 16:01:19 +0000 (UTC) (envelope-from bdrewery@FreeBSD.org) Received: from repo.freebsd.org ([127.0.1.70]) by repo.freebsd.org (8.15.2/8.15.2) with ESMTP id t93G1JWj079445; Sat, 3 Oct 2015 16:01:19 GMT (envelope-from bdrewery@FreeBSD.org) Received: (from bdrewery@localhost) by repo.freebsd.org (8.15.2/8.15.2/Submit) id t93G1G06079430; Sat, 3 Oct 2015 16:01:16 GMT (envelope-from bdrewery@FreeBSD.org) Message-Id: <201510031601.t93G1G06079430@repo.freebsd.org> X-Authentication-Warning: repo.freebsd.org: bdrewery set sender to bdrewery@FreeBSD.org using -f From: Bryan Drewery Date: Sat, 3 Oct 2015 16:01:16 +0000 (UTC) To: src-committers@freebsd.org, svn-src-all@freebsd.org, svn-src-stable@freebsd.org, svn-src-stable-10@freebsd.org Subject: svn commit: r288607 - in stable/10: gnu/usr.bin/binutils/libbfd kerberos5/libexec/kdigest kerberos5/usr.bin/hxtool kerberos5/usr.bin/kadmin kerberos5/usr.bin/kcc kerberos5/usr.sbin/iprop-log kerber... X-SVN-Group: stable-10 MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit X-BeenThere: svn-src-stable@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: SVN commit messages for all the -stable branches of the src tree List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 03 Oct 2015 16:01:20 -0000 Author: bdrewery Date: Sat Oct 3 16:01:16 2015 New Revision: 288607 URL: https://svnweb.freebsd.org/changeset/base/288607 Log: MFC r288199,r288246: Add missing CLEANFILES. Modified: stable/10/gnu/usr.bin/binutils/libbfd/Makefile.i386 stable/10/kerberos5/libexec/kdigest/Makefile stable/10/kerberos5/usr.bin/hxtool/Makefile stable/10/kerberos5/usr.bin/kadmin/Makefile stable/10/kerberos5/usr.bin/kcc/Makefile stable/10/kerberos5/usr.sbin/iprop-log/Makefile stable/10/kerberos5/usr.sbin/ktutil/Makefile stable/10/lib/clang/include/Makefile stable/10/lib/libc/tests/gen/posix_spawn/Makefile stable/10/lib/ncurses/ncurses/Makefile stable/10/usr.bin/yacc/tests/Makefile Directory Properties: stable/10/ (props changed) Modified: stable/10/gnu/usr.bin/binutils/libbfd/Makefile.i386 ============================================================================== --- stable/10/gnu/usr.bin/binutils/libbfd/Makefile.i386 Sat Oct 3 15:58:00 2015 (r288606) +++ stable/10/gnu/usr.bin/binutils/libbfd/Makefile.i386 Sat Oct 3 16:01:16 2015 (r288607) @@ -17,3 +17,5 @@ VECS= ${DEFAULT_VECTOR} \ peigen.c: peXXigen.c sed -e s/XX/pe/g ${.ALLSRC} > ${.TARGET} + +CLEANFILES+= peigen.c Modified: stable/10/kerberos5/libexec/kdigest/Makefile ============================================================================== --- stable/10/kerberos5/libexec/kdigest/Makefile Sat Oct 3 15:58:00 2015 (r288606) +++ stable/10/kerberos5/libexec/kdigest/Makefile Sat Oct 3 16:01:16 2015 (r288607) @@ -13,6 +13,8 @@ SRCS= kdigest.c \ kdigest-commands.c \ kdigest-commands.h +CLEANFILES= kdigest-commands.h kdigest-commands.c + kdigest-commands.h: kdigest-commands.in ${SLC} ${.ALLSRC:M*.in} Modified: stable/10/kerberos5/usr.bin/hxtool/Makefile ============================================================================== --- stable/10/kerberos5/usr.bin/hxtool/Makefile Sat Oct 3 15:58:00 2015 (r288606) +++ stable/10/kerberos5/usr.bin/hxtool/Makefile Sat Oct 3 16:01:16 2015 (r288607) @@ -10,6 +10,8 @@ DPADD= ${LIBHX509} ${LIBROKEN} ${LIBASN1 LDADD= -lhx509 -lroken -lasn1 -lcrypto -lcrypt ${LIBSL} ${LIBVERS} -ledit SRCS= hxtool.c hxtool-commands.c hxtool-commands.h +CLEANFILES= hxtool-commands.h hxtool-commands.c + hxtool-commands.h: hxtool-commands.in ${SLC} ${.ALLSRC:M*.in} Modified: stable/10/kerberos5/usr.bin/kadmin/Makefile ============================================================================== --- stable/10/kerberos5/usr.bin/kadmin/Makefile Sat Oct 3 15:58:00 2015 (r288606) +++ stable/10/kerberos5/usr.bin/kadmin/Makefile Sat Oct 3 16:01:16 2015 (r288607) @@ -36,6 +36,8 @@ LDADD= -lkadm5clnt -lkadm5srv -lhdb -lkr -ledit -lncurses ${LDAPLDADD} LDFLAGS=${LDAPLDFLAGS} +CLEANFILES= kadmin-commands.h kadmin-commands.c + .include kadmin-commands.h: ${KRB5DIR}/kadmin/kadmin-commands.in Modified: stable/10/kerberos5/usr.bin/kcc/Makefile ============================================================================== --- stable/10/kerberos5/usr.bin/kcc/Makefile Sat Oct 3 15:58:00 2015 (r288606) +++ stable/10/kerberos5/usr.bin/kcc/Makefile Sat Oct 3 16:01:16 2015 (r288607) @@ -19,6 +19,8 @@ SRCS= kcc.c \ kswitch.c \ copy_cred_cache.c +CLEANFILES= kcc-commands.h kcc-commands.c + kcc-commands.h: kcc-commands.in ${SLC} ${.ALLSRC:M*.in} Modified: stable/10/kerberos5/usr.sbin/iprop-log/Makefile ============================================================================== --- stable/10/kerberos5/usr.sbin/iprop-log/Makefile Sat Oct 3 15:58:00 2015 (r288606) +++ stable/10/kerberos5/usr.sbin/iprop-log/Makefile Sat Oct 3 16:01:16 2015 (r288607) @@ -14,6 +14,8 @@ LDADD= -lkadm5srv -lhdb -lkrb5 -lasn1 -l ${LIBVERS} -ledit LDFLAGS=${LDAPLDFLAGS} +CLEANFILES= iprop-commands.h iprop-commands.c + iprop-commands.h: iprop-commands.in ${SLC} ${.ALLSRC:M*.in} Modified: stable/10/kerberos5/usr.sbin/ktutil/Makefile ============================================================================== --- stable/10/kerberos5/usr.sbin/ktutil/Makefile Sat Oct 3 15:58:00 2015 (r288606) +++ stable/10/kerberos5/usr.sbin/ktutil/Makefile Sat Oct 3 16:01:16 2015 (r288607) @@ -22,6 +22,8 @@ DPADD= ${LIBKADM5CLNT} ${LIBKRB5} ${LIBS LDADD= -lkadm5clnt -lkrb5 ${LIBSL} -lroken ${LIBVERS} \ -lasn1 -lcrypto -lcrypt -ledit +CLEANFILES= ktutil-commands.h ktutil-commands.c + .include ktutil-commands.h: ${KRB5DIR}/admin/ktutil-commands.in Modified: stable/10/lib/clang/include/Makefile ============================================================================== --- stable/10/lib/clang/include/Makefile Sat Oct 3 15:58:00 2015 (r288606) +++ stable/10/lib/clang/include/Makefile Sat Oct 3 16:01:16 2015 (r288607) @@ -45,6 +45,6 @@ INCS= __wmmintrin_aes.h \ xopintrin.h \ ${GENINCS} GENINCS= arm_neon.h -CLEANFILES= ${GENINCS} +CLEANFILES= ${GENINCS} ${GENINCS:C/\.h$/.d/} .include Modified: stable/10/lib/libc/tests/gen/posix_spawn/Makefile ============================================================================== --- stable/10/lib/libc/tests/gen/posix_spawn/Makefile Sat Oct 3 15:58:00 2015 (r288606) +++ stable/10/lib/libc/tests/gen/posix_spawn/Makefile Sat Oct 3 16:01:16 2015 (r288607) @@ -21,6 +21,8 @@ PROGS+= h_spawnattr SCRIPTS= h_nonexec SCRIPTS+= h_zero +CLEANFILES+= h_nonexec + .include "../../Makefile.netbsd-tests" h_zero: Modified: stable/10/lib/ncurses/ncurses/Makefile ============================================================================== --- stable/10/lib/ncurses/ncurses/Makefile Sat Oct 3 15:58:00 2015 (r288606) +++ stable/10/lib/ncurses/ncurses/Makefile Sat Oct 3 16:01:16 2015 (r288607) @@ -282,6 +282,7 @@ CFLAGS+= -DFREEBSD_NATIVE -DTERMIOS # Installed HEADERS= curses.h term.h termcap.h unctrl.h SRCHDRS= ncurses_dll.h +CLEANFILES+= ncurses_dll.h .if defined(ENABLE_WIDEC) INCS= ${HEADERS} ${SRCHDRS} Modified: stable/10/usr.bin/yacc/tests/Makefile ============================================================================== --- stable/10/usr.bin/yacc/tests/Makefile Sat Oct 3 15:58:00 2015 (r288606) +++ stable/10/usr.bin/yacc/tests/Makefile Sat Oct 3 16:01:16 2015 (r288607) @@ -17,6 +17,8 @@ TEST_METADATA.yacc_tests+= required_user SCRIPTS= run_test SCRIPTSDIR= ${TESTSDIR} +CLEANFILES= run_test + FILESGROUPS= FILES FILEStest FILEStest_yacc FILEStestDIR= ${TESTSDIR} From owner-svn-src-stable@freebsd.org Sat Oct 3 16:21:55 2015 Return-Path: Delivered-To: svn-src-stable@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 2308CA0E479; Sat, 3 Oct 2015 16:21:55 +0000 (UTC) (envelope-from bdrewery@FreeBSD.org) Received: from repo.freebsd.org (repo.freebsd.org [IPv6:2001:1900:2254:2068::e6a:0]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id ED0CF1030; Sat, 3 Oct 2015 16:21:54 +0000 (UTC) (envelope-from bdrewery@FreeBSD.org) Received: from repo.freebsd.org ([127.0.1.70]) by repo.freebsd.org (8.15.2/8.15.2) with ESMTP id t93GLs8Z090007; Sat, 3 Oct 2015 16:21:54 GMT (envelope-from bdrewery@FreeBSD.org) Received: (from bdrewery@localhost) by repo.freebsd.org (8.15.2/8.15.2/Submit) id t93GLsT6089577; Sat, 3 Oct 2015 16:21:54 GMT (envelope-from bdrewery@FreeBSD.org) Message-Id: <201510031621.t93GLsT6089577@repo.freebsd.org> X-Authentication-Warning: repo.freebsd.org: bdrewery set sender to bdrewery@FreeBSD.org using -f From: Bryan Drewery Date: Sat, 3 Oct 2015 16:21:54 +0000 (UTC) To: src-committers@freebsd.org, svn-src-all@freebsd.org, svn-src-stable@freebsd.org, svn-src-stable-10@freebsd.org Subject: svn commit: r288610 - in stable/10: gnu/usr.bin/groff/src/utils/indxbib lib/libusb lib/libz X-SVN-Group: stable-10 MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit X-BeenThere: svn-src-stable@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: SVN commit messages for all the -stable branches of the src tree List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 03 Oct 2015 16:21:55 -0000 Author: bdrewery Date: Sat Oct 3 16:21:53 2015 New Revision: 288610 URL: https://svnweb.freebsd.org/changeset/base/288610 Log: MFC r273756,r287980: r273756: only install .pc files when we are not installing 32bit compat libs... r287980: Replace beforeinstall: handling with FILES. Modified: stable/10/gnu/usr.bin/groff/src/utils/indxbib/Makefile stable/10/lib/libusb/Makefile stable/10/lib/libz/Makefile Directory Properties: stable/10/ (props changed) Modified: stable/10/gnu/usr.bin/groff/src/utils/indxbib/Makefile ============================================================================== --- stable/10/gnu/usr.bin/groff/src/utils/indxbib/Makefile Sat Oct 3 16:21:06 2015 (r288609) +++ stable/10/gnu/usr.bin/groff/src/utils/indxbib/Makefile Sat Oct 3 16:21:53 2015 (r288610) @@ -5,9 +5,7 @@ SRCS= indxbib.cpp signal.c DPADD= ${LIBBIB} ${LIBGROFF} ${LIBM} LDADD= ${LIBBIB} ${LIBGROFF} -lm CLEANFILES= ${MAN} - -beforeinstall: - ${INSTALL} -o ${BINOWN} -g ${BINGRP} -m ${NOBINMODE} \ - ${DIST_DIR}/eign ${DESTDIR}${SHAREDIR}/dict/ +FILES= ${DIST_DIR}/eign +FILESDIR= ${SHAREDIR}/dict/ .include Modified: stable/10/lib/libusb/Makefile ============================================================================== --- stable/10/lib/libusb/Makefile Sat Oct 3 16:21:06 2015 (r288609) +++ stable/10/lib/libusb/Makefile Sat Oct 3 16:21:53 2015 (r288610) @@ -36,16 +36,11 @@ SRCS+= libusb10_io.c .if defined(COMPAT_32BIT) CFLAGS+= -DCOMPAT_32BIT +.else +FILES= libusb-0.1.pc libusb-1.0.pc libusb-2.0.pc +FILESDIR= ${LIBDATADIR}/pkgconfig .endif -beforeinstall: - ${INSTALL} -C -o ${LIBOWN} -g ${LIBGRP} -m ${LIBMODE} \ - ${.CURDIR}/libusb-0.1.pc ${DESTDIR}${LIBDATADIR}/pkgconfig - ${INSTALL} -C -o ${LIBOWN} -g ${LIBGRP} -m ${LIBMODE} \ - ${.CURDIR}/libusb-1.0.pc ${DESTDIR}${LIBDATADIR}/pkgconfig - ${INSTALL} -C -o ${LIBOWN} -g ${LIBGRP} -m ${LIBMODE} \ - ${.CURDIR}/libusb-2.0.pc ${DESTDIR}${LIBDATADIR}/pkgconfig - # # Cross platform support # Modified: stable/10/lib/libz/Makefile ============================================================================== --- stable/10/lib/libz/Makefile Sat Oct 3 16:21:06 2015 (r288609) +++ stable/10/lib/libz/Makefile Sat Oct 3 16:21:53 2015 (r288610) @@ -68,9 +68,10 @@ test: example minigzip (export LD_LIBRARY_PATH=. ; \ echo hello world | ./minigzip | ./minigzip -d ) -beforeinstall: - ${INSTALL} -C -o ${LIBOWN} -g ${LIBGRP} -m ${LIBMODE} \ - ${.CURDIR}/zlib.pc ${DESTDIR}${LIBDATADIR}/pkgconfig +.ifndef COMPAT_32BIT +FILES= zlib.pc +FILESDIR= ${LIBDATADIR}/pkgconfig +.endif .include From owner-svn-src-stable@freebsd.org Sat Oct 3 16:24:22 2015 Return-Path: Delivered-To: svn-src-stable@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 1E0D6A0E5C6; Sat, 3 Oct 2015 16:24:22 +0000 (UTC) (envelope-from bdrewery@FreeBSD.org) Received: from repo.freebsd.org (repo.freebsd.org [IPv6:2001:1900:2254:2068::e6a:0]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 0DBD511C3; Sat, 3 Oct 2015 16:24:22 +0000 (UTC) (envelope-from bdrewery@FreeBSD.org) Received: from repo.freebsd.org ([127.0.1.70]) by repo.freebsd.org (8.15.2/8.15.2) with ESMTP id t93GOLdv090320; Sat, 3 Oct 2015 16:24:21 GMT (envelope-from bdrewery@FreeBSD.org) Received: (from bdrewery@localhost) by repo.freebsd.org (8.15.2/8.15.2/Submit) id t93GOLww090318; Sat, 3 Oct 2015 16:24:21 GMT (envelope-from bdrewery@FreeBSD.org) Message-Id: <201510031624.t93GOLww090318@repo.freebsd.org> X-Authentication-Warning: repo.freebsd.org: bdrewery set sender to bdrewery@FreeBSD.org using -f From: Bryan Drewery Date: Sat, 3 Oct 2015 16:24:21 +0000 (UTC) To: src-committers@freebsd.org, svn-src-all@freebsd.org, svn-src-stable@freebsd.org, svn-src-stable-10@freebsd.org Subject: svn commit: r288611 - in stable/10: . secure/lib/libcrypto X-SVN-Group: stable-10 MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit X-BeenThere: svn-src-stable@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: SVN commit messages for all the -stable branches of the src tree List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 03 Oct 2015 16:24:22 -0000 Author: bdrewery Date: Sat Oct 3 16:24:20 2015 New Revision: 288611 URL: https://svnweb.freebsd.org/changeset/base/288611 Log: MFC r287981: Replace afterinstall: hack from r111083 with 'make delete-old' functionality. Modified: stable/10/ObsoleteFiles.inc stable/10/secure/lib/libcrypto/Makefile Directory Properties: stable/10/ (props changed) Modified: stable/10/ObsoleteFiles.inc ============================================================================== --- stable/10/ObsoleteFiles.inc Sat Oct 3 16:21:53 2015 (r288610) +++ stable/10/ObsoleteFiles.inc Sat Oct 3 16:24:20 2015 (r288611) @@ -6403,6 +6403,12 @@ OLD_LIBS+=usr/lib/libposix1e.so.2 OLD_LIBS+=usr/lib/libskey.so.2 OLD_LIBS+=usr/lib/libusbhid.so.0 OLD_LIBS+=usr/lib/libvgl.so.2 +# 20030218: OpenSSL 0.9.7 import +OLD_FILES+=usr/include/des.h +OLD_FILES+=usr/lib/libdes.a +OLD_FILES+=usr/lib/libdes.so +OLD_LIBS+=usr/lib/libdes.so.3 +OLD_FILES+=usr/lib/libdes_p.a # 200302XX OLD_LIBS+=usr/lib/libacl.so.3 OLD_LIBS+=usr/lib/libasn1.so.5 Modified: stable/10/secure/lib/libcrypto/Makefile ============================================================================== --- stable/10/secure/lib/libcrypto/Makefile Sat Oct 3 16:21:53 2015 (r288610) +++ stable/10/secure/lib/libcrypto/Makefile Sat Oct 3 16:24:20 2015 (r288611) @@ -402,14 +402,6 @@ opensslconf.h: opensslconf-${MACHINE_CPU .endif cp -f ${.ALLSRC} ${.TARGET} -OLDSYMLINKS+= libdes.a libdes.so libdes.so.3 libdes_p.a -afterinstall: - @${ECHO} "Removing stale symlinks." - rm -f ${DESTDIR}${INCLUDEDIR}/des.h -.for symlink in ${OLDSYMLINKS} - rm -f ${DESTDIR}${LIBDIR}/${symlink} -.endfor - .include .if ${MACHINE_CPUARCH} == "amd64" From owner-svn-src-stable@freebsd.org Sat Oct 3 16:25:48 2015 Return-Path: Delivered-To: svn-src-stable@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id A2DA4A0E679; Sat, 3 Oct 2015 16:25:48 +0000 (UTC) (envelope-from bdrewery@FreeBSD.org) Received: from repo.freebsd.org (repo.freebsd.org [IPv6:2001:1900:2254:2068::e6a:0]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 929CD1311; Sat, 3 Oct 2015 16:25:48 +0000 (UTC) (envelope-from bdrewery@FreeBSD.org) Received: from repo.freebsd.org ([127.0.1.70]) by repo.freebsd.org (8.15.2/8.15.2) with ESMTP id t93GPms3090469; Sat, 3 Oct 2015 16:25:48 GMT (envelope-from bdrewery@FreeBSD.org) Received: (from bdrewery@localhost) by repo.freebsd.org (8.15.2/8.15.2/Submit) id t93GPmEI090467; Sat, 3 Oct 2015 16:25:48 GMT (envelope-from bdrewery@FreeBSD.org) Message-Id: <201510031625.t93GPmEI090467@repo.freebsd.org> X-Authentication-Warning: repo.freebsd.org: bdrewery set sender to bdrewery@FreeBSD.org using -f From: Bryan Drewery Date: Sat, 3 Oct 2015 16:25:48 +0000 (UTC) To: src-committers@freebsd.org, svn-src-all@freebsd.org, svn-src-stable@freebsd.org, svn-src-stable-9@freebsd.org Subject: svn commit: r288612 - in stable/9: . secure/lib/libcrypto X-SVN-Group: stable-9 MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit X-BeenThere: svn-src-stable@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: SVN commit messages for all the -stable branches of the src tree List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 03 Oct 2015 16:25:48 -0000 Author: bdrewery Date: Sat Oct 3 16:25:47 2015 New Revision: 288612 URL: https://svnweb.freebsd.org/changeset/base/288612 Log: MFC r287981: Replace afterinstall: hack from r111083 with 'make delete-old' functionality. Modified: stable/9/ObsoleteFiles.inc (contents, props changed) stable/9/secure/lib/libcrypto/Makefile Directory Properties: stable/9/secure/lib/libcrypto/ (props changed) Modified: stable/9/ObsoleteFiles.inc ============================================================================== --- stable/9/ObsoleteFiles.inc Sat Oct 3 16:24:20 2015 (r288611) +++ stable/9/ObsoleteFiles.inc Sat Oct 3 16:25:47 2015 (r288612) @@ -5770,6 +5770,12 @@ OLD_LIBS+=usr/lib/libposix1e.so.2 OLD_LIBS+=usr/lib/libskey.so.2 OLD_LIBS+=usr/lib/libusbhid.so.0 OLD_LIBS+=usr/lib/libvgl.so.2 +# 20030218: OpenSSL 0.9.7 import +OLD_FILES+=usr/include/des.h +OLD_FILES+=usr/lib/libdes.a +OLD_FILES+=usr/lib/libdes.so +OLD_LIBS+=usr/lib/libdes.so.3 +OLD_FILES+=usr/lib/libdes_p.a # 200302XX OLD_LIBS+=usr/lib/libacl.so.3 OLD_LIBS+=usr/lib/libasn1.so.5 Modified: stable/9/secure/lib/libcrypto/Makefile ============================================================================== --- stable/9/secure/lib/libcrypto/Makefile Sat Oct 3 16:24:20 2015 (r288611) +++ stable/9/secure/lib/libcrypto/Makefile Sat Oct 3 16:25:47 2015 (r288612) @@ -378,14 +378,6 @@ fips_rand.h: CLEANFILES+= fips.h fips_rand.h -OLDSYMLINKS+= libdes.a libdes.so libdes.so.3 libdes_p.a -afterinstall: - @${ECHO} "Removing stale symlinks." - rm -f ${DESTDIR}${INCLUDEDIR}/des.h -.for symlink in ${OLDSYMLINKS} - rm -f ${DESTDIR}${LIBDIR}/${symlink} -.endfor - .include .if ${MACHINE_CPUARCH} == "i386" From owner-svn-src-stable@freebsd.org Sat Oct 3 16:26:14 2015 Return-Path: Delivered-To: svn-src-stable@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id D9A26A0E6EA; Sat, 3 Oct 2015 16:26:14 +0000 (UTC) (envelope-from bdrewery@FreeBSD.org) Received: from repo.freebsd.org (repo.freebsd.org [IPv6:2001:1900:2254:2068::e6a:0]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id C932914A0; Sat, 3 Oct 2015 16:26:14 +0000 (UTC) (envelope-from bdrewery@FreeBSD.org) Received: from repo.freebsd.org ([127.0.1.70]) by repo.freebsd.org (8.15.2/8.15.2) with ESMTP id t93GQELe090544; Sat, 3 Oct 2015 16:26:14 GMT (envelope-from bdrewery@FreeBSD.org) Received: (from bdrewery@localhost) by repo.freebsd.org (8.15.2/8.15.2/Submit) id t93GQEn9090541; Sat, 3 Oct 2015 16:26:14 GMT (envelope-from bdrewery@FreeBSD.org) Message-Id: <201510031626.t93GQEn9090541@repo.freebsd.org> X-Authentication-Warning: repo.freebsd.org: bdrewery set sender to bdrewery@FreeBSD.org using -f From: Bryan Drewery Date: Sat, 3 Oct 2015 16:26:14 +0000 (UTC) To: src-committers@freebsd.org, svn-src-all@freebsd.org, svn-src-stable@freebsd.org, svn-src-stable-10@freebsd.org Subject: svn commit: r288613 - in stable/10: . usr.sbin/ntp/ntpdc X-SVN-Group: stable-10 MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit X-BeenThere: svn-src-stable@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: SVN commit messages for all the -stable branches of the src tree List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 03 Oct 2015 16:26:15 -0000 Author: bdrewery Date: Sat Oct 3 16:26:13 2015 New Revision: 288613 URL: https://svnweb.freebsd.org/changeset/base/288613 Log: MFC r287982: Replace afterinstall: hack from r54681 with 'make delete-old' functionality. Modified: stable/10/ObsoleteFiles.inc stable/10/usr.sbin/ntp/ntpdc/Makefile Directory Properties: stable/10/ (props changed) Modified: stable/10/ObsoleteFiles.inc ============================================================================== --- stable/10/ObsoleteFiles.inc Sat Oct 3 16:25:47 2015 (r288612) +++ stable/10/ObsoleteFiles.inc Sat Oct 3 16:26:13 2015 (r288613) @@ -6467,6 +6467,8 @@ OLD_LIBS+=usr/lib/libtermcap.so.2 OLD_LIBS+=usr/lib/libutil.so.2 OLD_LIBS+=usr/lib/libvgl.so.1 OLD_LIBS+=usr/lib/libwrap.so.2 +# 19991216 +OLD_FILES+=usr/sbin/xntpdc # 199909XX OLD_LIBS+=usr/lib/libc_r.so.3 # ??? Modified: stable/10/usr.sbin/ntp/ntpdc/Makefile ============================================================================== --- stable/10/usr.sbin/ntp/ntpdc/Makefile Sat Oct 3 16:25:47 2015 (r288612) +++ stable/10/usr.sbin/ntp/ntpdc/Makefile Sat Oct 3 16:26:13 2015 (r288613) @@ -37,7 +37,4 @@ CLEANFILES+= .version version.c version.c: sh -e ${.CURDIR}/../scripts/mkver ntpdc -afterinstall: - rm -f ${DESTDIR}/usr/sbin/xntpdc - .include From owner-svn-src-stable@freebsd.org Sat Oct 3 16:26:48 2015 Return-Path: Delivered-To: svn-src-stable@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 24C8EA0E7AD; Sat, 3 Oct 2015 16:26:48 +0000 (UTC) (envelope-from bdrewery@FreeBSD.org) Received: from repo.freebsd.org (repo.freebsd.org [IPv6:2001:1900:2254:2068::e6a:0]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 146A91628; Sat, 3 Oct 2015 16:26:48 +0000 (UTC) (envelope-from bdrewery@FreeBSD.org) Received: from repo.freebsd.org ([127.0.1.70]) by repo.freebsd.org (8.15.2/8.15.2) with ESMTP id t93GQlvs090625; Sat, 3 Oct 2015 16:26:47 GMT (envelope-from bdrewery@FreeBSD.org) Received: (from bdrewery@localhost) by repo.freebsd.org (8.15.2/8.15.2/Submit) id t93GQlQi090622; Sat, 3 Oct 2015 16:26:47 GMT (envelope-from bdrewery@FreeBSD.org) Message-Id: <201510031626.t93GQlQi090622@repo.freebsd.org> X-Authentication-Warning: repo.freebsd.org: bdrewery set sender to bdrewery@FreeBSD.org using -f From: Bryan Drewery Date: Sat, 3 Oct 2015 16:26:47 +0000 (UTC) To: src-committers@freebsd.org, svn-src-all@freebsd.org, svn-src-stable@freebsd.org, svn-src-stable-9@freebsd.org Subject: svn commit: r288614 - in stable/9: . usr.sbin/ntp/ntpdc X-SVN-Group: stable-9 MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit X-BeenThere: svn-src-stable@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: SVN commit messages for all the -stable branches of the src tree List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 03 Oct 2015 16:26:48 -0000 Author: bdrewery Date: Sat Oct 3 16:26:46 2015 New Revision: 288614 URL: https://svnweb.freebsd.org/changeset/base/288614 Log: MFC r287982: Replace afterinstall: hack from r54681 with 'make delete-old' functionality. Modified: stable/9/ObsoleteFiles.inc (contents, props changed) stable/9/usr.sbin/ntp/ntpdc/Makefile Directory Properties: stable/9/usr.sbin/ntp/ (props changed) Modified: stable/9/ObsoleteFiles.inc ============================================================================== --- stable/9/ObsoleteFiles.inc Sat Oct 3 16:26:13 2015 (r288613) +++ stable/9/ObsoleteFiles.inc Sat Oct 3 16:26:46 2015 (r288614) @@ -5834,6 +5834,8 @@ OLD_LIBS+=usr/lib/libtermcap.so.2 OLD_LIBS+=usr/lib/libutil.so.2 OLD_LIBS+=usr/lib/libvgl.so.1 OLD_LIBS+=usr/lib/libwrap.so.2 +# 19991216 +OLD_FILES+=usr/sbin/xntpdc # 199909XX OLD_LIBS+=usr/lib/libc_r.so.3 # ??? Modified: stable/9/usr.sbin/ntp/ntpdc/Makefile ============================================================================== --- stable/9/usr.sbin/ntp/ntpdc/Makefile Sat Oct 3 16:26:13 2015 (r288613) +++ stable/9/usr.sbin/ntp/ntpdc/Makefile Sat Oct 3 16:26:46 2015 (r288614) @@ -25,7 +25,4 @@ CLEANFILES+= .version version.c version.c: sh -e ${.CURDIR}/../scripts/mkver ntpdc -afterinstall: - rm -f ${DESTDIR}/usr/sbin/xntpdc - .include From owner-svn-src-stable@freebsd.org Sat Oct 3 18:40:28 2015 Return-Path: Delivered-To: svn-src-stable@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 4DF8AA0E8F8; Sat, 3 Oct 2015 18:40:28 +0000 (UTC) (envelope-from bdrewery@FreeBSD.org) Received: from repo.freebsd.org (repo.freebsd.org [IPv6:2001:1900:2254:2068::e6a:0]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 34FC1186D; Sat, 3 Oct 2015 18:40:28 +0000 (UTC) (envelope-from bdrewery@FreeBSD.org) Received: from repo.freebsd.org ([127.0.1.70]) by repo.freebsd.org (8.15.2/8.15.2) with ESMTP id t93IeSiK053273; Sat, 3 Oct 2015 18:40:28 GMT (envelope-from bdrewery@FreeBSD.org) Received: (from bdrewery@localhost) by repo.freebsd.org (8.15.2/8.15.2/Submit) id t93IeSME053272; Sat, 3 Oct 2015 18:40:28 GMT (envelope-from bdrewery@FreeBSD.org) Message-Id: <201510031840.t93IeSME053272@repo.freebsd.org> X-Authentication-Warning: repo.freebsd.org: bdrewery set sender to bdrewery@FreeBSD.org using -f From: Bryan Drewery Date: Sat, 3 Oct 2015 18:40:28 +0000 (UTC) To: src-committers@freebsd.org, svn-src-all@freebsd.org, svn-src-stable@freebsd.org, svn-src-stable-10@freebsd.org Subject: svn commit: r288624 - stable/10/lib/ncurses/ncurses X-SVN-Group: stable-10 MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit X-BeenThere: svn-src-stable@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: SVN commit messages for all the -stable branches of the src tree List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 03 Oct 2015 18:40:28 -0000 Author: bdrewery Date: Sat Oct 3 18:40:27 2015 New Revision: 288624 URL: https://svnweb.freebsd.org/changeset/base/288624 Log: Partially revert r288607, stable/10 does not use ncurses_dll.h.in like head does, so this file should not be in CLEANFILES / .NOPATH. Modified: stable/10/lib/ncurses/ncurses/Makefile Modified: stable/10/lib/ncurses/ncurses/Makefile ============================================================================== --- stable/10/lib/ncurses/ncurses/Makefile Sat Oct 3 17:49:11 2015 (r288623) +++ stable/10/lib/ncurses/ncurses/Makefile Sat Oct 3 18:40:27 2015 (r288624) @@ -282,7 +282,6 @@ CFLAGS+= -DFREEBSD_NATIVE -DTERMIOS # Installed HEADERS= curses.h term.h termcap.h unctrl.h SRCHDRS= ncurses_dll.h -CLEANFILES+= ncurses_dll.h .if defined(ENABLE_WIDEC) INCS= ${HEADERS} ${SRCHDRS}