From owner-freebsd-amd64@freebsd.org Sun Mar 19 00:16:26 2017 Return-Path: Delivered-To: freebsd-amd64@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 5C3E0D080C1 for ; Sun, 19 Mar 2017 00:16:26 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2001:1900:2254:206a::16:76]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 3D5E51AA0 for ; Sun, 19 Mar 2017 00:16:26 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from bugs.freebsd.org ([127.0.1.118]) by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id v2J0GQPq071229 for ; Sun, 19 Mar 2017 00:16:26 GMT (envelope-from bugzilla-noreply@freebsd.org) From: bugzilla-noreply@freebsd.org To: freebsd-amd64@FreeBSD.org Subject: [Bug 217138] head (e.g.) -r314638 for arm64: sh vs. jemalloc asserts: include/jemalloc/internal/tsd.h:687: Failed assertion: "tsd_booted" once swapped in after being swapped out (comment 10) Date: Sun, 19 Mar 2017 00:16:26 +0000 X-Bugzilla-Reason: CC X-Bugzilla-Type: changed X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: Base System X-Bugzilla-Component: bin X-Bugzilla-Version: CURRENT X-Bugzilla-Keywords: X-Bugzilla-Severity: Affects Some People X-Bugzilla-Who: markmi@dsl-only.net X-Bugzilla-Status: New X-Bugzilla-Resolution: X-Bugzilla-Priority: --- X-Bugzilla-Assigned-To: freebsd-bugs@FreeBSD.org X-Bugzilla-Flags: X-Bugzilla-Changed-Fields: Message-ID: In-Reply-To: References: Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated MIME-Version: 1.0 X-BeenThere: freebsd-amd64@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: Porting FreeBSD to the AMD64 platform List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 19 Mar 2017 00:16:26 -0000 https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D217138 --- Comment #25 from Mark Millard --- Here are procstat -v results from while the two processes from the fork of my test program have already been swapped-out (before they run and fail during swap-in). I also show the results from looking at the matching core files. The start of the address range of that ends up all-zero when none of it should be is: (lldb) print dyn_region (region *volatile) $0 =3D 0x0000000040616000 # procstat -v 1954 PID START END PRT RES PRES REF SHD FLAG TP P= ATH 1954 0x10000 0x11000 r-- 1 51 5 1 CN-- vn /root/c_tests/a.out 1954 0x20000 0x21000 r-x 1 51 5 1 CN-- vn /root/c_tests/a.out 1954 0x30000 0x40000 rw- 16 0 1 0 C--- vn /root/c_tests/a.out 1954 0x40000 0x41000 r-- 0 0 2 0 CN-- sw=20 1954 0x41000 0x54000 rw- 0 0 2 0 CN-- sw=20 1954 0x40030000 0x4004a000 r-x 26 29 34 0 CN-- vn /libexec/ld-elf.so.1 1954 0x4004a000 0x40051000 rw- 3 0 1 0 C--- sw=20 1954 0x4005a000 0x4005c000 rw- 1 0 1 0 C--- sw=20 1954 0x4005c000 0x401b3000 r-x 343 384 61 27 CN-- vn /lib/libc.so.7 1954 0x401b3000 0x401c2000 --- 0 0 2 0 CN-- df=20 1954 0x401c2000 0x401cf000 rw- 13 0 2 0 CN-- vn /lib/libc.so.7 1954 0x401cf000 0x40201000 rw- 1 1 2 0 CN-- sw=20 1954 0x40400000 0x40800000 rw- 0 0 2 0 CN-- sw=20 1954 0xfffffffdf000 0xfffffffff000 rw- 0 0 1 0 C--D sw=20 1954 0xfffffffff000 0x1000000000000 r-x 1 1 37 0 ---- ph=20 # procstat -v 1955 PID START END PRT RES PRES REF SHD FLAG TP P= ATH 1955 0x10000 0x11000 r-- 1 51 5 1 CN-- vn /root/c_tests/a.out 1955 0x20000 0x21000 r-x 1 51 5 1 CN-- vn /root/c_tests/a.out 1955 0x30000 0x40000 rw- 16 0 1 0 C--- vn /root/c_tests/a.out 1955 0x40000 0x41000 r-- 0 0 2 0 CN-- sw=20 1955 0x41000 0x54000 rw- 0 0 2 0 CN-- sw=20 1955 0x40030000 0x4004a000 r-x 26 29 34 0 CN-- vn /libexec/ld-elf.so.1 1955 0x4004a000 0x40051000 rw- 3 0 1 0 C--- sw=20 1955 0x4005a000 0x4005c000 rw- 1 0 1 0 C--- sw=20 1955 0x4005c000 0x401b3000 r-x 343 384 61 27 CN-- vn /lib/libc.so.7 1955 0x401b3000 0x401c2000 --- 0 0 2 0 CN-- df=20 1955 0x401c2000 0x401cf000 rw- 13 0 2 0 CN-- vn /lib/libc.so.7 1955 0x401cf000 0x40201000 rw- 1 1 2 0 CN-- sw=20 1955 0x40400000 0x40800000 rw- 0 0 2 0 CN-- sw=20 1955 0xfffffffdf000 0xfffffffff000 rw- 0 0 1 0 C--D sw=20 1955 0xfffffffff000 0x1000000000000 r-x 1 1 37 0 ---- ph=20 The core file results are: # procstat -v /var/crash/a.out.1954.core PID START END PRT RES PRES REF SHD FLAG TP P= ATH 1954 0x10000 0x11000 r-- 1 51 3 1 CN-- vn /root/c_tests/a.out 1954 0x20000 0x21000 r-x 1 51 3 1 CN-- vn /root/c_tests/a.out 1954 0x30000 0x40000 rw- 16 0 1 0 C--- vn /root/c_tests/a.out 1954 0x40000 0x41000 r-- 1 1 1 0 CN-- sw=20 1954 0x41000 0x54000 rw- 4 4 1 0 C--- sw=20 1954 0x40030000 0x4004a000 r-x 26 29 30 0 CN-- vn /libexec/ld-elf.so.1 1954 0x4004a000 0x40051000 rw- 7 7 1 0 C--- sw=20 1954 0x4005a000 0x4005c000 rw- 2 2 1 0 C--- sw=20 1954 0x4005c000 0x401b3000 r-x 343 384 55 25 CN-- vn /lib/libc.so.7 1954 0x401b3000 0x401c2000 --- 0 0 1 0 CN-- df=20 1954 0x401c2000 0x401cf000 rw- 13 0 1 0 C--- vn /lib/libc.so.7 1954 0x401cf000 0x40201000 rw- 50 50 1 0 CN-- sw=20 1954 0x40400000 0x40800000 rw- 1024 1024 1 0 CN-- sw=20 1954 0xfffffffdf000 0xfffffffff000 rw- 3 3 1 0 C--D sw=20 1954 0xfffffffff000 0x1000000000000 r-x 1 1 33 0 ---- ph=20 # procstat -v /var/crash/a.out.1955.core PID START END PRT RES PRES REF SHD FLAG TP P= ATH 1955 0x10000 0x11000 r-- 1 51 5 1 CN-- vn /root/c_tests/a.out 1955 0x20000 0x21000 r-x 1 51 5 1 CN-- vn /root/c_tests/a.out 1955 0x30000 0x40000 rw- 16 0 1 0 C--- vn /root/c_tests/a.out 1955 0x40000 0x41000 r-- 0 0 2 0 CN-- sw=20 1955 0x41000 0x54000 rw- 4 0 1 0 C--- sw=20 1955 0x40030000 0x4004a000 r-x 26 29 31 0 CN-- vn /libexec/ld-elf.so.1 1955 0x4004a000 0x40051000 rw- 4 0 1 0 C--- sw=20 1955 0x4005a000 0x4005c000 rw- 2 0 1 0 C--- sw=20 1955 0x4005c000 0x401b3000 r-x 343 384 56 25 CN-- vn /lib/libc.so.7 1955 0x401b3000 0x401c2000 --- 0 0 2 0 CN-- df=20 1955 0x401c2000 0x401cf000 rw- 13 0 1 0 C--- vn /lib/libc.so.7 1955 0x401cf000 0x40201000 rw- 1 1 2 0 CN-- sw=20 1955 0x40400000 0x40800000 rw- 1 1 2 0 CN-- sw=20 1955 0xfffffffdf000 0xfffffffff000 rw- 1 0 1 0 C--D sw=20 1955 0xfffffffff000 0x1000000000000 r-x 1 1 34 0 ---- ph --=20 You are receiving this mail because: You are on the CC list for the bug.= From owner-freebsd-amd64@freebsd.org Sun Mar 19 00:29:32 2017 Return-Path: Delivered-To: freebsd-amd64@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 74C88D0854A for ; Sun, 19 Mar 2017 00:29:32 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2001:1900:2254:206a::16:76]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 5B5C5129D for ; Sun, 19 Mar 2017 00:29:32 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from bugs.freebsd.org ([127.0.1.118]) by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id v2J0TVk6098003 for ; Sun, 19 Mar 2017 00:29:32 GMT (envelope-from bugzilla-noreply@freebsd.org) From: bugzilla-noreply@freebsd.org To: freebsd-amd64@FreeBSD.org Subject: [Bug 217138] head (e.g.) -r314638 for arm64: sh vs. jemalloc asserts: include/jemalloc/internal/tsd.h:687: Failed assertion: "tsd_booted" once swapped in after being swapped out (comment 10) Date: Sun, 19 Mar 2017 00:29:32 +0000 X-Bugzilla-Reason: CC X-Bugzilla-Type: changed X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: Base System X-Bugzilla-Component: bin X-Bugzilla-Version: CURRENT X-Bugzilla-Keywords: X-Bugzilla-Severity: Affects Some People X-Bugzilla-Who: markmi@dsl-only.net X-Bugzilla-Status: New X-Bugzilla-Resolution: X-Bugzilla-Priority: --- X-Bugzilla-Assigned-To: freebsd-bugs@FreeBSD.org X-Bugzilla-Flags: X-Bugzilla-Changed-Fields: Message-ID: In-Reply-To: References: Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated MIME-Version: 1.0 X-BeenThere: freebsd-amd64@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: Porting FreeBSD to the AMD64 platform List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 19 Mar 2017 00:29:32 -0000 https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D217138 --- Comment #26 from Mark Millard --- Using procstat -v on one of the sh.*.core files from a process that directly reported the Failed assertion: "tds_booted" shows the following. But first the address of the tsd_booted storage: (lldb) print &__je_tsd_booted (bool *) $0 =3D 0x0000000040618520 # procstat -v /var/crash/sh.1766.core=20 PID START END PRT RES PRES REF SHD FLAG TP P= ATH 1766 0x400000 0x423000 r-x 35 37 7 0 CN-- vn /bin/sh 1766 0x433000 0x436000 rw- 3 3 1 0 C--- sw=20 1766 0x40433000 0x4044d000 r-x 26 29 32 0 CN-- vn /libexec/ld-elf.so.1 1766 0x4044d000 0x40454000 rw- 3 3 1 0 C--- sw=20 1766 0x4045d000 0x4045f000 rw- 2 2 1 0 C--- sw=20 1766 0x4045f000 0x4048b000 r-x 44 48 14 7 CN-- vn /lib/libedit.so.7 1766 0x4048b000 0x4049b000 --- 0 0 1 0 CN-- df=20 1766 0x4049b000 0x4049d000 rw- 2 0 1 0 CN-- vn /lib/libedit.so.7 1766 0x4049d000 0x404a1000 rw- 0 0 1 0 CN-- sw=20 1766 0x404a1000 0x405f8000 r-x 343 384 58 26 CN-- vn /lib/libc.so.7 1766 0x405f8000 0x40607000 --- 0 0 1 0 CN-- df=20 1766 0x40607000 0x40614000 rw- 13 0 1 0 C--- vn /lib/libc.so.7 1766 0x40614000 0x4062d000 rw- 5 5 1 0 C--- sw=20 1766 0x4062d000 0x4067a000 r-x 77 82 16 8 CN-- vn /lib/libncursesw.so.8 1766 0x4067a000 0x40689000 --- 0 0 1 0 CN-- df=20 1766 0x40689000 0x4068e000 rw- 5 0 1 0 CN-- vn /lib/libncursesw.so.8 1766 0x4068e000 0x406ad000 rw- 0 0 1 0 C--- sw=20 1766 0x40800000 0x40c00000 rw- 9 9 1 0 C--- sw=20 1766 0xfffffffdf000 0xfffffffff000 rw- 6 6 1 0 C--D sw=20 1766 0xfffffffff000 0x1000000000000 r-x 1 1 34 0 ---- ph=20 Similarly for an example of su.*.core that reported the assertion failure: (lldb) print &__je_tsd_booted (bool *) $0 =3D 0x000000004061d520 # procstat -v /var/crash/su.1765.core=20 PID START END PRT RES PRES REF SHD FLAG TP P= ATH 1765 0x400000 0x404000 r-x 4 5 3 0 CN-- vn /usr/bin/su 1765 0x413000 0x414000 rw- 1 1 1 0 C--- sw=20 1765 0x40413000 0x4042d000 r-x 26 29 30 0 CN-- vn /libexec/ld-elf.so.1 1765 0x4042d000 0x40435000 rw- 8 8 1 0 C--- sw=20 1765 0x4043d000 0x4043f000 rw- 2 2 1 0 C--- sw=20 1765 0x4043f000 0x4044e000 r-x 15 16 33 16 CN-- vn /lib/libutil.so.9 1765 0x4044e000 0x4045e000 --- 0 0 1 0 CN-- df=20 1765 0x4045e000 0x4045f000 rw- 1 0 1 0 CN-- vn /lib/libutil.so.9 1765 0x4045f000 0x40461000 rw- 0 0 1 0 CN-- sw=20 1765 0x40461000 0x4046c000 r-x 11 13 21 9 CN-- vn /usr/lib/libpam.so.6 1765 0x4046c000 0x4047c000 --- 0 0 1 0 CN-- df=20 1765 0x4047c000 0x4047d000 rw- 1 0 1 0 C--- vn /usr/lib/libpam.so.6 1765 0x4047d000 0x40494000 r-x 8 8 19 8 CN-- vn /usr/lib/libbsm.so.3 1765 0x40494000 0x404a4000 --- 0 0 1 0 CN-- df=20 1765 0x404a4000 0x404a6000 rw- 0 0 1 0 CN-- vn /usr/lib/libbsm.so.3 1765 0x404a6000 0x405fd000 r-x 343 384 54 24 CN-- vn /lib/libc.so.7 1765 0x405fd000 0x4060c000 --- 0 0 1 0 CN-- df=20 1765 0x4060c000 0x40619000 rw- 13 0 1 0 CN-- vn /lib/libc.so.7 1765 0x40619000 0x40653000 rw- 4 4 1 0 C--- sw=20 1765 0x40653000 0x40654000 r-x 0 0 6 3 CN-- vn /usr/lib/pam_rootok.so.6 1765 0x40654000 0x40663000 --- 0 0 1 0 CN-- df=20 1765 0x40663000 0x40664000 rw- 0 0 1 0 CN-- vn /usr/lib/pam_rootok.so.6 1765 0x40664000 0x40665000 r-x 0 0 8 4 CN-- vn /usr/lib/pam_self.so.6 1765 0x40665000 0x40674000 --- 0 0 1 0 CN-- df=20 1765 0x40674000 0x40675000 rw- 0 0 1 0 CN-- vn /usr/lib/pam_self.so.6 1765 0x40675000 0x40676000 r-x 0 0 6 3 CN-- vn /usr/lib/pam_group.so.6 1765 0x40676000 0x40685000 --- 0 0 1 0 CN-- df=20 1765 0x40685000 0x40686000 rw- 0 0 1 0 CN-- vn /usr/lib/pam_group.so.6 1765 0x40686000 0x40687000 r-x 0 0 17 7 CN-- vn /usr/lib/pam_opie.so.6 1765 0x40687000 0x40697000 --- 0 0 1 0 CN-- df=20 1765 0x40697000 0x40698000 rw- 0 0 1 0 CN-- vn /usr/lib/pam_opie.so.6 1765 0x40698000 0x4069e000 r-x 0 0 17 7 CN-- vn /usr/lib/libopie.so.8 1765 0x4069e000 0x406ae000 --- 0 0 1 0 CN-- df=20 1765 0x406ae000 0x406b1000 rw- 0 0 1 0 CN-- vn /usr/lib/libopie.so.8 1765 0x406b1000 0x406d1000 r-x 0 0 17 7 CN-- vn /lib/libmd.so.6 1765 0x406d1000 0x406e0000 --- 0 0 1 0 CN-- df=20 1765 0x406e0000 0x406e1000 rw- 0 0 1 0 CN-- vn /lib/libmd.so.6 1765 0x406e1000 0x406e2000 r-x 0 0 17 7 CN-- vn /usr/lib/pam_opieaccess.so.6 1765 0x406e2000 0x406f1000 --- 0 0 1 0 CN-- df=20 1765 0x406f1000 0x406f2000 rw- 0 0 1 0 CN-- vn /usr/lib/pam_opieaccess.so.6 1765 0x406f2000 0x406f5000 r-x 0 0 45 18 CN-- vn /usr/lib/pam_unix.so.6 1765 0x406f5000 0x40704000 --- 0 0 1 0 CN-- df=20 1765 0x40704000 0x40705000 rw- 0 0 1 0 CN-- vn /usr/lib/pam_unix.so.6 1765 0x40705000 0x40710000 r-x 0 0 19 8 CN-- vn /lib/libcrypt.so.5 1765 0x40710000 0x40720000 --- 0 0 1 0 CN-- df=20 1765 0x40720000 0x40721000 rw- 0 0 1 0 CN-- vn /lib/libcrypt.so.5 1765 0x40721000 0x40732000 rw- 0 0 0 0 ---- --=20 1765 0x40732000 0x40736000 r-x 0 0 17 7 CN-- vn /usr/lib/libypclnt.so.4 1765 0x40736000 0x40745000 --- 0 0 1 0 CN-- df=20 1765 0x40745000 0x40746000 rw- 0 0 1 0 CN-- vn /usr/lib/libypclnt.so.4 1765 0x40746000 0x4074f000 rw- 0 0 1 0 CN-- sw=20 1765 0x4074f000 0x40751000 r-x 0 0 17 7 CN-- vn /usr/lib/pam_login_access.so.6 1765 0x40751000 0x40760000 --- 0 0 1 0 CN-- df=20 1765 0x40760000 0x40761000 rw- 0 0 1 0 CN-- vn /usr/lib/pam_login_access.so.6 1765 0x40761000 0x40764000 r-x 0 0 45 18 CN-- vn /usr/lib/pam_unix.so.6 1765 0x40764000 0x40773000 --- 0 0 1 0 CN-- df=20 1765 0x40773000 0x40774000 rw- 0 0 1 0 CN-- vn /usr/lib/pam_unix.so.6 1765 0x40774000 0x40775000 r-x 0 0 21 9 CN-- vn /usr/lib/pam_permit.so.6 1765 0x40775000 0x40784000 --- 0 0 1 0 CN-- df=20 1765 0x40784000 0x40785000 rw- 0 0 1 0 CN-- vn /usr/lib/pam_permit.so.6 1765 0x40785000 0x40786000 r-x 0 0 21 9 CN-- vn /usr/lib/pam_permit.so.6 1765 0x40786000 0x40795000 --- 0 0 1 0 CN-- df=20 1765 0x40795000 0x40796000 rw- 0 0 1 0 CN-- vn /usr/lib/pam_permit.so.6 1765 0x40800000 0x40c00000 rw- 8 8 1 0 C--- sw=20 1765 0xfffffffdf000 0xfffffffff000 rw- 2 2 1 0 C--D sw=20 1765 0xfffffffff000 0x1000000000000 r-x 1 1 33 0 ---- ph --=20 You are receiving this mail because: You are on the CC list for the bug.= From owner-freebsd-amd64@freebsd.org Sun Mar 19 00:38:10 2017 Return-Path: Delivered-To: freebsd-amd64@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 1ECE6D08BE4 for ; Sun, 19 Mar 2017 00:38:10 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2001:1900:2254:206a::16:76]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 0D9361029 for ; Sun, 19 Mar 2017 00:38:10 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from bugs.freebsd.org ([127.0.1.118]) by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id v2J0c9fh019362 for ; Sun, 19 Mar 2017 00:38:09 GMT (envelope-from bugzilla-noreply@freebsd.org) From: bugzilla-noreply@freebsd.org To: freebsd-amd64@FreeBSD.org Subject: [Bug 217138] head (e.g.) -r314638 for arm64: sh vs. jemalloc asserts: include/jemalloc/internal/tsd.h:687: Failed assertion: "tsd_booted" once swapped in after being swapped out (comment 10) Date: Sun, 19 Mar 2017 00:38:10 +0000 X-Bugzilla-Reason: CC X-Bugzilla-Type: changed X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: Base System X-Bugzilla-Component: bin X-Bugzilla-Version: CURRENT X-Bugzilla-Keywords: X-Bugzilla-Severity: Affects Some People X-Bugzilla-Who: markmi@dsl-only.net X-Bugzilla-Status: New X-Bugzilla-Resolution: X-Bugzilla-Priority: --- X-Bugzilla-Assigned-To: freebsd-bugs@FreeBSD.org X-Bugzilla-Flags: X-Bugzilla-Changed-Fields: Message-ID: In-Reply-To: References: Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated MIME-Version: 1.0 X-BeenThere: freebsd-amd64@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: Porting FreeBSD to the AMD64 platform List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 19 Mar 2017 00:38:10 -0000 https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D217138 --- Comment #27 from Mark Millard --- The earlier procstat -v comments have something interesting in common other than the procstat results themselves: the 3 separate programs all end up with zeros in the same general memory area of each process: (lldb) print dyn_region (region *volatile) $0 =3D 0x0000000040616000 (lldb) print &__je_tsd_booted (bool *) $0 =3D 0x0000000040618520 (lldb) print &__je_tsd_booted (bool *) $0 =3D 0x0000000040618520 The first is from dynamic allocation ending up in the area. The other two are from libc.so.7 globals/statics ending up in the general area. It looks like something is trashing a specific memory area for some reason. --=20 You are receiving this mail because: You are on the CC list for the bug.= From owner-freebsd-amd64@freebsd.org Sun Mar 19 02:24:51 2017 Return-Path: Delivered-To: freebsd-amd64@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 002B8D08ABE for ; Sun, 19 Mar 2017 02:24:51 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2001:1900:2254:206a::16:76]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id E30E91086 for ; Sun, 19 Mar 2017 02:24:50 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from bugs.freebsd.org ([127.0.1.118]) by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id v2J2Oo0s012302 for ; Sun, 19 Mar 2017 02:24:50 GMT (envelope-from bugzilla-noreply@freebsd.org) From: bugzilla-noreply@freebsd.org To: freebsd-amd64@FreeBSD.org Subject: [Bug 217138] head (e.g.) -r314638 for arm64: sh vs. jemalloc asserts: include/jemalloc/internal/tsd.h:687: Failed assertion: "tsd_booted" once swapped in after being swapped out (comment 10) Date: Sun, 19 Mar 2017 02:24:50 +0000 X-Bugzilla-Reason: CC X-Bugzilla-Type: changed X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: Base System X-Bugzilla-Component: bin X-Bugzilla-Version: CURRENT X-Bugzilla-Keywords: X-Bugzilla-Severity: Affects Some People X-Bugzilla-Who: markmi@dsl-only.net X-Bugzilla-Status: New X-Bugzilla-Resolution: X-Bugzilla-Priority: --- X-Bugzilla-Assigned-To: freebsd-bugs@FreeBSD.org X-Bugzilla-Flags: X-Bugzilla-Changed-Fields: Message-ID: In-Reply-To: References: Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated MIME-Version: 1.0 X-BeenThere: freebsd-amd64@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: Porting FreeBSD to the AMD64 platform List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 19 Mar 2017 02:24:51 -0000 https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D217138 --- Comment #28 from Mark Millard --- (In reply to Mark Millard from comment #27) I incorrectly copy/pasted one of the addresses. The last one below is the correction: (lldb) print dyn_region (region *volatile) $0 =3D 0x0000000040616000 (lldb) print &__je_tsd_booted (bool *) $0 =3D 0x0000000040618520 (lldb) print &__je_tsd_booted (bool *) $0 =3D 0x000000004061d520 --=20 You are receiving this mail because: You are on the CC list for the bug.= From owner-freebsd-amd64@freebsd.org Wed Mar 22 02:27:20 2017 Return-Path: Delivered-To: freebsd-amd64@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 77342D17156 for ; Wed, 22 Mar 2017 02:27:20 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2001:1900:2254:206a::16:76]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 5D44110E4 for ; Wed, 22 Mar 2017 02:27:20 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from bugs.freebsd.org ([127.0.1.118]) by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id v2M2RK2S065435 for ; Wed, 22 Mar 2017 02:27:20 GMT (envelope-from bugzilla-noreply@freebsd.org) From: bugzilla-noreply@freebsd.org To: freebsd-amd64@FreeBSD.org Subject: [Bug 217138] head (e.g.) -r314638 for arm64: sh vs. jemalloc asserts: include/jemalloc/internal/tsd.h:687: Failed assertion: "tsd_booted" once swapped in after being swapped out (comment 10) Date: Wed, 22 Mar 2017 02:27:20 +0000 X-Bugzilla-Reason: CC X-Bugzilla-Type: changed X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: Base System X-Bugzilla-Component: bin X-Bugzilla-Version: CURRENT X-Bugzilla-Keywords: X-Bugzilla-Severity: Affects Some People X-Bugzilla-Who: markmi@dsl-only.net X-Bugzilla-Status: New X-Bugzilla-Resolution: X-Bugzilla-Priority: --- X-Bugzilla-Assigned-To: freebsd-bugs@FreeBSD.org X-Bugzilla-Flags: X-Bugzilla-Changed-Fields: attachments.created Message-ID: In-Reply-To: References: Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated MIME-Version: 1.0 X-BeenThere: freebsd-amd64@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: Porting FreeBSD to the AMD64 platform List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 22 Mar 2017 02:27:20 -0000 https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D217138 --- Comment #29 from Mark Millard --- Created attachment 181044 --> https://bugs.freebsd.org/bugzilla/attachment.cgi?id=3D181044&action= =3Dedit Program showing posix_madvise with POSIX_MADV_WILLNEED can prevent the fail= ure posix_madvise(.,.,POSIX_MADV_WILLNEED) in the child process after the fork but before the sleep/swap-out avoids the failure. In the parent process it makes no difference. The .,. is the address and size for a memory region that is to avoid the problem. --=20 You are receiving this mail because: You are on the CC list for the bug.= From owner-freebsd-amd64@freebsd.org Wed Mar 22 07:19:39 2017 Return-Path: Delivered-To: freebsd-amd64@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 3C1EED176F1 for ; Wed, 22 Mar 2017 07:19:39 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2001:1900:2254:206a::16:76]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 2AE82922 for ; Wed, 22 Mar 2017 07:19:39 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from bugs.freebsd.org ([127.0.1.118]) by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id v2M7JcVN035182 for ; Wed, 22 Mar 2017 07:19:39 GMT (envelope-from bugzilla-noreply@freebsd.org) From: bugzilla-noreply@freebsd.org To: freebsd-amd64@FreeBSD.org Subject: [Bug 217138] head (e.g.) -r314638 for arm64: sh vs. jemalloc asserts: include/jemalloc/internal/tsd.h:687: Failed assertion: "tsd_booted" once swapped in after being swapped out (comment 10) Date: Wed, 22 Mar 2017 07:19:39 +0000 X-Bugzilla-Reason: CC X-Bugzilla-Type: changed X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: Base System X-Bugzilla-Component: bin X-Bugzilla-Version: CURRENT X-Bugzilla-Keywords: X-Bugzilla-Severity: Affects Some People X-Bugzilla-Who: markmi@dsl-only.net X-Bugzilla-Status: New X-Bugzilla-Resolution: X-Bugzilla-Priority: --- X-Bugzilla-Assigned-To: freebsd-bugs@FreeBSD.org X-Bugzilla-Flags: X-Bugzilla-Changed-Fields: Message-ID: In-Reply-To: References: Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated MIME-Version: 1.0 X-BeenThere: freebsd-amd64@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: Porting FreeBSD to the AMD64 platform List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 22 Mar 2017 07:19:39 -0000 https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D217138 --- Comment #30 from Mark Millard --- (In reply to Mark Millard from comment #29) It turns out that POSIX_MADV_WILLNEED can instead be POSIX_MADV_DONTNEED and it still prevents the failures. --=20 You are receiving this mail because: You are on the CC list for the bug.= From owner-freebsd-amd64@freebsd.org Wed Mar 22 07:31:26 2017 Return-Path: Delivered-To: freebsd-amd64@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 41C98D17ACE for ; Wed, 22 Mar 2017 07:31:26 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2001:1900:2254:206a::16:76]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 30788FA4 for ; Wed, 22 Mar 2017 07:31:26 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from bugs.freebsd.org ([127.0.1.118]) by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id v2M7VP6n068289 for ; Wed, 22 Mar 2017 07:31:26 GMT (envelope-from bugzilla-noreply@freebsd.org) From: bugzilla-noreply@freebsd.org To: freebsd-amd64@FreeBSD.org Subject: [Bug 217138] head (e.g.) -r314638 for arm64: sh vs. jemalloc asserts: include/jemalloc/internal/tsd.h:687: Failed assertion: "tsd_booted" once swapped in after being swapped out (comment 10) Date: Wed, 22 Mar 2017 07:31:26 +0000 X-Bugzilla-Reason: CC X-Bugzilla-Type: changed X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: Base System X-Bugzilla-Component: bin X-Bugzilla-Version: CURRENT X-Bugzilla-Keywords: X-Bugzilla-Severity: Affects Some People X-Bugzilla-Who: markmi@dsl-only.net X-Bugzilla-Status: New X-Bugzilla-Resolution: X-Bugzilla-Priority: --- X-Bugzilla-Assigned-To: freebsd-bugs@FreeBSD.org X-Bugzilla-Flags: X-Bugzilla-Changed-Fields: Message-ID: In-Reply-To: References: Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated MIME-Version: 1.0 X-BeenThere: freebsd-amd64@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: Porting FreeBSD to the AMD64 platform List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 22 Mar 2017 07:31:26 -0000 https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D217138 --- Comment #31 from Mark Millard --- (In reply to Mark Millard from comment #30) Ignore the POSIX_MADV_DONTNEED: the example child had not reached the zero RES(ident memory) status. When it does it fails (and so to the ancestor processes if they also reached such). --=20 You are receiving this mail because: You are on the CC list for the bug.= From owner-freebsd-amd64@freebsd.org Wed Mar 22 20:30:26 2017 Return-Path: Delivered-To: freebsd-amd64@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 65214D17C10 for ; Wed, 22 Mar 2017 20:30:26 +0000 (UTC) (envelope-from truckman@FreeBSD.org) Received: from gw.catspoiler.org (unknown [IPv6:2602:304:b010:ef20::f2]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "gw.catspoiler.org", Issuer "gw.catspoiler.org" (not verified)) by mx1.freebsd.org (Postfix) with ESMTPS id 23A141D8E for ; Wed, 22 Mar 2017 20:30:26 +0000 (UTC) (envelope-from truckman@FreeBSD.org) Received: from FreeBSD.org (mousie.catspoiler.org [192.168.101.2]) by gw.catspoiler.org (8.15.2/8.15.2) with ESMTP id v2MKUJJs026400 for ; Wed, 22 Mar 2017 13:30:23 -0700 (PDT) (envelope-from truckman@FreeBSD.org) Message-Id: <201703222030.v2MKUJJs026400@gw.catspoiler.org> Date: Wed, 22 Mar 2017 13:30:19 -0700 (PDT) From: Don Lewis Subject: FreeBSD on Ryzen To: freebsd-amd64@FreeBSD.org MIME-Version: 1.0 Content-Type: TEXT/plain; charset=us-ascii X-BeenThere: freebsd-amd64@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: Porting FreeBSD to the AMD64 platform List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 22 Mar 2017 20:30:26 -0000 I put together a Ryzen 1700X machine over the weekend and installed the 12.0-CURRENT r315413 snapshot on it a couple of days ago. The RAM is DDR4 2400. First impression is that it's pretty zippy. Compared to my previous fastest machine: CPU: AMD FX-8320E Eight-Core Processor (3210.84-MHz K8-class CPU) make -j8 buildworld using tmpfs is a bit more than 2x faster. Since the Ryzen has SMT, it's eight cores look like 16 CPUs to FreeBSD, I get almost a 2.6x speedup with -j16 as compared to my old machine. I do see that the reported total CPU time increases quite a bit at -j16 (~19900u) as compared to -j8 (~13600u) so it is running into some hardware bottlenecks that are slowing down instruction execution. It could be the resources shared by both SMT threads that share each core, or it could be cache or memory bandwidth related. The Ryzen topology is a bit complicated. There are two groups of four cores, where each group of four cores shares half of the L3 cache, with a slowish interconnect bus between the groups. This probably causes some NUMA-like issues. I wonder if the ULE scheduler could be tweaked to handle this better. % sysctl kern.sched.topology_spec kern.sched.topology_spec: 0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15 0, 1, 2, 3, 4, 5, 6, 7 0, 1 2, 3 4, 5 6, 7 8, 9, 10, 11, 12, 13, 14, 15 8, 9 10, 11 12, 13 14, 15 I'll run some tests with SMT disabled when I get a chance. The amdtemp driver doesn't attach to this CPU, so I can't monitor the temperature at runtime. I've got ECC RAM, but unfortunately the motherboard I'm using doesn't enable ECC. The chipset is the B350. Copyright (c) 1992-2017 The FreeBSD Project. Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994 The Regents of the University of California. All rights reserved. FreeBSD is a registered trademark of The FreeBSD Foundation. FreeBSD 12.0-CURRENT #0 r315413: Thu Mar 16 17:23:31 UTC 2017 root@releng3.nyi.freebsd.org:/usr/obj/usr/src/sys/GENERIC amd64 FreeBSD clang version 4.0.0 (tags/RELEASE_400/final 297347) (based on LLVM 4.0.0) WARNING: WITNESS option enabled, expect reduced performance. SRAT: No memory found for CPU 0 VT(vga): resolution 640x480 CPU: AMD Ryzen 7 1700X Eight-Core Processor (3393.71-MHz K8-class CPU) Origin="AuthenticAMD" Id=0x800f11 Family=0x17 Model=0x1 Stepping=1 Features=0x178bfbff Features2=0x7ed8320b AMD Features=0x2e500800 AMD Features2=0x35c233ff Structured Extended Features=0x209c01a9 XSAVE Features=0xf SVM: NP,NRIP,VClean,AFlush,DAssist,NAsids=32768 TSC: P-state invariant, performance statistics real memory = 68719476736 (65536 MB) avail memory = 66671529984 (63582 MB) Event timer "LAPIC" quality 100 ACPI APIC Table: FreeBSD/SMP: Multiprocessor System Detected: 16 CPUs FreeBSD/SMP: 1 package(s) x 16 core(s) random: unblocking device. ACPI BIOS Warning (bug): Optional FADT field Pm2ControlBlock has valid Length but zero Address: 0x0000000000000000/0x1 (20170303/tbfadt-796) ioapic0: Changing APIC ID to 17 ioapic1: Changing APIC ID to 18 ioapic0 irqs 0-23 on motherboard ioapic1 irqs 24-55 on motherboard SMP: AP CPU #12 Launched! SMP: AP CPU #1 Launched! SMP: AP CPU #4 Launched! SMP: AP CPU #7 Launched! SMP: AP CPU #6 Launched! SMP: AP CPU #3 Launched! SMP: AP CPU #11 Launched! SMP: AP CPU #5 Launched! SMP: AP CPU #15 Launched! SMP: AP CPU #9 Launched! SMP: AP CPU #10 Launched! SMP: AP CPU #8 Launched! SMP: AP CPU #2 Launched! SMP: AP CPU #14 Launched! SMP: AP CPU #13 Launched! Timecounter "TSC-low" frequency 1696854133 Hz quality 1000 random: entropy device external interface netmap: loaded module module_register_init: MOD_LOAD (vesa, 0xffffffff80f49c70, 0) error 19 random: registering fast source Intel Secure Key RNG random: fast provider: "Intel Secure Key RNG" kbd1 at kbdmux0 nexus0 vtvga0: on motherboard cryptosoft0: on motherboard acpi0: on motherboard acpi0: Power Button (fixed) cpu0: on acpi0 cpu1: on acpi0 cpu2: on acpi0 cpu3: on acpi0 cpu4: on acpi0 cpu5: on acpi0 cpu6: on acpi0 cpu7: on acpi0 cpu8: on acpi0 cpu9: on acpi0 cpu10: on acpi0 cpu11: on acpi0 cpu12: on acpi0 cpu13: on acpi0 cpu14: on acpi0 cpu15: on acpi0 attimer0: port 0x40-0x43 irq 0 on acpi0 Timecounter "i8254" frequency 1193182 Hz quality 0 Event timer "i8254" frequency 1193182 Hz quality 100 atrtc0: port 0x70-0x71 on acpi0 Event timer "RTC" frequency 32768 Hz quality 0 hpet0: iomem 0xfed00000-0xfed003ff irq 0,8 on acpi0 Timecounter "HPET" frequency 14318180 Hz quality 950 Event timer "HPET" frequency 14318180 Hz quality 350 Event timer "HPET1" frequency 14318180 Hz quality 350 Event timer "HPET2" frequency 14318180 Hz quality 350 Timecounter "ACPI-fast" frequency 3579545 Hz quality 900 acpi_timer0: <32-bit timer at 3.579545MHz> port 0x808-0x80b on acpi0 pcib0: port 0xcf8-0xcff on acpi0 pci0: on pcib0 pci0: at device 0.2 (no driver attached) pcib1: at device 1.3 on pci0 pci1: on pcib1 xhci0: mem 0xfd6a0000-0xfd6a7fff irq 32 at device 0.0 on pci1 xhci0: 32 bytes context size, 64-bit DMA usbus0 on xhci0 usbus0: 5.0Gbps Super Speed USB v3.0 ahci0: mem 0xfd680000-0xfd69ffff irq 33 at device 0.1 on pci1 ahci0: AHCI v1.31 with 8 6Gbps ports, Port Multiplier supported ahcich0: at channel 0 on ahci0 ahcich1: at channel 1 on ahci0 ahcich4: at channel 4 on ahci0 ahcich5: at channel 5 on ahci0 pcib2: irq 34 at device 0.2 on pci1 pci2: on pcib2 pcib3: irq 32 at device 0.0 on pci2 pci3: on pcib3 re0: port 0xf000-0xf0ff mem 0xfd500000-0xfd500fff,0xf2100000-0xf2103fff irq 32 at device 0.0 on pci3 re0: Using 1 MSI-X message re0: Chip rev. 0x4c000000 re0: MAC rev. 0x00000000 miibus0: on re0 rgephy0: PHY 1 on miibus0 rgephy0: none, 10baseT, 10baseT-FDX, 10baseT-FDX-flow, 100baseTX, 100baseTX-FDX, 100baseTX-FDX-flow, 1000baseT-FDX, 1000baseT-FDX-master, 1000baseT-FDX-flow, 1000baseT-FDX-flow-master, auto, auto-flow re0: Using defaults for TSO: 65518/35/2048 re0: Ethernet address: 1c:1b:0d:9a:99:cb re0: netmap queues/slots: TX 1/256, RX 1/256 pcib4: irq 33 at device 1.0 on pci2 pci4: on pcib4 pcib5: irq 32 at device 4.0 on pci2 pci5: on pcib5 pcib6: at device 3.1 on pci0 pci6: on pcib6 vgapci0: port 0xe000-0xe07f mem 0xfc000000-0xfcffffff,0xe0000000-0xefffffff,0xf0000000-0xf1ffffff irq 54 at device 0.0 on pci6 vgapci0: Boot video device hdac0: mem 0xfd080000-0xfd083fff irq 55 at device 0.1 on pci6 pcib7: at device 7.1 on pci0 pci7: on pcib7 pci7: at device 0.0 (no driver attached) pci7: at device 0.2 (no driver attached) xhci1: mem 0xfd200000-0xfd2fffff irq 37 at device 0.3 on pci7 xhci1: 64 bytes context size, 64-bit DMA usbus1 on xhci1 usbus1: 5.0Gbps Super Speed USB v3.0 pcib8: at device 8.1 on pci0 pci8: on pcib8 pci8: at device 0.0 (no driver attached) ahci1: mem 0xfd708000-0xfd708fff irq 42 at device 0.2 on pci8 ahci1: AHCI v1.31 with 2 6Gbps ports, Port Multiplier supported with FBS ahcich8: at channel 0 on ahci1 ahcich9: at channel 1 on ahci1 hdac1: mem 0xfd700000-0xfd707fff irq 43 at device 0.3 on pci8 isab0: at device 20.3 on pci0 isa0: on isab0 acpi_button0: on acpi0 atkbdc0: port 0x60,0x64 irq 1 on acpi0 atkbd0: irq 1 on atkbdc0 kbd0 at atkbd0 atkbd0: [GIANT-LOCKED] uart0: <16550 or compatible> port 0x3f8-0x3ff irq 4 flags 0x10 on acpi0 ppc0: cannot reserve I/O port range hwpstate0: on cpu0 ZFS filesystem version: 5 ZFS storage pool version: features support (5000) Timecounters tick every 1.000 msec hdacc0: at cad 0 on hdac0 hdaa0: at nid 1 on hdacc0 pcm0: at nid 5 on hdaa0 hdacc1: at cad 1 on hdac0 hdaa1: at nid 1 on hdacc1 pcm1: at nid 5 on hdaa1 hdacc2: at cad 2 on hdac0 hdaa2: at nid 1 on hdacc2 pcm2: at nid 5 on hdaa2 hdacc3: at cad 3 on hdac0 hdaa3: at nid 1 on hdacc3 pcm3: at nid 5 on hdaa3 hdacc4: at cad 0 on hdac1 hdaa4: at nid 1 on hdacc4 pcm4: at nid 20 and 24,26 on hdaa4 pcm5: at nid 27 and 25 on hdaa4 pcm6: at nid 17 on hdaa4 ugen1.1: <0x1022 XHCI root HUB> at usbus1 ugen0.1: <0x1022 XHCI root HUB> at usbus0 uhub0: <0x1022 XHCI root HUB, class 9/0, rev 3.00/1.00, addr 1> on usbus1 uhub1: <0x1022 XHCI root HUB, class 9/0, rev 3.00/1.00, addr 1> on usbus0 ada0 at ahcich0 bus 0 scbus0 target 0 lun 0 ada0: ATA8-ACS SATA 2.x device ada0: Serial Number WD-WCASJ1778605 ada0: 300.000MB/s transfers (SATA 2.x, UDMA6, PIO 8192bytes) ada0: Command Queueing enabled ada0: 953869MB (1953525168 512 byte sectors) WARNING: WITNESS option enabled, expect reduced performance. Trying to mount root from zfs:zroot/ROOT/default []... Root mount waiting for: usbus1 usbus0 uhub0: 8 ports with 8 removable, self powered uhub1: 22 ports with 22 removable, self powered ugen1.2: at usbus1 umass0 on uhub0 umass0: on usbus1 umass0: SCSI over Bulk-Only; quirks = 0x4101 umass0:6:0: Attached to scbus6 da0 at umass-sim0 bus 0 scbus6 target 0 lun 0 da0: < USB Disk 8.07> Removable Direct Access SPC-2 SCSI device da0: Serial Number 5E2CDC72 da0: 40.000MB/s transfers da0: 3840MB (7864320 512 byte sectors) da0: quirks=0x2 GEOM: da0: the secondary GPT header is not in the last LBA. Root mount waiting for: usbus1 ugen1.3: at usbus1 re0: link state changed to DOWN ums0 on uhub0 ums0: on usbus1 ums0: 3 buttons and [XYZ] coordinates ID=0 From owner-freebsd-amd64@freebsd.org Wed Mar 22 20:50:36 2017 Return-Path: Delivered-To: freebsd-amd64@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id E9479D1884C for ; Wed, 22 Mar 2017 20:50:36 +0000 (UTC) (envelope-from fjwcash@gmail.com) Received: from mail-qk0-x236.google.com (mail-qk0-x236.google.com [IPv6:2607:f8b0:400d:c09::236]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id A5CBB1D74; Wed, 22 Mar 2017 20:50:36 +0000 (UTC) (envelope-from fjwcash@gmail.com) Received: by mail-qk0-x236.google.com with SMTP id p64so165614819qke.1; Wed, 22 Mar 2017 13:50:36 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=mime-version:in-reply-to:references:from:date:message-id:subject:to :cc; bh=NcyO8ph173stVV3zBS8MjyMpEOgouCGmlSUlso/Bbms=; b=h6eZAHdISu4P+0IpzdpYA2KGTDK4uD8mwyrl82irJhizg+HLhmAmB8atGsVTZVhRMV 63UqTBHqxO14LP9/Qs+N3IN6QWK1icEovWGg6E3d9FKXuVxGmyv+HKxoqRLtt8C8Hslu M+muDOOPOc9M62ShLcKgfieEqtr2Hv1HIbZ4Ltw/LIqFmL19k1s/dfxiZBF/geATiQwd PTKRTTEju0H7Byr0Tjp+NWREXZv7XsVv8k/9ycMGJ/u0gqYG7zDr7B7lcSZlzAZH8V37 stWOcSsCx5CbP3oRLAR1z6LE8vvFyA+vsJkMnUG7SdmQ+Sn+i34ItsZw5tTATSDSEvnA /LSw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:in-reply-to:references:from:date :message-id:subject:to:cc; bh=NcyO8ph173stVV3zBS8MjyMpEOgouCGmlSUlso/Bbms=; b=k5KbvjZTJyuIaY4lgVVzHtE+Y90CprYnuEcUt8mLcsr9zJ2ukhclJGsZunlYGEIvlh q30EwLZ3AO3pok+gKK9H+ksgA6zLmjBEXQIAncoyUdwrhzuF54A32yCHIzdSXT7+v3dq FalXRKqKKbz+Bu2bwE2aYz7DED6BIJM6Ug63Z8RDXyTOZoIeA1Ga811T+Jlw2ugFdn45 J8QU4P8fCmO0PyM5x0tyY95qAQUHx7/eExK6/d2FeCV3AX5KCYJWIb7GDAOtEAuH3Gap H3VDMGgaKi4wQJp5hxuwt6kULVKFbl4m1bUnmgpDx8KzIxhIsGQSV88RK8RPckqZVOwi EETQ== X-Gm-Message-State: AFeK/H2GDyOvpK43wzsdUP5fIKklo6yBMcvqI7blDcFlfsYPVg+fzpkoUe09b4DfYgSeAQtOCnkYaXXYjiLx9g== X-Received: by 10.55.157.146 with SMTP id g140mr36776617qke.30.1490215835757; Wed, 22 Mar 2017 13:50:35 -0700 (PDT) MIME-Version: 1.0 Received: by 10.140.28.202 with HTTP; Wed, 22 Mar 2017 13:50:35 -0700 (PDT) In-Reply-To: <201703222030.v2MKUJJs026400@gw.catspoiler.org> References: <201703222030.v2MKUJJs026400@gw.catspoiler.org> From: Freddie Cash Date: Wed, 22 Mar 2017 13:50:35 -0700 Message-ID: Subject: Re: FreeBSD on Ryzen To: Don Lewis Cc: freebsd-amd64@freebsd.org Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable X-Content-Filtered-By: Mailman/MimeDel 2.1.23 X-BeenThere: freebsd-amd64@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: Porting FreeBSD to the AMD64 platform List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 22 Mar 2017 20:50:37 -0000 On Wed, Mar 22, 2017 at 1:30 PM, Don Lewis wrote: > I put together a Ryzen 1700X machine over the weekend and installed the > 12.0-CURRENT r315413 snapshot on it a couple of days ago. The RAM is > DDR4 2400. > > First impression is that it's pretty zippy. Compared to my previous > fastest machine: > CPU: AMD FX-8320E Eight-Core Processor (3210.84-MHz K8-class CPU) > make -j8 buildworld using tmpfs is a bit more than 2x faster. Since the > Ryzen has SMT, it's eight cores look like 16 CPUs to FreeBSD, I get > almost a 2.6x speedup with -j16 as compared to my old machine. > > I do see that the reported total CPU time increases quite a bit at -j16 > (~19900u) as compared to -j8 (~13600u) so it is running into some > hardware bottlenecks that are slowing down instruction execution. It > could be the resources shared by both SMT threads that share each core, > or it could be cache or memory bandwidth related. The Ryzen topology is > a bit complicated. There are two groups of four cores, where each group > of four cores shares half of the L3 cache, with a slowish interconnect > bus between the groups. This probably causes some NUMA-like issues. I > wonder if the ULE scheduler could be tweaked to handle this better. > =E2=80=8BThe interconnect, aka Infinity Fabric, runs at the speed of the me= mory controller, so if you put faster RAM into the system, the fabric runs faster, and inter-CCX latency should drop to match. There's 2 MB of L3 cache shared between every two cores, but any core can access data in the L3 cache of any other core. Latency for those requests depends on whether it's within the same CCX (4-core cluster), or in the other CCX=E2=80=8B (going across the Infinity Fabric). There's a lot of finicky timing issues with L3 cache accesses, and with thread migration (in-CCX vs across the fabric). This is a whole other level of NUMA fun. And it'll get even more fun when the server version ships where you have 4 CCXes in a single CPU, with multiple sockets on a motherboard, and Infinity Fabric joining everything together. :) I feel sorry for the scheduler devs who get to figure all this out. :D Supposedly, the Linux folks have this mostly figured out in kernel 4.10, but I'll wait for the benchmarks to believe it. There's a bunch up on Phoronix ... but, well, it's Phoronix. :) --=20 Freddie Cash fjwcash@gmail.com From owner-freebsd-amd64@freebsd.org Wed Mar 22 22:37:02 2017 Return-Path: Delivered-To: freebsd-amd64@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 0DC21D181E3 for ; Wed, 22 Mar 2017 22:37:02 +0000 (UTC) (envelope-from truckman@FreeBSD.org) Received: from gw.catspoiler.org (unknown [IPv6:2602:304:b010:ef20::f2]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "gw.catspoiler.org", Issuer "gw.catspoiler.org" (not verified)) by mx1.freebsd.org (Postfix) with ESMTPS id E003F11F for ; Wed, 22 Mar 2017 22:37:01 +0000 (UTC) (envelope-from truckman@FreeBSD.org) Received: from FreeBSD.org (mousie.catspoiler.org [192.168.101.2]) by gw.catspoiler.org (8.15.2/8.15.2) with ESMTP id v2MMaqO9029200; Wed, 22 Mar 2017 15:36:56 -0700 (PDT) (envelope-from truckman@FreeBSD.org) Message-Id: <201703222236.v2MMaqO9029200@gw.catspoiler.org> Date: Wed, 22 Mar 2017 15:36:52 -0700 (PDT) From: Don Lewis Subject: Re: FreeBSD on Ryzen To: fjwcash@gmail.com cc: freebsd-amd64@freebsd.org In-Reply-To: MIME-Version: 1.0 Content-Type: TEXT/plain; charset=iso-8859-1 X-BeenThere: freebsd-amd64@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: Porting FreeBSD to the AMD64 platform List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 22 Mar 2017 22:37:02 -0000 On 22 Mar, Freddie Cash wrote: > On Wed, Mar 22, 2017 at 1:30 PM, Don Lewis wrote: > >> I put together a Ryzen 1700X machine over the weekend and installed the >> 12.0-CURRENT r315413 snapshot on it a couple of days ago. The RAM is >> DDR4 2400. >> >> First impression is that it's pretty zippy. Compared to my previous >> fastest machine: >> CPU: AMD FX-8320E Eight-Core Processor (3210.84-MHz K8-class CPU) >> make -j8 buildworld using tmpfs is a bit more than 2x faster. Since the >> Ryzen has SMT, it's eight cores look like 16 CPUs to FreeBSD, I get >> almost a 2.6x speedup with -j16 as compared to my old machine. >> >> I do see that the reported total CPU time increases quite a bit at -j16 >> (~19900u) as compared to -j8 (~13600u) so it is running into some >> hardware bottlenecks that are slowing down instruction execution. It >> could be the resources shared by both SMT threads that share each core, >> or it could be cache or memory bandwidth related. The Ryzen topology is >> a bit complicated. There are two groups of four cores, where each group >> of four cores shares half of the L3 cache, with a slowish interconnect >> bus between the groups. This probably causes some NUMA-like issues. I >> wonder if the ULE scheduler could be tweaked to handle this better. >> > > ?The interconnect, aka Infinity Fabric, runs at the speed of the memory > controller, so if you put faster RAM into the system, the fabric runs > faster, and inter-CCX latency should drop to match. Unfortunately ECC RAM seems to max out at DDR4 2400, so I'm already at the end of that road. > There's 2 MB of L3 cache shared between every two cores, but any core can > access data in the L3 cache of any other core. Latency for those requests > depends on whether it's within the same CCX (4-core cluster), or in the > other CCX? (going across the Infinity Fabric). I missed the extra level of L3 segmentation when I first read this: My "slowish" remark was about the speed of Infinity Fabric vs. QPI as mentioned in this article. > There's a lot of finicky timing issues with L3 cache accesses, and with > thread migration (in-CCX vs across the fabric). > > This is a whole other level of NUMA fun. And it'll get even more fun when > the server version ships where you have 4 CCXes in a single CPU, with > multiple sockets on a motherboard, and Infinity Fabric joining everything > together. :) Yeah, given that FreeBSD is pretty weak in terms of NUMA, I wasn't getting all that excited by the upcoming server stuff. > I feel sorry for the scheduler devs who get to figure all this out. :D > Supposedly, the Linux folks have this mostly figured out in kernel 4.10, > but I'll wait for the benchmarks to believe it. There's a bunch up on > Phoronix ... but, well, it's Phoronix. :) I saw it mentioned that the Linux change was to fix a Bulldozer optimization that de-optimizes performance on Ryzen. All in all, I'm pretty impressed with the performance improvement, especially once I noticed the relatively small clock speed difference between the 1700X (3394 MHz) vs. my FX-8320E (3211 MHz). Both have a 95W TDP. I'm not too thrilled with the 20C offset that AMD adds to Tctl on the 1700X and 1800X (but not the 1700). It makes the CPU fan sound like a vacuum cleaner even when the CPU is idle because the motherboard thinks the CPU is running in the mid-50 degree range. From owner-freebsd-amd64@freebsd.org Wed Mar 22 23:26:15 2017 Return-Path: Delivered-To: freebsd-amd64@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 69E1AD18071 for ; Wed, 22 Mar 2017 23:26:15 +0000 (UTC) (envelope-from truckman@FreeBSD.org) Received: from gw.catspoiler.org (unknown [IPv6:2602:304:b010:ef20::f2]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "gw.catspoiler.org", Issuer "gw.catspoiler.org" (not verified)) by mx1.freebsd.org (Postfix) with ESMTPS id 4C2771F8F; Wed, 22 Mar 2017 23:26:15 +0000 (UTC) (envelope-from truckman@FreeBSD.org) Received: from FreeBSD.org (mousie.catspoiler.org [192.168.101.2]) by gw.catspoiler.org (8.15.2/8.15.2) with ESMTP id v2MNQ7o1030213; Wed, 22 Mar 2017 16:26:11 -0700 (PDT) (envelope-from truckman@FreeBSD.org) Message-Id: <201703222326.v2MNQ7o1030213@gw.catspoiler.org> Date: Wed, 22 Mar 2017 16:26:07 -0700 (PDT) From: Don Lewis Subject: Re: FreeBSD on Ryzen To: avg@FreeBSD.org cc: freebsd-amd64@FreeBSD.org In-Reply-To: <4d09dde2-fef3-ac41-9d3a-6567c9b04029@FreeBSD.org> MIME-Version: 1.0 Content-Type: TEXT/plain; charset=us-ascii X-BeenThere: freebsd-amd64@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: Porting FreeBSD to the AMD64 platform List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 22 Mar 2017 23:26:15 -0000 On 23 Mar, Andriy Gapon wrote: > On 03/22/2017 22:30, Don Lewis wrote: >> I've got ECC RAM, but unfortunately the motherboard I'm using doesn't >> enable ECC. > > Are you absolutely sure of that? Yes. The motherboard is a Gigabyte GA-AB350 Gaming. On the motherboard specification page it specifically says: Support for ECC Un-buffered DIMM 1Rx8/2Rx8 memory modules (operate in non-ECC mode) and there are no knobs in the BIOS to enable ECC. I was hoping to get lucky though because the memory support list has an entry for the RAM that I purchased with a checkmark in the ECC column. I've got some older Gigabyte AM2 - AM3+ boards that support ECC even though it is not mentioned anywhere in the spec. I have heard rumors that some other motherboards silently support ECC if they detect it even though they don't have any mention of it in their BIOS configuration screens. To check this I installed the latest version of Fedora rawhide and there was no indication of ECC support in the boot messages, edac-utils didn't find it, and dmidecode said that the memory was 64 bits wide and not 72. > Could you please share dmesg from a verbose boot? > At least the portion where CPUs and topology are described. > Thanks! Yeah, I can send that later when I have a chance to reboot the box. From owner-freebsd-amd64@freebsd.org Wed Mar 22 22:41:17 2017 Return-Path: Delivered-To: freebsd-amd64@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id A93BBD182A9 for ; Wed, 22 Mar 2017 22:41:17 +0000 (UTC) (envelope-from avg@FreeBSD.org) Received: from citapm.icyb.net.ua (citapm.icyb.net.ua [212.40.38.140]) by mx1.freebsd.org (Postfix) with ESMTP id 87CD0243; Wed, 22 Mar 2017 22:41:15 +0000 (UTC) (envelope-from avg@FreeBSD.org) Received: from porto.starpoint.kiev.ua (porto-e.starpoint.kiev.ua [212.40.38.100]) by citapm.icyb.net.ua (8.8.8p3/ICyb-2.3exp) with ESMTP id AAA29872; Thu, 23 Mar 2017 00:41:14 +0200 (EET) (envelope-from avg@FreeBSD.org) Received: from localhost ([127.0.0.1]) by porto.starpoint.kiev.ua with esmtp (Exim 4.34 (FreeBSD)) id 1cqowM-000A2X-7H; Thu, 23 Mar 2017 00:41:14 +0200 Subject: Re: FreeBSD on Ryzen To: Don Lewis , freebsd-amd64@FreeBSD.org References: <201703222030.v2MKUJJs026400@gw.catspoiler.org> From: Andriy Gapon Message-ID: <4d09dde2-fef3-ac41-9d3a-6567c9b04029@FreeBSD.org> Date: Thu, 23 Mar 2017 00:40:18 +0200 User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:45.0) Gecko/20100101 Thunderbird/45.8.0 MIME-Version: 1.0 In-Reply-To: <201703222030.v2MKUJJs026400@gw.catspoiler.org> Content-Type: text/plain; charset=windows-1252 Content-Transfer-Encoding: 7bit X-Mailman-Approved-At: Thu, 23 Mar 2017 03:49:14 +0000 X-BeenThere: freebsd-amd64@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: Porting FreeBSD to the AMD64 platform List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 22 Mar 2017 22:41:17 -0000 On 03/22/2017 22:30, Don Lewis wrote: > I put together a Ryzen 1700X machine over the weekend and installed the > 12.0-CURRENT r315413 snapshot on it a couple of days ago. The RAM is > DDR4 2400. Very interesting, thank you for sharing! > First impression is that it's pretty zippy. Compared to my previous > fastest machine: > CPU: AMD FX-8320E Eight-Core Processor (3210.84-MHz K8-class CPU) > make -j8 buildworld using tmpfs is a bit more than 2x faster. Since the > Ryzen has SMT, it's eight cores look like 16 CPUs to FreeBSD, I get > almost a 2.6x speedup with -j16 as compared to my old machine. > > I do see that the reported total CPU time increases quite a bit at -j16 > (~19900u) as compared to -j8 (~13600u) so it is running into some > hardware bottlenecks that are slowing down instruction execution. It > could be the resources shared by both SMT threads that share each core, > or it could be cache or memory bandwidth related. The Ryzen topology is > a bit complicated. There are two groups of four cores, where each group > of four cores shares half of the L3 cache, with a slowish interconnect > bus between the groups. This probably causes some NUMA-like issues. I > wonder if the ULE scheduler could be tweaked to handle this better. > > % sysctl kern.sched.topology_spec > kern.sched.topology_spec: > > 0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15 > > > 0, 1, 2, 3, 4, 5, 6, 7 > > > 0, 1 > > > 2, 3 > > > 4, 5 > > > 6, 7 > > > > > 8, 9, 10, 11, 12, 13, 14, 15 > > > 8, 9 > > > 10, 11 > > > 12, 13 > > > 14, 15 > > > > > > The discovered topology looks like what I would expect. Each complex has its L3 cache and that's reflected above. L2 and L1 caches are per core (2 hardware "threads") and that's correct too. The only thing missing is the SMT flag per each SMT pair. I think that it does not have a dramatic effect on scheduling, but I can look into this shortcoming. > I'll run some tests with SMT disabled when I get a chance. > > The amdtemp driver doesn't attach to this CPU, so I can't monitor the > temperature at runtime. > > I've got ECC RAM, but unfortunately the motherboard I'm using doesn't > enable ECC. Are you absolutely sure of that? > The chipset is the B350. > Could you please share dmesg from a verbose boot? At least the portion where CPUs and topology are described. Thanks! > Copyright (c) 1992-2017 The FreeBSD Project. > Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994 > The Regents of the University of California. All rights reserved. > FreeBSD is a registered trademark of The FreeBSD Foundation. > FreeBSD 12.0-CURRENT #0 r315413: Thu Mar 16 17:23:31 UTC 2017 > root@releng3.nyi.freebsd.org:/usr/obj/usr/src/sys/GENERIC amd64 > FreeBSD clang version 4.0.0 (tags/RELEASE_400/final 297347) (based on LLVM 4.0.0) > WARNING: WITNESS option enabled, expect reduced performance. > SRAT: No memory found for CPU 0 > VT(vga): resolution 640x480 > CPU: AMD Ryzen 7 1700X Eight-Core Processor (3393.71-MHz K8-class CPU) > Origin="AuthenticAMD" Id=0x800f11 Family=0x17 Model=0x1 Stepping=1 > Features=0x178bfbff > Features2=0x7ed8320b > AMD Features=0x2e500800 > AMD Features2=0x35c233ff > Structured Extended Features=0x209c01a9 > XSAVE Features=0xf > SVM: NP,NRIP,VClean,AFlush,DAssist,NAsids=32768 > TSC: P-state invariant, performance statistics > real memory = 68719476736 (65536 MB) > avail memory = 66671529984 (63582 MB) > Event timer "LAPIC" quality 100 > ACPI APIC Table: > FreeBSD/SMP: Multiprocessor System Detected: 16 CPUs > FreeBSD/SMP: 1 package(s) x 16 core(s) > random: unblocking device. > ACPI BIOS Warning (bug): Optional FADT field Pm2ControlBlock has valid Length but zero Address: 0x0000000000000000/0x1 (20170303/tbfadt-796) > ioapic0: Changing APIC ID to 17 > ioapic1: Changing APIC ID to 18 > ioapic0 irqs 0-23 on motherboard > ioapic1 irqs 24-55 on motherboard > SMP: AP CPU #12 Launched! > SMP: AP CPU #1 Launched! > SMP: AP CPU #4 Launched! > SMP: AP CPU #7 Launched! > SMP: AP CPU #6 Launched! > SMP: AP CPU #3 Launched! > SMP: AP CPU #11 Launched! > SMP: AP CPU #5 Launched! > SMP: AP CPU #15 Launched! > SMP: AP CPU #9 Launched! > SMP: AP CPU #10 Launched! > SMP: AP CPU #8 Launched! > SMP: AP CPU #2 Launched! > SMP: AP CPU #14 Launched! > SMP: AP CPU #13 Launched! > Timecounter "TSC-low" frequency 1696854133 Hz quality 1000 > random: entropy device external interface > netmap: loaded module > module_register_init: MOD_LOAD (vesa, 0xffffffff80f49c70, 0) error 19 > random: registering fast source Intel Secure Key RNG > random: fast provider: "Intel Secure Key RNG" > kbd1 at kbdmux0 > nexus0 > vtvga0: on motherboard > cryptosoft0: on motherboard > acpi0: on motherboard > acpi0: Power Button (fixed) > cpu0: on acpi0 > cpu1: on acpi0 > cpu2: on acpi0 > cpu3: on acpi0 > cpu4: on acpi0 > cpu5: on acpi0 > cpu6: on acpi0 > cpu7: on acpi0 > cpu8: on acpi0 > cpu9: on acpi0 > cpu10: on acpi0 > cpu11: on acpi0 > cpu12: on acpi0 > cpu13: on acpi0 > cpu14: on acpi0 > cpu15: on acpi0 > attimer0: port 0x40-0x43 irq 0 on acpi0 > Timecounter "i8254" frequency 1193182 Hz quality 0 > Event timer "i8254" frequency 1193182 Hz quality 100 > atrtc0: port 0x70-0x71 on acpi0 > Event timer "RTC" frequency 32768 Hz quality 0 > hpet0: iomem 0xfed00000-0xfed003ff irq 0,8 on acpi0 > Timecounter "HPET" frequency 14318180 Hz quality 950 > Event timer "HPET" frequency 14318180 Hz quality 350 > Event timer "HPET1" frequency 14318180 Hz quality 350 > Event timer "HPET2" frequency 14318180 Hz quality 350 > Timecounter "ACPI-fast" frequency 3579545 Hz quality 900 > acpi_timer0: <32-bit timer at 3.579545MHz> port 0x808-0x80b on acpi0 > pcib0: port 0xcf8-0xcff on acpi0 > pci0: on pcib0 > pci0: at device 0.2 (no driver attached) > pcib1: at device 1.3 on pci0 > pci1: on pcib1 > xhci0: mem 0xfd6a0000-0xfd6a7fff irq 32 at device 0.0 on pci1 > xhci0: 32 bytes context size, 64-bit DMA > usbus0 on xhci0 > usbus0: 5.0Gbps Super Speed USB v3.0 > ahci0: mem 0xfd680000-0xfd69ffff irq 33 at device 0.1 on pci1 > ahci0: AHCI v1.31 with 8 6Gbps ports, Port Multiplier supported > ahcich0: at channel 0 on ahci0 > ahcich1: at channel 1 on ahci0 > ahcich4: at channel 4 on ahci0 > ahcich5: at channel 5 on ahci0 > pcib2: irq 34 at device 0.2 on pci1 > pci2: on pcib2 > pcib3: irq 32 at device 0.0 on pci2 > pci3: on pcib3 > re0: port 0xf000-0xf0ff mem 0xfd500000-0xfd500fff,0xf2100000-0xf2103fff irq 32 at device 0.0 on pci3 > re0: Using 1 MSI-X message > re0: Chip rev. 0x4c000000 > re0: MAC rev. 0x00000000 > miibus0: on re0 > rgephy0: PHY 1 on miibus0 > rgephy0: none, 10baseT, 10baseT-FDX, 10baseT-FDX-flow, 100baseTX, 100baseTX-FDX, 100baseTX-FDX-flow, 1000baseT-FDX, 1000baseT-FDX-master, 1000baseT-FDX-flow, 1000baseT-FDX-flow-master, auto, auto-flow > re0: Using defaults for TSO: 65518/35/2048 > re0: Ethernet address: 1c:1b:0d:9a:99:cb > re0: netmap queues/slots: TX 1/256, RX 1/256 > pcib4: irq 33 at device 1.0 on pci2 > pci4: on pcib4 > pcib5: irq 32 at device 4.0 on pci2 > pci5: on pcib5 > pcib6: at device 3.1 on pci0 > pci6: on pcib6 > vgapci0: port 0xe000-0xe07f mem 0xfc000000-0xfcffffff,0xe0000000-0xefffffff,0xf0000000-0xf1ffffff irq 54 at device 0.0 on pci6 > vgapci0: Boot video device > hdac0: mem 0xfd080000-0xfd083fff irq 55 at device 0.1 on pci6 > pcib7: at device 7.1 on pci0 > pci7: on pcib7 > pci7: at device 0.0 (no driver attached) > pci7: at device 0.2 (no driver attached) > xhci1: mem 0xfd200000-0xfd2fffff irq 37 at device 0.3 on pci7 > xhci1: 64 bytes context size, 64-bit DMA > usbus1 on xhci1 > usbus1: 5.0Gbps Super Speed USB v3.0 > pcib8: at device 8.1 on pci0 > pci8: on pcib8 > pci8: at device 0.0 (no driver attached) > ahci1: mem 0xfd708000-0xfd708fff irq 42 at device 0.2 on pci8 > ahci1: AHCI v1.31 with 2 6Gbps ports, Port Multiplier supported with FBS > ahcich8: at channel 0 on ahci1 > ahcich9: at channel 1 on ahci1 > hdac1: mem 0xfd700000-0xfd707fff irq 43 at device 0.3 on pci8 > isab0: at device 20.3 on pci0 > isa0: on isab0 > acpi_button0: on acpi0 > atkbdc0: port 0x60,0x64 irq 1 on acpi0 > atkbd0: irq 1 on atkbdc0 > kbd0 at atkbd0 > atkbd0: [GIANT-LOCKED] > uart0: <16550 or compatible> port 0x3f8-0x3ff irq 4 flags 0x10 on acpi0 > ppc0: cannot reserve I/O port range > hwpstate0: on cpu0 > ZFS filesystem version: 5 > ZFS storage pool version: features support (5000) > Timecounters tick every 1.000 msec > hdacc0: at cad 0 on hdac0 > hdaa0: at nid 1 on hdacc0 > pcm0: at nid 5 on hdaa0 > hdacc1: at cad 1 on hdac0 > hdaa1: at nid 1 on hdacc1 > pcm1: at nid 5 on hdaa1 > hdacc2: at cad 2 on hdac0 > hdaa2: at nid 1 on hdacc2 > pcm2: at nid 5 on hdaa2 > hdacc3: at cad 3 on hdac0 > hdaa3: at nid 1 on hdacc3 > pcm3: at nid 5 on hdaa3 > hdacc4: at cad 0 on hdac1 > hdaa4: at nid 1 on hdacc4 > pcm4: at nid 20 and 24,26 on hdaa4 > pcm5: at nid 27 and 25 on hdaa4 > pcm6: at nid 17 on hdaa4 > ugen1.1: <0x1022 XHCI root HUB> at usbus1 > ugen0.1: <0x1022 XHCI root HUB> at usbus0 > uhub0: <0x1022 XHCI root HUB, class 9/0, rev 3.00/1.00, addr 1> on usbus1 > uhub1: <0x1022 XHCI root HUB, class 9/0, rev 3.00/1.00, addr 1> on usbus0 > ada0 at ahcich0 bus 0 scbus0 target 0 lun 0 > ada0: ATA8-ACS SATA 2.x device > ada0: Serial Number WD-WCASJ1778605 > ada0: 300.000MB/s transfers (SATA 2.x, UDMA6, PIO 8192bytes) > ada0: Command Queueing enabled > ada0: 953869MB (1953525168 512 byte sectors) > WARNING: WITNESS option enabled, expect reduced performance. > Trying to mount root from zfs:zroot/ROOT/default []... > Root mount waiting for: usbus1 usbus0 > uhub0: 8 ports with 8 removable, self powered > uhub1: 22 ports with 22 removable, self powered > ugen1.2: at usbus1 > umass0 on uhub0 > umass0: on usbus1 > umass0: SCSI over Bulk-Only; quirks = 0x4101 > umass0:6:0: Attached to scbus6 > da0 at umass-sim0 bus 0 scbus6 target 0 lun 0 > da0: < USB Disk 8.07> Removable Direct Access SPC-2 SCSI device > da0: Serial Number 5E2CDC72 > da0: 40.000MB/s transfers > da0: 3840MB (7864320 512 byte sectors) > da0: quirks=0x2 > GEOM: da0: the secondary GPT header is not in the last LBA. > Root mount waiting for: usbus1 > ugen1.3: at usbus1 > re0: link state changed to DOWN > ums0 on uhub0 > ums0: on usbus1 > ums0: 3 buttons and [XYZ] coordinates ID=0 > -- Andriy Gapon From owner-freebsd-amd64@freebsd.org Thu Mar 23 04:06:07 2017 Return-Path: Delivered-To: freebsd-amd64@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 92661D188EC for ; Thu, 23 Mar 2017 04:06:07 +0000 (UTC) (envelope-from truckman@FreeBSD.org) Received: from gw.catspoiler.org (unknown [IPv6:2602:304:b010:ef20::f2]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "gw.catspoiler.org", Issuer "gw.catspoiler.org" (not verified)) by mx1.freebsd.org (Postfix) with ESMTPS id 1800B14E5; Thu, 23 Mar 2017 04:06:07 +0000 (UTC) (envelope-from truckman@FreeBSD.org) Received: from FreeBSD.org (mousie.catspoiler.org [192.168.101.2]) by gw.catspoiler.org (8.15.2/8.15.2) with ESMTP id v2N45xp8036004; Wed, 22 Mar 2017 21:06:03 -0700 (PDT) (envelope-from truckman@FreeBSD.org) Message-Id: <201703230406.v2N45xp8036004@gw.catspoiler.org> Date: Wed, 22 Mar 2017 21:05:59 -0700 (PDT) From: Don Lewis Subject: Re: FreeBSD on Ryzen To: avg@FreeBSD.org cc: freebsd-amd64@FreeBSD.org In-Reply-To: <201703222326.v2MNQ7o1030213@gw.catspoiler.org> MIME-Version: 1.0 Content-Type: TEXT/plain; charset=us-ascii X-BeenThere: freebsd-amd64@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: Porting FreeBSD to the AMD64 platform List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 23 Mar 2017 04:06:07 -0000 On 22 Mar, Don Lewis wrote: > On 23 Mar, Andriy Gapon wrote: >> On 03/22/2017 22:30, Don Lewis wrote: > >>> I've got ECC RAM, but unfortunately the motherboard I'm using doesn't >>> enable ECC. >> >> Are you absolutely sure of that? > > Yes. The motherboard is a Gigabyte GA-AB350 Gaming. On the motherboard > specification page it specifically says: > Support for ECC Un-buffered DIMM 1Rx8/2Rx8 memory modules (operate in > non-ECC mode) > and there are no knobs in the BIOS to enable ECC. I was hoping to get > lucky though because the memory support list has an entry for the RAM > that I purchased with a checkmark in the ECC column. I've got some older > Gigabyte AM2 - AM3+ boards that support ECC even though it is not > mentioned anywhere in the spec. > > I have heard rumors that some other motherboards silently support ECC if > they detect it even though they don't have any mention of it in their > BIOS configuration screens. To check this I installed the latest > version of Fedora rawhide and there was no indication of ECC support in > the boot messages, edac-utils didn't find it, and dmidecode said that > the memory was 64 bits wide and not 72. # dmidecode 3.0 Scanning /dev/mem for entry point. SMBIOS 3.0 present. Handle 0x0027, DMI type 16, 23 bytes Physical Memory Array Location: System Board Or Motherboard Use: System Memory Error Correction Type: None Maximum Capacity: 16 GB Error Information Handle: 0x0026 Number Of Devices: 4 Handle 0x002E, DMI type 17, 40 bytes Memory Device Array Handle: 0x0027 Error Information Handle: 0x002D Total Width: 128 bits Data Width: 64 bits Size: 16384 MB Form Factor: DIMM Set: None Locator: DIMM 0 Bank Locator: CHANNEL A Type: DDR4 Type Detail: Synchronous Unbuffered (Unregistered) Speed: 2400 MHz Manufacturer: Micron Technology Serial Number: 14C07593 Asset Tag: Not Specified Part Number: 18ASF2G7 Rank: 2 Configured Clock Speed: 1200 MHz Minimum Voltage: 1.2 V Maximum Voltage: 1.2 V Configured Voltage: 1.2 V Handle 0x0031, DMI type 17, 40 bytes Memory Device Array Handle: 0x0027 Error Information Handle: 0x0030 Total Width: 128 bits Data Width: 64 bits Size: 16384 MB Form Factor: DIMM Set: None Locator: DIMM 1 Bank Locator: CHANNEL A Type: DDR4 Type Detail: Synchronous Unbuffered (Unregistered) Speed: 2400 MHz Manufacturer: Micron Technology Serial Number: 14C0753E Asset Tag: Not Specified Part Number: 18ASF2G7 Rank: 2 Configured Clock Speed: 1200 MHz Minimum Voltage: 1.2 V Maximum Voltage: 1.2 V Configured Voltage: 1.2 V Handle 0x0034, DMI type 17, 40 bytes Memory Device Array Handle: 0x0027 Error Information Handle: 0x0033 Total Width: 128 bits Data Width: 64 bits Size: 16384 MB Form Factor: DIMM Set: None Locator: DIMM 0 Bank Locator: CHANNEL B Type: DDR4 Type Detail: Synchronous Unbuffered (Unregistered) Speed: 2400 MHz Manufacturer: Micron Technology Serial Number: 14C07579 Asset Tag: Not Specified Part Number: 18ASF2G7 Rank: 2 Configured Clock Speed: 1200 MHz Minimum Voltage: 1.2 V Maximum Voltage: 1.2 V Configured Voltage: 1.2 V Handle 0x0037, DMI type 17, 40 bytes Memory Device Array Handle: 0x0027 Error Information Handle: 0x0036 Total Width: 128 bits Data Width: 64 bits Size: 16384 MB Form Factor: DIMM Set: None Locator: DIMM 1 Bank Locator: CHANNEL B Type: DDR4 Type Detail: Synchronous Unbuffered (Unregistered) Speed: 2400 MHz Manufacturer: Micron Technology Serial Number: 14C07472 Asset Tag: Not Specified Part Number: 18ASF2G7 Rank: 2 Configured Clock Speed: 1200 MHz Minimum Voltage: 1.2 V Maximum Voltage: 1.2 V Configured Voltage: 1.2 V SMBIOS 3.0.0 present. Handle 0x0027, DMI type 16, 23 bytes Physical Memory Array Location: System Board Or Motherboard Use: System Memory Error Correction Type: None Maximum Capacity: 16 GB Error Information Handle: 0x0026 Number Of Devices: 4 Handle 0x002E, DMI type 17, 40 bytes Memory Device Array Handle: 0x0027 Error Information Handle: 0x002D Total Width: 128 bits Data Width: 64 bits Size: 16384 MB Form Factor: DIMM Set: None Locator: DIMM 0 Bank Locator: CHANNEL A Type: DDR4 Type Detail: Synchronous Unbuffered (Unregistered) Speed: 2400 MHz Manufacturer: Micron Technology Serial Number: 14C07593 Asset Tag: Not Specified Part Number: 18ASF2G7 Rank: 2 Configured Clock Speed: 1200 MHz Minimum Voltage: 1.2 V Maximum Voltage: 1.2 V Configured Voltage: 1.2 V Handle 0x0031, DMI type 17, 40 bytes Memory Device Array Handle: 0x0027 Error Information Handle: 0x0030 Total Width: 128 bits Data Width: 64 bits Size: 16384 MB Form Factor: DIMM Set: None Locator: DIMM 1 Bank Locator: CHANNEL A Type: DDR4 Type Detail: Synchronous Unbuffered (Unregistered) Speed: 2400 MHz Manufacturer: Micron Technology Serial Number: 14C0753E Asset Tag: Not Specified Part Number: 18ASF2G7 Rank: 2 Configured Clock Speed: 1200 MHz Minimum Voltage: 1.2 V Maximum Voltage: 1.2 V Configured Voltage: 1.2 V Handle 0x0034, DMI type 17, 40 bytes Memory Device Array Handle: 0x0027 Error Information Handle: 0x0033 Total Width: 128 bits Data Width: 64 bits Size: 16384 MB Form Factor: DIMM Set: None Locator: DIMM 0 Bank Locator: CHANNEL B Type: DDR4 Type Detail: Synchronous Unbuffered (Unregistered) Speed: 2400 MHz Manufacturer: Micron Technology Serial Number: 14C07579 Asset Tag: Not Specified Part Number: 18ASF2G7 Rank: 2 Configured Clock Speed: 1200 MHz Minimum Voltage: 1.2 V Maximum Voltage: 1.2 V Configured Voltage: 1.2 V Handle 0x0037, DMI type 17, 40 bytes Memory Device Array Handle: 0x0027 Error Information Handle: 0x0036 Total Width: 128 bits Data Width: 64 bits Size: 16384 MB Form Factor: DIMM Set: None Locator: DIMM 1 Bank Locator: CHANNEL B Type: DDR4 Type Detail: Synchronous Unbuffered (Unregistered) Speed: 2400 MHz Manufacturer: Micron Technology Serial Number: 14C07472 Asset Tag: Not Specified Part Number: 18ASF2G7 Rank: 2 Configured Clock Speed: 1200 MHz Minimum Voltage: 1.2 V Maximum Voltage: 1.2 V Configured Voltage: 1.2 V >> Could you please share dmesg from a verbose boot? >> At least the portion where CPUs and topology are described. >> Thanks! > > Yeah, I can send that later when I have a chance to reboot the box. Table 'FACP' at 0xdc0f8b80 Table 'APIC' at 0xdc0f8c98 APIC: Found table at 0xdc0f8c98 APIC: Using the MADT enumerator. MADT: Found CPU APIC ID 0 ACPI ID 1: enabled SMP: Added CPU 0 (AP) MADT: Found CPU APIC ID 1 ACPI ID 2: enabled SMP: Added CPU 1 (AP) MADT: Found CPU APIC ID 2 ACPI ID 3: enabled SMP: Added CPU 2 (AP) MADT: Found CPU APIC ID 3 ACPI ID 4: enabled SMP: Added CPU 3 (AP) MADT: Found CPU APIC ID 4 ACPI ID 5: enabled SMP: Added CPU 4 (AP) MADT: Found CPU APIC ID 5 ACPI ID 6: enabled SMP: Added CPU 5 (AP) MADT: Found CPU APIC ID 6 ACPI ID 7: enabled SMP: Added CPU 6 (AP) MADT: Found CPU APIC ID 7 ACPI ID 8: enabled SMP: Added CPU 7 (AP) MADT: Found CPU APIC ID 8 ACPI ID 9: enabled SMP: Added CPU 8 (AP) MADT: Found CPU APIC ID 9 ACPI ID 10: enabled SMP: Added CPU 9 (AP) MADT: Found CPU APIC ID 10 ACPI ID 11: enabled SMP: Added CPU 10 (AP) MADT: Found CPU APIC ID 11 ACPI ID 12: enabled SMP: Added CPU 11 (AP) MADT: Found CPU APIC ID 12 ACPI ID 13: enabled SMP: Added CPU 12 (AP) MADT: Found CPU APIC ID 13 ACPI ID 14: enabled SMP: Added CPU 13 (AP) MADT: Found CPU APIC ID 14 ACPI ID 15: enabled SMP: Added CPU 14 (AP) MADT: Found CPU APIC ID 15 ACPI ID 16: enabled SMP: Added CPU 15 (AP) Copyright (c) 1992-2017 The FreeBSD Project. Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994 The Regents of the University of California. All rights reserved. FreeBSD is a registered trademark of The FreeBSD Foundation. FreeBSD 12.0-CURRENT #0 r315413: Thu Mar 16 17:23:31 UTC 2017 root@releng3.nyi.freebsd.org:/usr/obj/usr/src/sys/GENERIC amd64 FreeBSD clang version 4.0.0 (tags/RELEASE_400/final 297347) (based on LLVM 4.0.0) WARNING: WITNESS option enabled, expect reduced performance. Table 'FACP' at 0xdc0f8b80 Table 'APIC' at 0xdc0f8c98 Table 'FPDT' at 0xdc0f8d78 Table 'SSDT' at 0xdc0f8dc0 Table 'FIDT' at 0xdc101a10 Table 'SSDT' at 0xdc101ab0 Table 'SRAT' at 0xdc103b98 SRAT: Found table at 0xdc103b98 SRAT: Found CPU APIC ID 0 domain 0: enabled SRAT: Found CPU APIC ID 1 domain 0: enabled SRAT: Found CPU APIC ID 2 domain 0: enabled SRAT: Found CPU APIC ID 3 domain 0: enabled SRAT: Found CPU APIC ID 4 domain 0: enabled SRAT: Found CPU APIC ID 5 domain 0: enabled SRAT: Found CPU APIC ID 6 domain 0: enabled SRAT: Found CPU APIC ID 7 domain 0: enabled SRAT: Found CPU APIC ID 8 domain 0: enabled SRAT: Found CPU APIC ID 9 domain 0: enabled SRAT: Found CPU APIC ID 10 domain 0: enabled SRAT: Found CPU APIC ID 11 domain 0: enabled SRAT: Found CPU APIC ID 12 domain 0: enabled SRAT: Found CPU APIC ID 13 domain 0: enabled SRAT: Found CPU APIC ID 14 domain 0: enabled SRAT: Found CPU APIC ID 15 domain 0: enabled SRAT: No memory found for CPU 0 PPIM 0: PA=0xa0000, VA=0xffffffff82810000, size=0x10000, mode=0 VT(vga): resolution 640x480 Preloaded elf kernel "/boot/kernel/kernel" at 0xffffffff8261b000. Preloaded /boot/entropy "/boot/entropy" at 0xffffffff8261bfd8. Preloaded elf obj module "/boot/kernel/zfs.ko" at 0xffffffff8261c028. Preloaded elf obj module "/boot/kernel/opensolaris.ko" at 0xffffffff8261c890. Calibrating TSC clock ... TSC clock: 3393707858 Hz CPU: AMD Ryzen 7 1700X Eight-Core Processor (3393.71-MHz K8-class CPU) Origin="AuthenticAMD" Id=0x800f11 Family=0x17 Model=0x1 Stepping=1 Features=0x178bfbff Features2=0x7ed8320b AMD Features=0x2e500800 AMD Features2=0x35c233ff Structured Extended Features=0x209c01a9 XSAVE Features=0xf SVM: Features=0x1bcff,PauseFilterThreshold,AVIC> Revision=1, ASIDs=32768 TSC: P-state invariant, performance statistics L1 2MB data TLB: 64 entries, fully associative L1 2MB instruction TLB: 64 entries, fully associative L1 4KB data TLB: 64 entries, fully associative L1 4KB instruction TLB: 64 entries, fully associative L1 data cache: 32 kbytes, 64 bytes/line, 1 lines/tag, 8-way associative L1 instruction cache: 64 kbytes, 64 bytes/line, 1 lines/tag, 4-way associative L2 2MB data TLB: 1536 entries, 2-way associative L2 2MB instruction TLB: 1024 entries, 2-way associative L2 4KB data TLB: 1536 entries, 8-way associative L2 4KB instruction TLB: 1536 entries, 8-way associative L2 unified cache: 512 kbytes, 64 bytes/line, 1 lines/tag, 8-way associative real memory = 68719476736 (65536 MB) Physical memory chunk(s): 0x0000000000010000 - 0x0000000000099fff, 565248 bytes (138 pages) 0x0000000000100000 - 0x00000000001fffff, 1048576 bytes (256 pages) 0x0000000002661000 - 0x0000000004cfffff, 40497152 bytes (9887 pages) 0x0000000005000000 - 0x00000000dbe2afff, 3605180416 bytes (880171 pages) 0x00000000dbfd0000 - 0x00000000dc0f2fff, 1191936 bytes (291 pages) 0x00000000dd011000 - 0x00000000deffffff, 33484800 bytes (8175 pages) 0x0000000100000000 - 0x0000000fb8a19fff, 63227142144 bytes (15436314 pages) avail memory = 66671529984 (63582 MB) Event timer "LAPIC" quality 100 LAPIC: ipi_wait() us multiplier 14 (r 23073012 tsc 3393707858) ACPI APIC Table: Package ID shift: 4 L3 cache ID shift: 3 L2 cache ID shift: 1 L1 cache ID shift: 1 Core ID shift: 0 INTR: Adding local APIC 1 as a target INTR: Adding local APIC 2 as a target INTR: Adding local APIC 3 as a target INTR: Adding local APIC 4 as a target INTR: Adding local APIC 5 as a target INTR: Adding local APIC 6 as a target INTR: Adding local APIC 7 as a target INTR: Adding local APIC 8 as a target INTR: Adding local APIC 9 as a target INTR: Adding local APIC 10 as a target INTR: Adding local APIC 11 as a target INTR: Adding local APIC 12 as a target INTR: Adding local APIC 13 as a target INTR: Adding local APIC 14 as a target INTR: Adding local APIC 15 as a target FreeBSD/SMP: Multiprocessor System Detected: 16 CPUs FreeBSD/SMP: 1 package(s) x 16 core(s) Package HW ID = 0 Core HW ID = 0 CPU0 (BSP): APIC ID: 0 Core HW ID = 1 CPU1 (AP): APIC ID: 1 Core HW ID = 2 CPU2 (AP): APIC ID: 2 Core HW ID = 3 CPU3 (AP): APIC ID: 3 Core HW ID = 4 CPU4 (AP): APIC ID: 4 Core HW ID = 5 CPU5 (AP): APIC ID: 5 Core HW ID = 6 CPU6 (AP): APIC ID: 6 Core HW ID = 7 CPU7 (AP): APIC ID: 7 Core HW ID = 8 CPU8 (AP): APIC ID: 8 Core HW ID = 9 CPU9 (AP): APIC ID: 9 Core HW ID = 10 CPU10 (AP): APIC ID: 10 Core HW ID = 11 CPU11 (AP): APIC ID: 11 Core HW ID = 12 CPU12 (AP): APIC ID: 12 Core HW ID = 13 CPU13 (AP): APIC ID: 13 Core HW ID = 14 CPU14 (AP): APIC ID: 14 Core HW ID = 15 CPU15 (AP): APIC ID: 15 APIC: CPU 0 has ACPI ID 1 APIC: CPU 1 has ACPI ID 2 APIC: CPU 2 has ACPI ID 3 APIC: CPU 3 has ACPI ID 4 APIC: CPU 4 has ACPI ID 5 APIC: CPU 5 has ACPI ID 6 APIC: CPU 6 has ACPI ID 7 APIC: CPU 7 has ACPI ID 8 APIC: CPU 8 has ACPI ID 9 APIC: CPU 9 has ACPI ID 10 APIC: CPU 10 has ACPI ID 11 APIC: CPU 11 has ACPI ID 12 APIC: CPU 12 has ACPI ID 13 APIC: CPU 13 has ACPI ID 14 APIC: CPU 14 has ACPI ID 15 APIC: CPU 15 has ACPI ID 16 x86bios: IVT 0x000000-0x0004ff at 0xfffff80000000000 x86bios: SSEG 0x098000-0x098fff at 0xfffffe100b1c4000 x86bios: EBDA 0x09d000-0x09ffff at 0xfffff8000009d000 x86bios: ROM 0x0a0000-0x0fefff at 0xfffff800000a0000 Pentium Pro MTRR support enabled random: read 4096 bytes from preloaded cache random: unblocking device. ULE: setup cpu 0 ULE: setup cpu 1 ULE: setup cpu 2 ULE: setup cpu 3 ULE: setup cpu 4 ULE: setup cpu 5 ULE: setup cpu 6 ULE: setup cpu 7 ULE: setup cpu 8 ULE: setup cpu 9 ULE: setup cpu 10 ULE: setup cpu 11 ULE: setup cpu 12 ULE: setup cpu 13 ULE: setup cpu 14 ULE: setup cpu 15 ACPI: RSDP 0x00000000000F05B0 000024 (v02 ALASKA) ACPI: XSDT 0x00000000DC0F3098 0000AC (v01 ALASKA A M I 01072009 AMI 00010013) ACPI: FACP 0x00000000DC0F8B80 000114 (v06 ALASKA A M I 01072009 AMI 00010013) ACPI BIOS Warning (bug): Optional FADT field Pm2ControlBlock has valid Length but zero Address: 0x0000000000000000/0x1 (20170303/tbfadt-796) ACPI: DSDT 0x00000000DC0F31D8 0059A5 (v02 ALASKA A M I 01072009 INTL 20120913) ACPI: FACS 0x00000000DC4CFC80 000040 ACPI: APIC 0x00000000DC0F8C98 0000DE (v03 ALASKA A M I 01072009 AMI 00010013) ACPI: FPDT 0x00000000DC0F8D78 000044 (v01 ALASKA A M I 01072009 AMI 00010013) ACPI: SSDT 0x00000000DC0F8DC0 008C4C (v02 00000002 MSFT 04000000) ACPI: FIDT 0x00000000DC101A10 00009C (v01 ALASKA A M I 01072009 AMI 00010013) ACPI: SSDT 0x00000000DC101AB0 0020E4 (v01 AMD AmdTable 00000001 AMD 00000001) ACPI: SRAT 0x00000000DC103B98 000130 (v03 AMD AmdTable 00000001 AMD 00000001) ACPI: CRAT 0x00000000DC103CC8 000F50 (v01 AMD AmdTable 00000001 AMD 00000001) ACPI: CDIT 0x00000000DC104C18 000029 (v01 AMD AmdTable 00000001 AMD 00000001) ACPI: SSDT 0x00000000DC104C48 0017A1 (v01 AMD AOD 00000001 INTL 20120913) ACPI: MCFG 0x00000000DC1063F0 00003C (v01 ALASKA A M I 01072009 MSFT 00010013) ACPI: HPET 0x00000000DC106430 000038 (v01 ALASKA A M I 01072009 AMI 00000005) ACPI: SSDT 0x00000000DC106468 000024 (v01 AMDFCH FCHZP 00001000 INTL 20120913) ACPI: UEFI 0x00000000DC106490 000042 (v01 00000000 00000000) ACPI: IVRS 0x00000000DC1064D8 0000D0 (v02 00000001 AMD 00000000) ACPI: SSDT 0x00000000DC1065A8 0000F8 (v01 AMDFCH FCHPT 00001000 INTL 20120913) ACPI: SSDT 0x00000000DC1066A0 001664 (v01 AMD CPMCMN 00000001 INTL 20120913) MADT: Found IO APIC ID 17, Interrupt 0 at 0xfec00000 ioapic0: Changing APIC ID to 17 ioapic0: ver 0x21 maxredir 0x17 ioapic0: Routing external 8259A's -> intpin 0 MADT: Found IO APIC ID 18, Interrupt 24 at 0xfec01000 ioapic1: Changing APIC ID to 18 ioapic1: ver 0x21 maxredir 0x1f lapic: Routing NMI -> LINT1 lapic: LINT1 trigger: edge lapic: LINT1 polarity: high MADT: Interrupt override: source 0, irq 2 ioapic0: Routing IRQ 0 -> intpin 2 MADT: Interrupt override: source 9, irq 9 ioapic0: intpin 9 trigger: level ioapic0: intpin 9 polarity: low ioapic0 irqs 0-23 on motherboard ioapic1 irqs 24-55 on motherboard cpu0 BSP: ID: 0x00000000 VER: 0x80050010 LDR: 0x00000000 DFR: 0xffffffff lint0: 0x00010700 lint1: 0x00000400 TPR: 0x00000000 SVR: 0x000001ff timer: 0x000100ef therm: 0x00010000 err: 0x000000f0 pmc: 0x00010400 AMD ext features: 0x00040007 AMD elvt0: 0x00010000 AMD elvt1: 0x00010000 AMD elvt2: 0x00010000 AMD elvt3: 0x00010000 SMP: AP CPU #15 Launched! cpu15 AP: ID: 0x0f000000 VER: 0x80050010 LDR: 0x00000000 DFR: 0xffffffff lint0: 0x00010700 lint1: 0x00000400 TPR: 0x00000000 SVR: 0x000001ff timer: 0x000100ef therm: 0x00010000 err: 0x000000f0 pmc: 0x00010400 AMD ext features: 0x00040007 AMD elvt0: 0x00010000 AMD elvt1: 0x00010000 AMD elvt2: 0x00010000 AMD elvt3: 0x00010000 SMP: AP CPU #2 Launched! cpu2 AP: ID: 0x02000000 VER: 0x80050010 LDR: 0x00000000 DFR: 0xffffffff lint0: 0x00010700 lint1: 0x00000400 TPR: 0x00000000 SVR: 0x000001ff timer: 0x000100ef therm: 0x00010000 err: 0x000000f0 pmc: 0x00010400 AMD ext features: 0x00040007 AMD elvt0: 0x00010000 AMD elvt1: 0x00010000 AMD elvt2: 0x00010000 AMD elvt3: 0x00010000 SMP: AP CPU #4 Launched! cpu4 AP: ID: 0x04000000 VER: 0x80050010 LDR: 0x00000000 DFR: 0xffffffff lint0: 0x00010700 lint1: 0x00000400 TPR: 0x00000000 SVR: 0x000001ff timer: 0x000100ef therm: 0x00010000 err: 0x000000f0 pmc: 0x00010400 AMD ext features: 0x00040007 AMD elvt0: 0x00010000 AMD elvt1: 0x00010000 AMD elvt2: 0x00010000 AMD elvt3: 0x00010000 SMP: AP CPU #7 Launched! cpu7 AP: ID: 0x07000000 VER: 0x80050010 LDR: 0x00000000 DFR: 0xffffffff lint0: 0x00010700 lint1: 0x00000400 TPR: 0x00000000 SVR: 0x000001ff timer: 0x000100ef therm: 0x00010000 err: 0x000000f0 pmc: 0x00010400 AMD ext features: 0x00040007 AMD elvt0: 0x00010000 AMD elvt1: 0x00010000 AMD elvt2: 0x00010000 AMD elvt3: 0x00010000 SMP: AP CPU #6 Launched! cpu6 AP: ID: 0x06000000 VER: 0x80050010 LDR: 0x00000000 DFR: 0xffffffff lint0: 0x00010700 lint1: 0x00000400 TPR: 0x00000000 SVR: 0x000001ff timer: 0x000100ef therm: 0x00010000 err: 0x000000f0 pmc: 0x00010400 AMD ext features: 0x00040007 AMD elvt0: 0x00010000 AMD elvt1: 0x00010000 AMD elvt2: 0x00010000 AMD elvt3: 0x00010000 SMP: AP CPU #5 Launched! cpu5 AP: ID: 0x05000000 VER: 0x80050010 LDR: 0x00000000 DFR: 0xffffffff lint0: 0x00010700 lint1: 0x00000400 TPR: 0x00000000 SVR: 0x000001ff timer: 0x000100ef therm: 0x00010000 err: 0x000000f0 pmc: 0x00010400 AMD ext features: 0x00040007 AMD elvt0: 0x00010000 AMD elvt1: 0x00010000 AMD elvt2: 0x00010000 AMD elvt3: 0x00010000 SMP: AP CPU #3 Launched! cpu3 AP: ID: 0x03000000 VER: 0x80050010 LDR: 0x00000000 DFR: 0xffffffff lint0: 0x00010700 lint1: 0x00000400 TPR: 0x00000000 SVR: 0x000001ff timer: 0x000100ef therm: 0x00010000 err: 0x000000f0 pmc: 0x00010400 AMD ext features: 0x00040007 AMD elvt0: 0x00010000 AMD elvt1: 0x00010000 AMD elvt2: 0x00010000 AMD elvt3: 0x00010000 SMP: AP CPU #1 Launched! cpu1 AP: ID: 0x01000000 VER: 0x80050010 LDR: 0x00000000 DFR: 0xffffffff lint0: 0x00010700 lint1: 0x00000400 TPR: 0x00000000 SVR: 0x000001ff timer: 0x000100ef therm: 0x00010000 err: 0x000000f0 pmc: 0x00010400 AMD ext features: 0x00040007 AMD elvt0: 0x00010000 AMD elvt1: 0x00010000 AMD elvt2: 0x00010000 AMD elvt3: 0x00010000 SMP: AP CPU #13 Launched! cpu13 AP: ID: 0x0d000000 VER: 0x80050010 LDR: 0x00000000 DFR: 0xffffffff lint0: 0x00010700 lint1: 0x00000400 TPR: 0x00000000 SVR: 0x000001ff timer: 0x000100ef therm: 0x00010000 err: 0x000000f0 pmc: 0x00010400 AMD ext features: 0x00040007 AMD elvt0: 0x00010000 AMD elvt1: 0x00010000 AMD elvt2: 0x00010000 AMD elvt3: 0x00010000 SMP: AP CPU #9 Launched! cpu9 AP: ID: 0x09000000 VER: 0x80050010 LDR: 0x00000000 DFR: 0xffffffff lint0: 0x00010700 lint1: 0x00000400 TPR: 0x00000000 SVR: 0x000001ff timer: 0x000100ef therm: 0x00010000 err: 0x000000f0 pmc: 0x00010400 AMD ext features: 0x00040007 AMD elvt0: 0x00010000 AMD elvt1: 0x00010000 AMD elvt2: 0x00010000 AMD elvt3: 0x00010000 SMP: AP CPU #11 Launched! cpu11 AP: ID: 0x0b000000 VER: 0x80050010 LDR: 0x00000000 DFR: 0xffffffff lint0: 0x00010700 lint1: 0x00000400 TPR: 0x00000000 SVR: 0x000001ff timer: 0x000100ef therm: 0x00010000 err: 0x000000f0 pmc: 0x00010400 AMD ext features: 0x00040007 AMD elvt0: 0x00010000 AMD elvt1: 0x00010000 AMD elvt2: 0x00010000 AMD elvt3: 0x00010000 SMP: AP CPU #12 Launched! cpu12 AP: ID: 0x0c000000 VER: 0x80050010 LDR: 0x00000000 DFR: 0xffffffff lint0: 0x00010700 lint1: 0x00000400 TPR: 0x00000000 SVR: 0x000001ff timer: 0x000100ef therm: 0x00010000 err: 0x000000f0 pmc: 0x00010400 AMD ext features: 0x00040007 AMD elvt0: 0x00010000 AMD elvt1: 0x00010000 AMD elvt2: 0x00010000 AMD elvt3: 0x00010000 SMP: AP CPU #8 Launched! cpu8 AP: ID: 0x08000000 VER: 0x80050010 LDR: 0x00000000 DFR: 0xffffffff lint0: 0x00010700 lint1: 0x00000400 TPR: 0x00000000 SVR: 0x000001ff timer: 0x000100ef therm: 0x00010000 err: 0x000000f0 pmc: 0x00010400 AMD ext features: 0x00040007 AMD elvt0: 0x00010000 AMD elvt1: 0x00010000 AMD elvt2: 0x00010000 AMD elvt3: 0x00010000 SMP: AP CPU #10 Launched! cpu10 AP: ID: 0x0a000000 VER: 0x80050010 LDR: 0x00000000 DFR: 0xffffffff lint0: 0x00010700 lint1: 0x00000400 TPR: 0x00000000 SVR: 0x000001ff timer: 0x000100ef therm: 0x00010000 err: 0x000000f0 pmc: 0x00010400 AMD ext features: 0x00040007 AMD elvt0: 0x00010000 AMD elvt1: 0x00010000 AMD elvt2: 0x00010000 AMD elvt3: 0x00010000 SMP: AP CPU #14 Launched! cpu14 AP: ID: 0x0e000000 VER: 0x80050010 LDR: 0x00000000 DFR: 0xffffffff lint0: 0x00010700 lint1: 0x00000400 TPR: 0x00000000 SVR: 0x000001ff timer: 0x000100ef therm: 0x00010000 err: 0x000000f0 pmc: 0x00010400 AMD ext features: 0x00040007 AMD elvt0: 0x00010000 AMD elvt1: 0x00010000 AMD elvt2: 0x00010000 AMD elvt3: 0x00010000 SMP: passed TSC synchronization test TSC timecounter discards lower 1 bit(s) Timecounter "TSC-low" frequency 1696853929 Hz quality 1000 snd_unit_init() u=0x00ff8000 [512] d=0x00007c00 [32] c=0x000003ff [1024] feeder_register: snd_unit=-1 snd_maxautovchans=16 latency=5 feeder_rate_min=1 feeder_rate_max=2016000 feeder_rate_round=25 wlan: <802.11 Link Layer> random: entropy device external interface netmap: loaded module null: nfslock: pseudo-device crypto: module_register_init: MOD_LOAD (vesa, 0xffffffff80f49c70, 0) error 19 io: random: registering fast source Intel Secure Key RNG random: fast provider: "Intel Secure Key RNG" kbd: new array size 4 kbd1 at kbdmux0 mem: hptnr: R750/DC7280 controller driver v1.1.4 hptrr: RocketRAID 17xx/2xxx SATA controller driver v1.2 hpt27xx: RocketRAID 27xx controller driver v1.2.7 nexus0 vtvga0: on motherboard random: harvesting attach, 8 bytes (4 bits) from vtvga0 random: harvesting attach, 8 bytes (4 bits) from ram0 cryptosoft0: on motherboard crypto: assign cryptosoft0 driver id 0, flags 100663296 crypto: cryptosoft0 registers alg 1 flags 0 maxoplen 0 crypto: cryptosoft0 registers alg 2 flags 0 maxoplen 0 crypto: cryptosoft0 registers alg 3 flags 0 maxoplen 0 crypto: cryptosoft0 registers alg 4 flags 0 maxoplen 0 crypto: cryptosoft0 registers alg 5 flags 0 maxoplen 0 crypto: cryptosoft0 registers alg 16 flags 0 maxoplen 0 crypto: cryptosoft0 registers alg 6 flags 0 maxoplen 0 crypto: cryptosoft0 registers alg 7 flags 0 maxoplen 0 crypto: cryptosoft0 registers alg 18 flags 0 maxoplen 0 crypto: cryptosoft0 registers alg 19 flags 0 maxoplen 0 crypto: cryptosoft0 registers alg 20 flags 0 maxoplen 0 crypto: cryptosoft0 registers alg 8 flags 0 maxoplen 0 crypto: cryptosoft0 registers alg 15 flags 0 maxoplen 0 crypto: cryptosoft0 registers alg 9 flags 0 maxoplen 0 crypto: cryptosoft0 registers alg 10 flags 0 maxoplen 0 crypto: cryptosoft0 registers alg 13 flags 0 maxoplen 0 crypto: cryptosoft0 registers alg 14 flags 0 maxoplen 0 crypto: cryptosoft0 registers alg 11 flags 0 maxoplen 0 crypto: cryptosoft0 registers alg 22 flags 0 maxoplen 0 crypto: cryptosoft0 registers alg 23 flags 0 maxoplen 0 crypto: cryptosoft0 registers alg 25 flags 0 maxoplen 0 crypto: cryptosoft0 registers alg 24 flags 0 maxoplen 0 crypto: cryptosoft0 registers alg 26 flags 0 maxoplen 0 crypto: cryptosoft0 registers alg 27 flags 0 maxoplen 0 crypto: cryptosoft0 registers alg 28 flags 0 maxoplen 0 crypto: cryptosoft0 registers alg 21 flags 0 maxoplen 0 crypto: cryptosoft0 registers alg 17 flags 0 maxoplen 0 random: harvesting attach, 8 bytes (4 bits) from cryptosoft0 acpi0: on motherboard ACPI: Executed 1 blocks of module-level executable AML code ACPI: Executed 1 blocks of module-level executable AML code ACPI: 7 ACPI AML tables successfully acquired and loaded PCIe: Memory Mapped configuration base @ 0xf8000000 ioapic0: routing intpin 9 (ISA IRQ 9) to lapic 0 vector 48 acpi0: Power Button (fixed) random: harvesting attach, 8 bytes (4 bits) from acpi_sysresource0 random: harvesting attach, 8 bytes (4 bits) from acpi_sysresource1 random: harvesting attach, 8 bytes (4 bits) from acpi_sysresource2 random: harvesting attach, 8 bytes (4 bits) from acpi_sysresource3 random: harvesting attach, 8 bytes (4 bits) from acpi_sysresource4 random: harvesting attach, 8 bytes (4 bits) from acpi_sysresource5 random: harvesting attach, 8 bytes (4 bits) from acpi_sysresource6 random: harvesting attach, 8 bytes (4 bits) from acpi_sysresource7 random: harvesting attach, 8 bytes (4 bits) from acpi_sysresource8 random: harvesting attach, 8 bytes (4 bits) from acpi_sysresource9 random: harvesting attach, 8 bytes (4 bits) from acpi_sysresource10 random: harvesting attach, 8 bytes (4 bits) from acpi_sysresource11 random: harvesting attach, 8 bytes (4 bits) from acpi_sysresource12 random: harvesting attach, 8 bytes (4 bits) from acpi_sysresource13 random: harvesting attach, 8 bytes (4 bits) from acpi_sysresource14 random: harvesting attach, 8 bytes (4 bits) from acpi_sysresource15 random: harvesting attach, 8 bytes (4 bits) from acpi_sysresource16 random: harvesting attach, 8 bytes (4 bits) from acpi_sysresource17 random: harvesting attach, 8 bytes (4 bits) from acpi_sysresource18 random: harvesting attach, 8 bytes (4 bits) from acpi_sysresource19 random: harvesting attach, 8 bytes (4 bits) from acpi_sysresource20 random: harvesting attach, 8 bytes (4 bits) from acpi_sysresource21 random: harvesting attach, 8 bytes (4 bits) from acpi_sysresource22 cpu0: Processor \134_PR_.P000 (ACPI ID 1) -> APIC ID 0 cpu0: on acpi0 random: harvesting attach, 8 bytes (4 bits) from cpu0 cpu1: Processor \134_PR_.P001 (ACPI ID 2) -> APIC ID 1 cpu1: on acpi0 random: harvesting attach, 8 bytes (4 bits) from cpu1 cpu2: Processor \134_PR_.P002 (ACPI ID 3) -> APIC ID 2 cpu2: on acpi0 random: harvesting attach, 8 bytes (4 bits) from cpu2 cpu3: Processor \134_PR_.P003 (ACPI ID 4) -> APIC ID 3 cpu3: on acpi0 random: harvesting attach, 8 bytes (4 bits) from cpu3 cpu4: Processor \134_PR_.P004 (ACPI ID 5) -> APIC ID 4 cpu4: on acpi0 random: harvesting attach, 8 bytes (4 bits) from cpu4 cpu5: Processor \134_PR_.P005 (ACPI ID 6) -> APIC ID 5 cpu5: on acpi0 random: harvesting attach, 8 bytes (4 bits) from cpu5 cpu6: Processor \134_PR_.P006 (ACPI ID 7) -> APIC ID 6 cpu6: on acpi0 random: harvesting attach, 8 bytes (4 bits) from cpu6 cpu7: Processor \134_PR_.P007 (ACPI ID 8) -> APIC ID 7 cpu7: on acpi0 random: harvesting attach, 8 bytes (4 bits) from cpu7 cpu8: Processor \134_PR_.P008 (ACPI ID 9) -> APIC ID 8 cpu8: on acpi0 random: harvesting attach, 8 bytes (4 bits) from cpu8 cpu9: Processor \134_PR_.P009 (ACPI ID 10) -> APIC ID 9 cpu9: on acpi0 random: harvesting attach, 8 bytes (4 bits) from cpu9 cpu10: Processor \134_PR_.P00A (ACPI ID 11) -> APIC ID 10 cpu10: on acpi0 random: harvesting attach, 8 bytes (4 bits) from cpu10 cpu11: Processor \134_PR_.P00B (ACPI ID 12) -> APIC ID 11 cpu11: on acpi0 random: harvesting attach, 8 bytes (4 bits) from cpu11 cpu12: Processor \134_PR_.P00C (ACPI ID 13) -> APIC ID 12 cpu12: on acpi0 random: harvesting attach, 8 bytes (4 bits) from cpu12 cpu13: Processor \134_PR_.P00D (ACPI ID 14) -> APIC ID 13 cpu13: on acpi0 random: harvesting attach, 8 bytes (4 bits) from cpu13 cpu14: Processor \134_PR_.P00E (ACPI ID 15) -> APIC ID 14 cpu14: on acpi0 random: harvesting attach, 8 bytes (4 bits) from cpu14 cpu15: Processor \134_PR_.P00F (ACPI ID 16) -> APIC ID 15 cpu15: on acpi0 random: harvesting attach, 8 bytes (4 bits) from cpu15 attimer0: port 0x40-0x43 irq 0 on acpi0 Timecounter "i8254" frequency 1193182 Hz quality 0 ioapic0: routing intpin 2 (ISA IRQ 0) to lapic 1 vector 48 Event timer "i8254" frequency 1193182 Hz quality 100 random: harvesting attach, 8 bytes (4 bits) from attimer0 atrtc0: port 0x70-0x71 on acpi0 atrtc0: registered as a time-of-day clock (resolution 1000000us, adjustment 0.500000000s) ioapic0: routing intpin 8 (ISA IRQ 8) to lapic 2 vector 48 ioapic0: routing intpin 8 (ISA IRQ 8) to lapic 0 vector 49 Event timer "RTC" frequency 32768 Hz quality 0 random: harvesting attach, 8 bytes (4 bits) from atrtc0 hpet0: iomem 0xfed00000-0xfed003ff irq 0,8 on acpi0 hpet0: vendor 0x1022, rev 0x1, 14318180Hz, 3 timers, legacy route hpet0: t0: irqs 0x00c00000 (0), MSI, periodic hpet0: t1: irqs 0x00c00000 (0), MSI, periodic hpet0: t2: irqs 0x00c00000 (0), MSI, periodic Timecounter "HPET" frequency 14318180 Hz quality 950 msi: routing MSI-X IRQ 256 to local APIC 3 vector 48 msi: routing MSI-X IRQ 257 to local APIC 4 vector 48 msi: routing MSI-X IRQ 258 to local APIC 5 vector 48 msi: Assigning MSI-X IRQ 256 to local APIC 0 vector 50 msi: Assigning MSI-X IRQ 257 to local APIC 0 vector 51 msi: Assigning MSI-X IRQ 258 to local APIC 0 vector 52 Event timer "HPET" frequency 14318180 Hz quality 350 Event timer "HPET1" frequency 14318180 Hz quality 350 Event timer "HPET2" frequency 14318180 Hz quality 350 random: harvesting attach, 8 bytes (4 bits) from hpet0 ACPI timer: 1/1 1/1 1/1 1/1 1/1 1/1 1/1 1/1 1/1 1/1 -> 10 Timecounter "ACPI-fast" frequency 3579545 Hz quality 900 acpi_timer0: <32-bit timer at 3.579545MHz> port 0x808-0x80b on acpi0 random: harvesting attach, 8 bytes (4 bits) from acpi_timer0 [SNIP] From owner-freebsd-amd64@freebsd.org Thu Mar 23 11:36:03 2017 Return-Path: Delivered-To: freebsd-amd64@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 1578CD1A402 for ; Thu, 23 Mar 2017 11:36:03 +0000 (UTC) (envelope-from avg@FreeBSD.org) Received: from citapm.icyb.net.ua (citapm.icyb.net.ua [212.40.38.140]) by mx1.freebsd.org (Postfix) with ESMTP id 372AE1FE2; Thu, 23 Mar 2017 11:36:01 +0000 (UTC) (envelope-from avg@FreeBSD.org) Received: from porto.starpoint.kiev.ua (porto-e.starpoint.kiev.ua [212.40.38.100]) by citapm.icyb.net.ua (8.8.8p3/ICyb-2.3exp) with ESMTP id NAA02175; Thu, 23 Mar 2017 13:36:00 +0200 (EET) (envelope-from avg@FreeBSD.org) Received: from localhost ([127.0.0.1]) by porto.starpoint.kiev.ua with esmtp (Exim 4.34 (FreeBSD)) id 1cr128-000Atf-45; Thu, 23 Mar 2017 13:36:00 +0200 Subject: Re: FreeBSD on Ryzen To: Don Lewis References: <201703222326.v2MNQ7o1030213@gw.catspoiler.org> Cc: freebsd-amd64@FreeBSD.org From: Andriy Gapon Message-ID: Date: Thu, 23 Mar 2017 13:35:03 +0200 User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:45.0) Gecko/20100101 Thunderbird/45.8.0 MIME-Version: 1.0 In-Reply-To: <201703222326.v2MNQ7o1030213@gw.catspoiler.org> Content-Type: text/plain; charset=windows-1252 Content-Transfer-Encoding: 7bit X-Mailman-Approved-At: Thu, 23 Mar 2017 12:36:46 +0000 X-BeenThere: freebsd-amd64@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: Porting FreeBSD to the AMD64 platform List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 23 Mar 2017 11:36:03 -0000 On 03/23/2017 01:26, Don Lewis wrote: > Yes. The motherboard is a Gigabyte GA-AB350 Gaming. On the motherboard > specification page it specifically says: > Support for ECC Un-buffered DIMM 1Rx8/2Rx8 memory modules (operate in > non-ECC mode) > and there are no knobs in the BIOS to enable ECC. I was hoping to get > lucky though because the memory support list has an entry for the RAM > that I purchased with a checkmark in the ECC column. I've got some older > Gigabyte AM2 - AM3+ boards that support ECC even though it is not > mentioned anywhere in the spec. > > I have heard rumors that some other motherboards silently support ECC if > they detect it even though they don't have any mention of it in their > BIOS configuration screens. Yeah, that's the reason I asked. > To check this I installed the latest > version of Fedora rawhide and there was no indication of ECC support in > the boot messages, edac-utils didn't find it, and dmidecode said that > the memory was 64 bits wide and not 72. -- Andriy Gapon From owner-freebsd-amd64@freebsd.org Thu Mar 23 11:42:45 2017 Return-Path: Delivered-To: freebsd-amd64@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 10D73D1A66E for ; Thu, 23 Mar 2017 11:42:45 +0000 (UTC) (envelope-from avg@FreeBSD.org) Received: from citapm.icyb.net.ua (citapm.icyb.net.ua [212.40.38.140]) by mx1.freebsd.org (Postfix) with ESMTP id 3363F661; Thu, 23 Mar 2017 11:42:44 +0000 (UTC) (envelope-from avg@FreeBSD.org) Received: from porto.starpoint.kiev.ua (porto-e.starpoint.kiev.ua [212.40.38.100]) by citapm.icyb.net.ua (8.8.8p3/ICyb-2.3exp) with ESMTP id NAA02190; Thu, 23 Mar 2017 13:42:42 +0200 (EET) (envelope-from avg@FreeBSD.org) Received: from localhost ([127.0.0.1]) by porto.starpoint.kiev.ua with esmtp (Exim 4.34 (FreeBSD)) id 1cr18c-000Au8-9X; Thu, 23 Mar 2017 13:42:42 +0200 Subject: Re: FreeBSD on Ryzen To: Don Lewis References: <201703230406.v2N45xp8036004@gw.catspoiler.org> Cc: freebsd-amd64@FreeBSD.org From: Andriy Gapon Message-ID: Date: Thu, 23 Mar 2017 13:41:46 +0200 User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:45.0) Gecko/20100101 Thunderbird/45.8.0 MIME-Version: 1.0 In-Reply-To: <201703230406.v2N45xp8036004@gw.catspoiler.org> Content-Type: text/plain; charset=windows-1252 Content-Transfer-Encoding: 7bit X-Mailman-Approved-At: Thu, 23 Mar 2017 12:37:02 +0000 X-BeenThere: freebsd-amd64@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: Porting FreeBSD to the AMD64 platform List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 23 Mar 2017 11:42:45 -0000 On 03/23/2017 06:05, Don Lewis wrote: > Package ID shift: 4 > L3 cache ID shift: 3 > L2 cache ID shift: 1 > L1 cache ID shift: 1 > Core ID shift: 0 > INTR: Adding local APIC 1 as a target > INTR: Adding local APIC 2 as a target > INTR: Adding local APIC 3 as a target > INTR: Adding local APIC 4 as a target > INTR: Adding local APIC 5 as a target > INTR: Adding local APIC 6 as a target > INTR: Adding local APIC 7 as a target > INTR: Adding local APIC 8 as a target > INTR: Adding local APIC 9 as a target > INTR: Adding local APIC 10 as a target > INTR: Adding local APIC 11 as a target > INTR: Adding local APIC 12 as a target > INTR: Adding local APIC 13 as a target > INTR: Adding local APIC 14 as a target > INTR: Adding local APIC 15 as a target > FreeBSD/SMP: Multiprocessor System Detected: 16 CPUs > FreeBSD/SMP: 1 package(s) x 16 core(s) Don, thank you! Could you please test this patch https://paste.debian.net/923675/ and see if it allows to detect SMT threads? -- Andriy Gapon From owner-freebsd-amd64@freebsd.org Thu Mar 23 18:22:59 2017 Return-Path: Delivered-To: freebsd-amd64@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 9193CD1AD77 for ; Thu, 23 Mar 2017 18:22:59 +0000 (UTC) (envelope-from truckman@FreeBSD.org) Received: from gw.catspoiler.org (unknown [IPv6:2602:304:b010:ef20::f2]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "gw.catspoiler.org", Issuer "gw.catspoiler.org" (not verified)) by mx1.freebsd.org (Postfix) with ESMTPS id 6E95F165A; Thu, 23 Mar 2017 18:22:59 +0000 (UTC) (envelope-from truckman@FreeBSD.org) Received: from FreeBSD.org (mousie.catspoiler.org [192.168.101.2]) by gw.catspoiler.org (8.15.2/8.15.2) with ESMTP id v2NIMp1e054314; Thu, 23 Mar 2017 11:22:55 -0700 (PDT) (envelope-from truckman@FreeBSD.org) Message-Id: <201703231822.v2NIMp1e054314@gw.catspoiler.org> Date: Thu, 23 Mar 2017 11:15:44 -0700 (PDT) From: Don Lewis Subject: Re: FreeBSD on Ryzen To: avg@FreeBSD.org cc: freebsd-amd64@FreeBSD.org MIME-Version: 1.0 Content-Type: TEXT/plain; charset=us-ascii X-BeenThere: freebsd-amd64@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: Porting FreeBSD to the AMD64 platform List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 23 Mar 2017 18:22:59 -0000 resending to the list ... On 23 Mar, Andriy Gapon wrote: > On 03/23/2017 06:05, Don Lewis wrote: >> Package ID shift: 4 >> L3 cache ID shift: 3 >> L2 cache ID shift: 1 >> L1 cache ID shift: 1 >> Core ID shift: 0 >> INTR: Adding local APIC 1 as a target >> INTR: Adding local APIC 2 as a target >> INTR: Adding local APIC 3 as a target >> INTR: Adding local APIC 4 as a target >> INTR: Adding local APIC 5 as a target >> INTR: Adding local APIC 6 as a target >> INTR: Adding local APIC 7 as a target >> INTR: Adding local APIC 8 as a target >> INTR: Adding local APIC 9 as a target >> INTR: Adding local APIC 10 as a target >> INTR: Adding local APIC 11 as a target >> INTR: Adding local APIC 12 as a target >> INTR: Adding local APIC 13 as a target >> INTR: Adding local APIC 14 as a target >> INTR: Adding local APIC 15 as a target >> FreeBSD/SMP: Multiprocessor System Detected: 16 CPUs >> FreeBSD/SMP: 1 package(s) x 16 core(s) > > Don, > > thank you! > Could you please test this patch https://paste.debian.net/923675/ and > see if it allows to detect SMT threads? It took some doing since we don't have the AMDID2_NODE_ID code, but with the patch applied, I do see SMT threads. FreeBSD/SMP: Multiprocessor System Detected: 16 CPUs FreeBSD/SMP: 1 package(s) x 8 core(s) x 2 hardware threads Package HW ID = 0 Core HW ID = 0 CPU0 (BSP): APIC ID: 0 CPU1 (AP/HT): APIC ID: 1 Core HW ID = 1 CPU2 (AP): APIC ID: 2 CPU3 (AP/HT): APIC ID: 3 Core HW ID = 2 CPU4 (AP): APIC ID: 4 CPU5 (AP/HT): APIC ID: 5 Core HW ID = 3 CPU6 (AP): APIC ID: 6 CPU7 (AP/HT): APIC ID: 7 Core HW ID = 4 CPU8 (AP): APIC ID: 8 CPU9 (AP/HT): APIC ID: 9 Core HW ID = 5 CPU10 (AP): APIC ID: 10 CPU11 (AP/HT): APIC ID: 11 Core HW ID = 6 CPU12 (AP): APIC ID: 12 CPU13 (AP/HT): APIC ID: 13 Core HW ID = 7 CPU14 (AP): APIC ID: 14 CPU15 (AP/HT): APIC ID: 15 % sysctl kern.sched.topology_spec kern.sched.topology_spec: 0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15 0, 1, 2, 3, 4, 5, 6, 7 0, 1 THREAD groupSMT group 2, 3 THREAD groupSMT group 4, 5 THREAD groupSMT group 6, 7 THREAD groupSMT group 8, 9, 10, 11, 12, 13, 14, 15 8, 9 THREAD groupSMT group 10, 11 THREAD groupSMT group 12, 13 THREAD groupSMT group 14, 15 THREAD groupSMT group BTW, with SMT disabled in the BIOS I see a 5% speed improvement in "make -j8 buildworld" as compared to SMT enabled with the stock kernel. From owner-freebsd-amd64@freebsd.org Thu Mar 23 19:14:09 2017 Return-Path: Delivered-To: freebsd-amd64@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id F2CB9CBA53C for ; Thu, 23 Mar 2017 19:14:09 +0000 (UTC) (envelope-from se@freebsd.org) Received: from mailout10.t-online.de (mailout10.t-online.de [194.25.134.21]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "mailout00.t-online.de", Issuer "TeleSec ServerPass DE-2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id B92A11EE2 for ; Thu, 23 Mar 2017 19:14:09 +0000 (UTC) (envelope-from se@freebsd.org) Received: from fwd27.aul.t-online.de (fwd27.aul.t-online.de [172.20.26.132]) by mailout10.t-online.de (Postfix) with SMTP id B855541F5C52 for ; Thu, 23 Mar 2017 20:14:06 +0100 (CET) Received: from Stefans-MBP.fritz.box (VOfuG-ZHghBmuo--NQ3YNBUzWG0stw9P429lyrRttpDQ3K63KE8lwCjKmOqnUeeZgQ@[84.154.122.135]) by fwd27.t-online.de with (TLSv1.2:ECDHE-RSA-AES256-SHA encrypted) esmtp id 1cr8BQ-2FCb0S0; Thu, 23 Mar 2017 20:14:04 +0100 Subject: Re: FreeBSD on Ryzen To: freebsd-amd64@freebsd.org References: <201703222030.v2MKUJJs026400@gw.catspoiler.org> From: Stefan Esser Message-ID: <51b6c5d5-fc66-f371-ef54-c3d85a6f2c2d@freebsd.org> Date: Thu, 23 Mar 2017 20:14:03 +0100 User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.12; rv:45.0) Gecko/20100101 Thunderbird/45.8.0 MIME-Version: 1.0 In-Reply-To: <201703222030.v2MKUJJs026400@gw.catspoiler.org> Content-Type: text/plain; charset=windows-1252 Content-Transfer-Encoding: 7bit X-ID: VOfuG-ZHghBmuo--NQ3YNBUzWG0stw9P429lyrRttpDQ3K63KE8lwCjKmOqnUeeZgQ X-TOI-MSGID: fd4e067d-b9c7-48ee-8019-37cb3776045d X-BeenThere: freebsd-amd64@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: Porting FreeBSD to the AMD64 platform List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 23 Mar 2017 19:14:10 -0000 Am 22.03.17 um 21:30 schrieb Don Lewis: > I put together a Ryzen 1700X machine over the weekend and installed the > 12.0-CURRENT r315413 snapshot on it a couple of days ago. The RAM is > DDR4 2400. > > First impression is that it's pretty zippy. Compared to my previous > fastest machine: > CPU: AMD FX-8320E Eight-Core Processor (3210.84-MHz K8-class CPU) > make -j8 buildworld using tmpfs is a bit more than 2x faster. Since the > Ryzen has SMT, it's eight cores look like 16 CPUs to FreeBSD, I get > almost a 2.6x speedup with -j16 as compared to my old machine. > > I do see that the reported total CPU time increases quite a bit at -j16 > (~19900u) as compared to -j8 (~13600u) so it is running into some > hardware bottlenecks that are slowing down instruction execution. It > could be the resources shared by both SMT threads that share each core, It is the resources shared by the cores. Under full CPU load, SMT makes a 3.3 GHz 8 core CPU "simulate" a ~2 GHz 16 core CPU. The throughput is (in 1st order) proportional to cores * CPU clock, and comes out as 8 * 3.3 = 26.4 vs. 16 * ~2 = ~32 (estimated) I'm positively surprised by the observed gain of +30% due to SMT. This seems to match the reported user times: 13,600 / 8 = 1,700 seconds user time per physical core (on average) 19,900 / 16 = 1,244 seconds per virtual (SMT) core vs. an estimate of the throughput with a CPU with SMT but without any gain in throughput: 27,200 / 16 = 1,700 seconds per virtual core with ineffective SMT (i.e. assuming SMT that does not increase effective IPC, resulting in identical real time compared to the non-SMT case) This result seems to match the increased performance when going from -j 8 to -j 16: 27,200 / 19,900 = 2.7 ~ 2.6 / 2.0 > or it could be cache or memory bandwidth related. The Ryzen topology is > a bit complicated. There are two groups of four cores, where each group > of four cores shares half of the L3 cache, with a slowish interconnect > bus between the groups. This probably causes some NUMA-like issues. I > wonder if the ULE scheduler could be tweaked to handle this better. I've been wondering whether it is possible to teach the scheduler about above mentioned effect, i.e. by distinguishing a SMT core that executes only 1 runnable thread from one that executes 2. The latter one should be assumed to run at an estimated 60% clock (which makes both threads proceed at 120% of the non-SMT speed). OTOH, the lower "effective clock rate" should be irrelevant under high load (when all cores are executing 2 threads), or under low load, when some cores are idle (assuming, that the scheduler prefers to assign only 1 thread per each core until there are more runnable threads then cores. If you assume that user time accounting is a raw measure of instructions executed, then assuming a reduced clock rate would lead to "fairer" results. From owner-freebsd-amd64@freebsd.org Thu Mar 23 22:33:16 2017 Return-Path: Delivered-To: freebsd-amd64@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 9697BD1BB00 for ; Thu, 23 Mar 2017 22:33:16 +0000 (UTC) (envelope-from truckman@FreeBSD.org) Received: from gw.catspoiler.org (unknown [IPv6:2602:304:b010:ef20::f2]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "gw.catspoiler.org", Issuer "gw.catspoiler.org" (not verified)) by mx1.freebsd.org (Postfix) with ESMTPS id 78BAC11EC; Thu, 23 Mar 2017 22:33:16 +0000 (UTC) (envelope-from truckman@FreeBSD.org) Received: from FreeBSD.org (mousie.catspoiler.org [192.168.101.2]) by gw.catspoiler.org (8.15.2/8.15.2) with ESMTP id v2NMX86u059606; Thu, 23 Mar 2017 15:33:12 -0700 (PDT) (envelope-from truckman@FreeBSD.org) Message-Id: <201703232233.v2NMX86u059606@gw.catspoiler.org> Date: Thu, 23 Mar 2017 15:33:08 -0700 (PDT) From: Don Lewis Subject: Re: FreeBSD on Ryzen To: avg@FreeBSD.org cc: freebsd-amd64@FreeBSD.org In-Reply-To: <201703231822.v2NIMp1e054314@gw.catspoiler.org> MIME-Version: 1.0 Content-Type: TEXT/plain; charset=us-ascii X-BeenThere: freebsd-amd64@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: Porting FreeBSD to the AMD64 platform List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 23 Mar 2017 22:33:16 -0000 On 23 Mar, Don Lewis wrote: > On 23 Mar, Andriy Gapon wrote: >> On 03/23/2017 06:05, Don Lewis wrote: >>> Package ID shift: 4 >>> L3 cache ID shift: 3 >>> L2 cache ID shift: 1 >>> L1 cache ID shift: 1 >>> Core ID shift: 0 >>> INTR: Adding local APIC 1 as a target >>> INTR: Adding local APIC 2 as a target >>> INTR: Adding local APIC 3 as a target >>> INTR: Adding local APIC 4 as a target >>> INTR: Adding local APIC 5 as a target >>> INTR: Adding local APIC 6 as a target >>> INTR: Adding local APIC 7 as a target >>> INTR: Adding local APIC 8 as a target >>> INTR: Adding local APIC 9 as a target >>> INTR: Adding local APIC 10 as a target >>> INTR: Adding local APIC 11 as a target >>> INTR: Adding local APIC 12 as a target >>> INTR: Adding local APIC 13 as a target >>> INTR: Adding local APIC 14 as a target >>> INTR: Adding local APIC 15 as a target >>> FreeBSD/SMP: Multiprocessor System Detected: 16 CPUs >>> FreeBSD/SMP: 1 package(s) x 16 core(s) >> >> Don, >> >> thank you! >> Could you please test this patch https://paste.debian.net/923675/ and >> see if it allows to detect SMT threads? > > It took some doing since we don't have the AMDID2_NODE_ID code, but with > the patch applied, I do see SMT threads. > > FreeBSD/SMP: Multiprocessor System Detected: 16 CPUs > FreeBSD/SMP: 1 package(s) x 8 core(s) x 2 hardware threads This patch improved make -j8 buildworld performance by about 0.9% and -j16 by about 1.3%. From owner-freebsd-amd64@freebsd.org Thu Mar 23 23:42:08 2017 Return-Path: Delivered-To: freebsd-amd64@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 8438CD1BD3A for ; Thu, 23 Mar 2017 23:42:08 +0000 (UTC) (envelope-from truckman@FreeBSD.org) Received: from gw.catspoiler.org (unknown [IPv6:2602:304:b010:ef20::f2]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "gw.catspoiler.org", Issuer "gw.catspoiler.org" (not verified)) by mx1.freebsd.org (Postfix) with ESMTPS id 6042E1127; Thu, 23 Mar 2017 23:42:08 +0000 (UTC) (envelope-from truckman@FreeBSD.org) Received: from FreeBSD.org (mousie.catspoiler.org [192.168.101.2]) by gw.catspoiler.org (8.15.2/8.15.2) with ESMTP id v2NNg0lT061089; Thu, 23 Mar 2017 16:42:04 -0700 (PDT) (envelope-from truckman@FreeBSD.org) Message-Id: <201703232342.v2NNg0lT061089@gw.catspoiler.org> Date: Thu, 23 Mar 2017 16:42:00 -0700 (PDT) From: Don Lewis Subject: Re: FreeBSD on Ryzen To: se@freebsd.org cc: freebsd-amd64@freebsd.org In-Reply-To: <51b6c5d5-fc66-f371-ef54-c3d85a6f2c2d@freebsd.org> MIME-Version: 1.0 Content-Type: TEXT/plain; charset=us-ascii X-BeenThere: freebsd-amd64@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: Porting FreeBSD to the AMD64 platform List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 23 Mar 2017 23:42:08 -0000 On 23 Mar, Stefan Esser wrote: > Am 22.03.17 um 21:30 schrieb Don Lewis: >> I put together a Ryzen 1700X machine over the weekend and installed the >> 12.0-CURRENT r315413 snapshot on it a couple of days ago. The RAM is >> DDR4 2400. >> >> First impression is that it's pretty zippy. Compared to my previous >> fastest machine: >> CPU: AMD FX-8320E Eight-Core Processor (3210.84-MHz K8-class CPU) >> make -j8 buildworld using tmpfs is a bit more than 2x faster. Since the >> Ryzen has SMT, it's eight cores look like 16 CPUs to FreeBSD, I get >> almost a 2.6x speedup with -j16 as compared to my old machine. >> >> I do see that the reported total CPU time increases quite a bit at -j16 >> (~19900u) as compared to -j8 (~13600u) so it is running into some >> hardware bottlenecks that are slowing down instruction execution. It >> could be the resources shared by both SMT threads that share each core, > > It is the resources shared by the cores. Under full CPU load, SMT makes > a 3.3 GHz 8 core CPU "simulate" a ~2 GHz 16 core CPU. > > The throughput is (in 1st order) proportional to cores * CPU clock, and > comes out as > > 8 * 3.3 = 26.4 vs. 16 * ~2 = ~32 (estimated) > > I'm positively surprised by the observed gain of +30% due to SMT. This Don't forget that the -j8 case is also paying some penalty for SMT. We don't currently recognize that Ryzen uses SMT and we think that there are 16 independent CPUs. In a test that I mentioned earlier today, I disabled SMT in the BIOS so that the chip only looks like it has 8 cores improved the performance in the -j8 case by 5%. > seems to match the reported user times: > > 13,600 / 8 = 1,700 seconds user time per physical core (on average) > 19,900 / 16 = 1,244 seconds per virtual (SMT) core > > vs. an estimate of the throughput with a CPU with SMT but without any > gain in throughput: > > 27,200 / 16 = 1,700 seconds per virtual core with ineffective SMT > > (i.e. assuming SMT that does not increase effective IPC, resulting > in identical real time compared to the non-SMT case) > > This result seems to match the increased performance when going from > -j 8 to -j 16: > > 27,200 / 19,900 = 2.7 ~ 2.6 / 2.0 > >> or it could be cache or memory bandwidth related. The Ryzen topology is >> a bit complicated. There are two groups of four cores, where each group >> of four cores shares half of the L3 cache, with a slowish interconnect >> bus between the groups. This probably causes some NUMA-like issues. I >> wonder if the ULE scheduler could be tweaked to handle this better. > > I've been wondering whether it is possible to teach the scheduler about > above mentioned effect, i.e. by distinguishing a SMT core that executes > only 1 runnable thread from one that executes 2. The latter one should > be assumed to run at an estimated 60% clock (which makes both threads > proceed at 120% of the non-SMT speed). > > OTOH, the lower "effective clock rate" should be irrelevant under high > load (when all cores are executing 2 threads), or under low load, when > some cores are idle (assuming, that the scheduler prefers to assign only > 1 thread per each core until there are more runnable threads then cores. > > If you assume that user time accounting is a raw measure of instructions > executed, then assuming a reduced clock rate would lead to "fairer" > results. Interesting, though it sounds complicated. Under light load, it seems like we would want to assign threads to idle cores rather than assigning a new thread to a core that already has one running thread. If there are no more than four threads on the current Ryzen chips, we would probably want to run them all on the same CCX to avoid the the Infinity Fabric overhead. Things get fuzzier with more than four threads. Should we try to keep them on the same CCX to avoid using the Infinity Fabric and pay the SMT overhead, or do the opposite? According to my first edition copy of _The Design and Implementation of the FreeBSD Operating System_, which covers FreeBSD 5.2, it seems that in the SMT case, the ULE scheduler prefers to migrate threads to another CPU in the same processor group. That would seem to indicate that on Ryzen it would prefer to keep threads on the same CPU core where they would compete, rather than spread them out across different cores. Is that (still) the case? From owner-freebsd-amd64@freebsd.org Fri Mar 24 03:08:24 2017 Return-Path: Delivered-To: freebsd-amd64@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id D1F8FD176B1 for ; Fri, 24 Mar 2017 03:08:24 +0000 (UTC) (envelope-from brde@optusnet.com.au) Received: from mail107.syd.optusnet.com.au (mail107.syd.optusnet.com.au [211.29.132.53]) by mx1.freebsd.org (Postfix) with ESMTP id 80FEC18E2; Fri, 24 Mar 2017 03:08:24 +0000 (UTC) (envelope-from brde@optusnet.com.au) Received: from besplex.bde.org (c122-106-153-191.carlnfd1.nsw.optusnet.com.au [122.106.153.191]) by mail107.syd.optusnet.com.au (Postfix) with ESMTPS id 578D0D4893B; Fri, 24 Mar 2017 14:06:49 +1100 (AEDT) Date: Fri, 24 Mar 2017 14:06:48 +1100 (EST) From: Bruce Evans X-X-Sender: bde@besplex.bde.org To: Stefan Esser cc: freebsd-amd64@freebsd.org Subject: Re: FreeBSD on Ryzen In-Reply-To: <51b6c5d5-fc66-f371-ef54-c3d85a6f2c2d@freebsd.org> Message-ID: <20170324122633.T1253@besplex.bde.org> References: <201703222030.v2MKUJJs026400@gw.catspoiler.org> <51b6c5d5-fc66-f371-ef54-c3d85a6f2c2d@freebsd.org> MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII; format=flowed X-Optus-CM-Score: 0 X-Optus-CM-Analysis: v=2.2 cv=KeqiiUQD c=1 sm=1 tr=0 a=Tj3pCpwHnMupdyZSltBt7Q==:117 a=Tj3pCpwHnMupdyZSltBt7Q==:17 a=kj9zAlcOel0A:10 a=Rf7QABXb2WDHDuOk828A:9 a=umuLrCqks1aB-5yt:21 a=Y3-H2PJT2owLGq65:21 a=CjuIK1q_8ugA:10 X-BeenThere: freebsd-amd64@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: Porting FreeBSD to the AMD64 platform List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 24 Mar 2017 03:08:24 -0000 On Thu, 23 Mar 2017, Stefan Esser wrote: > Am 22.03.17 um 21:30 schrieb Don Lewis: >> I put together a Ryzen 1700X machine over the weekend and installed the >> 12.0-CURRENT r315413 snapshot on it a couple of days ago. The RAM is >> DDR4 2400. >> >> First impression is that it's pretty zippy. Compared to my previous >> fastest machine: >> CPU: AMD FX-8320E Eight-Core Processor (3210.84-MHz K8-class CPU) >> make -j8 buildworld using tmpfs is a bit more than 2x faster. Since the >> Ryzen has SMT, it's eight cores look like 16 CPUs to FreeBSD, I get >> almost a 2.6x speedup with -j16 as compared to my old machine. >> >> I do see that the reported total CPU time increases quite a bit at -j16 >> (~19900u) as compared to -j8 (~13600u) so it is running into some >> hardware bottlenecks that are slowing down instruction execution. It >> could be the resources shared by both SMT threads that share each core, > > It is the resources shared by the cores. Under full CPU load, SMT makes > a 3.3 GHz 8 core CPU "simulate" a ~2 GHz 16 core CPU. This seems to be normal for all x86. See below. However, Don measured mixed methods by only using -j8. This gives less SMT use than -j16 or -j32, but still a lot, since make doesn't really understand scheduling doesn't limit the number of active threads to keep them on unshared cores. I use cpuset -l 0,2,4,6 to test this with 4x2 cores to test this manually. I also use SCHED_4BSD, which needs more manual scheduling to reduce sharing of cores. About 10 years ago, large -j for makeworld seemed to cost a lot for lock contention, but I didn't understand the overheads from core sharing at the time, and perhaps schedulers didn't either (SCHED_4BSD still doesn't, except I use a small hack for it), so the extra user time may have always been for SMT. Now large -j doesn't cost much for makeworld. I still try it (up to about 16 times as many threads as cores), but it just gives small pessimizations when it is larger than the number needed to keep all cores usually active. My slowest makeworld times on Haswell (4.08GHz no turbo 4x2) are 24 times faster than the above (572u instead of 13600u), but this is due to my world being ~5.2 and current worlds having clang and other slowness. The time scales almost perfectly inversely with the CPU clock and by a factor of about your 3.3/2 with SMT. E.g., on Haswell with no SMT (4x1), my makeworld time reduces to 354, which corresponds to a factor of 3.2/2. makeworld has a non-parallized install section near the end (about 10% of the real time of 140 seconds on Haswell), so this factor of 3.2/2 is smaller than the SMT factor (I would have expected even smaller). The SMT scaling for makeworld is similar on Sandybridge. The SMT scaling for buildworld is similar on most all of the FreeBSD-cluster's Xeons. However, not very long ago, the FreeBSD cluster and more trailing edge Xeons which had an SMT scaling factor of 4/2. The scaling for a simple benchmark that uses only integer resources for a countdown loop has an SMT scaling factor of only about 4/3. I think the scaling has precise factors like 4 and 3 since maximum throughput is 3 or 4 instructions/cycle, and when this throughput is achieved it uses all of a critical resource, leaving none to spare for SMT. > The throughput is (in 1st order) proportional to cores * CPU clock, and > comes out as > > 8 * 3.3 = 26.4 vs. 16 * ~2 = ~32 (estimated) This must be very CPU-dependent. x86 CPUs are still optimized for !SMT and/or low power, so they don't try for more than 3 or 4 instructions/cycle since this is rarely reached for a single thread, so they don't have many spare resources to use for SMT. > I'm positively surprised by the observed gain of +30% due to SMT. This > seems to match the reported user times: > > 13,600 / 8 = 1,700 seconds user time per physical core (on average) > 19,900 / 16 = 1,244 seconds per virtual (SMT) core This is probably from the mixed methods. I'm surprised that -j8 doesn't keep closer to >= 16 CPUs than 8 runnable most of the time, so that it uses SMT just as much as -j16 most of the time. > vs. an estimate of the throughput with a CPU with SMT but without any > gain in throughput: > > 27,200 / 16 = 1,700 seconds per virtual core with ineffective SMT > > (i.e. assuming SMT that does not increase effective IPC, resulting > in identical real time compared to the non-SMT case) > > This result seems to match the increased performance when going from > -j 8 to -j 16: > > 27,200 / 19,900 = 2.7 ~ 2.6 / 2.0 > >> or it could be cache or memory bandwidth related. The Ryzen topology is >> a bit complicated. There are two groups of four cores, where each group >> of four cores shares half of the L3 cache, with a slowish interconnect >> bus between the groups. This probably causes some NUMA-like issues. I >> wonder if the ULE scheduler could be tweaked to handle this better. > > I've been wondering whether it is possible to teach the scheduler about > above mentioned effect, i.e. by distinguishing a SMT core that executes > only 1 runnable thread from one that executes 2. The latter one should > be assumed to run at an estimated 60% clock (which makes both threads > proceed at 120% of the non-SMT speed). > > OTOH, the lower "effective clock rate" should be irrelevant under high > load (when all cores are executing 2 threads), or under low load, when > some cores are idle (assuming, that the scheduler prefers to assign only > 1 thread per each core until there are more runnable threads then cores. > > If you assume that user time accounting is a raw measure of instructions > executed, then assuming a reduced clock rate would lead to "fairer" > results. I thought that schedulers didn't understand SMT at all. SCHED_4BSD certainly doesn't. I use the following hack to reduce sharing in it. It is almost useless for the reasons that you state: - low load: makes little difference. A random choice of CPU from many free CPUs has a low chance of contending with an active CPU. - high load: makes little difference. There are no spare CPUs, and a random choice is less bad than a smart choice since it hard to do better but easy to do worse by making perfectly pessimal choices and sticking with them. X Index: sched_4bsd.c X =================================================================== X --- sched_4bsd.c (revision 315658) X +++ sched_4bsd.c (working copy) X @@ -1237,6 +1261,11 @@ X } X #endif X X +#ifdef SMP X +static int evenhack; X +SYSCTL_INT(_kern_sched, OID_AUTO, evenhack, CTLFLAG_RW, &evenhack, 0, ""); X +#endif X + X void X sched_add(struct thread *td, int flags) X #ifdef SMP X @@ -1307,6 +1336,23 @@ X td); X cpu = NOCPU; X ts->ts_runq = &runq; X +if (evenhack == mp_maxid) { X + int id; X + X + cpuid = PCPU_GET(cpuid); X + if (CPU_ISSET(cpuid ^ 1, &idle_cpus_mask)) X + goto found; X + for (id = 0; id <= mp_maxid; id += 2) { X + if (CPU_ISSET(id, &idle_cpus_mask) && X + CPU_ISSET(id ^ 1, &idle_cpus_mask)) { X + cpu = id; X + ts->ts_runq = &runq_pcpu[cpu]; X + single_cpu = 1; X + break; X + } X + } X +found: ; X +} X } X X if ((td->td_flags & TDF_NOLOAD) == 0) For makeworld, this seems to give improvements in the range of 0.1-0.5%, but it is hard to be sure since the variance in the real time is about 3% (and that is with some temperature control and closer to 100 than 10 other details to keep the test environment constant). System time is is ~80 seconds on Haswell and it is hard to get excited about improvements of even 1% in it (0.8 second divided by 8 cores = 0.1 second in real time). Using SCHED_ULE instead of SCHED_BSD gives improvements in the +-5% range (worse on older CPUs). Bruce From owner-freebsd-amd64@freebsd.org Fri Mar 24 08:16:06 2017 Return-Path: Delivered-To: freebsd-amd64@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id B13E4D1BF3F for ; Fri, 24 Mar 2017 08:16:06 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2001:1900:2254:206a::16:76]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 98F8C1C0A for ; Fri, 24 Mar 2017 08:16:06 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from bugs.freebsd.org ([127.0.1.118]) by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id v2O8G6XB040793 for ; Fri, 24 Mar 2017 08:16:06 GMT (envelope-from bugzilla-noreply@freebsd.org) From: bugzilla-noreply@freebsd.org To: freebsd-amd64@FreeBSD.org Subject: [Bug 217138] head (e.g.) -r315870 for arm64: sh vs. jemalloc asserts: include/jemalloc/internal/tsd.h:687: Failed assertion: "tsd_booted" once swapped in after being swapped out (comment 10) Date: Fri, 24 Mar 2017 08:16:06 +0000 X-Bugzilla-Reason: CC X-Bugzilla-Type: changed X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: Base System X-Bugzilla-Component: bin X-Bugzilla-Version: CURRENT X-Bugzilla-Keywords: X-Bugzilla-Severity: Affects Some People X-Bugzilla-Who: markmi@dsl-only.net X-Bugzilla-Status: New X-Bugzilla-Resolution: X-Bugzilla-Priority: --- X-Bugzilla-Assigned-To: freebsd-bugs@FreeBSD.org X-Bugzilla-Flags: X-Bugzilla-Changed-Fields: short_desc Message-ID: In-Reply-To: References: Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated MIME-Version: 1.0 X-BeenThere: freebsd-amd64@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: Porting FreeBSD to the AMD64 platform List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 24 Mar 2017 08:16:06 -0000 https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D217138 Mark Millard changed: What |Removed |Added ---------------------------------------------------------------------------- Summary|head (e.g.) -r314638 for |head (e.g.) -r315870 for |arm64: sh vs. jemalloc |arm64: sh vs. jemalloc |asserts: |asserts: |include/jemalloc/internal/t |include/jemalloc/internal/t |sd.h:687: Failed assertion: |sd.h:687: Failed assertion: |"tsd_booted" once swapped |"tsd_booted" once swapped |in after being swapped out |in after being swapped out |(comment 10) |(comment 10) --- Comment #32 from Mark Millard --- I have confirmed that -r315870 still trashes memory for the sequence: allocations (tcache and <=3D SMALL_MAXCLASS) fork sleep/wait then swap-out [zero RES(ident memory)] swap-in --=20 You are receiving this mail because: You are on the CC list for the bug.= From owner-freebsd-amd64@freebsd.org Fri Mar 24 23:15:23 2017 Return-Path: Delivered-To: freebsd-amd64@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 099E7D1C021 for ; Fri, 24 Mar 2017 23:15:23 +0000 (UTC) (envelope-from truckman@FreeBSD.org) Received: from gw.catspoiler.org (unknown [IPv6:2602:304:b010:ef20::f2]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "gw.catspoiler.org", Issuer "gw.catspoiler.org" (not verified)) by mx1.freebsd.org (Postfix) with ESMTPS id D38DD1A39 for ; Fri, 24 Mar 2017 23:15:22 +0000 (UTC) (envelope-from truckman@FreeBSD.org) Received: from FreeBSD.org (mousie.catspoiler.org [192.168.101.2]) by gw.catspoiler.org (8.15.2/8.15.2) with ESMTP id v2ONFF0J091423 for ; Fri, 24 Mar 2017 16:15:19 -0700 (PDT) (envelope-from truckman@FreeBSD.org) Message-Id: <201703242315.v2ONFF0J091423@gw.catspoiler.org> Date: Fri, 24 Mar 2017 16:15:15 -0700 (PDT) From: Don Lewis Subject: Re: FreeBSD on Ryzen To: freebsd-amd64@FreeBSD.org In-Reply-To: <201703222030.v2MKUJJs026400@gw.catspoiler.org> MIME-Version: 1.0 Content-Type: TEXT/plain; charset=us-ascii X-BeenThere: freebsd-amd64@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: Porting FreeBSD to the AMD64 platform List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 24 Mar 2017 23:15:23 -0000 On 22 Mar, Don Lewis wrote: > I put together a Ryzen 1700X machine over the weekend and installed the > 12.0-CURRENT r315413 snapshot on it a couple of days ago. The RAM is > DDR4 2400. > > First impression is that it's pretty zippy. Compared to my previous > fastest machine: > CPU: AMD FX-8320E Eight-Core Processor (3210.84-MHz K8-class CPU) > make -j8 buildworld using tmpfs is a bit more than 2x faster. Since the > Ryzen has SMT, it's eight cores look like 16 CPUs to FreeBSD, I get > almost a 2.6x speedup with -j16 as compared to my old machine. The reason that I put this machine together is for package building with poudriere. Unfortunately I don't see quite as much performance improvement there, possibly because I set ALLOW_MAKE_JOBS=yes, which can result in a *lot* of running processes at times and probably thrashes the caches pretty badly: last pid: 78855; load averages: 120.05, 106.95, 86.49 up 0+14:09:43 15:12:59 413 processes: 123 running, 281 sleeping, 1 stopped, 8 zombie CPU: 92.3% user, 0.0% nice, 7.7% system, 0.0% interrupt, 0.0% idle Mem: 5873M Active, 3662M Inact, 36G Laundry, 13G Wired, 4316M Free ARC: 9564M Total, 2577M MFU, 6739M MRU, 14M Anon, 162M Header, 72M Other Swap: 80G Total, 5336M Used, 75G Free, 6% Inuse, 132K In The high laundry and swap usage is due to cold data in tmpfs. For much of the run tmpfs helps avoid a lot of unnecessary I/O. It looks like I get about a 2X improvement vs. my FX-8320E, but I don't have an exact number because I'm seeing some consistent build breakage in a few of the ports that I build. I get SIGSEGV failures in lang/go and lang/ghc. I also get a compile failure in math/openblas due to what looks like an undefined preprocessor symbol. Continuing to investigate ... It would be nice if poudriere could balance threads between different build jails like make can do with parallel submakes ...