From owner-freebsd-amd64@freebsd.org Wed Mar 22 22:41:17 2017 Return-Path: Delivered-To: freebsd-amd64@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id A93BBD182A9 for ; Wed, 22 Mar 2017 22:41:17 +0000 (UTC) (envelope-from avg@FreeBSD.org) Received: from citapm.icyb.net.ua (citapm.icyb.net.ua [212.40.38.140]) by mx1.freebsd.org (Postfix) with ESMTP id 87CD0243; Wed, 22 Mar 2017 22:41:15 +0000 (UTC) (envelope-from avg@FreeBSD.org) Received: from porto.starpoint.kiev.ua (porto-e.starpoint.kiev.ua [212.40.38.100]) by citapm.icyb.net.ua (8.8.8p3/ICyb-2.3exp) with ESMTP id AAA29872; Thu, 23 Mar 2017 00:41:14 +0200 (EET) (envelope-from avg@FreeBSD.org) Received: from localhost ([127.0.0.1]) by porto.starpoint.kiev.ua with esmtp (Exim 4.34 (FreeBSD)) id 1cqowM-000A2X-7H; Thu, 23 Mar 2017 00:41:14 +0200 Subject: Re: FreeBSD on Ryzen To: Don Lewis , freebsd-amd64@FreeBSD.org References: <201703222030.v2MKUJJs026400@gw.catspoiler.org> From: Andriy Gapon Message-ID: <4d09dde2-fef3-ac41-9d3a-6567c9b04029@FreeBSD.org> Date: Thu, 23 Mar 2017 00:40:18 +0200 User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:45.0) Gecko/20100101 Thunderbird/45.8.0 MIME-Version: 1.0 In-Reply-To: <201703222030.v2MKUJJs026400@gw.catspoiler.org> Content-Type: text/plain; charset=windows-1252 Content-Transfer-Encoding: 7bit X-Mailman-Approved-At: Thu, 23 Mar 2017 03:49:14 +0000 X-BeenThere: freebsd-amd64@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: Porting FreeBSD to the AMD64 platform List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 22 Mar 2017 22:41:17 -0000 On 03/22/2017 22:30, Don Lewis wrote: > I put together a Ryzen 1700X machine over the weekend and installed the > 12.0-CURRENT r315413 snapshot on it a couple of days ago. The RAM is > DDR4 2400. Very interesting, thank you for sharing! > First impression is that it's pretty zippy. Compared to my previous > fastest machine: > CPU: AMD FX-8320E Eight-Core Processor (3210.84-MHz K8-class CPU) > make -j8 buildworld using tmpfs is a bit more than 2x faster. Since the > Ryzen has SMT, it's eight cores look like 16 CPUs to FreeBSD, I get > almost a 2.6x speedup with -j16 as compared to my old machine. > > I do see that the reported total CPU time increases quite a bit at -j16 > (~19900u) as compared to -j8 (~13600u) so it is running into some > hardware bottlenecks that are slowing down instruction execution. It > could be the resources shared by both SMT threads that share each core, > or it could be cache or memory bandwidth related. The Ryzen topology is > a bit complicated. There are two groups of four cores, where each group > of four cores shares half of the L3 cache, with a slowish interconnect > bus between the groups. This probably causes some NUMA-like issues. I > wonder if the ULE scheduler could be tweaked to handle this better. > > % sysctl kern.sched.topology_spec > kern.sched.topology_spec: > > 0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15 > > > 0, 1, 2, 3, 4, 5, 6, 7 > > > 0, 1 > > > 2, 3 > > > 4, 5 > > > 6, 7 > > > > > 8, 9, 10, 11, 12, 13, 14, 15 > > > 8, 9 > > > 10, 11 > > > 12, 13 > > > 14, 15 > > > > > > The discovered topology looks like what I would expect. Each complex has its L3 cache and that's reflected above. L2 and L1 caches are per core (2 hardware "threads") and that's correct too. The only thing missing is the SMT flag per each SMT pair. I think that it does not have a dramatic effect on scheduling, but I can look into this shortcoming. > I'll run some tests with SMT disabled when I get a chance. > > The amdtemp driver doesn't attach to this CPU, so I can't monitor the > temperature at runtime. > > I've got ECC RAM, but unfortunately the motherboard I'm using doesn't > enable ECC. Are you absolutely sure of that? > The chipset is the B350. > Could you please share dmesg from a verbose boot? At least the portion where CPUs and topology are described. Thanks! > Copyright (c) 1992-2017 The FreeBSD Project. > Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994 > The Regents of the University of California. All rights reserved. > FreeBSD is a registered trademark of The FreeBSD Foundation. > FreeBSD 12.0-CURRENT #0 r315413: Thu Mar 16 17:23:31 UTC 2017 > root@releng3.nyi.freebsd.org:/usr/obj/usr/src/sys/GENERIC amd64 > FreeBSD clang version 4.0.0 (tags/RELEASE_400/final 297347) (based on LLVM 4.0.0) > WARNING: WITNESS option enabled, expect reduced performance. > SRAT: No memory found for CPU 0 > VT(vga): resolution 640x480 > CPU: AMD Ryzen 7 1700X Eight-Core Processor (3393.71-MHz K8-class CPU) > Origin="AuthenticAMD" Id=0x800f11 Family=0x17 Model=0x1 Stepping=1 > Features=0x178bfbff > Features2=0x7ed8320b > AMD Features=0x2e500800 > AMD Features2=0x35c233ff > Structured Extended Features=0x209c01a9 > XSAVE Features=0xf > SVM: NP,NRIP,VClean,AFlush,DAssist,NAsids=32768 > TSC: P-state invariant, performance statistics > real memory = 68719476736 (65536 MB) > avail memory = 66671529984 (63582 MB) > Event timer "LAPIC" quality 100 > ACPI APIC Table: > FreeBSD/SMP: Multiprocessor System Detected: 16 CPUs > FreeBSD/SMP: 1 package(s) x 16 core(s) > random: unblocking device. > ACPI BIOS Warning (bug): Optional FADT field Pm2ControlBlock has valid Length but zero Address: 0x0000000000000000/0x1 (20170303/tbfadt-796) > ioapic0: Changing APIC ID to 17 > ioapic1: Changing APIC ID to 18 > ioapic0 irqs 0-23 on motherboard > ioapic1 irqs 24-55 on motherboard > SMP: AP CPU #12 Launched! > SMP: AP CPU #1 Launched! > SMP: AP CPU #4 Launched! > SMP: AP CPU #7 Launched! > SMP: AP CPU #6 Launched! > SMP: AP CPU #3 Launched! > SMP: AP CPU #11 Launched! > SMP: AP CPU #5 Launched! > SMP: AP CPU #15 Launched! > SMP: AP CPU #9 Launched! > SMP: AP CPU #10 Launched! > SMP: AP CPU #8 Launched! > SMP: AP CPU #2 Launched! > SMP: AP CPU #14 Launched! > SMP: AP CPU #13 Launched! > Timecounter "TSC-low" frequency 1696854133 Hz quality 1000 > random: entropy device external interface > netmap: loaded module > module_register_init: MOD_LOAD (vesa, 0xffffffff80f49c70, 0) error 19 > random: registering fast source Intel Secure Key RNG > random: fast provider: "Intel Secure Key RNG" > kbd1 at kbdmux0 > nexus0 > vtvga0: on motherboard > cryptosoft0: on motherboard > acpi0: on motherboard > acpi0: Power Button (fixed) > cpu0: on acpi0 > cpu1: on acpi0 > cpu2: on acpi0 > cpu3: on acpi0 > cpu4: on acpi0 > cpu5: on acpi0 > cpu6: on acpi0 > cpu7: on acpi0 > cpu8: on acpi0 > cpu9: on acpi0 > cpu10: on acpi0 > cpu11: on acpi0 > cpu12: on acpi0 > cpu13: on acpi0 > cpu14: on acpi0 > cpu15: on acpi0 > attimer0: port 0x40-0x43 irq 0 on acpi0 > Timecounter "i8254" frequency 1193182 Hz quality 0 > Event timer "i8254" frequency 1193182 Hz quality 100 > atrtc0: port 0x70-0x71 on acpi0 > Event timer "RTC" frequency 32768 Hz quality 0 > hpet0: iomem 0xfed00000-0xfed003ff irq 0,8 on acpi0 > Timecounter "HPET" frequency 14318180 Hz quality 950 > Event timer "HPET" frequency 14318180 Hz quality 350 > Event timer "HPET1" frequency 14318180 Hz quality 350 > Event timer "HPET2" frequency 14318180 Hz quality 350 > Timecounter "ACPI-fast" frequency 3579545 Hz quality 900 > acpi_timer0: <32-bit timer at 3.579545MHz> port 0x808-0x80b on acpi0 > pcib0: port 0xcf8-0xcff on acpi0 > pci0: on pcib0 > pci0: at device 0.2 (no driver attached) > pcib1: at device 1.3 on pci0 > pci1: on pcib1 > xhci0: mem 0xfd6a0000-0xfd6a7fff irq 32 at device 0.0 on pci1 > xhci0: 32 bytes context size, 64-bit DMA > usbus0 on xhci0 > usbus0: 5.0Gbps Super Speed USB v3.0 > ahci0: mem 0xfd680000-0xfd69ffff irq 33 at device 0.1 on pci1 > ahci0: AHCI v1.31 with 8 6Gbps ports, Port Multiplier supported > ahcich0: at channel 0 on ahci0 > ahcich1: at channel 1 on ahci0 > ahcich4: at channel 4 on ahci0 > ahcich5: at channel 5 on ahci0 > pcib2: irq 34 at device 0.2 on pci1 > pci2: on pcib2 > pcib3: irq 32 at device 0.0 on pci2 > pci3: on pcib3 > re0: port 0xf000-0xf0ff mem 0xfd500000-0xfd500fff,0xf2100000-0xf2103fff irq 32 at device 0.0 on pci3 > re0: Using 1 MSI-X message > re0: Chip rev. 0x4c000000 > re0: MAC rev. 0x00000000 > miibus0: on re0 > rgephy0: PHY 1 on miibus0 > rgephy0: none, 10baseT, 10baseT-FDX, 10baseT-FDX-flow, 100baseTX, 100baseTX-FDX, 100baseTX-FDX-flow, 1000baseT-FDX, 1000baseT-FDX-master, 1000baseT-FDX-flow, 1000baseT-FDX-flow-master, auto, auto-flow > re0: Using defaults for TSO: 65518/35/2048 > re0: Ethernet address: 1c:1b:0d:9a:99:cb > re0: netmap queues/slots: TX 1/256, RX 1/256 > pcib4: irq 33 at device 1.0 on pci2 > pci4: on pcib4 > pcib5: irq 32 at device 4.0 on pci2 > pci5: on pcib5 > pcib6: at device 3.1 on pci0 > pci6: on pcib6 > vgapci0: port 0xe000-0xe07f mem 0xfc000000-0xfcffffff,0xe0000000-0xefffffff,0xf0000000-0xf1ffffff irq 54 at device 0.0 on pci6 > vgapci0: Boot video device > hdac0: mem 0xfd080000-0xfd083fff irq 55 at device 0.1 on pci6 > pcib7: at device 7.1 on pci0 > pci7: on pcib7 > pci7: at device 0.0 (no driver attached) > pci7: at device 0.2 (no driver attached) > xhci1: mem 0xfd200000-0xfd2fffff irq 37 at device 0.3 on pci7 > xhci1: 64 bytes context size, 64-bit DMA > usbus1 on xhci1 > usbus1: 5.0Gbps Super Speed USB v3.0 > pcib8: at device 8.1 on pci0 > pci8: on pcib8 > pci8: at device 0.0 (no driver attached) > ahci1: mem 0xfd708000-0xfd708fff irq 42 at device 0.2 on pci8 > ahci1: AHCI v1.31 with 2 6Gbps ports, Port Multiplier supported with FBS > ahcich8: at channel 0 on ahci1 > ahcich9: at channel 1 on ahci1 > hdac1: mem 0xfd700000-0xfd707fff irq 43 at device 0.3 on pci8 > isab0: at device 20.3 on pci0 > isa0: on isab0 > acpi_button0: on acpi0 > atkbdc0: port 0x60,0x64 irq 1 on acpi0 > atkbd0: irq 1 on atkbdc0 > kbd0 at atkbd0 > atkbd0: [GIANT-LOCKED] > uart0: <16550 or compatible> port 0x3f8-0x3ff irq 4 flags 0x10 on acpi0 > ppc0: cannot reserve I/O port range > hwpstate0: on cpu0 > ZFS filesystem version: 5 > ZFS storage pool version: features support (5000) > Timecounters tick every 1.000 msec > hdacc0: at cad 0 on hdac0 > hdaa0: at nid 1 on hdacc0 > pcm0: at nid 5 on hdaa0 > hdacc1: at cad 1 on hdac0 > hdaa1: at nid 1 on hdacc1 > pcm1: at nid 5 on hdaa1 > hdacc2: at cad 2 on hdac0 > hdaa2: at nid 1 on hdacc2 > pcm2: at nid 5 on hdaa2 > hdacc3: at cad 3 on hdac0 > hdaa3: at nid 1 on hdacc3 > pcm3: at nid 5 on hdaa3 > hdacc4: at cad 0 on hdac1 > hdaa4: at nid 1 on hdacc4 > pcm4: at nid 20 and 24,26 on hdaa4 > pcm5: at nid 27 and 25 on hdaa4 > pcm6: at nid 17 on hdaa4 > ugen1.1: <0x1022 XHCI root HUB> at usbus1 > ugen0.1: <0x1022 XHCI root HUB> at usbus0 > uhub0: <0x1022 XHCI root HUB, class 9/0, rev 3.00/1.00, addr 1> on usbus1 > uhub1: <0x1022 XHCI root HUB, class 9/0, rev 3.00/1.00, addr 1> on usbus0 > ada0 at ahcich0 bus 0 scbus0 target 0 lun 0 > ada0: ATA8-ACS SATA 2.x device > ada0: Serial Number WD-WCASJ1778605 > ada0: 300.000MB/s transfers (SATA 2.x, UDMA6, PIO 8192bytes) > ada0: Command Queueing enabled > ada0: 953869MB (1953525168 512 byte sectors) > WARNING: WITNESS option enabled, expect reduced performance. > Trying to mount root from zfs:zroot/ROOT/default []... > Root mount waiting for: usbus1 usbus0 > uhub0: 8 ports with 8 removable, self powered > uhub1: 22 ports with 22 removable, self powered > ugen1.2: at usbus1 > umass0 on uhub0 > umass0: on usbus1 > umass0: SCSI over Bulk-Only; quirks = 0x4101 > umass0:6:0: Attached to scbus6 > da0 at umass-sim0 bus 0 scbus6 target 0 lun 0 > da0: < USB Disk 8.07> Removable Direct Access SPC-2 SCSI device > da0: Serial Number 5E2CDC72 > da0: 40.000MB/s transfers > da0: 3840MB (7864320 512 byte sectors) > da0: quirks=0x2 > GEOM: da0: the secondary GPT header is not in the last LBA. > Root mount waiting for: usbus1 > ugen1.3: at usbus1 > re0: link state changed to DOWN > ums0 on uhub0 > ums0: on usbus1 > ums0: 3 buttons and [XYZ] coordinates ID=0 > -- Andriy Gapon