Date: Wed, 25 Sep 2019 07:13:17 +0200 From: Peter Holm <pho@freebsd.org> To: Alexander Motin <mav@freebsd.org> Cc: src-committers@freebsd.org, svn-src-all@freebsd.org, svn-src-head@freebsd.org Subject: Re: svn commit: r352658 - head/sys/kern Message-ID: <20190925051317.GA77827@x8.osted.lan> In-Reply-To: <201909242001.x8OK1KAV099968@repo.freebsd.org> References: <201909242001.x8OK1KAV099968@repo.freebsd.org>
next in thread | previous in thread | raw e-mail | index | archive | help
On Tue, Sep 24, 2019 at 08:01:20PM +0000, Alexander Motin wrote: > Author: mav > Date: Tue Sep 24 20:01:20 2019 > New Revision: 352658 > URL: https://svnweb.freebsd.org/changeset/base/352658 > > Log: > Fix/improve interrupt threads scheduling. > > Doing some tests with very high interrupt rates I've noticed that one of > conditions I added in r232207 to make interrupt threads in most cases > run on local CPU never worked as expected (worked only if previous time > it was executed on some other CPU, that is quite opposite). It caused > additional CPU usage to run full CPU search and could schedule interrupt > threads to some other CPU. > > This patch removes that code and instead reuses existing non-interrupt > code path with some tweaks for interrupt case: > - On SMT systems, if current thread is idle, don't look on other threads. > Even if they are busy, it may take more time to do fill search and bounce > the interrupt thread to other core then execute it locally, even sharing > CPU resources. It is other threads should migrate, not bound interrupts. > - Try hard to keep interrupt threads within LLC of their original CPU. > This improves scheduling cost and supposedly cache and memory locality. > > On a test system with 72 threads doing 2.2M IOPS to NVMe this saves few > percents of CPU time while adding few percents to IOPS. > > MFC after: 1 month > Sponsored by: iXsystems, Inc. > > Modified: > head/sys/kern/sched_ule.c > > Modified: head/sys/kern/sched_ule.c > ============================================================================== > --- head/sys/kern/sched_ule.c Tue Sep 24 18:18:11 2019 (r352657) > +++ head/sys/kern/sched_ule.c Tue Sep 24 20:01:20 2019 (r352658) > @@ -1251,7 +1251,7 @@ sched_pickcpu(struct thread *td, int flags) Could this be yours? FreeBSD/SMP: Multiprocessor System Detected: 24 CPUs FreeBSD/SMP: 2 package(s) x 6 core(s) x 2 hardware threads random: unblocking device. Firmware Warning (ACPI): Invalid length for FADT/Pm1aControlBlock: 32, using default 16 (20190703/tbfadt-850) ioapic0 <Version 2.0> irqs 0-23 ioapic1 <Version 2.0> irqs 24-47 ioapic2 <Version 2.0> irqs 48-71 Launching APs: 13 6 18 17 5 9 8 19 7 10 1 11 2 12 14 15 20 4 21 16 22 3 23 panic: sched_pickcpu: Failed to find a cpu. cpuid = 0 time = 1 KDB: stack backtrace: db_trace_self_wrapper() at db_trace_self_wrapper+0x2b/frame 0xffffffff8254a830 vpanic() at vpanic+0x19d/frame 0xffffffff8254a880 panic() at panic+0x43/frame 0xffffffff8254a8e0 sched_pickcpu() at sched_pickcpu+0x4c1/frame 0xffffffff8254a990 sched_add() at sched_add+0x6e/frame 0xffffffff8254a9d0 gtaskqueue_start_threads() at gtaskqueue_start_threads+0x124/frame 0xffffffff8254aa70 taskqgroup_cpu_create() at taskqgroup_cpu_create+0x135/frame 0xffffffff8254aab0 taskqgroup_adjust() at taskqgroup_adjust+0x1ad/frame 0xffffffff8254ab20 mi_startup() at mi_startup+0x210/frame 0xffffffff8254ab70 btext() at btext+0x2c KDB: enter: panic [ thread pid 0 tid 100000 ] Stopped at kdb_enter+0x3b: movq $0,kdb_why db> - Peter
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?20190925051317.GA77827>