From owner-freebsd-current@freebsd.org Thu Feb 23 10:08:34 2017 Return-Path: Delivered-To: freebsd-current@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id E3105CE8A81 for ; Thu, 23 Feb 2017 10:08:34 +0000 (UTC) (envelope-from kostikbel@gmail.com) Received: from kib.kiev.ua (kib.kiev.ua [IPv6:2001:470:d5e7:1::1]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 7FE6ABD; Thu, 23 Feb 2017 10:08:34 +0000 (UTC) (envelope-from kostikbel@gmail.com) Received: from tom.home (kib@localhost [127.0.0.1]) by kib.kiev.ua (8.15.2/8.15.2) with ESMTPS id v1NA8T65075796 (version=TLSv1.2 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=NO); Thu, 23 Feb 2017 12:08:29 +0200 (EET) (envelope-from kostikbel@gmail.com) DKIM-Filter: OpenDKIM Filter v2.10.3 kib.kiev.ua v1NA8T65075796 Received: (from kostik@localhost) by tom.home (8.15.2/8.15.2/Submit) id v1NA8TpW075795; Thu, 23 Feb 2017 12:08:29 +0200 (EET) (envelope-from kostikbel@gmail.com) X-Authentication-Warning: tom.home: kostik set sender to kostikbel@gmail.com using -f Date: Thu, 23 Feb 2017 12:08:29 +0200 From: Konstantin Belousov To: Jia-Shiun Li Cc: freebsd-current , John Baldwin , Konstantin Belousov Subject: Re: TSC as timecounter makes system lag Message-ID: <20170223100829.GR2092@kib.kiev.ua> References: <20170113120534.GC2349@kib.kiev.ua> <20170115202008.GL2349@kib.kiev.ua> <20170116120021.GQ2349@kib.kiev.ua> <5f814a7a-5e46-057e-62ad-dde0a61bc1c8@selasky.org> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: User-Agent: Mutt/1.7.2 (2016-11-26) X-Spam-Status: No, score=-2.0 required=5.0 tests=ALL_TRUSTED,BAYES_00, DKIM_ADSP_CUSTOM_MED,FREEMAIL_FROM,NML_ADSP_CUSTOM_MED autolearn=no autolearn_force=no version=3.4.1 X-Spam-Checker-Version: SpamAssassin 3.4.1 (2015-04-28) on tom.home X-BeenThere: freebsd-current@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: Discussions about the use of FreeBSD-current List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 23 Feb 2017 10:08:35 -0000 On Thu, Feb 23, 2017 at 01:11:29AM +0800, Jia-Shiun Li wrote: > I got the impression that TSC was not preferred timecounter > if it is not C-state invariant. But this apparenly is not the case now. > Dig a bit and found r277900 chose to prefer TSC over saving power > by disabling C2 state when TSC is selected as timecounter. > > But with EARLY_AP_STARTUP, and TSC as timecounter, > CPU still enters C2 state. > (Observed by sysctl dev.cpu.0.cx_usage_counters) > With nooptions EARLY_AP_STARTUP, CPU correctly stays 100% > in C1 and no lower. > > I added a printf in tc_windup() to check. With EARLY_AP_STARTUP, > cpu_disable_c2_sleep is never increased, so it can not prevent CPU > from entering C2. > > Guess there's some kind of race or init order issue, but it is > beyond my understanding for now. This is a useful analysis. Yes, I think that there is an init ordering issue. Note that cpu_disable_c2_sleep is only changed in tc_windup() when timecounter is changed. If existing and already engadged timecounter suddenly gets TC_FLAG_C2STOP set, tc_windup() ignores the flag. And with the early AP startup, tsc seems to be set as timecounter too early. Just moving order of init_TSC_tc() would not help, since tsc checks smp consistency, which requires started APs. Try this for now, but might be John has better idea how to handle the issue. You might need to add some extern declarations for the patch to compile. diff --git a/sys/x86/x86/tsc.c b/sys/x86/x86/tsc.c index 3f36fbd9f8a..f8e33069c70 100644 --- a/sys/x86/x86/tsc.c +++ b/sys/x86/x86/tsc.c @@ -545,6 +545,8 @@ init_TSC_tc(void) if (cpu_deepest_sleep >= 2 && cpu_vendor_id == CPU_VENDOR_INTEL && (amd_pminfo & AMDPM_TSC_INVARIANT) == 0) { tsc_timecounter.tc_flags |= TC_FLAGS_C2STOP; + if (timecounter == &tsc_timecounter) + cpu_disable_c2_sleep++; if (bootverbose) printf("TSC timecounter disables C2 and C3.\n"); }