From owner-freebsd-stable@freebsd.org Fri Nov 25 12:12:54 2016 Return-Path: Delivered-To: freebsd-stable@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 364A0C5487E for ; Fri, 25 Nov 2016 12:12:54 +0000 (UTC) (envelope-from jason.harmening@gmail.com) Received: from mail-lf0-x22c.google.com (mail-lf0-x22c.google.com [IPv6:2a00:1450:4010:c07::22c]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id AD059124 for ; Fri, 25 Nov 2016 12:12:53 +0000 (UTC) (envelope-from jason.harmening@gmail.com) Received: by mail-lf0-x22c.google.com with SMTP id c13so49203490lfg.0 for ; Fri, 25 Nov 2016 04:12:53 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:in-reply-to:references:from:date:message-id:subject:to :cc; bh=39+V4UGcIpjosFtwLHOFGaeY/Od+vQtvRpbMmgGUUhk=; b=Syr6cBttsBpJQrlJwcEbOZ4MuvhmxKEDxc3YL2Tv08NU/aZRb92L9oEPcyaJotU6/W zigMK9X6zx+gbnHiuUS2Wf5m8jlQIxKqJExKTplZNaVR/x6y5HVko9ny/WQj7qJh1RlE veXXq0G0B35/wR8K98u1gvwIvkfX2G6gZEdYuOTywB02sWY4KqMXevt6tCvoyO6WyQkT deODzx+YooXeR/kevpmQc6682xwh9mCy1eSBmnKz0EEPf4sfO/HvBRpyoBtr/FzEwtfo 8m7LgHj++guLoHAFiURaXEmenOX3+/KRbs3PBEPVgLx5pxIHKWEVIzmyU4syusJkhR+5 M4DQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:mime-version:in-reply-to:references:from:date :message-id:subject:to:cc; bh=39+V4UGcIpjosFtwLHOFGaeY/Od+vQtvRpbMmgGUUhk=; b=U3CaWPl3hozILn7ftKLPgTJNX4/RxicaRq3Nvot1YsmLYAU4kNuhtZZObnO1C5M0if +syin/9sLnM4DCtd2bg4hbn0EDmuktbGrXh/S8gedH5dL+RrsxLQKLLB1CplxjZxVfws AvSQUQc+hzBKO7I459HLUrWUDFQz3cE+xNCKQcIrsApV3z74zsflZra3KTxWFO4isga0 EnuOQHtFahMgH/MYrnmpOyKJ6hOpmHGHZkxn4wP6tWiTLGFCnrWWw8aQmXqXMY9RbUm0 fCQGr9wEElL6KigtfIn1e91zopj+DbCZBurMkNhgNXb92HhUU/B8fCDRiDxr8X+XCsjj OKfw== X-Gm-Message-State: AKaTC01fXk9HuJ9BC6TyzuYPP3uJIDQkRGuJyre1ZGRoboB5ULDrj44bxrTY4dpmH6lRufs2bEKPtt0qp6YzaA== X-Received: by 10.46.1.93 with SMTP id 90mr3651230ljb.30.1480075971652; Fri, 25 Nov 2016 04:12:51 -0800 (PST) MIME-Version: 1.0 Received: by 10.25.193.16 with HTTP; Fri, 25 Nov 2016 04:12:50 -0800 (PST) In-Reply-To: <20161125092503.GZ54029@kib.kiev.ua> References: <6167392c-c37a-6e39-aa22-ca45435d6088@gmail.com> <20161102075509.GF54029@kib.kiev.ua> <3620f62e-0f4c-2d62-dcf8-e2fdff459250@gmail.com> <20161102162808.GI54029@kib.kiev.ua> <20161125092503.GZ54029@kib.kiev.ua> From: Jason Harmening Date: Fri, 25 Nov 2016 04:12:50 -0800 Message-ID: Subject: Re: huge nanosleep variance on 11-stable To: Konstantin Belousov Cc: FreeBSD-STABLE Mailing List Content-Type: text/plain; charset=UTF-8 X-Content-Filtered-By: Mailman/MimeDel 2.1.23 X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 25 Nov 2016 12:12:54 -0000 On Fri, Nov 25, 2016 at 1:25 AM, Konstantin Belousov wrote: > On Wed, Nov 02, 2016 at 06:28:08PM +0200, Konstantin Belousov wrote: > > On Wed, Nov 02, 2016 at 09:18:15AM -0700, Jason Harmening wrote: > > > I think you are probably right. Hacking out the Intel-specific > > > additions to C-state parsing in acpi_cpu_cx_cst() from r282678 (thus > > > going back to sti;hlt instead of monitor+mwait at C1) fixed the problem > > > for me. But r282678 also had the effect of enabling C2 and C3 on my > > > system, because ACPI only presents MWAIT entries for those states and > > > not p_lvlx. > > You can do the same with "debug.acpi.disabled=mwait" loader tunable > > without hacking the code. And set sysctl hw.acpi.cpu.cx_lowest to C1 to > > enforce use of hlt instruction even when mwait states were requested. > > I believe I now understood the problem. First, I got the definitive > confirmation that LAPIC timer on Nehalems is stopped in any C mode > higher than C1/C1E, i.e. even if C2 is enabled LAPIC eventtimer cannot > be used. This is consistent with the ARAT CPUID bit CPUID[0x6].eax[2] > reported zero. > > On SandyBridge and IvyBridge CPUs, it seems that ARAT might be both 0 > and 1 according to the same source, but all CPUs I saw have ARAT = 1. > And for Haswell and later generations, ARAT is claimed to be always > implemented. > > The actual issue is somewhat silly bug, I must admit: if ncpus >= 8, and > non-FSB interrupt routing from HPET, default HPET eventtimer quality 450 > is reduced by 100, i.e. it is 350. OTOH, LAPIC default quality is 600 > and it is reduced by 200 if ARAT is not reported. We end up with HPET > quality 350 < LAPIC quality 400, despite ARAT is not set. > > The patch below sets LAPIC eventtimer quality to 100 if not ARAT. Also > I realized that there is no reason to disable deadline mode regardless > of ARAT. > > diff --git a/sys/x86/x86/local_apic.c b/sys/x86/x86/local_apic.c > index d9a3453..1b1547d 100644 > --- a/sys/x86/x86/local_apic.c > +++ b/sys/x86/x86/local_apic.c > @@ -478,8 +478,9 @@ native_lapic_init(vm_paddr_t addr) > lapic_et.et_quality = 600; > if (!arat) { > lapic_et.et_flags |= ET_FLAGS_C3STOP; > - lapic_et.et_quality -= 200; > - } else if ((cpu_feature & CPUID_TSC) != 0 && > + lapic_et.et_quality = 100; > + } > + if ((cpu_feature & CPUID_TSC) != 0 && > (cpu_feature2 & CPUID2_TSCDLT) != 0 && > tsc_is_invariant && tsc_freq != 0) { > lapic_timer_tsc_deadline = 1; > > Ah, that makes sense. Thanks! I'll try the patch as soon as I get back from vacation. I've been able to verify that setting cx_lowest and disabling mwait fixes the problem without hacking the code. But I've been too busy at $(WORK) to check anything else, namely whether forcing HPET would also fix the problem.