From owner-freebsd-hackers@freebsd.org Sat Apr 21 10:35:48 2018 Return-Path: Delivered-To: freebsd-hackers@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 89D60F86E3A for ; Sat, 21 Apr 2018 10:35:48 +0000 (UTC) (envelope-from gljennjohn@gmail.com) Received: from mail-wr0-x230.google.com (mail-wr0-x230.google.com [IPv6:2a00:1450:400c:c0c::230]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 0055270A67; Sat, 21 Apr 2018 10:35:48 +0000 (UTC) (envelope-from gljennjohn@gmail.com) Received: by mail-wr0-x230.google.com with SMTP id q3-v6so18920196wrj.6; Sat, 21 Apr 2018 03:35:47 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=date:from:to:cc:subject:message-id:in-reply-to:references:reply-to :mime-version:content-transfer-encoding; bh=otAi6SMt9JmtcC8eASn2Lc57MDI3G9FJ6QpmI0lpC1k=; b=rq/l1sFktNjL2s/rmOC8bMOqPAb4RF5QDYgJsq24bO5pftoBrXxD/drKVPdcgMoMwi fn/jsV2fEBeBAFl00otgDLZ5H3g2SQW2GNpl3zijx2K9v247zCvxvxmq+QDmE7NGwNcs m+LrncesjI5vT5z4SiCM6KiqHpIe6uJMjqiAbv2UDdIQKZfusYp2GocwMLfZ2uu8OOz1 80GZUxlwAoo4gnjev47AqS6Eja09JJaDfw3i7UeOMsxALlePFdUqya8+qqxE94aHXQBe cNdDn9LWOGYOCrXP4gpnZEqxEprNkV+sV6wjW5DdAqrqvpSGeRph2Tm0eYdTcPPCimKY AXfg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:date:from:to:cc:subject:message-id:in-reply-to :references:reply-to:mime-version:content-transfer-encoding; bh=otAi6SMt9JmtcC8eASn2Lc57MDI3G9FJ6QpmI0lpC1k=; b=rgMZ4HFS5JKHwpEJgLZKR480VMdXwfetUHKuOY0TivU9wSt3XtoVeX3K4Qiy90B1yI uHEJHZk8XKpdmqVEn6+q/YuxrhuDNRpb3DLxoFvr9xqzfeQSKek9cLV9CeepCUfRjh5I 2mv4KyBAGDTjgeSBP9N/QFseeWZydVSejqTX7o1ktK7yfVy5bwjy/2wzy4OTMYlbPNvR crD9jiyqp6nX8XMQZ1RFB/SFGiYGB5wED5HA0g+PfZQ6VioWpWegv5qijTj2dg1NnZzt ylW0dml9DCr7ky2HpbrFQTJsdch5NpaqreC+rB0fx+s39EpMj4WqzifnBPlszBA0FdGX s5Tw== X-Gm-Message-State: ALQs6tBRhFqsDxqIEPDUv9mX+7WBdvmI7BNm3dhsNlBKU5pluF5sMPXg 54qyqUzEroF5Lw5fnrVgBVm5eA== X-Google-Smtp-Source: AIpwx4+SUs2e0eyaXbvFGsM1PWeuEUw98Aqm7z6T0ynptwfysu5ytdsNG8YxlorRi7oN7ioNinXEUQ== X-Received: by 2002:adf:9cc5:: with SMTP id h5-v6mr10182405wre.11.1524306946411; Sat, 21 Apr 2018 03:35:46 -0700 (PDT) Received: from ernst.home (pD9E23019.dip0.t-ipconnect.de. [217.226.48.25]) by smtp.gmail.com with ESMTPSA id 58-v6sm15558027wrv.41.2018.04.21.03.35.44 (version=TLS1_2 cipher=ECDHE-RSA-CHACHA20-POLY1305 bits=256/256); Sat, 21 Apr 2018 03:35:45 -0700 (PDT) Date: Sat, 21 Apr 2018 12:35:44 +0200 From: Gary Jennejohn To: Konstantin Belousov Cc: Colin Percival , "freebsd-hackers@freebsd.org" , Conrad Meyer Subject: Re: RFC: Hiding per-CPU kernel output behind bootverbose Message-ID: <20180421123544.56d7e690@ernst.home> In-Reply-To: <20180421092049.GM6887@kib.kiev.ua> References: <01000162df15f856-1e5d2641-2a72-4250-8d8e-adcd47bc5db4-000000@email.amazonses.com> <20180419204405.GE6887@kib.kiev.ua> <20180419214550.GF6887@kib.kiev.ua> <01000162e58a466e-98f0305b-1723-467a-bc49-342c3fa9fc5b-000000@email.amazonses.com> <20180421092049.GM6887@kib.kiev.ua> Reply-To: gljennjohn@gmail.com X-Mailer: Claws Mail 3.16.0 (GTK+ 2.24.32; amd64-portbld-freebsd12.0) MIME-Version: 1.0 Content-Type: text/plain; charset=US-ASCII Content-Transfer-Encoding: 7bit X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.25 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 21 Apr 2018 10:35:48 -0000 On Sat, 21 Apr 2018 12:20:49 +0300 Konstantin Belousov wrote: > On Sat, Apr 21, 2018 at 12:11:07AM +0000, Colin Percival wrote: > > On 04/19/18 14:45, Konstantin Belousov wrote: > > > On Thu, Apr 19, 2018 at 02:37:56PM -0700, Conrad Meyer wrote: > > >> On Thu, Apr 19, 2018 at 1:44 PM, Konstantin Belousov wrote: > > >>> The 'CPU XX Launched' messages are very useful for initial diagnostic > > >>> of the SMP startup failures. You need to enable bootverbose to see the > > >>> hang details, but for initial hint they are required. Unfortunately, AP > > >>> startup hangs occur too often to pretend that this can be delegated to > > >>> very specific circumstances. > > >> > > >> Really? I don't know that I've ever seen an AP startup hang. How > > >> often do they occur? > > > > > > It was epidemic with Sandy Bridge, mostly correlated to specific BIOS > > > supplier and its interaction with the x2APIC enablement, see madt.c:170 > > > and below. > > > > > > There were several recent reports of the issue with Broadwell Xeon > > > machines, no additional data or resolution. > > > > > > There are sporadic reports of the problem, where I do not see > > > a clear commonality. > > > > Would it be sufficient for debugging purposes if I change the !bootverbose > > case from printing many lines of > > > > SMP: AP CPU #N Launched! > > > > to instead have a single > > > > SMP: Launching AP CPUs: 86 73 111 21 8 77 100 28 57 42 10 60 87 88 41 113 36 > > 19 72 46 92 52 24 81 90 3 107 96 9 14 80 118 29 121 62 74 56 55 1 12 63 18 67 > > 13 45 102 33 94 69 68 93 83 48 31 30 32 51 89 71 78 64 84 123 61 40 47 37 22 > > 54 101 38 4 97 44 17 109 104 5 85 43 2 99 39 65 95 53 58 66 91 125 23 115 16 > > 35 79 112 103 82 7 75 11 6 98 15 126 127 20 70 34 105 27 50 116 120 49 25 108 > > 106 122 117 114 26 110 59 76 124 119 > > > > ? (With each AP printing its number as it reaches the appropriate point?) > > > > This yields almost the same gain as silencing the launch messages completely, > > while still allowing you to see each CPU announcing itself. > I am fine with the behaviour, but I am not sure how would you implement > this. printf(9) buffers the output, you need to flush it somehow. > And printf(9) calls vprintf(9) which in turn calls _vprintf which will allocate a buffer on the stack (bad idea) if the option PRINTF_BUFR_SIZE is set in the kernel config file. So it seems that output may even be double buffered. -- Gary Jennejohn