From owner-freebsd-hackers@freebsd.org Sat Apr 21 00:11:14 2018 Return-Path: Delivered-To: freebsd-hackers@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 1A1C9FADE59 for ; Sat, 21 Apr 2018 00:11:14 +0000 (UTC) (envelope-from 01000162e58a4670-66a9983e-c3ef-493a-a60f-c477645b5100-000000@amazonses.com) Received: from a8-56.smtp-out.amazonses.com (a8-56.smtp-out.amazonses.com [54.240.8.56]) (using TLSv1 with cipher ECDHE-RSA-AES128-SHA (128/128 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id AB7F885F1A for ; Sat, 21 Apr 2018 00:11:13 +0000 (UTC) (envelope-from 01000162e58a4670-66a9983e-c3ef-493a-a60f-c477645b5100-000000@amazonses.com) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/simple; s=dqtolf56kk3wpt62c3jnwboqvr7iedax; d=tarsnap.com; t=1524269467; h=Subject:To:Cc:References:From:Message-ID:Date:MIME-Version:In-Reply-To:Content-Type:Content-Transfer-Encoding; bh=1akbAon8rasTDWuShxU8zGH48iklbSYpCHWYA8+UU94=; b=OK4TmhPCxc1VPLJ+EURjVZy1EqCy0AmVfCj5GJ8BhXBTarO+84Vz1+8YTSxEFrZt OqdJ6ExdbJYw110jtsXVNREaVxLM4m7fFFbVtVpP6qrneIGcpN9CZpBdF2Hhq5SREv9 eIFo7bipDFghtoxw//w9x0HYnXm8EXoHWplYF478= DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/simple; s=ug7nbtf4gccmlpwj322ax3p6ow6yfsug; d=amazonses.com; t=1524269467; h=Subject:To:Cc:References:From:Message-ID:Date:MIME-Version:In-Reply-To:Content-Type:Content-Transfer-Encoding:Feedback-ID; bh=1akbAon8rasTDWuShxU8zGH48iklbSYpCHWYA8+UU94=; b=O+9TUry9boKkrtfe5Zbni5wZELPVrwTbahrTPfz9uxP1PbOScQHWD5Lx1TyxYXBx xUkzCs009zNolDkAS+58extcTCxwyWQtx+1KJxxWny0RTHL+MU4B9XlCVSy4hq8Sz+w +nvmsqQKYxBSjgEXW7x1UF88x33QwLcc4AN+exZk= Subject: Re: RFC: Hiding per-CPU kernel output behind bootverbose To: Konstantin Belousov , Conrad Meyer Cc: "freebsd-hackers@freebsd.org" References: <01000162df15f856-1e5d2641-2a72-4250-8d8e-adcd47bc5db4-000000@email.amazonses.com> <20180419204405.GE6887@kib.kiev.ua> <20180419214550.GF6887@kib.kiev.ua> From: Colin Percival Openpgp: preference=signencrypt Autocrypt: addr=cperciva@tarsnap.com; prefer-encrypt=mutual; keydata= xsDhBElrAAcRBACDfDys4ZtK+ErCJ1HAzYeteKpm3OEsvT/49AjUTLihkF79HhIKrCQU+1KC zv7BwHCMLb6hq30As9L7iFKG7n5QFLFC4Te/VcITUnWHMG/c3ViLOfJGvi+9/nOEHaM1dVJY D6tEp5yM1nHmVQpo9932j4KGuGFR0LhOK5IHXOSfGwCgxSFDPdgxe2OEjWxjGgY+oV3EafcD +JROXCTjlcQiG/OguQH4Vks3mhHfFnEppLxTkDuYgHZQiUtpcT9ssH5khgqoTyMar05OUdAj ZIhNbWDh4LgTj+7ZmvLhXT5Zxw8LX9d7T36aTB8XDQSenDqEtinMWOb0TCBBLbsB8EFG1WTT ESbZci9jJS5yhtktuZoY/eM8uXMD/3k4FWFO80VRRkELSp+XSy/VlSQjyi/rhl2nQq/oOA9F oJbDaB0yq9VNhxP+uFBzBWSqeIX0t1ZWLtNfVFr4TRP5hihI5ICrg/0OpqgisKsU2NFe9xyO hyJLYmfD8ebpDJ/9k30C7Iju9pVrwLm1QgS4S2fqJRcR+U4WbjvP7CgSzSVDb2xpbiBQZXJj aXZhbCA8Y3BlcmNpdmFAdGFyc25hcC5jb20+wmEEExECACEFAklrALYCGwMHCwkIBwMCAQQV AggDBBYCAwECHgECF4AACgkQOM7KaQxqam6/igCgn+z2k3V5ggNppmWrZstt1U2lugsAoL7L wS9V9yLtil3oWmHtwpUqYruEzsFNBElrAAcQCAD3ZLMIsP4CIDoJORg+YY0lqLVBgcnF7pFb 4Uy2+KvdWofN+DKH61rZLjgXXkNE9M4EQC1B4lGttBP8IY2gs41y3AUogGdyFbidq99rCBz7 LTsgARHwFxZoaHmXyiZLEU1QZuMqwPZV1mCviRhN5E3rRqYNXVcrnXAAuhBpvNyj/ntHvcDN 2/m+ochiuBYueU4kX3lHya7sOj+mTsndcWmQ9soOUyr8O0r/BG088bMn4qqtUw4dl5/pglXk jbl7uOOPinKf0WVd2r6M0wLPJCD4NPHrCWRLLLAjwfjrtoSRvXxDbXhCdgGBa72+K8eYLzVs hgq7tJOoBWzjVK6XRxR7AAMGB/9Mo3iJ2DxqDecd02KCB5BsFDICbJGhPltU7FwrtbC7djSb XUrwsEVLHi4st4cbdGNCWCrp0BRezXZKohKnNAPFOTK++ZfgeKxrV2sJod+Q9RILF86tQ4XF 7A7Yme5hy92t/WgiU4vc/fWbgP8gV/19f8nunaT2E9NSa70mZFjZNu4iuwThoUUO5CV3Wo0Y UISsnRK8XD1+LR3A2qVyLiFRwh/miC1hgLFCTGCQ3GLxZeZzIpYSlGdQJ0L5lixW5ZQD9r1I 8i/8zhE6qRFAM0upUMI3Gt1Oq2w03DiXrZU0Fu/R8Rm8rlnkQKA+95mRTUq1xL5P5NZIi4gJ Z569OPMFwkkEGBECAAkFAklrAAcCGwwACgkQOM7KaQxqam41igCfbaldnFTu5uAdrnrghESv EI3CAo8AoLkNMks1pThl2BJNRm4CtTK9xZeH Message-ID: <01000162e58a4670-66a9983e-c3ef-493a-a60f-c477645b5100-000000@email.amazonses.com> Date: Sat, 21 Apr 2018 00:11:07 +0000 User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:52.0) Gecko/20100101 Thunderbird/52.7.0 MIME-Version: 1.0 In-Reply-To: <20180419214550.GF6887@kib.kiev.ua> Content-Type: text/plain; charset=utf-8 Content-Language: en-US Content-Transfer-Encoding: 7bit X-SES-Outgoing: 2018.04.21-54.240.8.56 Feedback-ID: 1.us-east-1.Lv9FVjaNvvR5llaqfLoOVbo2VxOELl7cjN0AOyXnPlk=:AmazonSES X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.25 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 21 Apr 2018 00:11:14 -0000 On 04/19/18 14:45, Konstantin Belousov wrote: > On Thu, Apr 19, 2018 at 02:37:56PM -0700, Conrad Meyer wrote: >> On Thu, Apr 19, 2018 at 1:44 PM, Konstantin Belousov wrote: >>> The 'CPU XX Launched' messages are very useful for initial diagnostic >>> of the SMP startup failures. You need to enable bootverbose to see the >>> hang details, but for initial hint they are required. Unfortunately, AP >>> startup hangs occur too often to pretend that this can be delegated to >>> very specific circumstances. >> >> Really? I don't know that I've ever seen an AP startup hang. How >> often do they occur? > > It was epidemic with Sandy Bridge, mostly correlated to specific BIOS > supplier and its interaction with the x2APIC enablement, see madt.c:170 > and below. > > There were several recent reports of the issue with Broadwell Xeon > machines, no additional data or resolution. > > There are sporadic reports of the problem, where I do not see > a clear commonality. Would it be sufficient for debugging purposes if I change the !bootverbose case from printing many lines of SMP: AP CPU #N Launched! to instead have a single SMP: Launching AP CPUs: 86 73 111 21 8 77 100 28 57 42 10 60 87 88 41 113 36 19 72 46 92 52 24 81 90 3 107 96 9 14 80 118 29 121 62 74 56 55 1 12 63 18 67 13 45 102 33 94 69 68 93 83 48 31 30 32 51 89 71 78 64 84 123 61 40 47 37 22 54 101 38 4 97 44 17 109 104 5 85 43 2 99 39 65 95 53 58 66 91 125 23 115 16 35 79 112 103 82 7 75 11 6 98 15 126 127 20 70 34 105 27 50 116 120 49 25 108 106 122 117 114 26 110 59 76 124 119 ? (With each AP printing its number as it reaches the appropriate point?) This yields almost the same gain as silencing the launch messages completely, while still allowing you to see each CPU announcing itself. -- Colin Percival Security Officer Emeritus, FreeBSD | The power to serve Founder, Tarsnap | www.tarsnap.com | Online backups for the truly paranoid