From owner-freebsd-current@freebsd.org Fri Dec 22 09:44:54 2017 Return-Path: Delivered-To: freebsd-current@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 7BB0CE8C645 for ; Fri, 22 Dec 2017 09:44:54 +0000 (UTC) (envelope-from 010001607d9c59d5-15b6b788-a7ea-4edf-96e4-1235dd1a5c26-000000@amazonses.com) Received: from a8-52.smtp-out.amazonses.com (a8-52.smtp-out.amazonses.com [54.240.8.52]) (using TLSv1 with cipher ECDHE-RSA-AES128-SHA (128/128 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 4422D6FB7A for ; Fri, 22 Dec 2017 09:44:53 +0000 (UTC) (envelope-from 010001607d9c59d5-15b6b788-a7ea-4edf-96e4-1235dd1a5c26-000000@amazonses.com) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/simple; s=ae7m2yrxjw65l2cqdpjxuucyrvy564tn; d=tarsnap.com; t=1513935887; h=To:From:Subject:Message-ID:Date:MIME-Version:Content-Type:Content-Transfer-Encoding; bh=/YPAoY8MQG5P6rzyJKe9Rx/FXVWhcUCDZNhkL2UWcbM=; b=eBEUTdCcVRSOuFyuVZjcNbg1IPRs0vEvAsrzpxHK46KzG7H0/bsXWgC94k/Co1Ll +dHKJ/xYf7vAlDN8INc24kigIwx/bJNLoXyxgsbjcrPSUnjw49zjVw6oaYDQIKgjnCM nmze433Sjh0xmJtP5MH4DCR6YjEO/FRtDUxatyPM= DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/simple; s=224i4yxa5dv7c2xz3womw6peuasteono; d=amazonses.com; t=1513935887; h=To:From:Subject:Message-ID:Date:MIME-Version:Content-Type:Content-Transfer-Encoding:Feedback-ID; bh=/YPAoY8MQG5P6rzyJKe9Rx/FXVWhcUCDZNhkL2UWcbM=; b=ICozTx241FX9zQmEFPjQ5gZPILU2UjboMBcVUCS7c6XN3moQ5+QWmFv66xEzVt+l ARh9UEjzhLAALpAJaDkPr9GAjFPcdElKW0NqegNxXzOXOW9D/iqrNR/jLH1EcTtDitJ 3f/UwH5iKlkD0s6BDXj6tsmnPHGw5Fn/kkh9T5xU= To: "freebsd-current@freebsd.org" From: Colin Percival Subject: [Request for review] Profiling the FreeBSD kernel boot Message-ID: <010001607d9c59d5-15b6b788-a7ea-4edf-96e4-1235dd1a5c26-000000@email.amazonses.com> Date: Fri, 22 Dec 2017 09:44:46 +0000 User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:45.0) Gecko/20100101 Thunderbird/45.5.1 MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 7bit X-SES-Outgoing: 2017.12.22-54.240.8.52 Feedback-ID: 1.us-east-1.Lv9FVjaNvvR5llaqfLoOVbo2VxOELl7cjN0AOyXnPlk=:AmazonSES X-Mailman-Approved-At: Fri, 22 Dec 2017 11:39:44 +0000 X-BeenThere: freebsd-current@freebsd.org X-Mailman-Version: 2.1.25 Precedence: list List-Id: Discussions about the use of FreeBSD-current List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 22 Dec 2017 09:44:54 -0000 Hi everyone, For the past few months I've been working on code for profiling the FreeBSD "kernel boot", i.e., everything between when kernel code starts running and when we first enter userland as init(8). This is not trivial since it's impossible to use tools like dtrace to monitor things prior to when said tools are running. The goal of this exercise is to help me track down the places where we're wasting time during the boot, and then to fix them. The approach I've taken is to add some macros -- most notably TSENTER() and TSEXIT() -- which by default compile to nothing, but if the TSLOG kernel option is enabled they compile to code which logs the cycle count (e.g., on x86 the value from the RDTSC instruction) along with some other data (in the case of TSENTER and TSEXIT, the fact that we're entering/exiting a function). This can then be dumped via a sysctl (debug.tslog) and processed in userland to convert function entries/exits into stacks and to visualize the time spent in the boot process. Two examples: A flame chart of my laptop booting HEAD: http://www.daemonology.net/timestamping/tslog-laptop.svg A flame chart of an EC2 c5.4xlarge instance booting 11.1-RELEASE: http://www.daemonology.net/timestamping/tslog-c5.4xlarge.svg The patches (10 of them, to be applied in order), userland scripts, and very brief usage instructions are at: http://www.daemonology.net/timestamping/tslog.tgz I hope to commit the patches in the next week, since I'm planning on writing a paper to submit to AsiaBSDCon (which has a deadline of December 31st); so if anyone has interest/time to look at this in the near future (I mean, it's not like anyone is going to be busy this weekend, right?) I'd love to have some feedback before it goes into the tree. -- Colin Percival Security Officer Emeritus, FreeBSD | The power to serve Founder, Tarsnap | www.tarsnap.com | Online backups for the truly paranoid