Date: Wed, 10 Nov 2021 12:26:50 -0400 From: Mitchell Horne <mhorne@freebsd.org> To: freebsd-arch@freebsd.org, freebsd-hackers@freebsd.org Cc: cperciva@freebsd.org Subject: A new boot-time trace framework Message-ID: <deee2c46-c3ed-3c0f-1d7a-3321b4fe9a7c@freebsd.org>
index | next in thread | raw e-mail
[-- Attachment #1 --] Hello, For some time I have been working towards upstreaming the 'boottrace' feature, which captures boot-time and shutdown-time information as trace entries. This feature was developed by NetApp, and they have been using it internally to great success for several years. Its purpose is to aid in logging and identifying slow portions of boot or shutdown, spanning from kernel initialization through execution of rc scripts. It is driven by a simple sysctl interface, described in the main review (D30184). Boottrace has some functional overlap with the existing TSLOG framework, in that it captures timestamped entries of notable boot events in a buffer, for later inspection. This overlap has grown somewhat recently, as cperciva@'s work on reducing the overall system boot time extended TSLOG to capture trace data from userspace (see commit 46dd801acb23). Boottrace differs from TSLOG in the following ways: - Separate trace buffers for collecting run-time (post multiuser) and shutdown-time events - Does not cover the bootloader or early kernel initialization (tracing begins at SI_SUB_CPU) - Output log is human-readable, but not as suitable for generating flamegraphs (see the attached sample log) - Trace entries also record some resource usage of the invoking process (total CPU time, # of blocks in/out from disk) Given these differences, I believe the two can peacefully coexist. I would roughly characterize them as follows: boottrace may be most useful to a sysadmin/QA team, whereas TSLOG is a tool more suited to the needs of a developer and will provide more fine-grained and machine-readable information. Unlike TSLOG, I intend for this work to be compiled in to the kernel by default, but disabled behind a tunable (kern.boottrace.enabled). The cost of doing so should be minimal, only a couple of syscalls added to init(8) at most. The changes span several areas, including the following: - Core boottrace module - Kernel initialization and shutdown paths - init(8), shutdown(8), and reboot(8) utilities - addition of a new boottrace(1) wrapper utility - RC framework (rc.subr) If you have an interest in any of the above areas, please add yourself to the review. If you have other feedback, I would be interested in hearing it here. The reviews: - https://reviews.freebsd.org/D30184 - https://reviews.freebsd.org/D30187 - https://reviews.freebsd.org/D31928 - https://reviews.freebsd.org/D31929 - https://reviews.freebsd.org/D31930 If you are interested in trying this feature, I have made it available on the following git branch: https://github.com/mhorne/freebsd/tree/netapp_boottrace Finally, I've attached a sample output from `sysctl kern.boottrace.log`, for a small bhyve VM. Cheers, Mitchell [-- Attachment #2 --] kern.boottrace.log: CPU msecs delta process event PID CPUtime IBlks OBlks 0 44872811 0 kernel sysinit 0x2100001 0 0.00 0 0 0 44872812 1 kernel sysinit 0x2110000 0 0.00 0 0 0 44872812 0 kernel sysinit 0x2140000 0 0.00 0 0 0 44872812 0 kernel sysinit 0x2160000 0 0.00 0 0 0 44872813 1 kernel sysinit 0x2180000 0 0.00 0 0 0 44872813 0 kernel sysinit 0x21d0000 0 0.00 0 0 0 44872813 0 kernel sysinit 0x21e0000 0 0.00 0 0 0 44872813 0 kernel sysinit 0x2200000 0 0.00 0 0 0 44872814 1 kernel sysinit 0x2300000 0 0.00 0 0 0 44872817 3 kernel sysinit 0x2380000 0 0.00 0 0 0 44872817 0 kernel sysinit 0x2400000 0 0.00 0 0 0 44872817 0 kernel sysinit 0x2480000 0 0.00 0 0 0 44872817 0 kernel sysinit 0x24c0000 0 0.00 0 0 0 44872817 0 kernel sysinit 0x2500000 0 0.00 0 0 0 44872817 0 kernel sysinit 0x2600000 0 0.00 0 0 0 44872817 0 kernel sysinit 0x2700000 0 0.00 0 0 0 44872817 0 kernel sysinit 0x2800000 0 0.00 0 0 0 44873820 1003 kernel sysinit 0x2880000 0 0.00 0 0 0 44873820 0 kernel sysinit 0x2888000 0 0.00 0 0 0 44873820 0 kernel sysinit 0x2900000 0 0.00 0 0 1 44873852 32 kernel sysinit 0x2900001 0 0.00 0 0 1 44873852 0 kernel sysinit 0x2a00000 0 0.00 0 0 1 44873852 0 kernel sysinit 0x2f00000 0 0.00 0 0 1 44873852 0 kernel sysinit 0x3000000 0 0.00 0 0 1 44873852 0 kernel sysinit 0x3100000 0 0.00 0 0 1 44873855 3 kernel sysinit 0x3800000 0 0.00 0 0 1 44875723 1868 kernel sysinit 0x4000000 0 0.00 0 0 1 44875724 1 kernel sysinit 0x4800000 0 0.00 0 0 1 44875726 2 kernel sysinit 0x6400000 0 0.00 0 0 1 44875726 0 kernel sysinit 0x6800000 0 0.00 0 0 1 44875726 0 kernel sysinit 0x6c00000 0 0.00 0 0 1 44875726 0 kernel sysinit 0x6e00000 0 0.00 0 0 1 44875726 0 kernel sysinit 0x7000000 0 0.00 0 0 1 44875727 1 kernel sysinit 0x7400000 0 0.00 0 0 1 44875727 0 kernel sysinit 0x8100000 0 0.00 0 0 1 44875727 0 kernel sysinit 0x8400000 0 0.00 0 0 1 44875727 0 kernel sysinit 0x8600000 0 0.00 0 0 1 44875727 0 kernel sysinit 0x8700000 0 0.00 0 0 1 44875727 0 kernel sysinit 0x8800000 0 0.00 0 0 1 44875728 1 kernel sysinit 0x8808000 0 0.00 0 0 1 44875728 0 kernel sysinit 0xa000000 0 0.00 0 0 1 44875730 2 kernel sysinit 0xa800000 0 0.00 0 0 1 44875732 2 kernel sysinit 0xaffffff 0 0.00 0 0 1 44875732 0 kernel sysinit 0xb000000 0 0.00 0 0 1 44875732 0 kernel sysinit 0xd000000 0 0.00 0 0 1 44875732 0 kernel sysinit 0xd800000 0 0.00 0 0 1 44875732 0 kernel sysinit 0xdc00000 0 0.00 0 0 1 44875732 0 kernel sysinit 0xdffff9c 0 0.00 0 0 1 44875732 0 kernel sysinit 0xe000000 0 0.00 0 0 1 44875732 0 kernel sysinit 0xe400000 0 0.00 0 0 1 44875732 0 kernel sysinit 0xe800000 0 0.00 0 0 1 44875732 0 kernel sysinit 0xea00000 0 0.00 0 0 1 44875735 3 kernel sysinit 0xec00000 0 0.00 0 0 1 44875735 0 kernel sysinit 0xee00000 0 0.00 0 0 1 44875735 0 kernel sysinit 0xf100000 0 0.00 0 0 1 44875735 0 kernel sysinit 0xfffffff 0 0.00 0 0 1 44875735 0 swapper mi_startup done 0 0.00 0 0 0 44875750 15 init init(8) starting... 1 0.00 0 0 0 44875751 1 init /etc/rc starting... 1 0.00 0 0 0 44875831 80 boottrace /etc/rc.d/rctl start 26 0.00 0 0 1 44875839 8 boottrace /etc/rc.d/rctl done 26 0.00 2 0 1 44875839 0 boottrace /etc/rc.d/dumpon start 31 0.00 0 0 0 44875861 22 boottrace /etc/rc.d/dumpon done 31 0.01 37 0 0 44875861 0 boottrace /etc/rc.d/sysctl start 40 0.00 0 0 0 44875878 17 boottrace /etc/rc.d/sysctl done 40 0.01 17 0 0 44875878 0 boottrace /etc/rc.d/hostid start 49 0.00 0 0 1 44875892 14 boottrace /etc/rc.d/hostid done 49 0.01 12 0 1 44875893 1 boottrace /etc/rc.d/ddb start 60 0.00 0 0 0 44875907 14 boottrace /etc/rc.d/ddb done 60 0.01 0 0 0 44875908 1 boottrace /etc/rc.d/gbde start 68 0.00 0 0 0 44875914 6 boottrace /etc/rc.d/gbde done 68 0.00 0 0 0 44875914 0 boottrace /etc/rc.d/ccd start 74 0.00 0 0 0 44875919 5 boottrace /etc/rc.d/ccd done 74 0.00 0 0 0 44875920 1 boottrace /etc/rc.d/geli start 79 0.00 0 0 0 44875926 6 boottrace /etc/rc.d/geli done 79 0.00 0 0 0 44875926 0 boottrace /etc/rc.d/zpool start 85 0.00 0 0 0 44875931 5 boottrace /etc/rc.d/zpool done 85 0.00 0 0 0 44875932 1 boottrace /etc/rc.d/swap start 90 0.00 0 0 0 44875939 7 boottrace /etc/rc.d/swap done 90 0.00 1 0 0 44875939 0 boottrace /etc/rc.d/fsck start 96 0.00 0 0 0 44875944 5 boottrace /etc/rc.d/fsck done 96 0.00 0 0 0 44875945 1 boottrace /etc/rc.d/zvol start 101 0.00 0 0 0 44875950 5 boottrace /etc/rc.d/zvol done 101 0.00 0 0 0 44875951 1 boottrace /etc/rc.d/root start 106 0.00 0 0 1 44875964 13 boottrace /etc/rc.d/root done 106 0.01 7 2 1 44875965 1 boottrace /etc/rc.d/sppp start 115 0.00 0 0 1 44875973 8 boottrace /etc/rc.d/sppp done 115 0.00 0 0 1 44875974 1 boottrace /etc/rc.d/mdconfig start 120 0.00 0 0 0 44875985 11 boottrace /etc/rc.d/mdconfig done 120 0.01 0 0 0 44875986 1 boottrace /etc/rc.d/hostid_save start 128 0.00 0 0 1 44875992 6 boottrace /etc/rc.d/hostid_save done 128 0.00 0 0 1 44875993 1 boottrace /etc/rc.d/serial start 134 0.00 0 0 1 44875994 1 boottrace /etc/rc.d/serial done 134 0.00 0 0 1 44875995 1 boottrace /etc/rc.d/mountcritlocal start 136 0.00 0 0 1 44876008 13 boottrace /etc/rc.d/mountcritlocal done 136 0.00 67 1 1 44876009 1 boottrace /etc/rc.d/zfsbe start 143 0.00 0 0 1 44876014 5 boottrace /etc/rc.d/zfsbe done 143 0.00 0 0 1 44876015 1 boottrace /etc/rc.d/tmp start 148 0.00 0 0 0 44876024 9 boottrace /etc/rc.d/tmp done 148 0.00 5 0 0 44876025 1 boottrace /etc/rc.d/gssd start 158 0.00 0 0 1 44876039 14 boottrace /etc/rc.d/gssd done 158 0.01 0 0 1 44876040 1 boottrace /etc/rc.d/kldxref start 166 0.00 0 0 1 44876047 7 boottrace /etc/rc.d/kldxref done 166 0.00 5 0 1 44876048 1 boottrace /etc/rc.d/zfs start 172 0.00 0 0 1 44876054 6 boottrace /etc/rc.d/zfs done 172 0.00 0 0 1 44876054 0 boottrace /etc/rc.d/kld start 177 0.00 0 0 1 44876060 6 boottrace /etc/rc.d/kld done 177 0.00 0 0 1 44876061 1 boottrace /etc/rc.d/sysvipc start 182 0.00 0 0 1 44876066 5 boottrace /etc/rc.d/sysvipc done 182 0.00 0 0 1 44876067 1 boottrace /etc/rc.d/linux start 187 0.00 0 0 1 44876072 5 boottrace /etc/rc.d/linux done 187 0.00 0 0 1 44876073 1 boottrace /etc/rc.d/var start 192 0.00 0 0 1 44876081 8 boottrace /etc/rc.d/var done 192 0.00 1 0 1 44876081 0 boottrace /etc/rc.d/devmatch start 199 0.00 0 0 0 44876091 10 boottrace /etc/rc.d/devmatch done 199 0.00 9 0 0 44876092 1 boottrace /etc/rc.d/localpkg start 207 0.00 0 0 0 44876103 11 boottrace /etc/rc.d/localpkg done 207 0.00 7 0 0 44876104 1 boottrace /etc/rc.d/cleanvar start 214 0.00 0 0 0 44876117 13 boottrace /etc/rc.d/cleanvar done 214 0.00 15 0 0 44876118 1 boottrace /etc/rc.d/cfumass start 222 0.00 0 0 0 44876124 6 boottrace /etc/rc.d/cfumass done 222 0.00 0 0 0 44876125 1 boottrace /etc/rc.d/FILESYSTEMS start 227 0.00 0 0 0 44876126 1 boottrace /etc/rc.d/FILESYSTEMS done 227 0.00 0 0 0 44876133 7 boottrace /etc/rc.d/ippool start 231 0.00 0 0 1 44876144 11 boottrace /etc/rc.d/ippool done 231 0.01 0 0 1 44876145 1 boottrace /etc/rc.d/ldconfig start 239 0.00 0 0 0 44876163 18 boottrace /etc/rc.d/ldconfig done 239 0.01 17 0 0 44876163 0 boottrace /etc/rc.d/adjkerntz start 251 0.00 0 0 0 44876170 7 boottrace /etc/rc.d/adjkerntz done 251 0.00 1 0 0 44876171 1 boottrace /etc/rc.d/hostname start 257 0.00 0 0 0 44876183 12 boottrace /etc/rc.d/hostname done 257 0.01 3 0 0 44876184 1 boottrace /etc/rc.d/ip6addrctl start 265 0.00 0 0 0 44876199 15 boottrace /etc/rc.d/ip6addrctl done 265 0.01 3 0 0 44876200 1 boottrace /etc/rc.d/netoptions start 275 0.00 0 0 0 44876216 16 boottrace /etc/rc.d/netoptions done 275 0.01 0 0 0 44876217 1 boottrace /etc/rc.d/opensm start 290 0.00 0 0 1 44876228 11 boottrace /etc/rc.d/opensm done 290 0.01 0 0 1 44876228 0 boottrace /etc/rc.d/random start 298 0.00 0 0 1 44876356 128 boottrace /etc/rc.d/random done 298 0.02 7 9 1 44876356 0 boottrace /etc/rc.d/iovctl start 319 0.00 0 0 0 44876366 10 boottrace /etc/rc.d/iovctl done 319 0.01 0 0 0 44876367 1 boottrace /etc/rc.d/ipsec start 327 0.00 0 0 1 44876377 10 boottrace /etc/rc.d/ipsec done 327 0.01 0 0 1 44876378 1 boottrace /etc/rc.d/mixer start 335 0.00 0 0 0 44876385 7 boottrace /etc/rc.d/mixer done 335 0.00 1 0 0 44876386 1 boottrace /etc/rc.d/ugidfw start 341 0.00 0 0 0 44876391 5 boottrace /etc/rc.d/ugidfw done 341 0.00 0 0 0 44876392 1 boottrace /etc/rc.d/geli2 start 346 0.00 0 0 0 44876397 5 boottrace /etc/rc.d/geli2 done 346 0.00 0 0 0 44876398 1 boottrace /etc/rc.d/autounmountd start 352 0.00 0 0 0 44876404 6 boottrace /etc/rc.d/autounmountd done 352 0.00 0 0 0 44876404 0 boottrace /etc/rc.d/ipfilter start 358 0.00 0 0 1 44876414 10 boottrace /etc/rc.d/ipfilter done 358 0.01 0 0 1 44876415 1 boottrace /etc/rc.d/ipnat start 366 0.00 0 0 0 44876424 9 boottrace /etc/rc.d/ipnat done 366 0.01 0 0 0 44876425 1 boottrace /etc/rc.d/ipmon start 374 0.00 0 0 1 44876435 10 boottrace /etc/rc.d/ipmon done 374 0.01 0 0 1 44876436 1 boottrace /etc/rc.d/ipfs start 382 0.00 0 0 0 44876446 10 boottrace /etc/rc.d/ipfs done 382 0.01 0 0 0 44876446 0 boottrace /etc/rc.d/netif start 390 0.00 0 0 1 44881116 4670 boottrace /etc/rc.d/netif done 390 0.12 34 0 1 44881117 1 boottrace /etc/rc.d/pflog start 564 0.00 0 0 1 44881123 6 boottrace /etc/rc.d/pflog done 564 0.00 0 0 1 44881124 1 boottrace /etc/rc.d/rtsold start 570 0.00 0 0 1 44881131 7 boottrace /etc/rc.d/rtsold done 570 0.00 0 0 1 44881133 2 boottrace /etc/rc.d/static_ndp start 576 0.00 0 0 1 44881140 7 boottrace /etc/rc.d/static_ndp done 576 0.00 0 0 1 44881141 1 boottrace /etc/rc.d/static_arp start 581 0.00 0 0 1 44881148 7 boottrace /etc/rc.d/static_arp done 581 0.00 0 0 1 44881149 1 boottrace /etc/rc.d/stf start 586 0.00 0 0 1 44881156 7 boottrace /etc/rc.d/stf done 586 0.00 0 0 1 44881156 0 boottrace /etc/rc.d/resolv start 591 0.00 0 0 1 44881163 7 boottrace /etc/rc.d/resolv done 591 0.00 0 0 1 44881164 1 boottrace /etc/rc.d/devd start 597 0.00 0 0 1 44881539 375 boottrace /etc/rc.d/devd done 597 0.22 68 3 1 44881540 1 boottrace /etc/rc.d/ppp start 813 0.00 0 0 1 44881554 14 boottrace /etc/rc.d/ppp done 813 0.01 1 0 1 44881555 1 boottrace /etc/rc.d/pfsync start 822 0.00 0 0 1 44881562 7 boottrace /etc/rc.d/pfsync done 822 0.00 0 0 1 44881563 1 boottrace /etc/rc.d/routing start 827 0.00 0 0 1 44881606 43 boottrace /etc/rc.d/routing done 827 0.04 1 0 1 44881607 1 boottrace /etc/rc.d/defaultroute start 867 0.00 0 0 1 44881621 14 boottrace /etc/rc.d/defaultroute done 867 0.01 0 0 1 44881621 0 boottrace /etc/rc.d/ipfw start 881 0.00 0 0 1 44881630 9 boottrace /etc/rc.d/ipfw done 881 0.00 0 0 1 44881630 0 boottrace /etc/rc.d/zfsd start 886 0.00 0 0 0 44881641 11 boottrace /etc/rc.d/zfsd done 886 0.01 0 0 0 44881641 0 boottrace /etc/rc.d/bridge start 894 0.00 0 0 0 44881649 8 boottrace /etc/rc.d/bridge done 894 0.00 0 0 0 44881649 0 boottrace /etc/rc.d/ipfw_netflow start 899 0.00 0 0 0 44881656 7 boottrace /etc/rc.d/ipfw_netflow done 899 0.00 0 0 0 44881657 1 boottrace /etc/rc.d/pf start 904 0.00 0 0 1 44881670 13 boottrace /etc/rc.d/pf done 904 0.01 0 0 1 44881670 0 boottrace /etc/rc.d/routed start 912 0.00 0 0 0 44881682 12 boottrace /etc/rc.d/routed done 912 0.01 0 0 0 44881683 1 boottrace /etc/rc.d/route6d start 920 0.00 0 0 1 44881695 12 boottrace /etc/rc.d/route6d done 920 0.01 0 0 1 44881696 1 boottrace /etc/rc.d/netwait start 928 0.00 0 0 1 44881702 6 boottrace /etc/rc.d/netwait done 928 0.00 0 0 1 44881702 0 boottrace /etc/rc.d/blacklistd start 933 0.00 0 0 0 44881718 16 boottrace /etc/rc.d/blacklistd done 933 0.01 0 0 0 44881719 1 boottrace /etc/rc.d/local_unbound start 941 0.00 0 0 0 44881725 6 boottrace /etc/rc.d/local_unbound done 941 0.00 0 0 0 44881726 1 boottrace /etc/rc.d/NETWORKING start 947 0.00 0 0 1 44881727 1 boottrace /etc/rc.d/NETWORKING done 947 0.00 0 0 1 44881728 1 boottrace /etc/rc.d/nfsuserd start 949 0.00 0 0 0 44881738 10 boottrace /etc/rc.d/nfsuserd done 949 0.01 0 0 0 44881739 1 boottrace /etc/rc.d/tlsclntd start 957 0.00 0 0 0 44881747 8 boottrace /etc/rc.d/tlsclntd done 957 0.00 0 0 0 44881748 1 boottrace /etc/rc.d/kdc start 963 0.00 0 0 1 44881758 10 boottrace /etc/rc.d/kdc done 963 0.01 0 0 1 44881758 0 boottrace /etc/rc.d/ctld start 971 0.00 0 0 1 44881764 6 boottrace /etc/rc.d/ctld done 971 0.00 0 0 1 44881765 1 boottrace /etc/rc.d/iscsid start 977 0.00 0 0 1 44881771 6 boottrace /etc/rc.d/iscsid done 977 0.00 0 0 1 44881771 0 boottrace /etc/rc.d/tlsservd start 983 0.00 0 0 1 44881777 6 boottrace /etc/rc.d/tlsservd done 983 0.00 0 0 1 44881778 1 boottrace /etc/rc.d/pppoed start 989 0.00 0 0 0 44881783 5 boottrace /etc/rc.d/pppoed done 989 0.00 0 0 0 44881784 1 boottrace /etc/rc.d/kfd start 994 0.00 0 0 1 44881793 9 boottrace /etc/rc.d/kfd done 994 0.01 0 0 1 44881794 1 boottrace /etc/rc.d/kadmind start 1002 0.00 0 0 0 44881803 9 boottrace /etc/rc.d/kadmind done 1002 0.00 0 0 0 44881804 1 boottrace /etc/rc.d/kpasswdd start 1010 0.00 0 0 1 44881814 10 boottrace /etc/rc.d/kpasswdd done 1010 0.00 0 0 1 44881814 0 boottrace /etc/rc.d/iscsictl start 1018 0.00 0 0 0 44881824 10 boottrace /etc/rc.d/iscsictl done 1018 0.01 0 0 0 44881824 0 boottrace /etc/rc.d/nfscbd start 1026 0.00 0 0 1 44881834 10 boottrace /etc/rc.d/nfscbd done 1026 0.00 0 0 1 44881834 0 boottrace /etc/rc.d/ipropd_master start 1034 0.00 0 0 0 44881844 10 boottrace /etc/rc.d/ipropd_master done 1034 0.01 0 0 0 44881844 0 boottrace /etc/rc.d/ipropd_slave start 1042 0.00 0 0 1 44881854 10 boottrace /etc/rc.d/ipropd_slave done 1042 0.00 0 0 1 44881854 0 boottrace /etc/rc.d/mountcritremote start 1050 0.00 0 0 1 44881862 8 boottrace /etc/rc.d/mountcritremote done 1050 0.00 0 0 1 44881863 1 boottrace /etc/rc.d/accounting start 1058 0.00 0 0 1 44881868 5 boottrace /etc/rc.d/accounting done 1058 0.00 0 0 1 44881869 1 boottrace /etc/rc.d/virecover start 1063 0.00 0 0 0 44881877 8 boottrace /etc/rc.d/virecover done 1063 0.00 2 0 0 44881878 1 boottrace /etc/rc.d/cleartmp start 1069 0.00 0 0 0 44881886 8 boottrace /etc/rc.d/cleartmp done 1069 0.00 4 0 0 44881886 0 boottrace /etc/rc.d/dmesg start 1076 0.00 0 0 0 44881895 9 boottrace /etc/rc.d/dmesg done 1076 0.00 1 0 0 44881896 1 boottrace /etc/rc.d/devfs start 1083 0.00 0 0 1 44881947 51 boottrace /etc/rc.d/devfs done 1083 0.04 5 0 1 44881947 0 boottrace /etc/rc.d/os-release start 1157 0.00 0 0 0 44881961 14 boottrace /etc/rc.d/os-release done 1157 0.01 2 0 0 44881962 1 boottrace /etc/rc.d/motd start 1167 0.00 0 0 0 44881975 13 boottrace /etc/rc.d/motd done 1167 0.01 2 0 0 44881975 0 boottrace /etc/rc.d/newsyslog start 1178 0.00 0 0 0 44881996 21 boottrace /etc/rc.d/newsyslog done 1178 0.02 12 0 0 44881997 1 boottrace /etc/rc.d/gptboot start 1187 0.00 0 0 0 44882012 15 boottrace /etc/rc.d/gptboot done 1187 0.01 7 0 0 44882012 0 boottrace /etc/rc.d/hostapd start 1201 0.00 0 0 0 44882018 6 boottrace /etc/rc.d/hostapd done 1201 0.00 0 0 0 44882019 1 boottrace /etc/rc.d/mdconfig2 start 1207 0.00 0 0 1 44882027 8 boottrace /etc/rc.d/mdconfig2 done 1207 0.00 0 0 1 44882028 1 boottrace /etc/rc.d/syslogd start 1215 0.00 0 0 0 44882053 25 boottrace /etc/rc.d/syslogd done 1215 0.01 5 0 0 44882053 0 boottrace /etc/rc.d/ntpdate start 1232 0.00 0 0 1 44882064 11 boottrace /etc/rc.d/ntpdate done 1232 0.01 0 0 1 44882065 1 boottrace /etc/rc.d/auditd start 1240 0.00 0 0 1 44882071 6 boottrace /etc/rc.d/auditd done 1240 0.00 0 0 1 44882071 0 boottrace /etc/rc.d/bsnmpd start 1246 0.00 0 0 1 44882079 8 boottrace /etc/rc.d/bsnmpd done 1246 0.00 0 0 1 44882080 1 boottrace /etc/rc.d/savecore start 1252 0.00 0 0 1 44882102 22 boottrace /etc/rc.d/savecore done 1252 0.01 14 0 1 44882103 1 boottrace /etc/rc.d/watchdogd start 1266 0.00 0 0 1 44882111 8 boottrace /etc/rc.d/watchdogd done 1266 0.00 0 0 1 44882112 1 boottrace /etc/rc.d/pwcheck start 1272 0.00 0 0 1 44882117 5 boottrace /etc/rc.d/pwcheck done 1272 0.00 0 0 1 44882118 1 boottrace /etc/rc.d/SERVERS start 1277 0.00 0 0 1 44882119 1 boottrace /etc/rc.d/SERVERS done 1277 0.00 0 0 1 44882120 1 boottrace /etc/rc.d/auditdistd start 1279 0.00 0 0 1 44882126 6 boottrace /etc/rc.d/auditdistd done 1279 0.00 0 0 1 44882126 0 boottrace /etc/rc.d/rpcbind start 1285 0.00 0 0 0 44882137 11 boottrace /etc/rc.d/rpcbind done 1285 0.01 0 0 0 44882138 1 boottrace /usr/local/etc/rc.d/tpmd start 1293 0.00 0 0 1 44882147 9 boottrace /usr/local/etc/rc.d/tpmd done 1293 0.01 0 0 1 44882148 1 boottrace /etc/rc.d/nisdomain start 1301 0.00 0 0 1 44882154 6 boottrace /etc/rc.d/nisdomain done 1301 0.00 0 0 1 44882154 0 boottrace /etc/rc.d/nfsclient start 1306 0.00 0 0 1 44882160 6 boottrace /etc/rc.d/nfsclient done 1306 0.00 0 0 1 44882161 1 boottrace /etc/rc.d/ypserv start 1311 0.00 0 0 0 44882170 9 boottrace /etc/rc.d/ypserv done 1311 0.01 0 0 0 44882171 1 boottrace /usr/local/etc/rc.d/tcsd start 1319 0.00 0 0 1 44882181 10 boottrace /usr/local/etc/rc.d/tcsd done 1319 0.01 0 0 1 44882182 1 boottrace /etc/rc.d/ypxfrd start 1327 0.00 0 0 0 44882191 9 boottrace /etc/rc.d/ypxfrd done 1327 0.00 0 0 0 44882193 2 boottrace /etc/rc.d/ypldap start 1335 0.00 0 0 1 44882207 14 boottrace /etc/rc.d/ypldap done 1335 0.01 0 0 1 44882208 1 boottrace /etc/rc.d/ypbind start 1343 0.00 0 0 0 44882218 10 boottrace /etc/rc.d/ypbind done 1343 0.01 0 0 0 44882218 0 boottrace /etc/rc.d/ypupdated start 1351 0.00 0 0 1 44882228 10 boottrace /etc/rc.d/ypupdated done 1351 0.01 0 0 1 44882229 1 boottrace /etc/rc.d/hastd start 1359 0.00 0 0 1 44882235 6 boottrace /etc/rc.d/hastd done 1359 0.00 0 0 1 44882236 1 boottrace /etc/rc.d/ypset start 1365 0.00 0 0 0 44882245 9 boottrace /etc/rc.d/ypset done 1365 0.01 0 0 0 44882246 1 boottrace /etc/rc.d/yppasswdd start 1373 0.00 0 0 1 44882258 12 boottrace /etc/rc.d/yppasswdd done 1373 0.01 0 0 1 44882258 0 boottrace /etc/rc.d/quota start 1381 0.00 0 0 1 44882265 7 boottrace /etc/rc.d/quota done 1381 0.00 0 0 1 44882266 1 boottrace /etc/rc.d/keyserv start 1386 0.00 0 0 0 44882276 10 boottrace /etc/rc.d/keyserv done 1386 0.01 0 0 0 44882277 1 boottrace /etc/rc.d/automountd start 1394 0.00 0 0 1 44882283 6 boottrace /etc/rc.d/automountd done 1394 0.00 0 0 1 44882284 1 boottrace /etc/rc.d/automount start 1400 0.00 0 0 1 44882289 5 boottrace /etc/rc.d/automount done 1400 0.00 0 0 1 44882290 1 boottrace /etc/rc.d/mountd start 1405 0.00 0 0 1 44882296 6 boottrace /etc/rc.d/mountd done 1405 0.00 0 0 1 44882297 1 boottrace /etc/rc.d/nfsd start 1411 0.00 0 0 0 44882307 10 boottrace /etc/rc.d/nfsd done 1411 0.01 0 0 0 44882308 1 boottrace /etc/rc.d/statd start 1419 0.00 0 0 1 44882321 13 boottrace /etc/rc.d/statd done 1419 0.01 0 0 1 44882322 1 boottrace /etc/rc.d/lockd start 1427 0.00 0 0 0 44882336 14 boottrace /etc/rc.d/lockd done 1427 0.01 0 0 0 44882337 1 boottrace /etc/rc.d/DAEMON start 1435 0.00 0 0 0 44882338 1 boottrace /etc/rc.d/DAEMON done 1435 0.00 0 0 0 44882339 1 boottrace /etc/rc.d/rtadvd start 1437 0.00 0 0 1 44882354 15 boottrace /etc/rc.d/rtadvd done 1437 0.01 0 0 1 44882355 1 boottrace /etc/rc.d/rwho start 1445 0.00 0 0 0 44882365 10 boottrace /etc/rc.d/rwho done 1445 0.01 0 0 0 44882366 1 boottrace /etc/rc.d/rarpd start 1453 0.00 0 0 0 44882372 6 boottrace /etc/rc.d/rarpd done 1453 0.00 0 0 0 44882373 1 boottrace /etc/rc.d/apm start 1459 0.00 0 0 1 44882383 10 boottrace /etc/rc.d/apm done 1459 0.01 0 0 1 44882384 1 boottrace /etc/rc.d/ubthidhci start 1467 0.00 0 0 0 44882394 10 boottrace /etc/rc.d/ubthidhci done 1467 0.01 0 0 0 44882395 1 boottrace /etc/rc.d/utx start 1475 0.00 0 0 0 44882402 7 boottrace /etc/rc.d/utx done 1475 0.00 2 0 0 44882403 1 boottrace /etc/rc.d/sdpd start 1481 0.00 0 0 1 44882413 10 boottrace /etc/rc.d/sdpd done 1481 0.01 0 0 1 44882414 1 boottrace /etc/rc.d/local start 1489 0.00 0 0 1 44882419 5 boottrace /etc/rc.d/local done 1489 0.00 0 0 1 44882420 1 boottrace /etc/rc.d/mountlate start 1494 0.00 0 0 0 44882429 9 boottrace /etc/rc.d/mountlate done 1494 0.00 0 0 0 44882429 0 boottrace /etc/rc.d/nscd start 1502 0.00 0 0 1 44882442 13 boottrace /etc/rc.d/nscd done 1502 0.01 0 0 1 44882444 2 boottrace /etc/rc.d/ntpd start 1516 0.00 0 0 0 44882450 6 boottrace /etc/rc.d/ntpd done 1516 0.00 1 0 0 44882451 1 boottrace /etc/rc.d/ftp-proxy start 1522 0.00 0 0 1 44882462 11 boottrace /etc/rc.d/ftp-proxy done 1522 0.01 0 0 1 44882462 0 boottrace /etc/rc.d/bootparams start 1530 0.00 0 0 0 44882473 11 boottrace /etc/rc.d/bootparams done 1530 0.01 0 0 0 44882473 0 boottrace /etc/rc.d/hcsecd start 1538 0.00 0 0 0 44882480 7 boottrace /etc/rc.d/hcsecd done 1538 0.00 0 0 0 44882481 1 boottrace /etc/rc.d/powerd start 1544 0.00 0 0 1 44882491 10 boottrace /etc/rc.d/powerd done 1544 0.01 0 0 1 44882491 0 boottrace /etc/rc.d/lpd start 1552 0.00 0 0 0 44882501 10 boottrace /etc/rc.d/lpd done 1552 0.01 0 0 0 44882502 1 boottrace /etc/rc.d/moused start 1560 0.00 0 0 0 44882508 6 boottrace /etc/rc.d/moused done 1560 0.00 0 0 0 44882509 1 boottrace /etc/rc.d/bthidd start 1566 0.00 0 0 0 44882516 7 boottrace /etc/rc.d/bthidd done 1566 0.00 0 0 0 44882517 1 boottrace /etc/rc.d/rfcomm_pppd_server start 1573 0.00 0 0 1 44882527 10 boottrace /etc/rc.d/rfcomm_pppd_server done 1573 0.01 0 0 1 44882528 1 boottrace /etc/rc.d/swaplate start 1581 0.00 0 0 1 44882535 7 boottrace /etc/rc.d/swaplate done 1581 0.00 0 0 1 44882535 0 boottrace /etc/rc.d/LOGIN start 1587 0.00 0 0 1 44882537 2 boottrace /etc/rc.d/LOGIN done 1587 0.00 0 0 1 44882537 0 boottrace /etc/rc.d/msgs start 1589 0.00 0 0 1 44882543 6 boottrace /etc/rc.d/msgs done 1589 0.00 1 0 1 44882544 1 boottrace /etc/rc.d/sysctl_lastload start 1594 0.00 0 0 0 44882562 18 boottrace /etc/rc.d/sysctl_lastload done 1594 0.02 0 0 0 44882563 1 boottrace /etc/rc.d/syscons start 1607 0.00 0 0 0 44882573 10 boottrace /etc/rc.d/syscons done 1607 0.00 1 0 0 44882573 0 boottrace /etc/rc.d/sshd start 1614 0.00 0 0 0 44882580 7 boottrace /etc/rc.d/sshd done 1614 0.00 0 0 0 44882581 1 boottrace /etc/rc.d/sendmail start 1620 0.00 0 0 0 44882802 221 boottrace /etc/rc.d/sendmail done 1620 0.02 56 0 0 44882803 1 boottrace /etc/rc.d/cron start 1634 0.00 0 0 1 44882816 13 boottrace /etc/rc.d/cron done 1634 0.00 6 0 1 44882817 1 boottrace /etc/rc.d/othermta start 1642 0.00 0 0 0 44882822 5 boottrace /etc/rc.d/othermta done 1642 0.00 0 0 0 44882823 1 boottrace /etc/rc.d/inetd start 1647 0.00 0 0 0 44882831 8 boottrace /etc/rc.d/inetd done 1647 0.00 0 0 0 44882832 1 boottrace /etc/rc.d/ftpd start 1653 0.00 0 0 0 44882837 5 boottrace /etc/rc.d/ftpd done 1653 0.00 0 0 0 44882838 1 boottrace /etc/rc.d/bgfsck start 1659 0.00 0 0 0 44882848 10 boottrace /etc/rc.d/bgfsck done 1659 0.00 0 0 1 44882849 1 boottrace /etc/rc.d/jail start 1668 0.00 0 0 0 44882865 16 boottrace /etc/rc.d/jail done 1668 0.01 0 0 0 44882866 1 boottrace /etc/rc.d/securelevel start 1679 0.00 0 0 0 44882872 6 boottrace /etc/rc.d/securelevel done 1679 0.00 0 0 1 44882879 7 init /etc/rc finished 1 2.22 743 15 Total measured time: 10068 msecs CPU msecs delta process event PID CPUtime IBlks OBlks 1 44882880 0 init multi-user start 1 2.22 743 15 0 44918215 35335 kldload hwpmc.ko: sysinit 0xd800000 1698 0.00 0 0 Total measured time: 35335 msecshelp
Want to link to this message? Use this
URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?deee2c46-c3ed-3c0f-1d7a-3321b4fe9a7c>