Skip site navigation (1)Skip section navigation (2)
Date:      Thu, 10 Jul 2014 18:58:47 +0530
From:      Kashyap Desai <kashyap.desai@avagotech.com>
To:        Alexander Motin <mav@freebsd.org>
Cc:        FreeBSD-scsi <freebsd-scsi@freebsd.org>
Subject:   RE: SSDs peformance on head/freebsd-10 stable using FIO
Message-ID:  <9f138f242e278476e5c542d695e58bc8@mail.gmail.com>
In-Reply-To: <53BE8784.8060503@FreeBSD.org>
References:  <8fbe38cdad1e66717a9de7fdf63812c2@mail.gmail.com> <53BE8784.8060503@FreeBSD.org>

next in thread | previous in thread | raw e-mail | index | archive | help

[-- Attachment #1 --]
> -----Original Message-----
> From: Alexander Motin [mailto:mavbsd@gmail.com] On Behalf Of Alexander
> Motin
> Sent: Thursday, July 10, 2014 6:01 PM
> To: Kashyap Desai
> Cc: FreeBSD-scsi
> Subject: Re: SSDs peformance on head/freebsd-10 stable using FIO
>
> Hi, Kashyap.
>
> On 10.07.2014 15:00, Kashyap Desai wrote:
> > I am trying to collect IOPs and throughput using FIO on
> > FreeBSD-10-stable as below post mentioned that CAM can reach upto
> > 1,000,000 IOPS using Fine-Grained CAM locking.
> >
> > http://www.freebsd.org/news/status/report-2013-07-2013-
> 09.html#GEOM-Di
> > rect-Dispatch-and-Fine-Grained-CAM-Locking
> >
> > I am using below FIO parameter.
> >
> > [global]
> > ioengine=posixaio
> > buffered=0
> > rw=randread
> > bs=4K
> > iodepth=32
> > numjobs=2
> > direct=1
> > runtime=60s
> > thread
> > group_reporting=1
> > [job1]
> > filename=/dev/da0
> > [job2]
> > filename=/dev/da1
> > [job3]
> > filename=/dev/da2
> > [job4]
> > filename=/dev/da3
> > [job4]
> > filename=/dev/da4
> > ..
> >
> > I have 8 SSDs in my setup and all 8 SSDs are behind LSI’s 12Gp/s
> > MegaRaid Controller as JBOD. I also found FIO can be used in Async
> > mode after loading “aio” kernel module.
> >
> > Using single SSD, I am able to see  110K-130K IOPs.  This IOPs counts
> > are matching with what I see on Linux machine.
> >
> > Now, I am not able to scale IOPs on my machine after 200K.  I see CPU
> > is almost occupied and no idle time after IOPs reach to 200K.
> >
> > If you have any pointers to try with,  I can do some experiment on my
> setup.
>
> Getting such results I would immediately start doing profiling with
> pmcstat.
> Quite likely you are hitting some new lock congestion. Start with simple
> `pmcstat -n 100000000 -TS unhalted-cycles`. It it hard to say for sure
> what
> went wrong there without more data, so just couple
I have attached profile output for the command mentioned above. I will dig
further and see if this is what we have theoretical limit for CAM attached
HBA.
I am trying to isolate tools, tuning or Driver/FW issue at very first level.

> thoughts:
>
> First of all, I've never tried aio in my benchmarks, only synchronous
> ones. Try
> to run 8 instances of `dd if=/dev/daX of=/dev/null bs=512` per each SSD
> same time, just as I did. You may vary number of dd's, but keep total
> below
> 256, or you mad to increase nswbuf limit in kern_vfs_bio_buffer_alloc().

I ran multiple dd instance also and seeing IOPs throttle somewhere ~200K .

Do we have any mechanism to check CAM layer's max IOPs support without
involving actual Device ? Something like _null_ device driver which just
send the command back to CAM layer ?

>
> For second, you are using single HBA, that should create significant
> congestion around its CAM SIM lock.  Proper solution would be to add
> multiple queues support to the driver, and we discussed it with Scott Long
> for quite some time, but that requires more work (I hope you may be
> interested in it ;) ). Or you may just insert 3-4 HBAs. My million IOPS I
> was
> reaching with four 2008/2308 6Gbps HBAs and 16 SATA SSDs.

I remember this part and really good to contribute for this work.  As part
of this we have initiated multiple MSIx implementation in <mrsas>, which
will have multiple reply queue per MSI-x.

Do we really require to have multiple Submission queue at low level driver ?
I thought it will be a CAM interface for multi queue which _all_ low level
drivers need to hook into .

I just started gathering  performance numbers on FreeBSD with more SSDs on
LSI's 12Gbp/s card, so just getting familiar with tools and tunings.
Earlier I used just one SSDs and some HDDs so no issue found w.r.t IOPs
etc.. As part of this activity, I am trying to see if mrsas driver is able
to meet expected performance without considering any other component as
bottleneck.
I mean, are we able to meet IOPs which CAM layer can handle.

Kashyap

>
> Please tell me what you get, so we could continue investigation.
>
> --
> Alexander Motin

[-- Attachment #2 --]
@ CPU_CLK_UNHALTED_CORE [291 samples]

12.71%  [37]       cpu_idle @ /boot/kernel/kernel
 100.0%  [37]        sched_idletd
  100.0%  [37]         fork_exit

09.97%  [29]       Xinvlrng @ /boot/kernel/kernel

06.87%  [20]       invlrng_handler @ /boot/kernel/kernel

05.15%  [15]       smp_tlb_shootdown @ /boot/kernel/kernel
 100.0%  [15]        pmap_invalidate_range
  100.0%  [15]         biodone
   100.0%  [15]          dadone

04.47%  [13]       cpu_search_highest @ /boot/kernel/kernel
 84.62%  [11]        cpu_search_highest
  54.55%  [6]          sched_idletd
   100.0%  [6]           fork_exit
  45.45%  [5]          cpu_search_highest
   100.0%  [5]           sched_idletd
 15.38%  [2]         sched_idletd
  100.0%  [2]          fork_exit

04.12%  [12]       __mtx_lock_sleep @ /boot/kernel/kernel
 75.00%  [9]         __mtx_lock_flags
  100.0%  [9]          mrsas_cmd_done @ /boot/kernel/mrsas.ko
   100.0%  [9]           mrsas_complete_cmd
 08.33%  [1]         xpt_run_devq @ /boot/kernel/kernel
  100.0%  [1]          xpt_action_default
   100.0%  [1]           dastart
 08.33%  [1]         dastart
  100.0%  [1]          xpt_run_allocq
   100.0%  [1]           dastrategy
 08.33%  [1]         vmem_xalloc
  100.0%  [1]          vmem_alloc
   100.0%  [1]           g_io_check

02.41%  [7]        cpu_search_lowest @ /boot/kernel/kernel
 100.0%  [7]         cpu_search_lowest
  71.43%  [5]          cpu_search_lowest
   100.0%  [5]           sched_pickcpu
  28.57%  [2]          sched_pickcpu
   100.0%  [2]           sched_add

02.41%  [7]        cpu_switch @ /boot/kernel/kernel
 100.0%  [7]         mi_switch
  71.43%  [5]          sched_idletd
   100.0%  [5]           fork_exit
  14.29%  [1]          critical_exit
   100.0%  [1]           intr_event_handle
  14.29%  [1]          ithread_loop
   100.0%  [1]           fork_exit

01.72%  [5]        sched_idletd @ /boot/kernel/kernel
 100.0%  [5]         fork_exit

01.72%  [5]        _mtx_lock_spin_cookie @ /boot/kernel/kernel
 40.00%  [2]         turnstile_trywait
  100.0%  [2]          __mtx_lock_sleep
   100.0%  [2]           __mtx_lock_flags
 20.00%  [1]         tdq_lock_pair
  100.0%  [1]          sched_idletd
   100.0%  [1]           fork_exit
 20.00%  [1]         smp_tlb_shootdown
  100.0%  [1]          pmap_invalidate_range
   100.0%  [1]           biodone
 20.00%  [1]         sched_add
  100.0%  [1]          setrunnable
   100.0%  [1]           sleepq_broadcast

01.37%  [4]        sched_switch @ /boot/kernel/kernel
 100.0%  [4]         mi_switch
  75.00%  [3]          ithread_loop
   100.0%  [3]           fork_exit
  25.00%  [1]          sched_idletd
   100.0%  [1]           fork_exit

01.37%  [4]        xpt_done @ /boot/kernel/kernel
 100.0%  [4]         mrsas_cmd_done @ /boot/kernel/mrsas.ko
  100.0%  [4]          mrsas_complete_cmd
   100.0%  [4]           intr_event_execute_handlers @ /boot/kernel/kernel

01.37%  [4]        Xapic_isr1 @ /boot/kernel/kernel

01.37%  [4]        g_io_check @ /boot/kernel/kernel
 100.0%  [4]         g_io_request
  100.0%  [4]          dev_strategy_csw
   100.0%  [4]           physio

01.37%  [4]        vmem_xalloc @ /boot/kernel/kernel
 100.0%  [4]         vmem_alloc
  100.0%  [4]          g_io_check
   100.0%  [4]           g_io_request

01.37%  [4]        dastart @ /boot/kernel/kernel
 100.0%  [4]         xpt_run_allocq
  100.0%  [4]          dastrategy
   100.0%  [4]           g_disk_start

01.37%  [4]        spinlock_exit @ /boot/kernel/kernel
 50.00%  [2]         sched_idletd
  100.0%  [2]          fork_exit
 25.00%  [1]         wakeup
  100.0%  [1]          bdone
   100.0%  [1]           bufdone
 25.00%  [1]         vmem_xfree
  100.0%  [1]          biodone
   100.0%  [1]           dadone

01.37%  [4]        mrsas_complete_cmd @ /boot/kernel/mrsas.ko
 100.0%  [4]         intr_event_execute_handlers @ /boot/kernel/kernel
  100.0%  [4]          ithread_loop
   100.0%  [4]           fork_exit

01.37%  [4]        amd64_syscall @ /boot/kernel/kernel

01.03%  [3]        vm_fault_quick_hold_pages @ /boot/kernel/kernel
 100.0%  [3]         vmapbuf
  100.0%  [3]          physio
   100.0%  [3]           devfs_read_f

01.03%  [3]        ithread_loop @ /boot/kernel/kernel
 100.0%  [3]         fork_exit

01.03%  [3]        critical_enter @ /boot/kernel/kernel
 33.33%  [1]         thread_lock_flags_
  100.0%  [1]          critical_exit
   100.0%  [1]           intr_event_handle
 33.33%  [1]         uma_zfree_arg
  100.0%  [1]          g_dev_done
   100.0%  [1]           g_io_deliver
 33.33%  [1]         smp_tlb_shootdown
  100.0%  [1]          pmap_invalidate_range
   100.0%  [1]           biodone

01.03%  [3]        sleepq_broadcast @ /boot/kernel/kernel
 100.0%  [3]         wakeup
  100.0%  [3]          bdone
   100.0%  [3]           bufdone

01.03%  [3]        pmap_invalidate_range @ /boot/kernel/kernel
 100.0%  [3]         biodone
  100.0%  [3]          dadone
   100.0%  [3]           xpt_done_process

01.03%  [3]        xpt_done_process @ /boot/kernel/kernel
 100.0%  [3]         xpt_done_td
  100.0%  [3]          fork_exit

01.03%  [3]        relpbuf @ /boot/kernel/kernel
 100.0%  [3]         physio
  100.0%  [3]          devfs_read_f
   100.0%  [3]           dofileread

01.03%  [3]        xpt_run_devq @ /boot/kernel/kernel
 100.0%  [3]         xpt_action_default
  100.0%  [3]          dastart
   100.0%  [3]           xpt_run_allocq

01.03%  [3]        __mtx_unlock_flags @ /boot/kernel/kernel
 66.67%  [2]         xpt_run_devq
  100.0%  [2]          xpt_action_default
   100.0%  [2]           dastart
 33.33%  [1]         mrsas_cmd_done @ /boot/kernel/mrsas.ko
  100.0%  [1]          mrsas_complete_cmd
   100.0%  [1]           intr_event_execute_handlers @ /boot/kernel/kernel

00.69%  [2]        uma_zfree_arg @ /boot/kernel/kernel
 50.00%  [1]         g_dev_done
  100.0%  [1]          g_io_deliver
   100.0%  [1]           g_disk_done_single
 50.00%  [1]         g_io_deliver
  100.0%  [1]          g_disk_done_single
   100.0%  [1]           dadone

00.69%  [2]        getpbuf @ /boot/kernel/kernel
 100.0%  [2]         physio
  100.0%  [2]          devfs_read_f
   100.0%  [2]           dofileread

00.69%  [2]        xpt_action_default @ /boot/kernel/kernel
 100.0%  [2]         dastart
  100.0%  [2]          xpt_run_allocq
   100.0%  [2]           dastrategy

00.69%  [2]        mrsas_action @ /boot/kernel/mrsas.ko
 100.0%  [2]         xpt_run_devq @ /boot/kernel/kernel
  100.0%  [2]          xpt_action_default
   100.0%  [2]           dastart

00.69%  [2]        sleepq_lock @ /boot/kernel/kernel
 50.00%  [1]         wakeup
  100.0%  [1]          bdone
   100.0%  [1]           bufdone
 50.00%  [1]         cv_broadcastpri
  100.0%  [1]          vmem_xfree
   100.0%  [1]           biodone

00.69%  [2]        doreti @ /boot/kernel/kernel

00.69%  [2]        xpt_run_allocq @ /boot/kernel/kernel
 100.0%  [2]         dastrategy
  100.0%  [2]          g_disk_start
   100.0%  [2]           g_io_request

00.69%  [2]        bwait @ /boot/kernel/kernel
 100.0%  [2]         physio
  100.0%  [2]          devfs_read_f
   100.0%  [2]           dofileread

00.69%  [2]        lock_mtx @ /boot/kernel/kernel
 100.0%  [2]         _sleep
  100.0%  [2]          bwait
   100.0%  [2]           physio

00.69%  [2]        __sys_read @ /lib/libc.so.7

00.69%  [2]        cpu_set_syscall_retval @ /boot/kernel/kernel
 100.0%  [2]         amd64_syscall

00.69%  [2]        vmem_xfree @ /boot/kernel/kernel
 100.0%  [2]         biodone
  100.0%  [2]          dadone
   100.0%  [2]           xpt_done_process

00.69%  [2]        thread_lock_flags_ @ /boot/kernel/kernel
 50.00%  [1]         sleepq_add
  100.0%  [1]          _sleep
   100.0%  [1]           bwait
 50.00%  [1]         sched_idletd
  100.0%  [1]          fork_exit

00.69%  [2]        sched_choose @ /boot/kernel/kernel
 100.0%  [2]         choosethread
  100.0%  [2]          sched_switch
   100.0%  [2]           mi_switch

00.69%  [2]        pmap_extract_and_hold @ /boot/kernel/kernel
 100.0%  [2]         vm_fault_quick_hold_pages
  100.0%  [2]          vmapbuf
   100.0%  [2]           physio

00.69%  [2]        lapic_ipi_vectored @ /boot/kernel/kernel
 100.0%  [2]         smp_tlb_shootdown
  100.0%  [2]          pmap_invalidate_range
   100.0%  [2]           biodone

00.69%  [2]        __mtx_lock_flags @ /boot/kernel/kernel
 50.00%  [1]         mrsas_unmap_request @ /boot/kernel/mrsas.ko
  100.0%  [1]          mrsas_cmd_done
   100.0%  [1]           mrsas_complete_cmd
 50.00%  [1]         mrsas_map_request
  100.0%  [1]          mrsas_build_dcdb
   100.0%  [1]           mrsas_action

00.69%  [2]        tdq_move @ /boot/kernel/kernel
 100.0%  [2]         sched_idletd
  100.0%  [2]          fork_exit

00.34%  [1]        bdone @ /boot/kernel/kernel
 100.0%  [1]         bufdone
  100.0%  [1]          bufdonebio
   100.0%  [1]           g_io_deliver

00.34%  [1]        devvn_refthread @ /boot/kernel/kernel
 100.0%  [1]         devfs_read_f
  100.0%  [1]          dofileread
   100.0%  [1]           kern_readv

00.34%  [1]        Xfast_syscall @ /boot/kernel/kernel

00.34%  [1]        hardclock_cnt @ /boot/kernel/kernel
 100.0%  [1]         handleevents
  100.0%  [1]          timercb
   100.0%  [1]           lapic_handle_timer

00.34%  [1]        devstat_end_transaction @ /boot/kernel/kernel
 100.0%  [1]         devstat_end_transaction_bio_bt
  100.0%  [1]          g_disk_done_single
   100.0%  [1]           dadone

00.34%  [1]        pmap_qenter @ /boot/kernel/kernel
 100.0%  [1]         g_io_check
  100.0%  [1]          g_io_request
   100.0%  [1]           dev_strategy_csw

00.34%  [1]        dastrategy @ /boot/kernel/kernel
 100.0%  [1]         g_disk_start
  100.0%  [1]          g_io_request
   100.0%  [1]           dev_strategy_csw

00.34%  [1]        kern_readv @ /boot/kernel/kernel
 100.0%  [1]         sys_read
  100.0%  [1]          amd64_syscall

00.34%  [1]        vfs_timestamp @ /boot/kernel/kernel
 100.0%  [1]         devfs_write_f
  100.0%  [1]          dofilewrite
   100.0%  [1]           kern_writev

00.34%  [1]        g_io_deliver @ /boot/kernel/kernel
 100.0%  [1]         g_disk_done_single
  100.0%  [1]          dadone
   100.0%  [1]           xpt_done_process

00.34%  [1]        g_dev_strategy @ /boot/kernel/kernel
 100.0%  [1]         dev_strategy_csw
  100.0%  [1]          physio
   100.0%  [1]           devfs_read_f

00.34%  [1]        malloc @ /boot/kernel/kernel
 100.0%  [1]         xpt_run_allocq
  100.0%  [1]          dastrategy
   100.0%  [1]           g_disk_start

00.34%  [1]        sched_add @ /boot/kernel/kernel
 100.0%  [1]         setrunnable
  100.0%  [1]          sleepq_broadcast
   100.0%  [1]           wakeup

00.34%  [1]        __lockmgr_args @ /boot/kernel/kernel
 100.0%  [1]         relpbuf
  100.0%  [1]          physio
   100.0%  [1]           devfs_read_f

00.34%  [1]        _sleep @ /boot/kernel/kernel
 100.0%  [1]         bwait
  100.0%  [1]          physio
   100.0%  [1]           devfs_read_f

00.34%  [1]        __cap_rights_is_set @ /boot/kernel/kernel
 100.0%  [1]         fget_unlocked
  100.0%  [1]          _fget
   100.0%  [1]           kern_writev

00.34%  [1]        bounce_bus_dmamap_load_buffer @ /boot/kernel/kernel
 100.0%  [1]         bus_dmamap_load
  100.0%  [1]          mrsas_map_request @ /boot/kernel/mrsas.ko
   100.0%  [1]           mrsas_build_dcdb

00.34%  [1]        sched_pickcpu @ /boot/kernel/kernel
 100.0%  [1]         sched_add
  100.0%  [1]          setrunnable
   100.0%  [1]           sleepq_broadcast

00.34%  [1]        intr_event_handle @ /boot/kernel/kernel
 100.0%  [1]         intr_execute_handlers
  100.0%  [1]          lapic_handle_intr

00.34%  [1]        free @ /boot/kernel/kernel
 100.0%  [1]         xpt_release_ccb
  100.0%  [1]          dadone
   100.0%  [1]           xpt_done_process

00.34%  [1]        callout_lock @ /boot/kernel/kernel
 100.0%  [1]         _callout_stop_safe
  100.0%  [1]          mrsas_cmd_done @ /boot/kernel/mrsas.ko
   100.0%  [1]           mrsas_complete_cmd

00.34%  [1]        dev_relthread @ /boot/kernel/kernel
 100.0%  [1]         devfs_read_f
  100.0%  [1]          dofileread
   100.0%  [1]           kern_readv

00.34%  [1]        runq_add @ /boot/kernel/kernel
 100.0%  [1]         tdq_move
  100.0%  [1]          sched_idletd
   100.0%  [1]           fork_exit

00.34%  [1]        critical_exit @ /boot/kernel/kernel
 100.0%  [1]         spinlock_exit
  100.0%  [1]          _sleep
   100.0%  [1]           bwait

00.34%  [1]        msi_vector @ /boot/kernel/kernel
 100.0%  [1]         intr_execute_handlers
  100.0%  [1]          lapic_handle_intr

00.34%  [1]        foffset_unlock @ /boot/kernel/kernel
 100.0%  [1]         devfs_write_f
  100.0%  [1]          dofilewrite
   100.0%  [1]           kern_writev

00.34%  [1]        sched_sleep @ /boot/kernel/kernel
 100.0%  [1]         sleepq_switch
  100.0%  [1]          sleepq_wait
   100.0%  [1]           _sleep

00.34%  [1]        unlock_mtx @ /boot/kernel/kernel
 100.0%  [1]         _sleep
  100.0%  [1]          bwait
   100.0%  [1]           physio

00.34%  [1]        callout_reset_sbt_on @ /boot/kernel/kernel
 100.0%  [1]         mrsas_action @ /boot/kernel/mrsas.ko
  100.0%  [1]          xpt_run_devq @ /boot/kernel/kernel
   100.0%  [1]           xpt_action_default

00.34%  [1]        dadone @ /boot/kernel/kernel
 100.0%  [1]         xpt_done_process
  100.0%  [1]          xpt_done_td
   100.0%  [1]           fork_exit

00.34%  [1]        tsc_get_timecount_low_lfence @ /boot/kernel/kernel
 100.0%  [1]         binuptime
  100.0%  [1]          devstat_start_transaction_bio
   100.0%  [1]           g_disk_start

00.34%  [1]        sleepq_resume_thread @ /boot/kernel/kernel
 100.0%  [1]         sleepq_broadcast
  100.0%  [1]          wakeup
   100.0%  [1]           bdone

00.34%  [1]        physio @ /boot/kernel/kernel
 100.0%  [1]         devfs_read_f
  100.0%  [1]          dofileread
   100.0%  [1]           kern_readv

00.34%  [1]        runq_choose @ /boot/kernel/kernel
 100.0%  [1]         sched_choose
  100.0%  [1]          choosethread
   100.0%  [1]           sched_switch

00.34%  [1]        xpt_done_td @ /boot/kernel/kernel
 100.0%  [1]         fork_exit

00.34%  [1]        g_new_bio @ /boot/kernel/kernel
 100.0%  [1]         dev_strategy_csw
  100.0%  [1]          physio
   100.0%  [1]           devfs_read_f

00.34%  [1]        dev_strategy_csw @ /boot/kernel/kernel
 100.0%  [1]         physio
  100.0%  [1]          devfs_read_f
   100.0%  [1]           dofileread

00.34%  [1]        mrsas_fire_cmd @ /boot/kernel/mrsas.ko
 100.0%  [1]         xpt_run_devq @ /boot/kernel/kernel
  100.0%  [1]          xpt_action_default
   100.0%  [1]           dastart

00.34%  [1]        runq_remove @ /boot/kernel/kernel
 100.0%  [1]         sched_choose
  100.0%  [1]          choosethread
   100.0%  [1]           sched_switch

00.34%  [1]        uma_zalloc_arg @ /boot/kernel/kernel
 100.0%  [1]         malloc
  100.0%  [1]          xpt_run_allocq
   100.0%  [1]           dastrategy

00.34%  [1]        spinlock_enter @ /boot/kernel/kernel
 100.0%  [1]         smp_tlb_shootdown
  100.0%  [1]          pmap_invalidate_range
   100.0%  [1]           biodone

00.34%  [1]        g_io_request @ /boot/kernel/kernel
 100.0%  [1]         dev_strategy_csw
  100.0%  [1]          physio
   100.0%  [1]           devfs_read_f

@ CPU_CLK_UNHALTED_CORE [295 samples]

13.56%  [40]       cpu_idle @ /boot/kernel/kernel
 100.0%  [40]        sched_idletd
  100.0%  [40]         fork_exit

09.49%  [28]       Xinvlrng @ /boot/kernel/kernel

06.44%  [19]       invlrng_handler @ /boot/kernel/kernel

04.07%  [12]       smp_tlb_shootdown @ /boot/kernel/kernel
 100.0%  [12]        pmap_invalidate_range
  100.0%  [12]         biodone
   100.0%  [12]          dadone

03.73%  [11]       cpu_search_highest @ /boot/kernel/kernel
 100.0%  [11]        cpu_search_highest
  63.64%  [7]          sched_idletd
   100.0%  [7]           fork_exit
  36.36%  [4]          cpu_search_highest
   100.0%  [4]           sched_idletd

03.05%  [9]        sched_idletd @ /boot/kernel/kernel
 100.0%  [9]         fork_exit

02.71%  [8]        spinlock_exit @ /boot/kernel/kernel
 37.50%  [3]         sched_idletd
  100.0%  [3]          fork_exit
 12.50%  [1]         callout_reset_sbt_on
  100.0%  [1]          mrsas_action @ /boot/kernel/mrsas.ko
   100.0%  [1]           xpt_run_devq @ /boot/kernel/kernel
 12.50%  [1]         wakeup
  100.0%  [1]          bdone
   100.0%  [1]           bufdone
 12.50%  [1]         thread_lock_flags_
  100.0%  [1]          ithread_loop
   100.0%  [1]           fork_exit
 12.50%  [1]         __mtx_lock_sleep
  100.0%  [1]          __mtx_lock_flags
   100.0%  [1]           mrsas_cmd_done @ /boot/kernel/mrsas.ko
 12.50%  [1]         ithread_loop @ /boot/kernel/kernel
  100.0%  [1]          fork_exit

02.71%  [8]        __mtx_lock_sleep @ /boot/kernel/kernel
 62.50%  [5]         __mtx_lock_flags
  100.0%  [5]          mrsas_cmd_done @ /boot/kernel/mrsas.ko
   100.0%  [5]           mrsas_complete_cmd
 12.50%  [1]         xpt_run_devq @ /boot/kernel/kernel
  100.0%  [1]          xpt_action_default
   100.0%  [1]           dastart
 12.50%  [1]         vmem_xfree
  100.0%  [1]          biodone
   100.0%  [1]           dadone
 12.50%  [1]         dadone
  100.0%  [1]          xpt_done_process
   100.0%  [1]           xpt_done_td

02.37%  [7]        cpu_switch @ /boot/kernel/kernel
 100.0%  [7]         mi_switch
  42.86%  [3]          sleepq_wait
   100.0%  [3]           _sleep
  28.57%  [2]          sched_idletd
   100.0%  [2]           fork_exit
  14.29%  [1]          ithread_loop
   100.0%  [1]           fork_exit
  14.29%  [1]          critical_exit
   100.0%  [1]           intr_event_handle

02.37%  [7]        thread_lock_flags_ @ /boot/kernel/kernel
 42.86%  [3]         sleepq_add
  100.0%  [3]          _sleep
   100.0%  [3]           bwait
 42.86%  [3]         sched_idletd
  100.0%  [3]          fork_exit
 14.29%  [1]         turnstile_wait
  100.0%  [1]          __mtx_lock_sleep
   100.0%  [1]           __mtx_lock_flags

02.37%  [7]        vmem_xalloc @ /boot/kernel/kernel
 100.0%  [7]         vmem_alloc
  100.0%  [7]          g_io_check
   100.0%  [7]           g_io_request

02.03%  [6]        cpu_search_lowest @ /boot/kernel/kernel
 100.0%  [6]         cpu_search_lowest
  66.67%  [4]          cpu_search_lowest
   100.0%  [4]           sched_pickcpu
  33.33%  [2]          sched_pickcpu
   100.0%  [2]           sched_add

01.69%  [5]        xpt_run_devq @ /boot/kernel/kernel
 100.0%  [5]         xpt_action_default
  100.0%  [5]          dastart
   100.0%  [5]           xpt_run_allocq

01.36%  [4]        callout_lock @ /boot/kernel/kernel
 100.0%  [4]         _callout_stop_safe
  100.0%  [4]          mrsas_cmd_done @ /boot/kernel/mrsas.ko
   100.0%  [4]           mrsas_complete_cmd

01.36%  [4]        amd64_syscall @ /boot/kernel/kernel

01.36%  [4]        __lockmgr_args @ /boot/kernel/kernel
 75.00%  [3]         relpbuf
  100.0%  [3]          physio
   100.0%  [3]           devfs_read_f
 25.00%  [1]         getpbuf
  100.0%  [1]          physio
   100.0%  [1]           devfs_read_f

01.36%  [4]        physio @ /boot/kernel/kernel
 100.0%  [4]         devfs_read_f
  100.0%  [4]          dofileread
   100.0%  [4]           kern_readv

01.36%  [4]        sched_pickcpu @ /boot/kernel/kernel
 100.0%  [4]         sched_add
  50.00%  [2]          setrunnable
   100.0%  [2]           sleepq_broadcast
  50.00%  [2]          intr_event_schedule_thread
   100.0%  [2]           intr_event_handle

01.36%  [4]        sched_switch @ /boot/kernel/kernel
 100.0%  [4]         mi_switch
  100.0%  [4]          sleepq_wait
   100.0%  [4]           _sleep

01.36%  [4]        __mtx_unlock_flags @ /boot/kernel/kernel
 50.00%  [2]         mrsas_complete_cmd @ /boot/kernel/mrsas.ko
  100.0%  [2]          intr_event_execute_handlers @ /boot/kernel/kernel
   100.0%  [2]           ithread_loop
 50.00%  [2]         xpt_run_devq
  100.0%  [2]          xpt_action_default
   100.0%  [2]           dastart

01.36%  [4]        vm_fault_quick_hold_pages @ /boot/kernel/kernel
 100.0%  [4]         vmapbuf
  100.0%  [4]          physio
   100.0%  [4]           devfs_read_f

01.02%  [3]        __mtx_lock_flags @ /boot/kernel/kernel
 33.33%  [1]         mrsas_fire_cmd @ /boot/kernel/mrsas.ko
  100.0%  [1]          xpt_run_devq @ /boot/kernel/kernel
   100.0%  [1]           xpt_action_default
 33.33%  [1]         mrsas_get_mpt_cmd @ /boot/kernel/mrsas.ko
  100.0%  [1]          mrsas_action
   100.0%  [1]           xpt_run_devq @ /boot/kernel/kernel
 33.33%  [1]         mrsas_unmap_request @ /boot/kernel/mrsas.ko
  100.0%  [1]          mrsas_cmd_done
   100.0%  [1]           mrsas_complete_cmd

01.02%  [3]        sched_runnable @ /boot/kernel/kernel
 100.0%  [3]         cpu_idle
  100.0%  [3]          sched_idletd
   100.0%  [3]           fork_exit

01.02%  [3]        critical_exit @ /boot/kernel/kernel
 66.67%  [2]         spinlock_exit
  50.00%  [1]          thread_lock_block
   100.0%  [1]           sched_add
  50.00%  [1]          intr_event_schedule_thread
   100.0%  [1]           intr_event_handle
 33.33%  [1]         malloc
  100.0%  [1]          xpt_run_allocq
   100.0%  [1]           dastrategy

01.02%  [3]        vm_page_unhold_pages @ /boot/kernel/kernel
 100.0%  [3]         vunmapbuf
  100.0%  [3]          physio
   100.0%  [3]           devfs_read_f

01.02%  [3]        vmem_xfree @ /boot/kernel/kernel
 100.0%  [3]         biodone
  100.0%  [3]          dadone
   100.0%  [3]           xpt_done_process

01.02%  [3]        doreti @ /boot/kernel/kernel

00.68%  [2]        dastrategy @ /boot/kernel/kernel
 100.0%  [2]         g_disk_start
  100.0%  [2]          g_io_request
   100.0%  [2]           dev_strategy_csw

00.68%  [2]        xpt_done_td @ /boot/kernel/kernel
 100.0%  [2]         fork_exit

00.68%  [2]        pmap_extract_and_hold @ /boot/kernel/kernel
 100.0%  [2]         vm_fault_quick_hold_pages
  100.0%  [2]          vmapbuf
   100.0%  [2]           physio

00.68%  [2]        devstat_end_transaction @ /boot/kernel/kernel
 100.0%  [2]         devstat_end_transaction_bio_bt
  100.0%  [2]          g_disk_done_single
   100.0%  [2]           dadone

00.68%  [2]        _fget @ /boot/kernel/kernel
 100.0%  [2]         kern_readv
  100.0%  [2]          sys_read
   100.0%  [2]           amd64_syscall

00.68%  [2]        mrsas_action @ /boot/kernel/mrsas.ko
 100.0%  [2]         xpt_run_devq @ /boot/kernel/kernel
  100.0%  [2]          xpt_action_default
   100.0%  [2]           dastart

00.68%  [2]        mrsas_complete_cmd @ /boot/kernel/mrsas.ko
 100.0%  [2]         intr_event_execute_handlers @ /boot/kernel/kernel
  100.0%  [2]          ithread_loop
   100.0%  [2]           fork_exit

00.68%  [2]        intr_event_handle @ /boot/kernel/kernel
 100.0%  [2]         intr_execute_handlers
  100.0%  [2]          lapic_handle_intr

00.68%  [2]        getpbuf @ /boot/kernel/kernel
 100.0%  [2]         physio
  100.0%  [2]          devfs_read_f
   100.0%  [2]           dofileread

00.68%  [2]        uma_zalloc_arg @ /boot/kernel/kernel
 50.00%  [1]         malloc
  100.0%  [1]          xpt_run_allocq
   100.0%  [1]           dastrategy
 50.00%  [1]         dev_strategy_csw
  100.0%  [1]          physio
   100.0%  [1]           devfs_read_f

00.68%  [2]        spinlock_enter @ /boot/kernel/kernel
 50.00%  [1]         callout_lock
  100.0%  [1]          _callout_stop_safe
   100.0%  [1]           mrsas_cmd_done @ /boot/kernel/mrsas.ko
 50.00%  [1]         sleepq_lock @ /boot/kernel/kernel
  100.0%  [1]          _sleep
   100.0%  [1]           bwait

00.68%  [2]        pmap_invalidate_range @ /boot/kernel/kernel
 100.0%  [2]         biodone
  100.0%  [2]          dadone
   100.0%  [2]           xpt_done_process

00.68%  [2]        devfs_read_f @ /boot/kernel/kernel
 100.0%  [2]         dofileread
  100.0%  [2]          kern_readv
   100.0%  [2]           sys_read

00.68%  [2]        bdone @ /boot/kernel/kernel
 100.0%  [2]         bufdone
  100.0%  [2]          bufdonebio
   100.0%  [2]           g_io_deliver

00.68%  [2]        bcopy @ /boot/kernel/kernel
 100.0%  [2]         xpt_run_devq
  100.0%  [2]          xpt_action_default
   100.0%  [2]           dastart

00.34%  [1]        relpbuf @ /boot/kernel/kernel
 100.0%  [1]         physio
  100.0%  [1]          devfs_read_f
   100.0%  [1]           dofileread

00.34%  [1]        _mtx_lock_spin_cookie @ /boot/kernel/kernel
 100.0%  [1]         turnstile_trywait
  100.0%  [1]          __mtx_lock_sleep
   100.0%  [1]           vmem_xalloc

00.34%  [1]        cam_ccbq_remove_ccb @ /boot/kernel/kernel
 100.0%  [1]         xpt_run_devq
  100.0%  [1]          xpt_action_default
   100.0%  [1]           dastart

00.34%  [1]        _callout_stop_safe @ /boot/kernel/kernel
 100.0%  [1]         mrsas_cmd_done @ /boot/kernel/mrsas.ko
  100.0%  [1]          mrsas_complete_cmd
   100.0%  [1]           intr_event_execute_handlers @ /boot/kernel/kernel

00.34%  [1]        fget_unlocked @ /boot/kernel/kernel
 100.0%  [1]         _fget
  100.0%  [1]          kern_writev
   100.0%  [1]           sys_write

00.34%  [1]        cpu_fetch_syscall_args @ /boot/kernel/kernel
 100.0%  [1]         amd64_syscall

00.34%  [1]        tdq_notify @ /boot/kernel/kernel
 100.0%  [1]         sched_add
  100.0%  [1]          setrunnable
   100.0%  [1]           sleepq_broadcast

00.34%  [1]        sched_choose @ /boot/kernel/kernel
 100.0%  [1]         choosethread
  100.0%  [1]          sched_switch
   100.0%  [1]           mi_switch

00.34%  [1]        xpt_action_default @ /boot/kernel/kernel
 100.0%  [1]         dastart
  100.0%  [1]          xpt_run_allocq
   100.0%  [1]           dastrategy

00.34%  [1]        ithread_loop @ /boot/kernel/kernel
 100.0%  [1]         fork_exit

00.34%  [1]        vm_page_pa_tryrelock @ /boot/kernel/kernel
 100.0%  [1]         pmap_extract_and_hold
  100.0%  [1]          vm_fault_quick_hold_pages
   100.0%  [1]           vmapbuf

00.34%  [1]        sleepq_broadcast @ /boot/kernel/kernel
 100.0%  [1]         wakeup
  100.0%  [1]          bdone
   100.0%  [1]           bufdone

00.34%  [1]        syscall_thread_enter @ /boot/kernel/kernel
 100.0%  [1]         amd64_syscall

00.34%  [1]        xpt_run_allocq @ /boot/kernel/kernel
 100.0%  [1]         dastrategy
  100.0%  [1]          g_disk_start
   100.0%  [1]           g_io_request

00.34%  [1]        Xapic_isr1 @ /boot/kernel/kernel

00.34%  [1]        kern_readv @ /boot/kernel/kernel
 100.0%  [1]         sys_read
  100.0%  [1]          amd64_syscall

00.34%  [1]        rdtsc @ /boot/kernel/kernel
 100.0%  [1]         mi_switch
  100.0%  [1]          sleepq_wait
   100.0%  [1]           _sleep

00.34%  [1]        mrsas_data_load_cb @ /boot/kernel/mrsas.ko
 100.0%  [1]         bus_dmamap_load @ /boot/kernel/kernel
  100.0%  [1]          mrsas_map_request @ /boot/kernel/mrsas.ko
   100.0%  [1]           mrsas_build_dcdb

00.34%  [1]        xpt_release_ccb @ /boot/kernel/kernel
 100.0%  [1]         dadone
  100.0%  [1]          xpt_done_process
   100.0%  [1]           xpt_done_td

00.34%  [1]        mtx_pool_find @ /boot/kernel/kernel
 100.0%  [1]         bdone
  100.0%  [1]          bufdone
   100.0%  [1]           bufdonebio

00.34%  [1]        critical_enter @ /boot/kernel/kernel
 100.0%  [1]         thread_lock_flags_
  100.0%  [1]          tdq_move
   100.0%  [1]           sched_idletd

00.34%  [1]        intr_event_execute_handlers @ /boot/kernel/kernel
 100.0%  [1]         ithread_loop
  100.0%  [1]          fork_exit

00.34%  [1]        dastart @ /boot/kernel/kernel
 100.0%  [1]         xpt_run_allocq
  100.0%  [1]          dastrategy
   100.0%  [1]           g_disk_start

00.34%  [1]        atomic_fetchadd_int @ /boot/kernel/kernel
 100.0%  [1]         mrsas_action @ /boot/kernel/mrsas.ko
  100.0%  [1]          xpt_run_devq @ /boot/kernel/kernel
   100.0%  [1]           xpt_action_default

00.34%  [1]        sched_rem @ /boot/kernel/kernel
 100.0%  [1]         tdq_move
  100.0%  [1]          sched_idletd
   100.0%  [1]           fork_exit

00.34%  [1]        sched_add @ /boot/kernel/kernel
 100.0%  [1]         setrunnable
  100.0%  [1]          sleepq_broadcast
   100.0%  [1]           wakeup

00.34%  [1]        lapic_handle_intr @ /boot/kernel/kernel

00.34%  [1]        _sleep @ /boot/kernel/kernel
 100.0%  [1]         bwait
  100.0%  [1]          physio
   100.0%  [1]           devfs_read_f

00.34%  [1]        dofilewrite @ /boot/kernel/kernel
 100.0%  [1]         kern_writev
  100.0%  [1]          sys_write
   100.0%  [1]           amd64_syscall

00.34%  [1]        free @ /boot/kernel/kernel
 100.0%  [1]         xpt_release_ccb
  100.0%  [1]          dadone
   100.0%  [1]           xpt_done_process

00.34%  [1]        intr_event_schedule_thread @ /boot/kernel/kernel
 100.0%  [1]         intr_event_handle
  100.0%  [1]          intr_execute_handlers
   100.0%  [1]           lapic_handle_intr

00.34%  [1]        mrsas_cmd_done @ /boot/kernel/mrsas.ko
 100.0%  [1]         mrsas_complete_cmd
  100.0%  [1]          intr_event_execute_handlers @ /boot/kernel/kernel
   100.0%  [1]           ithread_loop

00.34%  [1]        foffset_lock_uio @ /boot/kernel/kernel
 100.0%  [1]         devfs_write_f
  100.0%  [1]          dofilewrite
   100.0%  [1]           kern_writev

00.34%  [1]        pmap_kextract @ /boot/kernel/kernel
 100.0%  [1]         bounce_bus_dmamap_load_buffer
  100.0%  [1]          bus_dmamap_load
   100.0%  [1]           mrsas_map_request @ /boot/kernel/mrsas.ko

00.34%  [1]        __sys_read @ /lib/libc.so.7

00.34%  [1]        dofileread @ /boot/kernel/kernel
 100.0%  [1]         kern_readv
  100.0%  [1]          sys_read
   100.0%  [1]           amd64_syscall

00.34%  [1]        tdq_lock_pair @ /boot/kernel/kernel
 100.0%  [1]         sched_idletd
  100.0%  [1]          fork_exit

00.34%  [1]        g_clone_bio @ /boot/kernel/kernel
 100.0%  [1]         g_dev_strategy
  100.0%  [1]          dev_strategy_csw
   100.0%  [1]           physio

00.34%  [1]        foffset_lock @ /boot/kernel/kernel
 100.0%  [1]         foffset_lock_uio
  100.0%  [1]          devfs_write_f
   100.0%  [1]           dofilewrite

00.34%  [1]        mrsas_map_request @ /boot/kernel/mrsas.ko
 100.0%  [1]         mrsas_build_dcdb
  100.0%  [1]          mrsas_action
   100.0%  [1]           xpt_run_devq @ /boot/kernel/kernel

00.34%  [1]        lock_mtx @ /boot/kernel/kernel
 100.0%  [1]         _sleep
  100.0%  [1]          bwait
   100.0%  [1]           physio

00.34%  [1]        dadone @ /boot/kernel/kernel
 100.0%  [1]         xpt_done_process
  100.0%  [1]          xpt_done_td
   100.0%  [1]           fork_exit

00.34%  [1]        tsc_get_timecount_low_lfence @ /boot/kernel/kernel
 100.0%  [1]         binuptime
  100.0%  [1]          devstat_start_transaction_bio
   100.0%  [1]           g_disk_start

00.34%  [1]        choosethread @ /boot/kernel/kernel
 100.0%  [1]         sched_switch
  100.0%  [1]          mi_switch
   100.0%  [1]           sched_idletd

00.34%  [1]        uma_zfree_arg @ /boot/kernel/kernel
 100.0%  [1]         g_io_deliver
  100.0%  [1]          g_disk_done_single
   100.0%  [1]           dadone

00.34%  [1]        Xtimerint @ /boot/kernel/kernel

00.34%  [1]        devstat_start_transaction_bio @ /boot/kernel/kernel
 100.0%  [1]         g_disk_start
  100.0%  [1]          g_io_request
   100.0%  [1]           dev_strategy_csw

00.34%  [1]        runq_choose @ /boot/kernel/kernel
 100.0%  [1]         sched_choose
  100.0%  [1]          choosethread
   100.0%  [1]           sched_switch

@ CPU_CLK_UNHALTED_CORE [288 samples]

13.19%  [38]       cpu_idle @ /boot/kernel/kernel
 100.0%  [38]        sched_idletd
  100.0%  [38]         fork_exit

10.07%  [29]       Xinvlrng @ /boot/kernel/kernel

08.68%  [25]       invlrng_handler @ /boot/kernel/kernel

05.21%  [15]       cpu_search_highest @ /boot/kernel/kernel
 66.67%  [10]        cpu_search_highest
  80.00%  [8]          sched_idletd
   100.0%  [8]           fork_exit
  20.00%  [2]          cpu_search_highest
   100.0%  [2]           sched_idletd
 33.33%  [5]         sched_idletd
  100.0%  [5]          fork_exit

03.82%  [11]       cpu_switch @ /boot/kernel/kernel
 100.0%  [11]        mi_switch
  72.73%  [8]          sched_idletd
   100.0%  [8]           fork_exit
  18.18%  [2]          ithread_loop
   100.0%  [2]           fork_exit
  09.09%  [1]          critical_exit
   100.0%  [1]           intr_event_handle

02.43%  [7]        cpu_search_lowest @ /boot/kernel/kernel
 100.0%  [7]         cpu_search_lowest
  85.71%  [6]          cpu_search_lowest
   100.0%  [6]           sched_pickcpu
  14.29%  [1]          sched_pickcpu
   100.0%  [1]           sched_add

02.08%  [6]        smp_tlb_shootdown @ /boot/kernel/kernel
 100.0%  [6]         pmap_invalidate_range
  100.0%  [6]          biodone
   100.0%  [6]           dadone

02.08%  [6]        mrsas_complete_cmd @ /boot/kernel/mrsas.ko
 100.0%  [6]         intr_event_execute_handlers @ /boot/kernel/kernel
  100.0%  [6]          ithread_loop
   100.0%  [6]           fork_exit

02.08%  [6]        sched_idletd @ /boot/kernel/kernel
 100.0%  [6]         fork_exit

02.08%  [6]        __mtx_lock_sleep @ /boot/kernel/kernel
 66.67%  [4]         __mtx_lock_flags
  100.0%  [4]          mrsas_cmd_done @ /boot/kernel/mrsas.ko
   100.0%  [4]           mrsas_complete_cmd
 16.67%  [1]         xpt_run_devq @ /boot/kernel/kernel
  100.0%  [1]          xpt_action_default
   100.0%  [1]           dastart
 16.67%  [1]         vmem_xfree
  100.0%  [1]          biodone
   100.0%  [1]           dadone

02.08%  [6]        __sys_read @ /lib/libc.so.7

01.74%  [5]        spinlock_exit @ /boot/kernel/kernel
 40.00%  [2]         wakeup
  100.0%  [2]          bdone
   100.0%  [2]           bufdone
 20.00%  [1]         ithread_loop
  100.0%  [1]          fork_exit
 20.00%  [1]         turnstile_unpend
  100.0%  [1]          __mtx_unlock_sleep
   100.0%  [1]           xpt_run_devq
 20.00%  [1]         sched_idletd
  100.0%  [1]          fork_exit

01.74%  [5]        __lockmgr_args @ /boot/kernel/kernel
 60.00%  [3]         relpbuf
  100.0%  [3]          physio
   100.0%  [3]           devfs_read_f
 40.00%  [2]         getpbuf
  100.0%  [2]          physio
   100.0%  [2]           devfs_read_f

01.74%  [5]        doreti @ /boot/kernel/kernel

01.39%  [4]        vmem_xalloc @ /boot/kernel/kernel
 100.0%  [4]         vmem_alloc
  100.0%  [4]          g_io_check
   100.0%  [4]           g_io_request

01.39%  [4]        __mtx_unlock_flags @ /boot/kernel/kernel
 50.00%  [2]         mrsas_complete_cmd @ /boot/kernel/mrsas.ko
  100.0%  [2]          intr_event_execute_handlers @ /boot/kernel/kernel
   100.0%  [2]           ithread_loop
 25.00%  [1]         xpt_run_devq
  100.0%  [1]          xpt_action_default
   100.0%  [1]           dastart
 25.00%  [1]         mrsas_action @ /boot/kernel/mrsas.ko
  100.0%  [1]          xpt_run_devq @ /boot/kernel/kernel
   100.0%  [1]           xpt_action_default

01.39%  [4]        bzero @ /boot/kernel/kernel
 50.00%  [2]         dev_strategy_csw
  100.0%  [2]          physio
   100.0%  [2]           devfs_read_f
 25.00%  [1]         kern_writev
  100.0%  [1]          sys_write
   100.0%  [1]           amd64_syscall
 25.00%  [1]         mrsas_action @ /boot/kernel/mrsas.ko
  100.0%  [1]          xpt_run_devq @ /boot/kernel/kernel
   100.0%  [1]           xpt_action_default

01.39%  [4]        tdq_notify @ /boot/kernel/kernel
 100.0%  [4]         sched_add
  50.00%  [2]          setrunnable
   100.0%  [2]           sleepq_broadcast
  25.00%  [1]          intr_event_schedule_thread
   100.0%  [1]           intr_event_handle
  25.00%  [1]          turnstile_unpend
   100.0%  [1]           __mtx_unlock_sleep

01.04%  [3]        g_disk_done_single @ /boot/kernel/kernel
 100.0%  [3]         dadone
  100.0%  [3]          xpt_done_process
   100.0%  [3]           xpt_done_td

01.04%  [3]        sched_switch @ /boot/kernel/kernel
 100.0%  [3]         mi_switch
  100.0%  [3]          sleepq_wait
   100.0%  [3]           _sleep

01.04%  [3]        lapic_ipi_vectored @ /boot/kernel/kernel
 100.0%  [3]         smp_tlb_shootdown
  100.0%  [3]          pmap_invalidate_range
   100.0%  [3]           biodone

01.04%  [3]        xpt_action_default @ /boot/kernel/kernel
 100.0%  [3]         dastart
  100.0%  [3]          xpt_run_allocq
   100.0%  [3]           dastrategy

01.04%  [3]        sleepq_lock @ /boot/kernel/kernel
 66.67%  [2]         wakeup
  100.0%  [2]          bdone
   100.0%  [2]           bufdone
 33.33%  [1]         cv_broadcastpri
  100.0%  [1]          vmem_xfree
   100.0%  [1]           biodone

01.04%  [3]        sched_choose @ /boot/kernel/kernel
 100.0%  [3]         choosethread
  100.0%  [3]          sched_switch
   100.0%  [3]           mi_switch

01.04%  [3]        uma_zalloc_arg @ /boot/kernel/kernel
 33.33%  [1]         g_clone_bio
  100.0%  [1]          g_dev_strategy
   100.0%  [1]           dev_strategy_csw
 33.33%  [1]         dev_strategy_csw
  100.0%  [1]          physio
   100.0%  [1]           devfs_read_f
 33.33%  [1]         malloc
  100.0%  [1]          xpt_run_allocq
   100.0%  [1]           dastrategy

01.04%  [3]        bdone @ /boot/kernel/kernel
 100.0%  [3]         bufdone
  100.0%  [3]          bufdonebio
   100.0%  [3]           g_io_deliver

00.69%  [2]        thread_lock_flags_ @ /boot/kernel/kernel
 50.00%  [1]         intr_event_schedule_thread
  100.0%  [1]          intr_event_handle
   100.0%  [1]           intr_execute_handlers
 50.00%  [1]         sched_idletd
  100.0%  [1]          fork_exit

00.69%  [2]        kern_writev @ /boot/kernel/kernel
 100.0%  [2]         sys_write
  100.0%  [2]          amd64_syscall

00.69%  [2]        bt_freetrim @ /boot/kernel/kernel
 100.0%  [2]         biodone
  100.0%  [2]          dadone
   100.0%  [2]           xpt_done_process

00.69%  [2]        devstat_end_transaction @ /boot/kernel/kernel
 100.0%  [2]         devstat_end_transaction_bio_bt
  100.0%  [2]          g_disk_done_single
   100.0%  [2]           dadone

00.69%  [2]        Xapic_isr1 @ /boot/kernel/kernel

00.69%  [2]        pmap_qenter @ /boot/kernel/kernel
 100.0%  [2]         g_io_check
  100.0%  [2]          g_io_request
   100.0%  [2]           dev_strategy_csw

00.69%  [2]        __sys_write @ /lib/libc.so.7

00.69%  [2]        g_dev_strategy @ /boot/kernel/kernel
 100.0%  [2]         dev_strategy_csw
  100.0%  [2]          physio
   100.0%  [2]           devfs_read_f

00.69%  [2]        critical_enter @ /boot/kernel/kernel
 50.00%  [1]         sched_switch
  100.0%  [1]          mi_switch
   100.0%  [1]           ithread_loop
 50.00%  [1]         uma_zalloc_arg
  100.0%  [1]          dev_strategy_csw
   100.0%  [1]           physio

00.69%  [2]        dastart @ /boot/kernel/kernel
 100.0%  [2]         xpt_run_allocq
  100.0%  [2]          dastrategy
   100.0%  [2]           g_disk_start

00.69%  [2]        dofilewrite @ /boot/kernel/kernel
 100.0%  [2]         kern_writev
  100.0%  [2]          sys_write
   100.0%  [2]           amd64_syscall

00.69%  [2]        __mtx_lock_flags @ /boot/kernel/kernel
 100.0%  [2]         mrsas_cmd_done @ /boot/kernel/mrsas.ko
  100.0%  [2]          mrsas_complete_cmd
   100.0%  [2]           intr_event_execute_handlers @ /boot/kernel/kernel

00.69%  [2]        callout_lock @ /boot/kernel/kernel
 100.0%  [2]         _callout_stop_safe
  100.0%  [2]          mrsas_cmd_done @ /boot/kernel/mrsas.ko
   100.0%  [2]           mrsas_complete_cmd

00.69%  [2]        runq_add @ /boot/kernel/kernel
 50.00%  [1]         sched_switch
  100.0%  [1]          mi_switch
   100.0%  [1]           critical_exit
 50.00%  [1]         tdq_move
  100.0%  [1]          sched_idletd
   100.0%  [1]           fork_exit

00.69%  [2]        dadone @ /boot/kernel/kernel
 100.0%  [2]         xpt_done_process
  100.0%  [2]          xpt_done_td
   100.0%  [2]           fork_exit

00.69%  [2]        runq_choose @ /boot/kernel/kernel
 100.0%  [2]         sched_choose
  100.0%  [2]          choosethread
   100.0%  [2]           sched_switch

00.69%  [2]        amd64_syscall @ /boot/kernel/kernel

00.69%  [2]        spinlock_enter @ /boot/kernel/kernel
 50.00%  [1]         thread_lock_flags_
  100.0%  [1]          sched_idletd
   100.0%  [1]           fork_exit
 50.00%  [1]         callout_lock
  100.0%  [1]          _callout_stop_safe
   100.0%  [1]           mrsas_cmd_done @ /boot/kernel/mrsas.ko

00.35%  [1]        atomic_fetchadd_int @ /boot/kernel/kernel
 100.0%  [1]         mrsas_action @ /boot/kernel/mrsas.ko
  100.0%  [1]          xpt_run_devq @ /boot/kernel/kernel
   100.0%  [1]           xpt_action_default

00.35%  [1]        pmap_kextract @ /boot/kernel/kernel
 100.0%  [1]         free
  100.0%  [1]          xpt_release_ccb
   100.0%  [1]           dadone

00.35%  [1]        g_io_request @ /boot/kernel/kernel
 100.0%  [1]         dev_strategy_csw
  100.0%  [1]          physio
   100.0%  [1]           devfs_read_f

00.35%  [1]        camq_remove @ /boot/kernel/kernel
 100.0%  [1]         cam_ccbq_remove_ccb
  100.0%  [1]          xpt_run_devq
   100.0%  [1]           xpt_action_default

00.35%  [1]        bus_dmamap_load @ /boot/kernel/kernel
 100.0%  [1]         mrsas_map_request @ /boot/kernel/mrsas.ko
  100.0%  [1]          mrsas_build_dcdb
   100.0%  [1]           mrsas_action

00.35%  [1]        g_dev_done @ /boot/kernel/kernel
 100.0%  [1]         g_io_deliver
  100.0%  [1]          g_disk_done_single
   100.0%  [1]           dadone

00.35%  [1]        thread_lock_set @ /boot/kernel/kernel
 100.0%  [1]         sleepq_switch
  100.0%  [1]          sleepq_wait
   100.0%  [1]           _sleep

00.35%  [1]        devvn_refthread @ /boot/kernel/kernel
 100.0%  [1]         devfs_read_f
  100.0%  [1]          dofileread
   100.0%  [1]           kern_readv

00.35%  [1]        bioq_takefirst @ /boot/kernel/kernel
 100.0%  [1]         dastart
  100.0%  [1]          xpt_run_allocq
   100.0%  [1]           dastrategy

00.35%  [1]        vfs_timestamp @ /boot/kernel/kernel
 100.0%  [1]         devfs_write_f
  100.0%  [1]          dofilewrite
   100.0%  [1]           kern_writev

00.35%  [1]        xpt_done_process @ /boot/kernel/kernel
 100.0%  [1]         xpt_done_td
  100.0%  [1]          fork_exit

00.35%  [1]        tdq_lock_pair @ /boot/kernel/kernel
 100.0%  [1]         sched_idletd
  100.0%  [1]          fork_exit

00.35%  [1]        scsi_read_write @ /boot/kernel/kernel
 100.0%  [1]         dastart
  100.0%  [1]          xpt_run_allocq
   100.0%  [1]           dastrategy

00.35%  [1]        unlock_mtx @ /boot/kernel/kernel
 100.0%  [1]         _sleep
  100.0%  [1]          bwait
   100.0%  [1]           physio

00.35%  [1]        ithread_loop @ /boot/kernel/kernel
 100.0%  [1]         fork_exit

00.35%  [1]        callout_reset_sbt_on @ /boot/kernel/kernel
 100.0%  [1]         mrsas_action @ /boot/kernel/mrsas.ko
  100.0%  [1]          xpt_run_devq @ /boot/kernel/kernel
   100.0%  [1]           xpt_action_default

00.35%  [1]        mrsas_isr @ /boot/kernel/mrsas.ko
 100.0%  [1]         intr_event_execute_handlers @ /boot/kernel/kernel
  100.0%  [1]          ithread_loop
   100.0%  [1]           fork_exit

00.35%  [1]        pmap_invalidate_range @ /boot/kernel/kernel
 100.0%  [1]         biodone
  100.0%  [1]          dadone
   100.0%  [1]           xpt_done_process

00.35%  [1]        devfs_read_f @ /boot/kernel/kernel
 100.0%  [1]         dofileread
  100.0%  [1]          kern_readv
   100.0%  [1]           sys_read

00.35%  [1]        getpbuf @ /boot/kernel/kernel
 100.0%  [1]         physio
  100.0%  [1]          devfs_read_f
   100.0%  [1]           dofileread

00.35%  [1]        xpt_done @ /boot/kernel/kernel
 100.0%  [1]         mrsas_cmd_done @ /boot/kernel/mrsas.ko
  100.0%  [1]          mrsas_complete_cmd
   100.0%  [1]           intr_event_execute_handlers @ /boot/kernel/kernel

00.35%  [1]        devfs_write_f @ /boot/kernel/kernel
 100.0%  [1]         dofilewrite
  100.0%  [1]          kern_writev
   100.0%  [1]           sys_write

00.35%  [1]        vmem_xfree @ /boot/kernel/kernel
 100.0%  [1]         biodone
  100.0%  [1]          dadone
   100.0%  [1]           xpt_done_process

00.35%  [1]        pcpu_find @ /boot/kernel/kernel
 100.0%  [1]         cpu_idle_wakeup
  100.0%  [1]          tdq_notify
   100.0%  [1]           sched_add

00.35%  [1]        relpbuf @ /boot/kernel/kernel
 100.0%  [1]         physio
  100.0%  [1]          devfs_read_f
   100.0%  [1]           dofileread

00.35%  [1]        _mtx_lock_spin_cookie @ /boot/kernel/kernel
 100.0%  [1]         tdq_lock_pair
  100.0%  [1]          sched_idletd
   100.0%  [1]           fork_exit

00.35%  [1]        tdq_move @ /boot/kernel/kernel
 100.0%  [1]         sched_idletd
  100.0%  [1]          fork_exit

00.35%  [1]        mrsas_unmap_request @ /boot/kernel/mrsas.ko
 100.0%  [1]         mrsas_cmd_done
  100.0%  [1]          mrsas_complete_cmd
   100.0%  [1]           intr_event_execute_handlers @ /boot/kernel/kernel

00.35%  [1]        _callout_stop_safe @ /boot/kernel/kernel
 100.0%  [1]         mrsas_cmd_done @ /boot/kernel/mrsas.ko
  100.0%  [1]          mrsas_complete_cmd
   100.0%  [1]           intr_event_execute_handlers @ /boot/kernel/kernel

00.35%  [1]        fget_unlocked @ /boot/kernel/kernel
 100.0%  [1]         _fget
  100.0%  [1]          kern_writev
   100.0%  [1]           sys_write

00.35%  [1]        sleepq_release @ /boot/kernel/kernel
 100.0%  [1]         wakeup
  100.0%  [1]          bdone
   100.0%  [1]           bufdone

00.35%  [1]        xpt_run_allocq @ /boot/kernel/kernel
 100.0%  [1]         dastrategy
  100.0%  [1]          g_disk_start
   100.0%  [1]           g_io_request

00.35%  [1]        turnstile_trywait @ /boot/kernel/kernel
 100.0%  [1]         __mtx_lock_sleep
  100.0%  [1]          relpbuf
   100.0%  [1]           physio

00.35%  [1]        g_is_geom_thread @ /boot/kernel/kernel
 100.0%  [1]         g_io_deliver
  100.0%  [1]          g_disk_done_single
   100.0%  [1]           dadone

00.35%  [1]        userret @ /boot/kernel/kernel
 100.0%  [1]         amd64_syscall

00.35%  [1]        xpt_run_devq @ /boot/kernel/kernel
 100.0%  [1]         xpt_action_default
  100.0%  [1]          dastart
   100.0%  [1]           xpt_run_allocq

00.35%  [1]        sched_pickcpu @ /boot/kernel/kernel
 100.0%  [1]         sched_add
  100.0%  [1]          setrunnable
   100.0%  [1]           sleepq_broadcast

00.35%  [1]        intr_event_handle @ /boot/kernel/kernel
 100.0%  [1]         intr_execute_handlers
  100.0%  [1]          lapic_handle_intr

00.35%  [1]        bounce_bus_dmamap_load_buffer @ /boot/kernel/kernel
 100.0%  [1]         bus_dmamap_load
  100.0%  [1]          mrsas_map_request @ /boot/kernel/mrsas.ko
   100.0%  [1]           mrsas_build_dcdb

00.35%  [1]        sched_priority @ /boot/kernel/kernel
 100.0%  [1]         sched_add
  100.0%  [1]          setrunnable
   100.0%  [1]           sleepq_broadcast

00.35%  [1]        mrsas_cmd_done @ /boot/kernel/mrsas.ko
 100.0%  [1]         mrsas_complete_cmd
  100.0%  [1]          intr_event_execute_handlers @ /boot/kernel/kernel
   100.0%  [1]           ithread_loop

00.35%  [1]        biodone @ /boot/kernel/kernel
 100.0%  [1]         dadone
  100.0%  [1]          xpt_done_process
   100.0%  [1]           xpt_done_td

@ CPU_CLK_UNHALTED_CORE [299 samples]

12.37%  [37]       cpu_idle @ /boot/kernel/kernel
 100.0%  [37]        sched_idletd
  100.0%  [37]         fork_exit

10.37%  [31]       invlrng_handler @ /boot/kernel/kernel

10.03%  [30]       Xinvlrng @ /boot/kernel/kernel

06.35%  [19]       smp_tlb_shootdown @ /boot/kernel/kernel
 100.0%  [19]        pmap_invalidate_range
  100.0%  [19]         biodone
   100.0%  [19]          dadone

04.68%  [14]       cpu_search_highest @ /boot/kernel/kernel
 78.57%  [11]        cpu_search_highest
  63.64%  [7]          sched_idletd
   100.0%  [7]           fork_exit
  36.36%  [4]          cpu_search_highest
   100.0%  [4]           sched_idletd
 21.43%  [3]         sched_idletd
  100.0%  [3]          fork_exit

03.01%  [9]        cpu_switch @ /boot/kernel/kernel
 100.0%  [9]         mi_switch
  66.67%  [6]          sched_idletd
   100.0%  [6]           fork_exit
  22.22%  [2]          sleepq_wait
   100.0%  [2]           _sleep
  11.11%  [1]          critical_exit
   100.0%  [1]           intr_event_handle

03.01%  [9]        cpu_search_lowest @ /boot/kernel/kernel
 100.0%  [9]         cpu_search_lowest
  88.89%  [8]          cpu_search_lowest
   100.0%  [8]           sched_pickcpu
  11.11%  [1]          sched_pickcpu
   100.0%  [1]           sched_add

02.34%  [7]        __mtx_lock_sleep @ /boot/kernel/kernel
 28.57%  [2]         __mtx_lock_flags
  100.0%  [2]          mrsas_cmd_done @ /boot/kernel/mrsas.ko
   100.0%  [2]           mrsas_complete_cmd
 14.29%  [1]         vmem_xfree @ /boot/kernel/kernel
  100.0%  [1]          biodone
   100.0%  [1]           dadone
 14.29%  [1]         dadone
  100.0%  [1]          xpt_done_process
   100.0%  [1]           xpt_done_td
 14.29%  [1]         getpbuf
  100.0%  [1]          physio
   100.0%  [1]           devfs_read_f
 14.29%  [1]         _sleep
  100.0%  [1]          bwait
   100.0%  [1]           physio
 14.29%  [1]         vmem_xalloc
  100.0%  [1]          vmem_alloc
   100.0%  [1]           g_io_check

02.01%  [6]        sched_switch @ /boot/kernel/kernel
 100.0%  [6]         mi_switch
  33.33%  [2]          sleepq_wait
   100.0%  [2]           _sleep
  33.33%  [2]          sched_idletd
   100.0%  [2]           fork_exit
  16.67%  [1]          turnstile_wait
   100.0%  [1]           __mtx_lock_sleep
  16.67%  [1]          ithread_loop
   100.0%  [1]           fork_exit

01.67%  [5]        thread_lock_flags_ @ /boot/kernel/kernel
 40.00%  [2]         ithread_loop
  100.0%  [2]          fork_exit
 40.00%  [2]         sched_idletd
  100.0%  [2]          fork_exit
 20.00%  [1]         intr_event_schedule_thread
  100.0%  [1]          intr_event_handle
   100.0%  [1]           intr_execute_handlers

01.67%  [5]        vmem_xalloc @ /boot/kernel/kernel
 100.0%  [5]         vmem_alloc
  100.0%  [5]          g_io_check
   100.0%  [5]           g_io_request

01.67%  [5]        xpt_run_devq @ /boot/kernel/kernel
 60.00%  [3]         xpt_action_default
  100.0%  [3]          dastart
   100.0%  [3]           xpt_run_allocq
 40.00%  [2]         xpt_done_process
  100.0%  [2]          xpt_done_td
   100.0%  [2]           fork_exit

01.67%  [5]        __mtx_lock_flags @ /boot/kernel/kernel
 40.00%  [2]         mrsas_fire_cmd @ /boot/kernel/mrsas.ko
  100.0%  [2]          xpt_run_devq @ /boot/kernel/kernel
   100.0%  [2]           xpt_action_default
 40.00%  [2]         mrsas_action @ /boot/kernel/mrsas.ko
  100.0%  [2]          xpt_run_devq @ /boot/kernel/kernel
   100.0%  [2]           xpt_action_default
 20.00%  [1]         mrsas_get_mpt_cmd @ /boot/kernel/mrsas.ko
  100.0%  [1]          mrsas_action
   100.0%  [1]           xpt_run_devq @ /boot/kernel/kernel

01.34%  [4]        xpt_done_process @ /boot/kernel/kernel
 100.0%  [4]         xpt_done_td
  100.0%  [4]          fork_exit

01.34%  [4]        ithread_loop @ /boot/kernel/kernel
 100.0%  [4]         fork_exit

01.34%  [4]        sched_idletd @ /boot/kernel/kernel
 100.0%  [4]         fork_exit

01.34%  [4]        Xapic_isr1 @ /boot/kernel/kernel

01.34%  [4]        spinlock_exit @ /boot/kernel/kernel
 75.00%  [3]         sched_idletd
  100.0%  [3]          fork_exit
 25.00%  [1]         smp_tlb_shootdown
  100.0%  [1]          pmap_invalidate_range
   100.0%  [1]           biodone

01.00%  [3]        __lockmgr_args @ /boot/kernel/kernel
 66.67%  [2]         relpbuf
  100.0%  [2]          physio
   100.0%  [2]           devfs_read_f
 33.33%  [1]         getpbuf
  100.0%  [1]          physio
   100.0%  [1]           devfs_read_f

01.00%  [3]        vmem_xfree @ /boot/kernel/kernel
 100.0%  [3]         biodone
  100.0%  [3]          dadone
   100.0%  [3]           xpt_done_process

01.00%  [3]        kern_writev @ /boot/kernel/kernel
 100.0%  [3]         sys_write
  100.0%  [3]          amd64_syscall

01.00%  [3]        doreti @ /boot/kernel/kernel

01.00%  [3]        dastrategy @ /boot/kernel/kernel
 100.0%  [3]         g_disk_start
  100.0%  [3]          g_io_request
   100.0%  [3]           dev_strategy_csw

01.00%  [3]        __sys_read @ /lib/libc.so.7

01.00%  [3]        getpbuf @ /boot/kernel/kernel
 100.0%  [3]         physio
  100.0%  [3]          devfs_read_f
   100.0%  [3]           dofileread

01.00%  [3]        physio @ /boot/kernel/kernel
 100.0%  [3]         devfs_read_f
  100.0%  [3]          dofileread
   100.0%  [3]           kern_readv

01.00%  [3]        _mtx_lock_spin_cookie @ /boot/kernel/kernel
 33.33%  [1]         smp_tlb_shootdown
  100.0%  [1]          pmap_invalidate_range
   100.0%  [1]           biodone
 33.33%  [1]         sched_add
  100.0%  [1]          setrunnable
   100.0%  [1]           sleepq_broadcast
 33.33%  [1]         sched_idletd
  100.0%  [1]          fork_exit

01.00%  [3]        g_trace @ /boot/kernel/kernel
 33.33%  [1]         g_io_request
  100.0%  [1]          dev_strategy_csw
   100.0%  [1]           physio
 33.33%  [1]         g_io_deliver
  100.0%  [1]          g_disk_done_single
   100.0%  [1]           dadone
 33.33%  [1]         g_dev_strategy
  100.0%  [1]          dev_strategy_csw
   100.0%  [1]           physio

01.00%  [3]        _callout_stop_safe @ /boot/kernel/kernel
 100.0%  [3]         mrsas_cmd_done @ /boot/kernel/mrsas.ko
  100.0%  [3]          mrsas_complete_cmd
   100.0%  [3]           intr_event_execute_handlers @ /boot/kernel/kernel

00.67%  [2]        atomic_fetchadd_int @ /boot/kernel/kernel
 50.00%  [1]         mrsas_action @ /boot/kernel/mrsas.ko
  100.0%  [1]          xpt_run_devq @ /boot/kernel/kernel
   100.0%  [1]           xpt_action_default
 50.00%  [1]         mrsas_complete_cmd @ /boot/kernel/mrsas.ko
  100.0%  [1]          intr_event_execute_handlers @ /boot/kernel/kernel
   100.0%  [1]           ithread_loop

00.67%  [2]        free @ /boot/kernel/kernel
 100.0%  [2]         xpt_release_ccb
  100.0%  [2]          dadone
   100.0%  [2]           xpt_done_process

00.67%  [2]        kern_readv @ /boot/kernel/kernel
 100.0%  [2]         sys_read
  100.0%  [2]          amd64_syscall

00.67%  [2]        lock_mtx @ /boot/kernel/kernel
 100.0%  [2]         _sleep
  100.0%  [2]          bwait
   100.0%  [2]           physio

00.67%  [2]        __mtx_unlock_flags @ /boot/kernel/kernel
 50.00%  [1]         mrsas_cmd_done @ /boot/kernel/mrsas.ko
  100.0%  [1]          mrsas_complete_cmd
   100.0%  [1]           intr_event_execute_handlers @ /boot/kernel/kernel
 50.00%  [1]         xpt_run_devq
  100.0%  [1]          xpt_action_default
   100.0%  [1]           dastart

00.67%  [2]        dadone @ /boot/kernel/kernel
 100.0%  [2]         xpt_done_process
  100.0%  [2]          xpt_done_td
   100.0%  [2]           fork_exit

00.67%  [2]        Xfast_syscall @ /boot/kernel/kernel

00.67%  [2]        amd64_syscall @ /boot/kernel/kernel

00.67%  [2]        sched_add @ /boot/kernel/kernel
 100.0%  [2]         setrunnable
  100.0%  [2]          sleepq_broadcast
   100.0%  [2]           wakeup

00.67%  [2]        g_io_check @ /boot/kernel/kernel
 100.0%  [2]         g_io_request
  100.0%  [2]          dev_strategy_csw
   100.0%  [2]           physio

00.67%  [2]        pmap_extract_and_hold @ /boot/kernel/kernel
 100.0%  [2]         vm_fault_quick_hold_pages
  100.0%  [2]          vmapbuf
   100.0%  [2]           physio

00.67%  [2]        critical_enter @ /boot/kernel/kernel
 50.00%  [1]         thread_lock_flags_
  100.0%  [1]          sleepq_add
   100.0%  [1]           _sleep
 50.00%  [1]         uma_zalloc_arg
  100.0%  [1]          dev_strategy_csw
   100.0%  [1]           physio

00.67%  [2]        lapic_ipi_vectored @ /boot/kernel/kernel
 100.0%  [2]         smp_tlb_shootdown
  100.0%  [2]          pmap_invalidate_range
   100.0%  [2]           biodone

00.67%  [2]        setrunnable @ /boot/kernel/kernel
 100.0%  [2]         sleepq_broadcast
  100.0%  [2]          wakeup
   100.0%  [2]           bdone

00.67%  [2]        callout_lock @ /boot/kernel/kernel
 100.0%  [2]         callout_reset_sbt_on
  100.0%  [2]          mrsas_action @ /boot/kernel/mrsas.ko
   100.0%  [2]           xpt_run_devq @ /boot/kernel/kernel

00.33%  [1]        devfs_read_f @ /boot/kernel/kernel
 100.0%  [1]         dofileread
  100.0%  [1]          kern_readv
   100.0%  [1]           sys_read

00.33%  [1]        sleepq_wait @ /boot/kernel/kernel
 100.0%  [1]         _sleep
  100.0%  [1]          bwait
   100.0%  [1]           physio

00.33%  [1]        mrsas_map_request @ /boot/kernel/mrsas.ko
 100.0%  [1]         mrsas_build_dcdb
  100.0%  [1]          mrsas_action
   100.0%  [1]           xpt_run_devq @ /boot/kernel/kernel

00.33%  [1]        fget_read @ /boot/kernel/kernel
 100.0%  [1]         kern_readv
  100.0%  [1]          sys_read
   100.0%  [1]           amd64_syscall

00.33%  [1]        xpt_run_allocq @ /boot/kernel/kernel
 100.0%  [1]         dastrategy
  100.0%  [1]          g_disk_start
   100.0%  [1]           g_io_request

00.33%  [1]        bwait @ /boot/kernel/kernel
 100.0%  [1]         physio
  100.0%  [1]          devfs_read_f
   100.0%  [1]           dofileread

00.33%  [1]        devstat_end_transaction @ /boot/kernel/kernel
 100.0%  [1]         devstat_end_transaction_bio_bt
  100.0%  [1]          g_disk_done_single
   100.0%  [1]           dadone

00.33%  [1]        vm_fault_quick_hold_pages @ /boot/kernel/kernel
 100.0%  [1]         vmapbuf
  100.0%  [1]          physio
   100.0%  [1]           devfs_read_f

00.33%  [1]        dev_strategy_csw @ /boot/kernel/kernel
 100.0%  [1]         physio
  100.0%  [1]          devfs_read_f
   100.0%  [1]           dofileread

00.33%  [1]        xpt_done @ /boot/kernel/kernel
 100.0%  [1]         mrsas_cmd_done @ /boot/kernel/mrsas.ko
  100.0%  [1]          mrsas_complete_cmd
   100.0%  [1]           intr_event_execute_handlers @ /boot/kernel/kernel

00.33%  [1]        sleepq_switch @ /boot/kernel/kernel
 100.0%  [1]         sleepq_wait
  100.0%  [1]          _sleep
   100.0%  [1]           bwait

00.33%  [1]        PHYS_TO_VM_PAGE @ /boot/kernel/kernel
 100.0%  [1]         free
  100.0%  [1]          xpt_release_ccb
   100.0%  [1]           dadone

00.33%  [1]        _fget @ /boot/kernel/kernel
 100.0%  [1]         kern_readv
  100.0%  [1]          sys_read
   100.0%  [1]           amd64_syscall

00.33%  [1]        g_is_geom_thread @ /boot/kernel/kernel
 100.0%  [1]         g_io_deliver
  100.0%  [1]          g_disk_done_single
   100.0%  [1]           dadone

00.33%  [1]        g_io_request @ /boot/kernel/kernel
 100.0%  [1]         dev_strategy_csw
  100.0%  [1]          physio
   100.0%  [1]           devfs_read_f

00.33%  [1]        xpt_release_ccb @ /boot/kernel/kernel
 100.0%  [1]         dadone
  100.0%  [1]          xpt_done_process
   100.0%  [1]           xpt_done_td

00.33%  [1]        __cap_rights_init @ /boot/kernel/kernel
 100.0%  [1]         kern_writev
  100.0%  [1]          sys_write
   100.0%  [1]           amd64_syscall

00.33%  [1]        dastart @ /boot/kernel/kernel
 100.0%  [1]         xpt_run_allocq
  100.0%  [1]          dastrategy
   100.0%  [1]           g_disk_start

00.33%  [1]        callout_cc_add @ /boot/kernel/kernel
 100.0%  [1]         callout_reset_sbt_on
  100.0%  [1]          mrsas_action @ /boot/kernel/mrsas.ko
   100.0%  [1]           xpt_run_devq @ /boot/kernel/kernel

00.33%  [1]        mrsas_action @ /boot/kernel/mrsas.ko
 100.0%  [1]         xpt_run_devq @ /boot/kernel/kernel
  100.0%  [1]          xpt_action_default
   100.0%  [1]           dastart

00.33%  [1]        userret @ /boot/kernel/kernel
 100.0%  [1]         amd64_syscall

00.33%  [1]        vm_page_unhold_pages @ /boot/kernel/kernel
 100.0%  [1]         vunmapbuf
  100.0%  [1]          physio
   100.0%  [1]           devfs_read_f

00.33%  [1]        __sys_write @ /lib/libc.so.7

00.33%  [1]        bufdone @ /boot/kernel/kernel
 100.0%  [1]         bufdonebio
  100.0%  [1]          g_io_deliver
   100.0%  [1]           g_disk_done_single

00.33%  [1]        dofilewrite @ /boot/kernel/kernel
 100.0%  [1]         kern_writev
  100.0%  [1]          sys_write
   100.0%  [1]           amd64_syscall

00.33%  [1]        sched_pickcpu @ /boot/kernel/kernel
 100.0%  [1]         sched_add
  100.0%  [1]          intr_event_schedule_thread
   100.0%  [1]           intr_event_handle

00.33%  [1]        mrsas_data_load_cb @ /boot/kernel/mrsas.ko
 100.0%  [1]         bus_dmamap_load @ /boot/kernel/kernel
  100.0%  [1]          mrsas_map_request @ /boot/kernel/mrsas.ko
   100.0%  [1]           mrsas_build_dcdb

00.33%  [1]        g_disk_start @ /boot/kernel/kernel
 100.0%  [1]         g_io_request
  100.0%  [1]          dev_strategy_csw
   100.0%  [1]           physio

00.33%  [1]        g_disk_done_single @ /boot/kernel/kernel
 100.0%  [1]         dadone
  100.0%  [1]          xpt_done_process
   100.0%  [1]           xpt_done_td

00.33%  [1]        mrsas_cmd_done @ /boot/kernel/mrsas.ko
 100.0%  [1]         mrsas_complete_cmd
  100.0%  [1]          intr_event_execute_handlers @ /boot/kernel/kernel
   100.0%  [1]           ithread_loop

00.33%  [1]        g_dev_done @ /boot/kernel/kernel
 100.0%  [1]         g_io_deliver
  100.0%  [1]          g_disk_done_single
   100.0%  [1]           dadone

00.33%  [1]        runq_add @ /boot/kernel/kernel
 100.0%  [1]         sched_add
  100.0%  [1]          intr_event_schedule_thread
   100.0%  [1]           intr_event_handle

00.33%  [1]        critical_exit @ /boot/kernel/kernel
 100.0%  [1]         spinlock_exit
  100.0%  [1]          thread_lock_block
   100.0%  [1]           sched_switch

00.33%  [1]        sleepq_lock @ /boot/kernel/kernel
 100.0%  [1]         wakeup
  100.0%  [1]          bdone
   100.0%  [1]           bufdone

@ CPU_CLK_UNHALTED_CORE [294 samples]

13.95%  [41]       Xinvlrng @ /boot/kernel/kernel

09.86%  [29]       cpu_idle @ /boot/kernel/kernel
 100.0%  [29]        sched_idletd
  100.0%  [29]         fork_exit

08.50%  [25]       invlrng_handler @ /boot/kernel/kernel

05.78%  [17]       smp_tlb_shootdown @ /boot/kernel/kernel
 100.0%  [17]        pmap_invalidate_range
  100.0%  [17]         biodone
   100.0%  [17]          dadone

03.74%  [11]       cpu_switch @ /boot/kernel/kernel
 100.0%  [11]        mi_switch
  81.82%  [9]          sched_idletd
   100.0%  [9]           fork_exit
  18.18%  [2]          sleepq_wait
   100.0%  [2]           _sleep

03.40%  [10]       cpu_search_highest @ /boot/kernel/kernel
 80.00%  [8]         cpu_search_highest
  50.00%  [4]          cpu_search_highest
   100.0%  [4]           sched_idletd
  50.00%  [4]          sched_idletd
   100.0%  [4]           fork_exit
 20.00%  [2]         sched_idletd
  100.0%  [2]          fork_exit

02.72%  [8]        cpu_search_lowest @ /boot/kernel/kernel
 100.0%  [8]         cpu_search_lowest
  87.50%  [7]          cpu_search_lowest
   100.0%  [7]           sched_pickcpu
  12.50%  [1]          sched_pickcpu
   100.0%  [1]           sched_add

02.72%  [8]        __mtx_lock_sleep @ /boot/kernel/kernel
 75.00%  [6]         __mtx_lock_flags
  83.33%  [5]          mrsas_cmd_done @ /boot/kernel/mrsas.ko
   100.0%  [5]           mrsas_complete_cmd
  16.67%  [1]          mrsas_unmap_request
   100.0%  [1]           mrsas_cmd_done
 12.50%  [1]         xpt_run_devq @ /boot/kernel/kernel
  100.0%  [1]          xpt_action_default
   100.0%  [1]           dastart
 12.50%  [1]         vmem_xfree
  100.0%  [1]          biodone
   100.0%  [1]           dadone

02.72%  [8]        sched_switch @ /boot/kernel/kernel
 100.0%  [8]         mi_switch
  37.50%  [3]          sleepq_wait
   100.0%  [3]           _sleep
  37.50%  [3]          ithread_loop
   100.0%  [3]           fork_exit
  25.00%  [2]          sched_idletd
   100.0%  [2]           fork_exit

02.72%  [8]        spinlock_exit @ /boot/kernel/kernel
 62.50%  [5]         sched_idletd
  100.0%  [5]          fork_exit
 12.50%  [1]         _mtx_lock_spin_cookie
  100.0%  [1]          smp_tlb_shootdown
   100.0%  [1]           pmap_invalidate_range
 12.50%  [1]         _sleep
  100.0%  [1]          bwait
   100.0%  [1]           physio
 12.50%  [1]         callout_reset_sbt_on
  100.0%  [1]          mrsas_action @ /boot/kernel/mrsas.ko
   100.0%  [1]           xpt_run_devq @ /boot/kernel/kernel

02.04%  [6]        __sys_read @ /lib/libc.so.7

01.36%  [4]        g_dev_strategy @ /boot/kernel/kernel
 100.0%  [4]         dev_strategy_csw
  100.0%  [4]          physio
   100.0%  [4]           devfs_read_f

01.36%  [4]        sched_idletd @ /boot/kernel/kernel
 100.0%  [4]         fork_exit

01.36%  [4]        xpt_done_process @ /boot/kernel/kernel
 100.0%  [4]         xpt_done_td
  100.0%  [4]          fork_exit

01.36%  [4]        Xapic_isr1 @ /boot/kernel/kernel

01.36%  [4]        __mtx_lock_flags @ /boot/kernel/kernel
 50.00%  [2]         mrsas_action @ /boot/kernel/mrsas.ko
  100.0%  [2]          xpt_run_devq @ /boot/kernel/kernel
   100.0%  [2]           xpt_action_default
 25.00%  [1]         mrsas_get_mpt_cmd @ /boot/kernel/mrsas.ko
  100.0%  [1]          mrsas_action
   100.0%  [1]           xpt_run_devq @ /boot/kernel/kernel
 25.00%  [1]         mrsas_unmap_request @ /boot/kernel/mrsas.ko
  100.0%  [1]          mrsas_cmd_done
   100.0%  [1]           mrsas_complete_cmd

01.02%  [3]        sched_pickcpu @ /boot/kernel/kernel
 100.0%  [3]         sched_add
  66.67%  [2]          intr_event_schedule_thread
   100.0%  [2]           intr_event_handle
  33.33%  [1]          turnstile_unpend
   100.0%  [1]           __mtx_unlock_sleep

01.02%  [3]        _mtx_lock_spin_cookie @ /boot/kernel/kernel
 66.67%  [2]         turnstile_trywait
  100.0%  [2]          __mtx_lock_sleep
   50.00%  [1]           xpt_done_process
   50.00%  [1]           vmem_xalloc
 33.33%  [1]         sched_add
  100.0%  [1]          intr_event_schedule_thread
   100.0%  [1]           intr_event_handle

01.02%  [3]        lapic_ipi_vectored @ /boot/kernel/kernel
 100.0%  [3]         smp_tlb_shootdown
  100.0%  [3]          pmap_invalidate_range
   100.0%  [3]           biodone

01.02%  [3]        uma_zfree_arg @ /boot/kernel/kernel
 100.0%  [3]         free
  100.0%  [3]          xpt_release_ccb
   100.0%  [3]           dadone

01.02%  [3]        thread_lock_flags_ @ /boot/kernel/kernel
 66.67%  [2]         intr_event_schedule_thread
  100.0%  [2]          intr_event_handle
   100.0%  [2]           intr_execute_handlers
 33.33%  [1]         sleepq_wait
  100.0%  [1]          _sleep
   100.0%  [1]           bwait

01.02%  [3]        mrsas_action @ /boot/kernel/mrsas.ko
 100.0%  [3]         xpt_run_devq @ /boot/kernel/kernel
  100.0%  [3]          xpt_action_default
   100.0%  [3]           dastart

01.02%  [3]        xpt_run_allocq @ /boot/kernel/kernel
 100.0%  [3]         dastrategy
  100.0%  [3]          g_disk_start
   100.0%  [3]           g_io_request

01.02%  [3]        __mtx_unlock_flags @ /boot/kernel/kernel
 66.67%  [2]         xpt_run_devq
  100.0%  [2]          xpt_action_default
   100.0%  [2]           dastart
 33.33%  [1]         mrsas_cmd_done @ /boot/kernel/mrsas.ko
  100.0%  [1]          mrsas_complete_cmd
   100.0%  [1]           intr_event_execute_handlers @ /boot/kernel/kernel

01.02%  [3]        vmem_xalloc @ /boot/kernel/kernel
 100.0%  [3]         vmem_alloc
  100.0%  [3]          g_io_check
   100.0%  [3]           g_io_request

01.02%  [3]        dastart @ /boot/kernel/kernel
 100.0%  [3]         xpt_run_allocq
  100.0%  [3]          dastrategy
   100.0%  [3]           g_disk_start

01.02%  [3]        amd64_syscall @ /boot/kernel/kernel

01.02%  [3]        xpt_run_devq @ /boot/kernel/kernel
 100.0%  [3]         xpt_action_default
  100.0%  [3]          dastart
   100.0%  [3]           xpt_run_allocq

00.68%  [2]        sched_choose @ /boot/kernel/kernel
 100.0%  [2]         choosethread
  100.0%  [2]          sched_switch
   100.0%  [2]           mi_switch

00.68%  [2]        doreti @ /boot/kernel/kernel

00.68%  [2]        scsi_read_write @ /boot/kernel/kernel
 100.0%  [2]         dastart
  100.0%  [2]          xpt_run_allocq
   100.0%  [2]           dastrategy

00.68%  [2]        _mtx_trylock_flags_ @ /boot/kernel/kernel
 100.0%  [2]         vm_page_pa_tryrelock
  100.0%  [2]          pmap_extract_and_hold
   100.0%  [2]           vm_fault_quick_hold_pages

00.68%  [2]        _fget @ /boot/kernel/kernel
 50.00%  [1]         kern_writev
  100.0%  [1]          sys_write
   100.0%  [1]           amd64_syscall
 50.00%  [1]         kern_readv
  100.0%  [1]          sys_read
   100.0%  [1]           amd64_syscall

00.68%  [2]        physio @ /boot/kernel/kernel
 100.0%  [2]         devfs_read_f
  100.0%  [2]          dofileread
   100.0%  [2]           kern_readv

00.68%  [2]        runq_choose @ /boot/kernel/kernel
 100.0%  [2]         sched_choose
  100.0%  [2]          choosethread
   100.0%  [2]           sched_switch

00.68%  [2]        bcopy @ /boot/kernel/kernel
 50.00%  [1]         xpt_run_devq
  100.0%  [1]          xpt_action_default
   100.0%  [1]           dastart
 50.00%  [1]         amd64_syscall

00.68%  [2]        vmem_xfree @ /boot/kernel/kernel
 100.0%  [2]         biodone
  100.0%  [2]          dadone
   100.0%  [2]           xpt_done_process

00.68%  [2]        bt_freetrim @ /boot/kernel/kernel
 100.0%  [2]         biodone
  100.0%  [2]          dadone
   100.0%  [2]           xpt_done_process

00.68%  [2]        devstat_start_transaction_bio @ /boot/kernel/kernel
 100.0%  [2]         g_disk_start
  100.0%  [2]          g_io_request
   100.0%  [2]           dev_strategy_csw

00.68%  [2]        bzero @ /boot/kernel/kernel
 50.00%  [1]         mrsas_action @ /boot/kernel/mrsas.ko
  100.0%  [1]          xpt_run_devq @ /boot/kernel/kernel
   100.0%  [1]           xpt_action_default
 50.00%  [1]         __cap_rights_init
  100.0%  [1]          kern_readv
   100.0%  [1]           sys_read

00.68%  [2]        spinlock_enter @ /boot/kernel/kernel
 50.00%  [1]         sched_idletd
  100.0%  [1]          fork_exit
 50.00%  [1]         callout_lock
  100.0%  [1]          _callout_stop_safe
   100.0%  [1]           mrsas_cmd_done @ /boot/kernel/mrsas.ko

00.68%  [2]        sleepq_lock @ /boot/kernel/kernel
 100.0%  [2]         _sleep
  100.0%  [2]          bwait
   100.0%  [2]           physio

00.68%  [2]        critical_exit @ /boot/kernel/kernel
 100.0%  [2]         spinlock_exit
  50.00%  [1]          turnstile_wait
   100.0%  [1]           __mtx_lock_sleep
  50.00%  [1]          intr_event_schedule_thread
   100.0%  [1]           intr_event_handle

00.68%  [2]        pmap_invalidate_range @ /boot/kernel/kernel
 100.0%  [2]         biodone
  100.0%  [2]          dadone
   100.0%  [2]           xpt_done_process

00.68%  [2]        dastrategy @ /boot/kernel/kernel
 100.0%  [2]         g_disk_start
  100.0%  [2]          g_io_request
   100.0%  [2]           dev_strategy_csw

00.34%  [1]        tdq_notify @ /boot/kernel/kernel
 100.0%  [1]         sched_add
  100.0%  [1]          setrunnable
   100.0%  [1]           sleepq_broadcast

00.34%  [1]        intr_execute_handlers @ /boot/kernel/kernel
 100.0%  [1]         lapic_handle_intr

00.34%  [1]        ithread_loop @ /boot/kernel/kernel
 100.0%  [1]         fork_exit

00.34%  [1]        mrsas_get_mpt_cmd @ /boot/kernel/mrsas.ko
 100.0%  [1]         mrsas_action
  100.0%  [1]          xpt_run_devq @ /boot/kernel/kernel
   100.0%  [1]           xpt_action_default

00.34%  [1]        setrunnable @ /boot/kernel/kernel
 100.0%  [1]         sleepq_broadcast
  100.0%  [1]          wakeup
   100.0%  [1]           bdone

00.34%  [1]        mi_switch @ /boot/kernel/kernel
 100.0%  [1]         sleepq_wait
  100.0%  [1]          _sleep
   100.0%  [1]           bwait

00.34%  [1]        ipi_bitmap_handler @ /boot/kernel/kernel

00.34%  [1]        devfs_read_f @ /boot/kernel/kernel
 100.0%  [1]         dofileread
  100.0%  [1]          kern_readv
   100.0%  [1]           sys_read

00.34%  [1]        g_disk_done_single @ /boot/kernel/kernel
 100.0%  [1]         dadone
  100.0%  [1]          xpt_done_process
   100.0%  [1]           xpt_done_td

00.34%  [1]        kern_readv @ /boot/kernel/kernel
 100.0%  [1]         sys_read
  100.0%  [1]          amd64_syscall

00.34%  [1]        sched_runnable @ /boot/kernel/kernel
 100.0%  [1]         cpu_idle
  100.0%  [1]          sched_idletd
   100.0%  [1]           fork_exit

00.34%  [1]        __lockmgr_args @ /boot/kernel/kernel
 100.0%  [1]         relpbuf
  100.0%  [1]          physio
   100.0%  [1]           devfs_read_f

00.34%  [1]        vm_page_unhold_pages @ /boot/kernel/kernel
 100.0%  [1]         vunmapbuf
  100.0%  [1]          physio
   100.0%  [1]           devfs_read_f

00.34%  [1]        __cap_rights_is_set @ /boot/kernel/kernel
 100.0%  [1]         fget_unlocked
  100.0%  [1]          _fget
   100.0%  [1]           kern_readv

00.34%  [1]        bounce_bus_dmamap_load_buffer @ /boot/kernel/kernel
 100.0%  [1]         bus_dmamap_load
  100.0%  [1]          mrsas_map_request @ /boot/kernel/mrsas.ko
   100.0%  [1]           mrsas_build_dcdb

00.34%  [1]        mrsas_complete_cmd @ /boot/kernel/mrsas.ko
 100.0%  [1]         intr_event_execute_handlers @ /boot/kernel/kernel
  100.0%  [1]          ithread_loop
   100.0%  [1]           fork_exit

00.34%  [1]        free @ /boot/kernel/kernel
 100.0%  [1]         xpt_release_ccb
  100.0%  [1]          dadone
   100.0%  [1]           xpt_done_process

00.34%  [1]        intr_event_schedule_thread @ /boot/kernel/kernel
 100.0%  [1]         intr_event_handle
  100.0%  [1]          intr_execute_handlers
   100.0%  [1]           lapic_handle_intr

00.34%  [1]        mrsas_cmd_done @ /boot/kernel/mrsas.ko
 100.0%  [1]         mrsas_complete_cmd
  100.0%  [1]          intr_event_execute_handlers @ /boot/kernel/kernel
   100.0%  [1]           ithread_loop

00.34%  [1]        callout_lock @ /boot/kernel/kernel
 100.0%  [1]         _callout_stop_safe
  100.0%  [1]          mrsas_cmd_done @ /boot/kernel/mrsas.ko
   100.0%  [1]           mrsas_complete_cmd

00.34%  [1]        bioq_takefirst @ /boot/kernel/kernel
 100.0%  [1]         dastart
  100.0%  [1]          xpt_run_allocq
   100.0%  [1]           dastrategy

00.34%  [1]        foffset_unlock @ /boot/kernel/kernel
 100.0%  [1]         devfs_write_f
  100.0%  [1]          dofilewrite
   100.0%  [1]           kern_writev

00.34%  [1]        dadone @ /boot/kernel/kernel
 100.0%  [1]         xpt_done_process
  100.0%  [1]          xpt_done_td
   100.0%  [1]           fork_exit

00.34%  [1]        getpbuf @ /boot/kernel/kernel
 100.0%  [1]         physio
  100.0%  [1]          devfs_read_f
   100.0%  [1]           dofileread

00.34%  [1]        Xtimerint @ /boot/kernel/kernel

00.34%  [1]        sleepq_add @ /boot/kernel/kernel
 100.0%  [1]         _sleep
  100.0%  [1]          bwait
   100.0%  [1]           physio

00.34%  [1]        relpbuf @ /boot/kernel/kernel
 100.0%  [1]         physio
  100.0%  [1]          devfs_read_f
   100.0%  [1]           dofileread

00.34%  [1]        devstat_end_transaction_bio_bt @ /boot/kernel/kernel
 100.0%  [1]         g_io_deliver
  100.0%  [1]          g_disk_done_single
   100.0%  [1]           dadone

00.34%  [1]        fget_unlocked @ /boot/kernel/kernel
 100.0%  [1]         _fget
  100.0%  [1]          kern_writev
   100.0%  [1]           sys_write

00.34%  [1]        dev_strategy_csw @ /boot/kernel/kernel
 100.0%  [1]         physio
  100.0%  [1]          devfs_read_f
   100.0%  [1]           dofileread

00.34%  [1]        uma_zalloc_arg @ /boot/kernel/kernel
 100.0%  [1]         malloc
  100.0%  [1]          xpt_run_allocq
   100.0%  [1]           dastrategy

00.34%  [1]        camq_remove @ /boot/kernel/kernel
 100.0%  [1]         xpt_run_devq
  100.0%  [1]          xpt_action_default
   100.0%  [1]           dastart

00.34%  [1]        cpu_fetch_syscall_args @ /boot/kernel/kernel
 100.0%  [1]         amd64_syscall

@ CPU_CLK_UNHALTED_CORE [293 samples]

13.65%  [40]       cpu_idle @ /boot/kernel/kernel
 100.0%  [40]        sched_idletd
  100.0%  [40]         fork_exit

11.60%  [34]       Xinvlrng @ /boot/kernel/kernel

06.48%  [19]       smp_tlb_shootdown @ /boot/kernel/kernel
 100.0%  [19]        pmap_invalidate_range
  100.0%  [19]         biodone
   100.0%  [19]          dadone

06.14%  [18]       invlrng_handler @ /boot/kernel/kernel

03.07%  [9]        cpu_search_highest @ /boot/kernel/kernel
 88.89%  [8]         cpu_search_highest
  87.50%  [7]          sched_idletd
   100.0%  [7]           fork_exit
  12.50%  [1]          cpu_search_highest
   100.0%  [1]           sched_idletd
 11.11%  [1]         sched_idletd
  100.0%  [1]          fork_exit

02.73%  [8]        cpu_search_lowest @ /boot/kernel/kernel
 100.0%  [8]         cpu_search_lowest
  87.50%  [7]          cpu_search_lowest
   100.0%  [7]           sched_pickcpu
  12.50%  [1]          sched_pickcpu
   100.0%  [1]           sched_add

02.39%  [7]        __mtx_lock_sleep @ /boot/kernel/kernel
 85.71%  [6]         __mtx_lock_flags
  83.33%  [5]          mrsas_cmd_done @ /boot/kernel/mrsas.ko
   100.0%  [5]           mrsas_complete_cmd
  16.67%  [1]          mrsas_unmap_request
   100.0%  [1]           mrsas_cmd_done
 14.29%  [1]         vmem_xfree @ /boot/kernel/kernel
  100.0%  [1]          biodone
   100.0%  [1]           dadone

02.39%  [7]        cpu_switch @ /boot/kernel/kernel
 100.0%  [7]         mi_switch
  100.0%  [7]          sched_idletd
   100.0%  [7]           fork_exit

02.05%  [6]        sched_idletd @ /boot/kernel/kernel
 100.0%  [6]         fork_exit

02.05%  [6]        __sys_read @ /lib/libc.so.7

01.71%  [5]        __mtx_lock_flags @ /boot/kernel/kernel
 60.00%  [3]         mrsas_cmd_done @ /boot/kernel/mrsas.ko
  100.0%  [3]          mrsas_complete_cmd
   100.0%  [3]           intr_event_execute_handlers @ /boot/kernel/kernel
 20.00%  [1]         mrsas_get_mpt_cmd @ /boot/kernel/mrsas.ko
  100.0%  [1]          mrsas_action
   100.0%  [1]           xpt_run_devq @ /boot/kernel/kernel
 20.00%  [1]         mrsas_unmap_request @ /boot/kernel/mrsas.ko
  100.0%  [1]          mrsas_cmd_done
   100.0%  [1]           mrsas_complete_cmd

01.71%  [5]        spinlock_exit @ /boot/kernel/kernel
 20.00%  [1]         _sleep
  100.0%  [1]          bwait
   100.0%  [1]           physio
 20.00%  [1]         ithread_loop
  100.0%  [1]          fork_exit
 20.00%  [1]         vmem_xfree
  100.0%  [1]          biodone
   100.0%  [1]           dadone
 20.00%  [1]         intr_event_schedule_thread
  100.0%  [1]          intr_event_handle
   100.0%  [1]           intr_execute_handlers
 20.00%  [1]         sched_idletd
  100.0%  [1]          fork_exit

01.37%  [4]        thread_lock_flags_ @ /boot/kernel/kernel
 25.00%  [1]         sched_idletd
  100.0%  [1]          fork_exit
 25.00%  [1]         ithread_loop
  100.0%  [1]          fork_exit
 25.00%  [1]         critical_exit
  100.0%  [1]          intr_event_handle
   100.0%  [1]           intr_execute_handlers
 25.00%  [1]         sleepq_add
  100.0%  [1]          _sleep
   100.0%  [1]           bwait

01.37%  [4]        xpt_run_devq @ /boot/kernel/kernel
 100.0%  [4]         xpt_action_default
  100.0%  [4]          dastart
   100.0%  [4]           xpt_run_allocq

01.37%  [4]        pmap_kextract @ /boot/kernel/kernel
 100.0%  [4]         bounce_bus_dmamap_load_buffer
  100.0%  [4]          bus_dmamap_load
   100.0%  [4]           mrsas_map_request @ /boot/kernel/mrsas.ko

01.37%  [4]        sleepq_lock @ /boot/kernel/kernel
 50.00%  [2]         wakeup
  100.0%  [2]          bdone
   100.0%  [2]           bufdone
 25.00%  [1]         cv_broadcastpri
  100.0%  [1]          vmem_xfree
   100.0%  [1]           biodone
 25.00%  [1]         _sleep
  100.0%  [1]          bwait
   100.0%  [1]           physio

01.37%  [4]        __mtx_unlock_flags @ /boot/kernel/kernel
 100.0%  [4]         xpt_run_devq
  100.0%  [4]          xpt_action_default
   100.0%  [4]           dastart

01.37%  [4]        sched_switch @ /boot/kernel/kernel
 100.0%  [4]         mi_switch
  75.00%  [3]          sleepq_wait
   100.0%  [3]           _sleep
  25.00%  [1]          sched_idletd
   100.0%  [1]           fork_exit

01.02%  [3]        spinlock_enter @ /boot/kernel/kernel
 33.33%  [1]         sched_idletd
  100.0%  [1]          fork_exit
 33.33%  [1]         callout_lock
  100.0%  [1]          callout_reset_sbt_on
   100.0%  [1]           mrsas_action @ /boot/kernel/mrsas.ko
 33.33%  [1]         sleepq_lock @ /boot/kernel/kernel
  100.0%  [1]          _sleep
   100.0%  [1]           bwait

01.02%  [3]        critical_exit @ /boot/kernel/kernel
 66.67%  [2]         spinlock_exit
  50.00%  [1]          thread_lock_block
   100.0%  [1]           sched_add
  50.00%  [1]          intr_event_schedule_thread
   100.0%  [1]           intr_event_handle
 33.33%  [1]         intr_event_handle
  100.0%  [1]          intr_execute_handlers
   100.0%  [1]           lapic_handle_intr

01.02%  [3]        vmem_xfree @ /boot/kernel/kernel
 100.0%  [3]         biodone
  100.0%  [3]          dadone
   100.0%  [3]           xpt_done_process

01.02%  [3]        devstat_end_transaction @ /boot/kernel/kernel
 100.0%  [3]         devstat_end_transaction_bio_bt
  100.0%  [3]          g_disk_done_single
   100.0%  [3]           dadone

01.02%  [3]        lapic_ipi_vectored @ /boot/kernel/kernel
 100.0%  [3]         smp_tlb_shootdown
  100.0%  [3]          pmap_invalidate_range
   100.0%  [3]           biodone

01.02%  [3]        Xapic_isr1 @ /boot/kernel/kernel

00.68%  [2]        xpt_run_allocq @ /boot/kernel/kernel
 100.0%  [2]         dastrategy
  100.0%  [2]          g_disk_start
   100.0%  [2]           g_io_request

00.68%  [2]        g_dev_strategy @ /boot/kernel/kernel
 100.0%  [2]         dev_strategy_csw
  100.0%  [2]          physio
   100.0%  [2]           devfs_read_f

00.68%  [2]        physio @ /boot/kernel/kernel
 100.0%  [2]         devfs_read_f
  100.0%  [2]          dofileread
   100.0%  [2]           kern_readv

00.68%  [2]        g_io_check @ /boot/kernel/kernel
 100.0%  [2]         g_io_request
  100.0%  [2]          dev_strategy_csw
   100.0%  [2]           physio

00.68%  [2]        callout_lock @ /boot/kernel/kernel
 100.0%  [2]         _callout_stop_safe
  100.0%  [2]          mrsas_cmd_done @ /boot/kernel/mrsas.ko
   100.0%  [2]           mrsas_complete_cmd

00.68%  [2]        relpbuf @ /boot/kernel/kernel
 100.0%  [2]         physio
  100.0%  [2]          devfs_read_f
   100.0%  [2]           dofileread

00.68%  [2]        ithread_loop @ /boot/kernel/kernel
 100.0%  [2]         fork_exit

00.68%  [2]        tdq_move @ /boot/kernel/kernel
 100.0%  [2]         sched_idletd
  100.0%  [2]          fork_exit

00.68%  [2]        vmem_xalloc @ /boot/kernel/kernel
 100.0%  [2]         vmem_alloc
  100.0%  [2]          g_io_check
   100.0%  [2]           g_io_request

00.68%  [2]        mrsas_action @ /boot/kernel/mrsas.ko
 100.0%  [2]         xpt_run_devq @ /boot/kernel/kernel
  100.0%  [2]          xpt_action_default
   100.0%  [2]           dastart

00.68%  [2]        mrsas_isr @ /boot/kernel/mrsas.ko
 100.0%  [2]         intr_event_execute_handlers @ /boot/kernel/kernel
  100.0%  [2]          ithread_loop
   100.0%  [2]           fork_exit

00.68%  [2]        propagate_priority @ /boot/kernel/kernel
 100.0%  [2]         turnstile_wait
  100.0%  [2]          __mtx_lock_sleep
   50.00%  [1]           g_disk_start
   50.00%  [1]           relpbuf

00.68%  [2]        sched_pickcpu @ /boot/kernel/kernel
 100.0%  [2]         sched_add
  50.00%  [1]          setrunnable
   100.0%  [1]           sleepq_broadcast
  50.00%  [1]          intr_event_schedule_thread
   100.0%  [1]           intr_event_handle

00.68%  [2]        sched_choose @ /boot/kernel/kernel
 100.0%  [2]         choosethread
  100.0%  [2]          sched_switch
   100.0%  [2]           mi_switch

00.68%  [2]        g_dev_done @ /boot/kernel/kernel
 100.0%  [2]         g_io_deliver
  100.0%  [2]          g_disk_done_single
   100.0%  [2]           dadone

00.68%  [2]        doreti @ /boot/kernel/kernel

00.68%  [2]        mrsas_get_mpt_cmd @ /boot/kernel/mrsas.ko
 100.0%  [2]         mrsas_action
  100.0%  [2]          xpt_run_devq @ /boot/kernel/kernel
   100.0%  [2]           xpt_action_default

00.68%  [2]        dastrategy @ /boot/kernel/kernel
 100.0%  [2]         g_disk_start
  100.0%  [2]          g_io_request
   100.0%  [2]           dev_strategy_csw

00.68%  [2]        sleepq_broadcast @ /boot/kernel/kernel
 100.0%  [2]         wakeup
  100.0%  [2]          bdone
   100.0%  [2]           bufdone

00.68%  [2]        biodone @ /boot/kernel/kernel
 100.0%  [2]         dadone
  100.0%  [2]          xpt_done_process
   100.0%  [2]           xpt_done_td

00.34%  [1]        g_io_request @ /boot/kernel/kernel
 100.0%  [1]         dev_strategy_csw
  100.0%  [1]          physio
   100.0%  [1]           devfs_read_f

00.34%  [1]        camq_remove @ /boot/kernel/kernel
 100.0%  [1]         cam_ccbq_remove_ccb
  100.0%  [1]          xpt_run_devq
   100.0%  [1]           xpt_action_default

00.34%  [1]        cpu_fetch_syscall_args @ /boot/kernel/kernel
 100.0%  [1]         amd64_syscall

00.34%  [1]        mrsas_unmap_request @ /boot/kernel/mrsas.ko
 100.0%  [1]         mrsas_cmd_done
  100.0%  [1]          mrsas_complete_cmd
   100.0%  [1]           intr_event_execute_handlers @ /boot/kernel/kernel

00.34%  [1]        userret @ /boot/kernel/kernel
 100.0%  [1]         amd64_syscall

00.34%  [1]        mrsas_map_mpt_cmd_status @ /boot/kernel/mrsas.ko
 100.0%  [1]         mrsas_complete_cmd
  100.0%  [1]          intr_event_execute_handlers @ /boot/kernel/kernel
   100.0%  [1]           ithread_loop

00.34%  [1]        devvn_refthread @ /boot/kernel/kernel
 100.0%  [1]         devfs_write_f
  100.0%  [1]          dofilewrite
   100.0%  [1]           kern_writev

00.34%  [1]        setrunnable @ /boot/kernel/kernel
 100.0%  [1]         sleepq_broadcast
  100.0%  [1]          wakeup
   100.0%  [1]           bdone

00.34%  [1]        Xfast_syscall @ /boot/kernel/kernel

00.34%  [1]        pmap_extract_and_hold @ /boot/kernel/kernel
 100.0%  [1]         vm_fault_quick_hold_pages
  100.0%  [1]          vmapbuf
   100.0%  [1]           physio

00.34%  [1]        xpt_path_sim @ /boot/kernel/kernel
 100.0%  [1]         mrsas_map_request @ /boot/kernel/mrsas.ko
  100.0%  [1]          mrsas_build_dcdb
   100.0%  [1]           mrsas_action

00.34%  [1]        scsi_read_write @ /boot/kernel/kernel
 100.0%  [1]         dastart
  100.0%  [1]          xpt_run_allocq
   100.0%  [1]           dastrategy

00.34%  [1]        kern_readv @ /boot/kernel/kernel
 100.0%  [1]         sys_read
  100.0%  [1]          amd64_syscall

00.34%  [1]        __sys_write @ /lib/libc.so.7

00.34%  [1]        g_io_deliver @ /boot/kernel/kernel
 100.0%  [1]         g_disk_done_single
  100.0%  [1]          dadone
   100.0%  [1]           xpt_done_process

00.34%  [1]        critical_enter @ /boot/kernel/kernel
 100.0%  [1]         callout_lock
  100.0%  [1]          _callout_stop_safe
   100.0%  [1]           mrsas_cmd_done @ /boot/kernel/mrsas.ko

00.34%  [1]        dastart @ /boot/kernel/kernel
 100.0%  [1]         xpt_run_allocq
  100.0%  [1]          dastrategy
   100.0%  [1]           g_disk_start

00.34%  [1]        _mtx_trylock_flags_ @ /boot/kernel/kernel
 100.0%  [1]         vm_page_pa_tryrelock
  100.0%  [1]          pmap_extract_and_hold
   100.0%  [1]           vm_fault_quick_hold_pages

00.34%  [1]        callout_cc_add @ /boot/kernel/kernel
 100.0%  [1]         callout_reset_sbt_on
  100.0%  [1]          mrsas_action @ /boot/kernel/mrsas.ko
   100.0%  [1]           xpt_run_devq @ /boot/kernel/kernel

00.34%  [1]        sys_read @ /boot/kernel/kernel
 100.0%  [1]         amd64_syscall

00.34%  [1]        sched_rem @ /boot/kernel/kernel
 100.0%  [1]         tdq_move
  100.0%  [1]          sched_idletd
   100.0%  [1]           fork_exit

00.34%  [1]        sched_add @ /boot/kernel/kernel
 100.0%  [1]         setrunnable
  100.0%  [1]          sleepq_broadcast
   100.0%  [1]           wakeup

00.34%  [1]        __lockmgr_args @ /boot/kernel/kernel
 100.0%  [1]         getpbuf
  100.0%  [1]          physio
   100.0%  [1]           devfs_read_f

00.34%  [1]        sched_priority @ /boot/kernel/kernel
 100.0%  [1]         sched_add
  100.0%  [1]          setrunnable
   100.0%  [1]           sleepq_broadcast

00.34%  [1]        bounce_bus_dmamap_load_buffer @ /boot/kernel/kernel
 100.0%  [1]         bus_dmamap_load
  100.0%  [1]          mrsas_map_request @ /boot/kernel/mrsas.ko
   100.0%  [1]           mrsas_build_dcdb

00.34%  [1]        mrsas_complete_cmd @ /boot/kernel/mrsas.ko
 100.0%  [1]         intr_event_execute_handlers @ /boot/kernel/kernel
  100.0%  [1]          ithread_loop
   100.0%  [1]           fork_exit

00.34%  [1]        free @ /boot/kernel/kernel
 100.0%  [1]         xpt_release_ccb
  100.0%  [1]          dadone
   100.0%  [1]           xpt_done_process

00.34%  [1]        dev_relthread @ /boot/kernel/kernel
 100.0%  [1]         devfs_read_f
  100.0%  [1]          dofileread
   100.0%  [1]           kern_readv

00.34%  [1]        thread_lock_set @ /boot/kernel/kernel
 100.0%  [1]         sleepq_switch
  100.0%  [1]          sleepq_wait
   100.0%  [1]           _sleep

00.34%  [1]        runq_add @ /boot/kernel/kernel
 100.0%  [1]         sched_add
  100.0%  [1]          intr_event_schedule_thread
   100.0%  [1]           intr_event_handle

00.34%  [1]        tdq_lock_pair @ /boot/kernel/kernel
 100.0%  [1]         sched_idletd
  100.0%  [1]          fork_exit

00.34%  [1]        sleepq_wait @ /boot/kernel/kernel
 100.0%  [1]         _sleep
  100.0%  [1]          bwait
   100.0%  [1]           physio

00.34%  [1]        thread_lock_block @ /boot/kernel/kernel
 100.0%  [1]         sched_switch
  100.0%  [1]          mi_switch
   100.0%  [1]           sleepq_wait

00.34%  [1]        xpt_done @ /boot/kernel/kernel
 100.0%  [1]         mrsas_cmd_done @ /boot/kernel/mrsas.ko
  100.0%  [1]          mrsas_complete_cmd
   100.0%  [1]           intr_event_execute_handlers @ /boot/kernel/kernel

00.34%  [1]        uma_zfree_arg @ /boot/kernel/kernel
 100.0%  [1]         g_dev_done
  100.0%  [1]          g_io_deliver
   100.0%  [1]           g_disk_done_single

00.34%  [1]        Xtimerint @ /boot/kernel/kernel

00.34%  [1]        devstat_start_transaction_bio @ /boot/kernel/kernel
 100.0%  [1]         g_disk_start
  100.0%  [1]          g_io_request
   100.0%  [1]           dev_strategy_csw

00.34%  [1]        devfs_write_f @ /boot/kernel/kernel
 100.0%  [1]         dofilewrite
  100.0%  [1]          kern_writev
   100.0%  [1]           sys_write

00.34%  [1]        vunmapbuf @ /boot/kernel/kernel
 100.0%  [1]         physio
  100.0%  [1]          devfs_read_f
   100.0%  [1]           dofileread

00.34%  [1]        sleepq_resume_thread @ /boot/kernel/kernel
 100.0%  [1]         sleepq_broadcast
  100.0%  [1]          wakeup
   100.0%  [1]           bdone

00.34%  [1]        sleepq_add @ /boot/kernel/kernel
 100.0%  [1]         _sleep
  100.0%  [1]          bwait
   100.0%  [1]           physio

00.34%  [1]        _mtx_lock_spin_cookie @ /boot/kernel/kernel
 100.0%  [1]         sched_idletd
  100.0%  [1]          fork_exit

00.34%  [1]        amd64_syscall @ /boot/kernel/kernel

00.34%  [1]        _callout_stop_safe @ /boot/kernel/kernel
 100.0%  [1]         mrsas_cmd_done @ /boot/kernel/mrsas.ko
  100.0%  [1]          mrsas_complete_cmd
   100.0%  [1]           intr_event_execute_handlers @ /boot/kernel/kernel

00.34%  [1]        runq_remove @ /boot/kernel/kernel
 100.0%  [1]         sched_choose
  100.0%  [1]          choosethread
   100.0%  [1]           sched_switch

00.34%  [1]        bzero @ /boot/kernel/kernel
 100.0%  [1]         mrsas_action @ /boot/kernel/mrsas.ko
  100.0%  [1]          xpt_run_devq @ /boot/kernel/kernel
   100.0%  [1]           xpt_action_default

00.34%  [1]        uma_zalloc_arg @ /boot/kernel/kernel
 100.0%  [1]         dev_strategy_csw
  100.0%  [1]          physio
   100.0%  [1]           devfs_read_f

@ CPU_CLK_UNHALTED_CORE [294 samples]

11.90%  [35]       cpu_idle @ /boot/kernel/kernel
 100.0%  [35]        sched_idletd
  100.0%  [35]         fork_exit

09.86%  [29]       invlrng_handler @ /boot/kernel/kernel

08.84%  [26]       Xinvlrng @ /boot/kernel/kernel

05.78%  [17]       __mtx_lock_sleep @ /boot/kernel/kernel
 58.82%  [10]        __mtx_lock_flags
  100.0%  [10]         mrsas_cmd_done @ /boot/kernel/mrsas.ko
   100.0%  [10]          mrsas_complete_cmd
 17.65%  [3]         vmem_xalloc @ /boot/kernel/kernel
  100.0%  [3]          vmem_alloc
   100.0%  [3]           g_io_check
 05.88%  [1]         xpt_run_devq
  100.0%  [1]          xpt_action_default
   100.0%  [1]           dastart
 05.88%  [1]         xpt_done_process
  100.0%  [1]          xpt_done_td
   100.0%  [1]           fork_exit
 05.88%  [1]         vmem_xfree
  100.0%  [1]          biodone
   100.0%  [1]           dadone
 05.88%  [1]         _sleep
  100.0%  [1]          bwait
   100.0%  [1]           physio

04.42%  [13]       smp_tlb_shootdown @ /boot/kernel/kernel
 100.0%  [13]        pmap_invalidate_range
  100.0%  [13]         biodone
   100.0%  [13]          dadone

03.40%  [10]       cpu_search_highest @ /boot/kernel/kernel
 90.00%  [9]         cpu_search_highest
  66.67%  [6]          sched_idletd
   100.0%  [6]           fork_exit
  33.33%  [3]          cpu_search_highest
   100.0%  [3]           sched_idletd
 10.00%  [1]         sched_idletd
  100.0%  [1]          fork_exit

03.06%  [9]        cpu_search_lowest @ /boot/kernel/kernel
 100.0%  [9]         cpu_search_lowest
  88.89%  [8]          cpu_search_lowest
   100.0%  [8]           sched_pickcpu
  11.11%  [1]          sched_pickcpu
   100.0%  [1]           sched_add

02.38%  [7]        cpu_switch @ /boot/kernel/kernel
 71.43%  [5]         mi_switch
  40.00%  [2]          sched_idletd
   100.0%  [2]           fork_exit
  20.00%  [1]          critical_exit
   100.0%  [1]           intr_event_handle
  20.00%  [1]          ithread_loop
   100.0%  [1]           fork_exit
  20.00%  [1]          sleepq_wait
   100.0%  [1]           _sleep
 28.57%  [2]         sched_switch
  100.0%  [2]          mi_switch
   50.00%  [1]           sleepq_wait
   50.00%  [1]           ithread_loop

01.70%  [5]        thread_lock_flags_ @ /boot/kernel/kernel
 80.00%  [4]         sched_idletd
  100.0%  [4]          fork_exit
 20.00%  [1]         sleepq_add
  100.0%  [1]          _sleep
   100.0%  [1]           bwait

01.70%  [5]        doreti @ /boot/kernel/kernel

01.70%  [5]        sched_switch @ /boot/kernel/kernel
 100.0%  [5]         mi_switch
  60.00%  [3]          sched_idletd
   100.0%  [3]           fork_exit
  20.00%  [1]          sleepq_wait
   100.0%  [1]           _sleep
  20.00%  [1]          ithread_loop
   100.0%  [1]           fork_exit

01.70%  [5]        sched_idletd @ /boot/kernel/kernel
 100.0%  [5]         fork_exit

01.70%  [5]        vmem_xalloc @ /boot/kernel/kernel
 100.0%  [5]         vmem_alloc
  100.0%  [5]          g_io_check
   100.0%  [5]           g_io_request

01.70%  [5]        __mtx_unlock_flags @ /boot/kernel/kernel
 40.00%  [2]         xpt_run_devq
  100.0%  [2]          xpt_action_default
   100.0%  [2]           dastart
 40.00%  [2]         mrsas_cmd_done @ /boot/kernel/mrsas.ko
  100.0%  [2]          mrsas_complete_cmd
   100.0%  [2]           intr_event_execute_handlers @ /boot/kernel/kernel
 20.00%  [1]         mrsas_get_mpt_cmd @ /boot/kernel/mrsas.ko
  100.0%  [1]          mrsas_action
   100.0%  [1]           xpt_run_devq @ /boot/kernel/kernel

01.70%  [5]        xpt_run_devq @ /boot/kernel/kernel
 100.0%  [5]         xpt_action_default
  100.0%  [5]          dastart
   100.0%  [5]           xpt_run_allocq

01.36%  [4]        critical_exit @ /boot/kernel/kernel
 25.00%  [1]         uma_zalloc_arg
  100.0%  [1]          dev_strategy_csw
   100.0%  [1]           physio
 25.00%  [1]         intr_event_handle
  100.0%  [1]          intr_execute_handlers
   100.0%  [1]           lapic_handle_intr
 25.00%  [1]         malloc
  100.0%  [1]          xpt_run_allocq
   100.0%  [1]           dastrategy
 25.00%  [1]         spinlock_exit
  100.0%  [1]          turnstile_wait
   100.0%  [1]           __mtx_lock_sleep

01.36%  [4]        __mtx_lock_flags @ /boot/kernel/kernel
 50.00%  [2]         mrsas_fire_cmd @ /boot/kernel/mrsas.ko
  100.0%  [2]          xpt_run_devq @ /boot/kernel/kernel
   100.0%  [2]           xpt_action_default
 50.00%  [2]         mrsas_unmap_request @ /boot/kernel/mrsas.ko
  100.0%  [2]          mrsas_cmd_done
   100.0%  [2]           mrsas_complete_cmd

01.36%  [4]        dastart @ /boot/kernel/kernel
 100.0%  [4]         xpt_run_allocq
  100.0%  [4]          dastrategy
   100.0%  [4]           g_disk_start

01.36%  [4]        spinlock_exit @ /boot/kernel/kernel
 25.00%  [1]         sched_idletd
  100.0%  [1]          fork_exit
 25.00%  [1]         intr_event_schedule_thread
  100.0%  [1]          intr_event_handle
   100.0%  [1]           intr_execute_handlers
 25.00%  [1]         ithread_loop
  100.0%  [1]          fork_exit
 25.00%  [1]         callout_reset_sbt_on
  100.0%  [1]          mrsas_action @ /boot/kernel/mrsas.ko
   100.0%  [1]           xpt_run_devq @ /boot/kernel/kernel

01.02%  [3]        g_dev_strategy @ /boot/kernel/kernel
 100.0%  [3]         dev_strategy_csw
  100.0%  [3]          physio
   100.0%  [3]           devfs_read_f

01.02%  [3]        xpt_action_default @ /boot/kernel/kernel
 100.0%  [3]         dastart
  100.0%  [3]          xpt_run_allocq
   100.0%  [3]           dastrategy

01.02%  [3]        xpt_done_process @ /boot/kernel/kernel
 100.0%  [3]         xpt_done_td
  100.0%  [3]          fork_exit

01.02%  [3]        mrsas_complete_cmd @ /boot/kernel/mrsas.ko
 100.0%  [3]         intr_event_execute_handlers @ /boot/kernel/kernel
  100.0%  [3]          ithread_loop
   100.0%  [3]           fork_exit

01.02%  [3]        pmap_invalidate_range @ /boot/kernel/kernel
 100.0%  [3]         biodone
  100.0%  [3]          dadone
   100.0%  [3]           xpt_done_process

01.02%  [3]        xpt_done @ /boot/kernel/kernel
 100.0%  [3]         mrsas_cmd_done @ /boot/kernel/mrsas.ko
  100.0%  [3]          mrsas_complete_cmd
   100.0%  [3]           intr_event_execute_handlers @ /boot/kernel/kernel

01.02%  [3]        physio @ /boot/kernel/kernel
 100.0%  [3]         devfs_read_f
  100.0%  [3]          dofileread
   100.0%  [3]           kern_readv

01.02%  [3]        _mtx_lock_spin_cookie @ /boot/kernel/kernel
 66.67%  [2]         sched_idletd
  100.0%  [2]          fork_exit
 33.33%  [1]         smp_tlb_shootdown
  100.0%  [1]          pmap_invalidate_range
   100.0%  [1]           biodone

01.02%  [3]        amd64_syscall @ /boot/kernel/kernel

00.68%  [2]        g_io_deliver @ /boot/kernel/kernel
 100.0%  [2]         g_disk_done_single
  100.0%  [2]          dadone
   100.0%  [2]           xpt_done_process

00.68%  [2]        mi_switch @ /boot/kernel/kernel
 50.00%  [1]         sched_idletd
  100.0%  [1]          fork_exit
 50.00%  [1]         critical_exit
  100.0%  [1]          intr_event_handle
   100.0%  [1]           intr_execute_handlers

00.68%  [2]        devstat_end_transaction @ /boot/kernel/kernel
 100.0%  [2]         devstat_end_transaction_bio_bt
  100.0%  [2]          g_disk_done_single
   100.0%  [2]           dadone

00.68%  [2]        setrunnable @ /boot/kernel/kernel
 100.0%  [2]         sleepq_broadcast
  100.0%  [2]          wakeup
   100.0%  [2]           bdone

00.68%  [2]        vm_page_unhold_pages @ /boot/kernel/kernel
 100.0%  [2]         vunmapbuf
  100.0%  [2]          physio
   100.0%  [2]           devfs_read_f

00.68%  [2]        ithread_loop @ /boot/kernel/kernel
 100.0%  [2]         fork_exit

00.68%  [2]        __sys_read @ /lib/libc.so.7

00.68%  [2]        g_disk_start @ /boot/kernel/kernel
 100.0%  [2]         g_io_request
  100.0%  [2]          dev_strategy_csw
   100.0%  [2]           physio

00.68%  [2]        mrsas_action @ /boot/kernel/mrsas.ko
 100.0%  [2]         xpt_run_devq @ /boot/kernel/kernel
  100.0%  [2]          xpt_action_default
   100.0%  [2]           dastart

00.68%  [2]        spinlock_enter @ /boot/kernel/kernel
 50.00%  [1]         thread_lock_flags_
  100.0%  [1]          sleepq_broadcast
   100.0%  [1]           wakeup
 50.00%  [1]         callout_lock
  100.0%  [1]          callout_reset_sbt_on
   100.0%  [1]           mrsas_action @ /boot/kernel/mrsas.ko

00.68%  [2]        devvn_refthread @ /boot/kernel/kernel
 100.0%  [2]         devfs_read_f
  100.0%  [2]          dofileread
   100.0%  [2]           kern_readv

00.68%  [2]        uma_zalloc_arg @ /boot/kernel/kernel
 50.00%  [1]         dev_strategy_csw
  100.0%  [1]          physio
   100.0%  [1]           devfs_read_f
 50.00%  [1]         malloc
  100.0%  [1]          xpt_run_allocq
   100.0%  [1]           dastrategy

00.68%  [2]        dastrategy @ /boot/kernel/kernel
 100.0%  [2]         g_disk_start
  100.0%  [2]          g_io_request
   100.0%  [2]           dev_strategy_csw

00.68%  [2]        g_disk_done_single @ /boot/kernel/kernel
 100.0%  [2]         dadone
  100.0%  [2]          xpt_done_process
   100.0%  [2]           xpt_done_td

00.68%  [2]        bzero @ /boot/kernel/kernel
 100.0%  [2]         mrsas_action @ /boot/kernel/mrsas.ko
  100.0%  [2]          xpt_run_devq @ /boot/kernel/kernel
   100.0%  [2]           xpt_action_default

00.68%  [2]        xpt_run_allocq @ /boot/kernel/kernel
 100.0%  [2]         dastrategy
  100.0%  [2]          g_disk_start
   100.0%  [2]           g_io_request

00.68%  [2]        devstat_start_transaction_bio @ /boot/kernel/kernel
 100.0%  [2]         g_disk_start
  100.0%  [2]          g_io_request
   100.0%  [2]           dev_strategy_csw

00.34%  [1]        Xfast_syscall @ /boot/kernel/kernel

00.34%  [1]        sleepq_wait @ /boot/kernel/kernel
 100.0%  [1]         _sleep
  100.0%  [1]          bwait
   100.0%  [1]           physio

00.34%  [1]        lock_mtx @ /boot/kernel/kernel
 100.0%  [1]         _sleep
  100.0%  [1]          bwait
   100.0%  [1]           physio

00.34%  [1]        syscall_thread_exit @ /boot/kernel/kernel
 100.0%  [1]         amd64_syscall

00.34%  [1]        kern_writev @ /boot/kernel/kernel
 100.0%  [1]         sys_write
  100.0%  [1]          amd64_syscall

00.34%  [1]        vm_fault_quick_hold_pages @ /boot/kernel/kernel
 100.0%  [1]         vmapbuf
  100.0%  [1]          physio
   100.0%  [1]           devfs_read_f

00.34%  [1]        g_io_request @ /boot/kernel/kernel
 100.0%  [1]         dev_strategy_csw
  100.0%  [1]          physio
   100.0%  [1]           devfs_read_f

00.34%  [1]        uma_zfree_arg @ /boot/kernel/kernel
 100.0%  [1]         g_io_deliver
  100.0%  [1]          g_disk_done_single
   100.0%  [1]           dadone

00.34%  [1]        sleepq_add @ /boot/kernel/kernel
 100.0%  [1]         _sleep
  100.0%  [1]          bwait
   100.0%  [1]           physio

00.34%  [1]        _fget @ /boot/kernel/kernel
 100.0%  [1]         kern_writev
  100.0%  [1]          sys_write
   100.0%  [1]           amd64_syscall

00.34%  [1]        camq_remove @ /boot/kernel/kernel
 100.0%  [1]         xpt_run_devq
  100.0%  [1]          xpt_action_default
   100.0%  [1]           dastart

00.34%  [1]        bioq_insert_tail @ /boot/kernel/kernel
 100.0%  [1]         dastrategy
  100.0%  [1]          g_disk_start
   100.0%  [1]           g_io_request

00.34%  [1]        vmem_xfree @ /boot/kernel/kernel
 100.0%  [1]         biodone
  100.0%  [1]          dadone
   100.0%  [1]           xpt_done_process

00.34%  [1]        sched_unlend_prio @ /boot/kernel/kernel
 100.0%  [1]         turnstile_unpend
  100.0%  [1]          __mtx_unlock_sleep
   100.0%  [1]           bufdone

00.34%  [1]        fget_unlocked @ /boot/kernel/kernel
 100.0%  [1]         _fget
  100.0%  [1]          kern_writev
   100.0%  [1]           sys_write

00.34%  [1]        mrsas_isr @ /boot/kernel/mrsas.ko
 100.0%  [1]         intr_event_execute_handlers @ /boot/kernel/kernel
  100.0%  [1]          ithread_loop
   100.0%  [1]           fork_exit

00.34%  [1]        pmap_qenter @ /boot/kernel/kernel
 100.0%  [1]         g_io_check
  100.0%  [1]          g_io_request
   100.0%  [1]           dev_strategy_csw

00.34%  [1]        mrsas_data_load_cb @ /boot/kernel/mrsas.ko
 100.0%  [1]         bus_dmamap_load @ /boot/kernel/kernel
  100.0%  [1]          mrsas_map_request @ /boot/kernel/mrsas.ko
   100.0%  [1]           mrsas_build_dcdb

00.34%  [1]        __cap_rights_init @ /boot/kernel/kernel
 100.0%  [1]         kern_readv
  100.0%  [1]          sys_read
   100.0%  [1]           amd64_syscall

00.34%  [1]        xpt_release_ccb @ /boot/kernel/kernel
 100.0%  [1]         dadone
  100.0%  [1]          xpt_done_process
   100.0%  [1]           xpt_done_td

00.34%  [1]        sched_choose @ /boot/kernel/kernel
 100.0%  [1]         choosethread
  100.0%  [1]          sched_switch
   100.0%  [1]           mi_switch

00.34%  [1]        critical_enter @ /boot/kernel/kernel
 100.0%  [1]         sched_switch
  100.0%  [1]          mi_switch
   100.0%  [1]           ithread_loop

00.34%  [1]        lapic_ipi_vectored @ /boot/kernel/kernel
 100.0%  [1]         smp_tlb_shootdown
  100.0%  [1]          pmap_invalidate_range
   100.0%  [1]           biodone

00.34%  [1]        callout_cc_add @ /boot/kernel/kernel
 100.0%  [1]         callout_reset_sbt_on
  100.0%  [1]          mrsas_action @ /boot/kernel/mrsas.ko
   100.0%  [1]           xpt_run_devq @ /boot/kernel/kernel

00.34%  [1]        sched_add @ /boot/kernel/kernel
 100.0%  [1]         turnstile_unpend
  100.0%  [1]          __mtx_unlock_sleep
   100.0%  [1]           __mtx_unlock_flags

00.34%  [1]        __lockmgr_args @ /boot/kernel/kernel
 100.0%  [1]         getpbuf
  100.0%  [1]          physio
   100.0%  [1]           devfs_read_f

00.34%  [1]        Xapic_isr1 @ /boot/kernel/kernel

00.34%  [1]        _sleep @ /boot/kernel/kernel
 100.0%  [1]         bwait
  100.0%  [1]          physio
   100.0%  [1]           devfs_read_f

00.34%  [1]        g_io_check @ /boot/kernel/kernel
 100.0%  [1]         g_io_request
  100.0%  [1]          dev_strategy_csw
   100.0%  [1]           physio

00.34%  [1]        sched_pickcpu @ /boot/kernel/kernel
 100.0%  [1]         sched_add
  100.0%  [1]          intr_event_schedule_thread
   100.0%  [1]           intr_event_handle

00.34%  [1]        sys_write @ /boot/kernel/kernel
 100.0%  [1]         amd64_syscall

00.34%  [1]        sleepq_broadcast @ /boot/kernel/kernel
 100.0%  [1]         wakeup
  100.0%  [1]          bdone
   100.0%  [1]           bufdone

00.34%  [1]        bt_freetrim @ /boot/kernel/kernel
 100.0%  [1]         biodone
  100.0%  [1]          dadone
   100.0%  [1]           xpt_done_process

00.34%  [1]        g_dev_done @ /boot/kernel/kernel
 100.0%  [1]         g_io_deliver
  100.0%  [1]          g_disk_done_single
   100.0%  [1]           dadone

00.34%  [1]        runq_add @ /boot/kernel/kernel
 100.0%  [1]         sched_add
  100.0%  [1]          setrunnable
   100.0%  [1]           sleepq_broadcast

00.34%  [1]        pmap_extract_and_hold @ /boot/kernel/kernel
 100.0%  [1]         vm_fault_quick_hold_pages
  100.0%  [1]          vmapbuf
   100.0%  [1]           physio

@ CPU_CLK_UNHALTED_CORE [294 samples]

10.54%  [31]       invlrng_handler @ /boot/kernel/kernel

09.52%  [28]       Xinvlrng @ /boot/kernel/kernel

08.16%  [24]       cpu_idle @ /boot/kernel/kernel
 100.0%  [24]        sched_idletd
  100.0%  [24]         fork_exit

06.12%  [18]       smp_tlb_shootdown @ /boot/kernel/kernel
 100.0%  [18]        pmap_invalidate_range
  100.0%  [18]         biodone
   100.0%  [18]          dadone

05.10%  [15]       cpu_search_highest @ /boot/kernel/kernel
 80.00%  [12]        cpu_search_highest
  66.67%  [8]          sched_idletd
   100.0%  [8]           fork_exit
  33.33%  [4]          cpu_search_highest
   100.0%  [4]           sched_idletd
 20.00%  [3]         sched_idletd
  100.0%  [3]          fork_exit

03.40%  [10]       cpu_search_lowest @ /boot/kernel/kernel
 100.0%  [10]        cpu_search_lowest
  70.00%  [7]          cpu_search_lowest
   100.0%  [7]           sched_pickcpu
  30.00%  [3]          sched_pickcpu
   100.0%  [3]           sched_add

02.72%  [8]        vmem_xfree @ /boot/kernel/kernel
 100.0%  [8]         biodone
  100.0%  [8]          dadone
   100.0%  [8]           xpt_done_process

02.38%  [7]        cpu_switch @ /boot/kernel/kernel
 100.0%  [7]         mi_switch
  57.14%  [4]          sched_idletd
   100.0%  [4]           fork_exit
  42.86%  [3]          sleepq_wait
   100.0%  [3]           _sleep

02.38%  [7]        sched_switch @ /boot/kernel/kernel
 100.0%  [7]         mi_switch
  42.86%  [3]          ithread_loop
   100.0%  [3]           fork_exit
  28.57%  [2]          sched_idletd
   100.0%  [2]           fork_exit
  14.29%  [1]          critical_exit
   100.0%  [1]           intr_event_handle
  14.29%  [1]          sleepq_wait
   100.0%  [1]           _sleep

02.04%  [6]        __mtx_unlock_flags @ /boot/kernel/kernel
 33.33%  [2]         xpt_run_devq
  100.0%  [2]          xpt_action_default
   100.0%  [2]           dastart
 16.67%  [1]         mrsas_action @ /boot/kernel/mrsas.ko
  100.0%  [1]          xpt_run_devq @ /boot/kernel/kernel
   100.0%  [1]           xpt_action_default
 16.67%  [1]         mrsas_cmd_done @ /boot/kernel/mrsas.ko
  100.0%  [1]          mrsas_complete_cmd
   100.0%  [1]           intr_event_execute_handlers @ /boot/kernel/kernel
 16.67%  [1]         mrsas_get_mpt_cmd @ /boot/kernel/mrsas.ko
  100.0%  [1]          mrsas_action
   100.0%  [1]           xpt_run_devq @ /boot/kernel/kernel
 16.67%  [1]         mrsas_map_request @ /boot/kernel/mrsas.ko
  100.0%  [1]          mrsas_build_dcdb
   100.0%  [1]           mrsas_action

01.70%  [5]        __mtx_lock_sleep @ /boot/kernel/kernel
 60.00%  [3]         __mtx_lock_flags
  100.0%  [3]          mrsas_cmd_done @ /boot/kernel/mrsas.ko
   100.0%  [3]           mrsas_complete_cmd
 20.00%  [1]         xpt_run_devq @ /boot/kernel/kernel
  100.0%  [1]          xpt_action_default
   100.0%  [1]           dastart
 20.00%  [1]         vmem_xalloc
  100.0%  [1]          vmem_alloc
   100.0%  [1]           g_io_check

01.70%  [5]        ithread_loop @ /boot/kernel/kernel
 100.0%  [5]         fork_exit

01.70%  [5]        pmap_invalidate_range @ /boot/kernel/kernel
 100.0%  [5]         biodone
  100.0%  [5]          dadone
   100.0%  [5]           xpt_done_process

01.70%  [5]        doreti @ /boot/kernel/kernel

01.70%  [5]        spinlock_exit @ /boot/kernel/kernel
 40.00%  [2]         sched_idletd
  100.0%  [2]          fork_exit
 20.00%  [1]         smp_tlb_shootdown
  100.0%  [1]          pmap_invalidate_range
   100.0%  [1]           biodone
 20.00%  [1]         callout_reset_sbt_on
  100.0%  [1]          mrsas_action @ /boot/kernel/mrsas.ko
   100.0%  [1]           xpt_run_devq @ /boot/kernel/kernel
 20.00%  [1]         wakeup
  100.0%  [1]          bdone
   100.0%  [1]           bufdone

01.70%  [5]        thread_lock_flags_ @ /boot/kernel/kernel
 20.00%  [1]         sleepq_add
  100.0%  [1]          _sleep
   100.0%  [1]           bwait
 20.00%  [1]         sleepq_wait
  100.0%  [1]          _sleep
   100.0%  [1]           bwait
 20.00%  [1]         intr_event_schedule_thread
  100.0%  [1]          intr_event_handle
   100.0%  [1]           intr_execute_handlers
 20.00%  [1]         propagate_priority
  100.0%  [1]          turnstile_wait
   100.0%  [1]           __mtx_lock_sleep
 20.00%  [1]         sleepq_broadcast
  100.0%  [1]          wakeup
   100.0%  [1]           bdone

01.36%  [4]        __sys_read @ /lib/libc.so.7

01.36%  [4]        xpt_run_devq @ /boot/kernel/kernel
 100.0%  [4]         xpt_action_default
  100.0%  [4]          dastart
   100.0%  [4]           xpt_run_allocq

01.36%  [4]        sched_idletd @ /boot/kernel/kernel
 100.0%  [4]         fork_exit

01.36%  [4]        Xapic_isr1 @ /boot/kernel/kernel

01.36%  [4]        critical_enter @ /boot/kernel/kernel
 25.00%  [1]         uma_zalloc_arg
  100.0%  [1]          dev_strategy_csw
   100.0%  [1]           physio
 25.00%  [1]         smp_tlb_shootdown
  100.0%  [1]          pmap_invalidate_range
   100.0%  [1]           biodone
 25.00%  [1]         sleepq_lock
  100.0%  [1]          cv_broadcastpri
   100.0%  [1]           vmem_xfree
 25.00%  [1]         thread_lock_flags_
  100.0%  [1]          sleepq_add
   100.0%  [1]           _sleep

01.36%  [4]        uma_zfree_arg @ /boot/kernel/kernel
 75.00%  [3]         g_dev_done
  100.0%  [3]          g_io_deliver
   100.0%  [3]           g_disk_done_single
 25.00%  [1]         g_io_deliver
  100.0%  [1]          g_disk_done_single
   100.0%  [1]           dadone

01.02%  [3]        g_dev_done @ /boot/kernel/kernel
 100.0%  [3]         g_io_deliver
  100.0%  [3]          g_disk_done_single
   100.0%  [3]           dadone

01.02%  [3]        dadone @ /boot/kernel/kernel
 100.0%  [3]         xpt_done_process
  100.0%  [3]          xpt_done_td
   100.0%  [3]           fork_exit

01.02%  [3]        _mtx_lock_spin_cookie @ /boot/kernel/kernel
 66.67%  [2]         sched_idletd
  100.0%  [2]          fork_exit
 33.33%  [1]         sched_add
  100.0%  [1]          setrunnable
   100.0%  [1]           sleepq_broadcast

01.02%  [3]        Xfast_syscall @ /boot/kernel/kernel

01.02%  [3]        __mtx_lock_flags @ /boot/kernel/kernel
 100.0%  [3]         mrsas_cmd_done @ /boot/kernel/mrsas.ko
  100.0%  [3]          mrsas_complete_cmd
   100.0%  [3]           intr_event_execute_handlers @ /boot/kernel/kernel

00.68%  [2]        dofilewrite @ /boot/kernel/kernel
 100.0%  [2]         kern_writev
  100.0%  [2]          sys_write
   100.0%  [2]           amd64_syscall

00.68%  [2]        bzero @ /boot/kernel/kernel
 50.00%  [1]         dev_strategy_csw
  100.0%  [1]          physio
   100.0%  [1]           devfs_read_f
 50.00%  [1]         xpt_run_devq
  100.0%  [1]          xpt_action_default
   100.0%  [1]           dastart

00.68%  [2]        amd64_syscall @ /boot/kernel/kernel

00.68%  [2]        g_is_geom_thread @ /boot/kernel/kernel
 50.00%  [1]         g_io_deliver
  100.0%  [1]          g_disk_done_single
   100.0%  [1]           dadone
 50.00%  [1]         g_io_request
  100.0%  [1]          dev_strategy_csw
   100.0%  [1]           physio

00.68%  [2]        g_io_request @ /boot/kernel/kernel
 100.0%  [2]         dev_strategy_csw
  100.0%  [2]          physio
   100.0%  [2]           devfs_read_f

00.68%  [2]        __lockmgr_args @ /boot/kernel/kernel
 50.00%  [1]         relpbuf
  100.0%  [1]          physio
   100.0%  [1]           devfs_read_f
 50.00%  [1]         getpbuf
  100.0%  [1]          physio
   100.0%  [1]           devfs_read_f

00.68%  [2]        free @ /boot/kernel/kernel
 100.0%  [2]         xpt_release_ccb
  100.0%  [2]          dadone
   100.0%  [2]           xpt_done_process

00.68%  [2]        physio @ /boot/kernel/kernel
 100.0%  [2]         devfs_read_f
  100.0%  [2]          dofileread
   100.0%  [2]           kern_readv

00.68%  [2]        _fget @ /boot/kernel/kernel
 50.00%  [1]         kern_readv
  100.0%  [1]          sys_read
   100.0%  [1]           amd64_syscall
 50.00%  [1]         kern_writev
  100.0%  [1]          sys_write
   100.0%  [1]           amd64_syscall

00.68%  [2]        devfs_write_f @ /boot/kernel/kernel
 100.0%  [2]         dofilewrite
  100.0%  [2]          kern_writev
   100.0%  [2]           sys_write

00.68%  [2]        devfs_read_f @ /boot/kernel/kernel
 100.0%  [2]         dofileread
  100.0%  [2]          kern_readv
   100.0%  [2]           sys_read

00.68%  [2]        atomic_fetchadd_int @ /boot/kernel/kernel
 50.00%  [1]         mrsas_complete_cmd @ /boot/kernel/mrsas.ko
  100.0%  [1]          intr_event_execute_handlers @ /boot/kernel/kernel
   100.0%  [1]           ithread_loop
 50.00%  [1]         mrsas_action @ /boot/kernel/mrsas.ko
  100.0%  [1]          xpt_run_devq @ /boot/kernel/kernel
   100.0%  [1]           xpt_action_default

00.68%  [2]        vmem_xalloc @ /boot/kernel/kernel
 100.0%  [2]         vmem_alloc
  100.0%  [2]          g_io_check
   100.0%  [2]           g_io_request

00.68%  [2]        xpt_done_process @ /boot/kernel/kernel
 100.0%  [2]         xpt_done_td
  100.0%  [2]          fork_exit

00.68%  [2]        dastart @ /boot/kernel/kernel
 100.0%  [2]         xpt_run_allocq
  100.0%  [2]          dastrategy
   100.0%  [2]           g_disk_start

00.68%  [2]        g_io_check @ /boot/kernel/kernel
 100.0%  [2]         g_io_request
  100.0%  [2]          dev_strategy_csw
   100.0%  [2]           physio

00.68%  [2]        sched_pickcpu @ /boot/kernel/kernel
 100.0%  [2]         sched_add
  100.0%  [2]          intr_event_schedule_thread
   100.0%  [2]           intr_event_handle

00.34%  [1]        mrsas_unmap_request @ /boot/kernel/mrsas.ko
 100.0%  [1]         mrsas_cmd_done
  100.0%  [1]          mrsas_complete_cmd
   100.0%  [1]           intr_event_execute_handlers @ /boot/kernel/kernel

00.34%  [1]        userret @ /boot/kernel/kernel
 100.0%  [1]         amd64_syscall

00.34%  [1]        xpt_action_default @ /boot/kernel/kernel
 100.0%  [1]         dastart
  100.0%  [1]          xpt_run_allocq
   100.0%  [1]           dastrategy

00.34%  [1]        kern_writev @ /boot/kernel/kernel
 100.0%  [1]         sys_write
  100.0%  [1]          amd64_syscall

00.34%  [1]        devstat_end_transaction @ /boot/kernel/kernel
 100.0%  [1]         devstat_end_transaction_bio_bt
  100.0%  [1]          g_disk_done_single
   100.0%  [1]           dadone

00.34%  [1]        sleepq_broadcast @ /boot/kernel/kernel
 100.0%  [1]         wakeup
  100.0%  [1]          bdone
   100.0%  [1]           bufdone

00.34%  [1]        sleepq_lock @ /boot/kernel/kernel
 100.0%  [1]         wakeup
  100.0%  [1]          bdone
   100.0%  [1]           bufdone

00.34%  [1]        sys_write @ /boot/kernel/kernel
 100.0%  [1]         amd64_syscall

00.34%  [1]        dastrategy @ /boot/kernel/kernel
 100.0%  [1]         g_disk_start
  100.0%  [1]          g_io_request
   100.0%  [1]           dev_strategy_csw

00.34%  [1]        rdtsc @ /boot/kernel/kernel
 100.0%  [1]         mi_switch
  100.0%  [1]          sleepq_wait
   100.0%  [1]           _sleep

00.34%  [1]        Xxen_intr_upcall @ /boot/kernel/kernel

00.34%  [1]        __cap_rights_init @ /boot/kernel/kernel
 100.0%  [1]         kern_readv
  100.0%  [1]          sys_read
   100.0%  [1]           amd64_syscall

00.34%  [1]        biodone @ /boot/kernel/kernel
 100.0%  [1]         dadone
  100.0%  [1]          xpt_done_process
   100.0%  [1]           xpt_done_td

00.34%  [1]        sched_add @ /boot/kernel/kernel
 100.0%  [1]         intr_event_schedule_thread
  100.0%  [1]          intr_event_handle
   100.0%  [1]           intr_execute_handlers

00.34%  [1]        sched_runnable @ /boot/kernel/kernel
 100.0%  [1]         cpu_idle
  100.0%  [1]          sched_idletd
   100.0%  [1]           fork_exit

00.34%  [1]        _sleep @ /boot/kernel/kernel
 100.0%  [1]         bwait
  100.0%  [1]          physio
   100.0%  [1]           devfs_read_f

00.34%  [1]        intr_event_schedule_thread @ /boot/kernel/kernel
 100.0%  [1]         intr_event_handle
  100.0%  [1]          intr_execute_handlers
   100.0%  [1]           lapic_handle_intr

00.34%  [1]        callout_lock @ /boot/kernel/kernel
 100.0%  [1]         _callout_stop_safe
  100.0%  [1]          mrsas_cmd_done @ /boot/kernel/mrsas.ko
   100.0%  [1]           mrsas_complete_cmd

00.34%  [1]        scsi_action @ /boot/kernel/kernel
 100.0%  [1]         dastart
  100.0%  [1]          xpt_run_allocq
   100.0%  [1]           dastrategy

00.34%  [1]        critical_exit @ /boot/kernel/kernel
 100.0%  [1]         spinlock_exit
  100.0%  [1]          sched_idletd
   100.0%  [1]           fork_exit

00.34%  [1]        dofileread @ /boot/kernel/kernel
 100.0%  [1]         kern_readv
  100.0%  [1]          sys_read
   100.0%  [1]           amd64_syscall

00.34%  [1]        tdq_lock_pair @ /boot/kernel/kernel
 100.0%  [1]         sched_idletd
  100.0%  [1]          fork_exit

00.34%  [1]        sleepq_wait @ /boot/kernel/kernel
 100.0%  [1]         _sleep
  100.0%  [1]          bwait
   100.0%  [1]           physio

00.34%  [1]        syscall_thread_exit @ /boot/kernel/kernel
 100.0%  [1]         amd64_syscall

00.34%  [1]        vm_fault_quick_hold_pages @ /boot/kernel/kernel
 100.0%  [1]         vmapbuf
  100.0%  [1]          physio
   100.0%  [1]           devfs_read_f

00.34%  [1]        xpt_done @ /boot/kernel/kernel
 100.0%  [1]         mrsas_cmd_done @ /boot/kernel/mrsas.ko
  100.0%  [1]          mrsas_complete_cmd
   100.0%  [1]           intr_event_execute_handlers @ /boot/kernel/kernel

00.34%  [1]        runq_choose @ /boot/kernel/kernel
 100.0%  [1]         sched_choose
  100.0%  [1]          choosethread
   100.0%  [1]           sched_switch

00.34%  [1]        bcopy @ /boot/kernel/kernel
 100.0%  [1]         xpt_run_devq
  100.0%  [1]          xpt_action_default
   100.0%  [1]           dastart

00.34%  [1]        relpbuf @ /boot/kernel/kernel
 100.0%  [1]         physio
  100.0%  [1]          devfs_read_f
   100.0%  [1]           dofileread

00.34%  [1]        g_trace @ /boot/kernel/kernel
 100.0%  [1]         g_dev_strategy
  100.0%  [1]          dev_strategy_csw
   100.0%  [1]           physio

00.34%  [1]        g_new_bio @ /boot/kernel/kernel
 100.0%  [1]         dev_strategy_csw
  100.0%  [1]          physio
   100.0%  [1]           devfs_read_f

00.34%  [1]        mrsas_action @ /boot/kernel/mrsas.ko
 100.0%  [1]         xpt_run_devq @ /boot/kernel/kernel
  100.0%  [1]          xpt_action_default
   100.0%  [1]           dastart

00.34%  [1]        dev_strategy_csw @ /boot/kernel/kernel
 100.0%  [1]         physio
  100.0%  [1]          devfs_read_f
   100.0%  [1]           dofileread

00.34%  [1]        uma_zalloc_arg @ /boot/kernel/kernel
 100.0%  [1]         dev_strategy_csw
  100.0%  [1]          physio
   100.0%  [1]           devfs_read_f

00.34%  [1]        camq_remove @ /boot/kernel/kernel
 100.0%  [1]         xpt_run_devq
  100.0%  [1]          xpt_action_default
   100.0%  [1]           dastart

00.34%  [1]        cpu_set_syscall_retval @ /boot/kernel/kernel
 100.0%  [1]         amd64_syscall

00.34%  [1]        sched_choose @ /boot/kernel/kernel
 100.0%  [1]         choosethread
  100.0%  [1]          sched_switch
   100.0%  [1]           mi_switch

@ CPU_CLK_UNHALTED_CORE [291 samples]

11.68%  [34]       Xinvlrng @ /boot/kernel/kernel

09.97%  [29]       invlrng_handler @ /boot/kernel/kernel

08.93%  [26]       cpu_idle @ /boot/kernel/kernel
 100.0%  [26]        sched_idletd
  100.0%  [26]         fork_exit

05.15%  [15]       __mtx_lock_sleep @ /boot/kernel/kernel
 53.33%  [8]         __mtx_lock_flags
  75.00%  [6]          mrsas_cmd_done @ /boot/kernel/mrsas.ko
   100.0%  [6]           mrsas_complete_cmd
  12.50%  [1]          mrsas_get_mpt_cmd
   100.0%  [1]           mrsas_action
  12.50%  [1]          mrsas_unmap_request
   100.0%  [1]           mrsas_cmd_done
 20.00%  [3]         vmem_xalloc @ /boot/kernel/kernel
  100.0%  [3]          vmem_alloc
   100.0%  [3]           g_io_check
 06.67%  [1]         g_disk_start
  100.0%  [1]          g_io_request
   100.0%  [1]           dev_strategy_csw
 06.67%  [1]         vmem_xfree
  100.0%  [1]          biodone
   100.0%  [1]           dadone
 06.67%  [1]         xpt_action_default
  100.0%  [1]          dastart
   100.0%  [1]           xpt_run_allocq
 06.67%  [1]         xpt_run_devq
  100.0%  [1]          xpt_action_default
   100.0%  [1]           dastart

04.12%  [12]       cpu_search_lowest @ /boot/kernel/kernel
 100.0%  [12]        cpu_search_lowest
  75.00%  [9]          cpu_search_lowest
   100.0%  [9]           sched_pickcpu
  25.00%  [3]          sched_pickcpu
   100.0%  [3]           sched_add

03.78%  [11]       cpu_search_highest @ /boot/kernel/kernel
 81.82%  [9]         cpu_search_highest
  77.78%  [7]          sched_idletd
   100.0%  [7]           fork_exit
  22.22%  [2]          cpu_search_highest
   100.0%  [2]           sched_idletd
 18.18%  [2]         sched_idletd
  100.0%  [2]          fork_exit

03.09%  [9]        cpu_switch @ /boot/kernel/kernel
 88.89%  [8]         mi_switch
  50.00%  [4]          sched_idletd
   100.0%  [4]           fork_exit
  50.00%  [4]          sleepq_wait
   100.0%  [4]           _sleep

02.75%  [8]        smp_tlb_shootdown @ /boot/kernel/kernel
 100.0%  [8]         pmap_invalidate_range
  100.0%  [8]          biodone
   100.0%  [8]           dadone

02.41%  [7]        sched_switch @ /boot/kernel/kernel
 100.0%  [7]         mi_switch
  85.71%  [6]          sleepq_wait
   100.0%  [6]           _sleep
  14.29%  [1]          sched_idletd
   100.0%  [1]           fork_exit

02.41%  [7]        doreti @ /boot/kernel/kernel

02.06%  [6]        xpt_run_devq @ /boot/kernel/kernel
 83.33%  [5]         xpt_action_default
  100.0%  [5]          dastart
   100.0%  [5]           xpt_run_allocq
 16.67%  [1]         xpt_done_process
  100.0%  [1]          xpt_done_td
   100.0%  [1]           fork_exit

02.06%  [6]        thread_lock_flags_ @ /boot/kernel/kernel
 33.33%  [2]         sched_idletd
  100.0%  [2]          fork_exit
 33.33%  [2]         propagate_priority
  100.0%  [2]          turnstile_wait
   100.0%  [2]           __mtx_lock_sleep
 16.67%  [1]         sleepq_wait
  100.0%  [1]          _sleep
   100.0%  [1]           bwait
 16.67%  [1]         intr_event_schedule_thread
  100.0%  [1]          intr_event_handle
   100.0%  [1]           intr_execute_handlers

01.72%  [5]        __lockmgr_args @ /boot/kernel/kernel
 100.0%  [5]         getpbuf
  100.0%  [5]          physio
   100.0%  [5]           devfs_read_f

01.72%  [5]        ithread_loop @ /boot/kernel/kernel
 100.0%  [5]         fork_exit

01.37%  [4]        spinlock_exit @ /boot/kernel/kernel
 25.00%  [1]         sched_idletd
  100.0%  [1]          fork_exit
 25.00%  [1]         _mtx_lock_spin_cookie
  100.0%  [1]          smp_tlb_shootdown
   100.0%  [1]           pmap_invalidate_range
 25.00%  [1]         wakeup
  100.0%  [1]          bdone
   100.0%  [1]           bufdone
 25.00%  [1]         ithread_loop
  100.0%  [1]          fork_exit

01.03%  [3]        devfs_write_f @ /boot/kernel/kernel
 100.0%  [3]         dofilewrite
  100.0%  [3]          kern_writev
   100.0%  [3]           sys_write

01.03%  [3]        mi_switch @ /boot/kernel/kernel
 66.67%  [2]         sleepq_wait
  100.0%  [2]          _sleep
   100.0%  [2]           bwait
 33.33%  [1]         ithread_loop
  100.0%  [1]          fork_exit

01.03%  [3]        g_io_request @ /boot/kernel/kernel
 100.0%  [3]         dev_strategy_csw
  100.0%  [3]          physio
   100.0%  [3]           devfs_read_f

01.03%  [3]        uma_zalloc_arg @ /boot/kernel/kernel
 33.33%  [1]         malloc
  100.0%  [1]          xpt_run_allocq
   100.0%  [1]           dastrategy
 33.33%  [1]         dev_strategy_csw
  100.0%  [1]          physio
   100.0%  [1]           devfs_read_f
 33.33%  [1]         g_clone_bio
  100.0%  [1]          g_dev_strategy
   100.0%  [1]           dev_strategy_csw

01.03%  [3]        bzero @ /boot/kernel/kernel
 66.67%  [2]         g_clone_bio
  100.0%  [2]          g_dev_strategy
   100.0%  [2]           dev_strategy_csw
 33.33%  [1]         dev_strategy_csw
  100.0%  [1]          physio
   100.0%  [1]           devfs_read_f

01.03%  [3]        _mtx_lock_spin_cookie @ /boot/kernel/kernel
 66.67%  [2]         smp_tlb_shootdown
  100.0%  [2]          pmap_invalidate_range
   100.0%  [2]           biodone
 33.33%  [1]         sched_idletd
  100.0%  [1]          fork_exit

01.03%  [3]        g_dev_strategy @ /boot/kernel/kernel
 100.0%  [3]         dev_strategy_csw
  100.0%  [3]          physio
   100.0%  [3]           devfs_read_f

01.03%  [3]        pmap_invalidate_range @ /boot/kernel/kernel
 100.0%  [3]         biodone
  100.0%  [3]          dadone
   100.0%  [3]           xpt_done_process

01.03%  [3]        devvn_refthread @ /boot/kernel/kernel
 66.67%  [2]         devfs_write_f
  100.0%  [2]          dofilewrite
   100.0%  [2]           kern_writev
 33.33%  [1]         devfs_read_f
  100.0%  [1]          dofileread
   100.0%  [1]           kern_readv

01.03%  [3]        xpt_done_process @ /boot/kernel/kernel
 100.0%  [3]         xpt_done_td
  100.0%  [3]          fork_exit

01.03%  [3]        sched_idletd @ /boot/kernel/kernel
 100.0%  [3]         fork_exit

01.03%  [3]        critical_exit @ /boot/kernel/kernel
 66.67%  [2]         spinlock_exit
  50.00%  [1]          sched_idletd
   100.0%  [1]           fork_exit
  50.00%  [1]          _sleep
   100.0%  [1]           bwait
 33.33%  [1]         intr_event_handle
  100.0%  [1]          intr_execute_handlers
   100.0%  [1]           lapic_handle_intr

01.03%  [3]        callout_lock @ /boot/kernel/kernel
 66.67%  [2]         callout_reset_sbt_on
  100.0%  [2]          mrsas_action @ /boot/kernel/mrsas.ko
   100.0%  [2]           xpt_run_devq @ /boot/kernel/kernel
 33.33%  [1]         _callout_stop_safe
  100.0%  [1]          mrsas_cmd_done @ /boot/kernel/mrsas.ko
   100.0%  [1]           mrsas_complete_cmd

00.69%  [2]        sched_choose @ /boot/kernel/kernel
 100.0%  [2]         choosethread
  100.0%  [2]          sched_switch
   100.0%  [2]           mi_switch

00.69%  [2]        xpt_action_default @ /boot/kernel/kernel
 100.0%  [2]         dastart
  100.0%  [2]          xpt_run_allocq
   100.0%  [2]           dastrategy

00.69%  [2]        Xfast_syscall @ /boot/kernel/kernel

00.69%  [2]        lapic_ipi_vectored @ /boot/kernel/kernel
 100.0%  [2]         smp_tlb_shootdown
  100.0%  [2]          pmap_invalidate_range
   100.0%  [2]           biodone

00.69%  [2]        dastrategy @ /boot/kernel/kernel
 100.0%  [2]         g_disk_start
  100.0%  [2]          g_io_request
   100.0%  [2]           dev_strategy_csw

00.69%  [2]        critical_enter @ /boot/kernel/kernel
 50.00%  [1]         thread_lock_flags_
  100.0%  [1]          sleepq_add
   100.0%  [1]           _sleep
 50.00%  [1]         uma_zfree_arg
  100.0%  [1]          g_dev_done
   100.0%  [1]           g_io_deliver

00.69%  [2]        atomic_fetchadd_int @ /boot/kernel/kernel
 50.00%  [1]         mrsas_complete_cmd @ /boot/kernel/mrsas.ko
  100.0%  [1]          intr_event_execute_handlers @ /boot/kernel/kernel
   100.0%  [1]           ithread_loop
 50.00%  [1]         mrsas_action @ /boot/kernel/mrsas.ko
  100.0%  [1]          xpt_run_devq @ /boot/kernel/kernel
   100.0%  [1]           xpt_action_default

00.69%  [2]        sched_add @ /boot/kernel/kernel
 50.00%  [1]         turnstile_unpend
  100.0%  [1]          __mtx_unlock_sleep
   100.0%  [1]           xpt_run_devq
 50.00%  [1]         setrunnable
  100.0%  [1]          sleepq_broadcast
   100.0%  [1]           wakeup

00.69%  [2]        sched_runnable @ /boot/kernel/kernel
 100.0%  [2]         cpu_idle
  100.0%  [2]          sched_idletd
   100.0%  [2]           fork_exit

00.69%  [2]        __mtx_unlock_flags @ /boot/kernel/kernel
 50.00%  [1]         mrsas_get_mpt_cmd @ /boot/kernel/mrsas.ko
  100.0%  [1]          mrsas_action
   100.0%  [1]           xpt_run_devq @ /boot/kernel/kernel
 50.00%  [1]         mrsas_complete_cmd @ /boot/kernel/mrsas.ko
  100.0%  [1]          intr_event_execute_handlers @ /boot/kernel/kernel
   100.0%  [1]           ithread_loop

00.69%  [2]        __sys_read @ /lib/libc.so.7

00.69%  [2]        dev_relthread @ /boot/kernel/kernel
 100.0%  [2]         devfs_read_f
  100.0%  [2]          dofileread
   100.0%  [2]           kern_readv

00.69%  [2]        xpt_done @ /boot/kernel/kernel
 100.0%  [2]         mrsas_cmd_done @ /boot/kernel/mrsas.ko
  100.0%  [2]          mrsas_complete_cmd
   100.0%  [2]           intr_event_execute_handlers @ /boot/kernel/kernel

00.69%  [2]        uma_zfree_arg @ /boot/kernel/kernel
 100.0%  [2]         g_dev_done
  100.0%  [2]          g_io_deliver
   100.0%  [2]           g_disk_done_single

00.34%  [1]        intr_event_handle @ /boot/kernel/kernel
 100.0%  [1]         intr_execute_handlers
  100.0%  [1]          lapic_handle_intr

00.34%  [1]        syscall_thread_enter @ /boot/kernel/kernel
 100.0%  [1]         amd64_syscall

00.34%  [1]        __cap_rights_init @ /boot/kernel/kernel
 100.0%  [1]         kern_writev
  100.0%  [1]          sys_write
   100.0%  [1]           amd64_syscall

00.34%  [1]        mrsas_get_mpt_cmd @ /boot/kernel/mrsas.ko
 100.0%  [1]         mrsas_action
  100.0%  [1]          xpt_run_devq @ /boot/kernel/kernel
   100.0%  [1]           xpt_action_default

00.34%  [1]        cpu_set_syscall_retval @ /boot/kernel/kernel
 100.0%  [1]         amd64_syscall

00.34%  [1]        g_dev_done @ /boot/kernel/kernel
 100.0%  [1]         g_io_deliver
  100.0%  [1]          g_disk_done_single
   100.0%  [1]           dadone

00.34%  [1]        thread_lock_set @ /boot/kernel/kernel
 100.0%  [1]         sleepq_switch
  100.0%  [1]          sleepq_wait
   100.0%  [1]           _sleep

00.34%  [1]        __sys_write @ /lib/libc.so.7

00.34%  [1]        mrsas_data_load_cb @ /boot/kernel/mrsas.ko
 100.0%  [1]         bus_dmamap_load @ /boot/kernel/kernel
  100.0%  [1]          mrsas_map_request @ /boot/kernel/mrsas.ko
   100.0%  [1]           mrsas_build_dcdb

00.34%  [1]        kern_readv @ /boot/kernel/kernel
 100.0%  [1]         sys_read
  100.0%  [1]          amd64_syscall

00.34%  [1]        sleepq_wait @ /boot/kernel/kernel
 100.0%  [1]         _sleep
  100.0%  [1]          bwait
   100.0%  [1]           physio

00.34%  [1]        pmap_qenter @ /boot/kernel/kernel
 100.0%  [1]         g_io_check
  100.0%  [1]          g_io_request
   100.0%  [1]           dev_strategy_csw

00.34%  [1]        lock_mtx @ /boot/kernel/kernel
 100.0%  [1]         _sleep
  100.0%  [1]          bwait
   100.0%  [1]           physio

00.34%  [1]        dadone @ /boot/kernel/kernel
 100.0%  [1]         xpt_done_process
  100.0%  [1]          xpt_done_td
   100.0%  [1]           fork_exit

00.34%  [1]        bwait @ /boot/kernel/kernel
 100.0%  [1]         physio
  100.0%  [1]          devfs_read_f
   100.0%  [1]           dofileread

00.34%  [1]        mrsas_isr @ /boot/kernel/mrsas.ko
 100.0%  [1]         intr_event_execute_handlers @ /boot/kernel/kernel
  100.0%  [1]          ithread_loop
   100.0%  [1]           fork_exit

00.34%  [1]        g_io_check @ /boot/kernel/kernel
 100.0%  [1]         g_io_request
  100.0%  [1]          dev_strategy_csw
   100.0%  [1]           physio

00.34%  [1]        g_disk_done_single @ /boot/kernel/kernel
 100.0%  [1]         dadone
  100.0%  [1]          xpt_done_process
   100.0%  [1]           xpt_done_td

00.34%  [1]        turnstile_broadcast @ /boot/kernel/kernel
 100.0%  [1]         __mtx_unlock_sleep
  100.0%  [1]          xpt_run_devq
   100.0%  [1]           xpt_action_default

00.34%  [1]        vm_fault_quick_hold_pages @ /boot/kernel/kernel
 100.0%  [1]         vmapbuf
  100.0%  [1]          physio
   100.0%  [1]           devfs_read_f

00.34%  [1]        getpbuf @ /boot/kernel/kernel
 100.0%  [1]         physio
  100.0%  [1]          devfs_read_f
   100.0%  [1]           dofileread

00.34%  [1]        vmem_xfree @ /boot/kernel/kernel
 100.0%  [1]         biodone
  100.0%  [1]          dadone
   100.0%  [1]           xpt_done_process

00.34%  [1]        smp_invlpg_range @ /boot/kernel/kernel
 100.0%  [1]         pmap_invalidate_range
  100.0%  [1]          biodone
   100.0%  [1]           dadone

00.34%  [1]        bdone @ /boot/kernel/kernel
 100.0%  [1]         bufdone
  100.0%  [1]          bufdonebio
   100.0%  [1]           g_io_deliver

00.34%  [1]        Xapic_isr1 @ /boot/kernel/kernel

00.34%  [1]        physio @ /boot/kernel/kernel
 100.0%  [1]         devfs_read_f
  100.0%  [1]          dofileread
   100.0%  [1]           kern_readv

00.34%  [1]        sleepq_resume_thread @ /boot/kernel/kernel
 100.0%  [1]         sleepq_broadcast
  100.0%  [1]          wakeup
   100.0%  [1]           bdone

00.34%  [1]        bcopy @ /boot/kernel/kernel
 100.0%  [1]         amd64_syscall

00.34%  [1]        bioq_insert_tail @ /boot/kernel/kernel
 100.0%  [1]         dastrategy
  100.0%  [1]          g_disk_start
   100.0%  [1]           g_io_request

00.34%  [1]        binuptime @ /boot/kernel/kernel
 100.0%  [1]         g_disk_done_single
  100.0%  [1]          dadone
   100.0%  [1]           xpt_done_process

00.34%  [1]        cam_ccbq_remove_ccb @ /boot/kernel/kernel
 100.0%  [1]         xpt_run_devq
  100.0%  [1]          xpt_action_default
   100.0%  [1]           dastart

00.34%  [1]        mrsas_action @ /boot/kernel/mrsas.ko
 100.0%  [1]         xpt_run_devq @ /boot/kernel/kernel
  100.0%  [1]          xpt_action_default
   100.0%  [1]           dastart

00.34%  [1]        fget_unlocked @ /boot/kernel/kernel
 100.0%  [1]         _fget
  100.0%  [1]          kern_readv
   100.0%  [1]           sys_read

00.34%  [1]        sleepq_lock @ /boot/kernel/kernel
 100.0%  [1]         _sleep
  100.0%  [1]          bwait
   100.0%  [1]           physio

00.34%  [1]        devfs_read_f @ /boot/kernel/kernel
 100.0%  [1]         dofileread
  100.0%  [1]          kern_readv
   100.0%  [1]           sys_read

00.34%  [1]        sched_pickcpu @ /boot/kernel/kernel
 100.0%  [1]         sched_add
  100.0%  [1]          turnstile_unpend
   100.0%  [1]           __mtx_unlock_sleep

00.34%  [1]        g_disk_start @ /boot/kernel/kernel
 100.0%  [1]         g_io_request
  100.0%  [1]          dev_strategy_csw
   100.0%  [1]           physio

00.34%  [1]        dastart @ /boot/kernel/kernel
 100.0%  [1]         xpt_run_allocq
  100.0%  [1]          dastrategy
   100.0%  [1]           g_disk_start

00.34%  [1]        mrsas_complete_cmd @ /boot/kernel/mrsas.ko
 100.0%  [1]         intr_event_execute_handlers @ /boot/kernel/kernel
  100.0%  [1]          ithread_loop
   100.0%  [1]           fork_exit

00.34%  [1]        xpt_release_ccb @ /boot/kernel/kernel
 100.0%  [1]         dadone
  100.0%  [1]          xpt_done_process
   100.0%  [1]           xpt_done_td

@ CPU_CLK_UNHALTED_CORE [290 samples]

12.76%  [37]       invlrng_handler @ /boot/kernel/kernel

09.31%  [27]       cpu_idle @ /boot/kernel/kernel
 100.0%  [27]        sched_idletd
  100.0%  [27]         fork_exit

07.93%  [23]       Xinvlrng @ /boot/kernel/kernel

06.21%  [18]       smp_tlb_shootdown @ /boot/kernel/kernel
 100.0%  [18]        pmap_invalidate_range
  100.0%  [18]         biodone
   100.0%  [18]          dadone

03.45%  [10]       __mtx_lock_sleep @ /boot/kernel/kernel
 60.00%  [6]         __mtx_lock_flags
  100.0%  [6]          mrsas_cmd_done @ /boot/kernel/mrsas.ko
   100.0%  [6]           mrsas_complete_cmd
 20.00%  [2]         xpt_run_devq @ /boot/kernel/kernel
  100.0%  [2]          xpt_action_default
   100.0%  [2]           dastart
 10.00%  [1]         vmem_xalloc
  100.0%  [1]          vmem_alloc
   100.0%  [1]           g_io_check
 10.00%  [1]         xpt_done_process
  100.0%  [1]          xpt_done_td
   100.0%  [1]           fork_exit

03.10%  [9]        cpu_search_highest @ /boot/kernel/kernel
 66.67%  [6]         cpu_search_highest
  66.67%  [4]          sched_idletd
   100.0%  [4]           fork_exit
  33.33%  [2]          cpu_search_highest
   100.0%  [2]           sched_idletd
 33.33%  [3]         sched_idletd
  100.0%  [3]          fork_exit

02.76%  [8]        cpu_search_lowest @ /boot/kernel/kernel
 100.0%  [8]         cpu_search_lowest
  75.00%  [6]          cpu_search_lowest
   100.0%  [6]           sched_pickcpu
  25.00%  [2]          sched_pickcpu
   100.0%  [2]           sched_add

02.41%  [7]        cpu_switch @ /boot/kernel/kernel
 100.0%  [7]         mi_switch
  42.86%  [3]          sleepq_wait
   100.0%  [3]           _sleep
  42.86%  [3]          sched_idletd
   100.0%  [3]           fork_exit
  14.29%  [1]          critical_exit
   100.0%  [1]           intr_event_handle

02.41%  [7]        sched_switch @ /boot/kernel/kernel
 100.0%  [7]         mi_switch
  71.43%  [5]          sleepq_wait
   100.0%  [5]           _sleep
  14.29%  [1]          sched_idletd
   100.0%  [1]           fork_exit
  14.29%  [1]          critical_exit
   100.0%  [1]           intr_event_handle

02.41%  [7]        doreti @ /boot/kernel/kernel

02.07%  [6]        spinlock_exit @ /boot/kernel/kernel
 33.33%  [2]         sched_idletd
  100.0%  [2]          fork_exit
 33.33%  [2]         _sleep
  100.0%  [2]          bwait
   100.0%  [2]           physio
 16.67%  [1]         wakeup
  100.0%  [1]          bdone
   100.0%  [1]           bufdone
 16.67%  [1]         smp_tlb_shootdown
  100.0%  [1]          pmap_invalidate_range
   100.0%  [1]           biodone

02.07%  [6]        __mtx_unlock_flags @ /boot/kernel/kernel
 16.67%  [1]         mrsas_action @ /boot/kernel/mrsas.ko
  100.0%  [1]          xpt_run_devq @ /boot/kernel/kernel
   100.0%  [1]           xpt_action_default
 16.67%  [1]         xpt_run_devq
  100.0%  [1]          xpt_action_default
   100.0%  [1]           dastart
 16.67%  [1]         mrsas_map_request @ /boot/kernel/mrsas.ko
  100.0%  [1]          mrsas_build_dcdb
   100.0%  [1]           mrsas_action
 16.67%  [1]         mrsas_get_mpt_cmd
  100.0%  [1]          mrsas_action
   100.0%  [1]           xpt_run_devq @ /boot/kernel/kernel
 16.67%  [1]         mrsas_complete_cmd @ /boot/kernel/mrsas.ko
  100.0%  [1]          intr_event_execute_handlers @ /boot/kernel/kernel
   100.0%  [1]           ithread_loop
 16.67%  [1]         mrsas_cmd_done @ /boot/kernel/mrsas.ko
  100.0%  [1]          mrsas_complete_cmd
   100.0%  [1]           intr_event_execute_handlers @ /boot/kernel/kernel

01.72%  [5]        sched_idletd @ /boot/kernel/kernel
 100.0%  [5]         fork_exit

01.38%  [4]        critical_exit @ /boot/kernel/kernel
 75.00%  [3]         spinlock_exit
  33.33%  [1]          sched_idletd
   100.0%  [1]           fork_exit
  33.33%  [1]          _sleep
   100.0%  [1]           bwait
  33.33%  [1]          thread_lock_flags_
   100.0%  [1]           intr_event_schedule_thread
 25.00%  [1]         uma_zalloc_arg
  100.0%  [1]          malloc
   100.0%  [1]           xpt_run_allocq

01.03%  [3]        _mtx_lock_spin_cookie @ /boot/kernel/kernel
 66.67%  [2]         tdq_lock_pair
  100.0%  [2]          sched_idletd
   100.0%  [2]           fork_exit
 33.33%  [1]         smp_tlb_shootdown
  100.0%  [1]          pmap_invalidate_range
   100.0%  [1]           biodone

01.03%  [3]        vmem_xalloc @ /boot/kernel/kernel
 100.0%  [3]         vmem_alloc
  100.0%  [3]          g_io_check
   100.0%  [3]           g_io_request

01.03%  [3]        __sys_read @ /lib/libc.so.7

01.03%  [3]        thread_lock_flags_ @ /boot/kernel/kernel
 66.67%  [2]         sleepq_add
  100.0%  [2]          _sleep
   100.0%  [2]           bwait
 33.33%  [1]         sched_idletd
  100.0%  [1]          fork_exit

01.03%  [3]        biodone @ /boot/kernel/kernel
 66.67%  [2]         dadone
  100.0%  [2]          xpt_done_process
   100.0%  [2]           xpt_done_td
 33.33%  [1]         g_io_deliver
  100.0%  [1]          g_disk_done_single
   100.0%  [1]           dadone

01.03%  [3]        critical_enter @ /boot/kernel/kernel
 66.67%  [2]         smp_tlb_shootdown
  100.0%  [2]          pmap_invalidate_range
   100.0%  [2]           biodone
 33.33%  [1]         uma_zalloc_arg
  100.0%  [1]          dev_strategy_csw
   100.0%  [1]           physio

01.03%  [3]        dastart @ /boot/kernel/kernel
 100.0%  [3]         xpt_run_allocq
  100.0%  [3]          dastrategy
   100.0%  [3]           g_disk_start

01.03%  [3]        devvn_refthread @ /boot/kernel/kernel
 66.67%  [2]         devfs_read_f
  100.0%  [2]          dofileread
   100.0%  [2]           kern_readv
 33.33%  [1]         devfs_write_f
  100.0%  [1]          dofilewrite
   100.0%  [1]           kern_writev

01.03%  [3]        vmem_xfree @ /boot/kernel/kernel
 100.0%  [3]         biodone
  100.0%  [3]          dadone
   100.0%  [3]           xpt_done_process

01.03%  [3]        amd64_syscall @ /boot/kernel/kernel

01.03%  [3]        xpt_run_devq @ /boot/kernel/kernel
 100.0%  [3]         xpt_action_default
  100.0%  [3]          dastart
   100.0%  [3]           xpt_run_allocq

00.69%  [2]        xpt_action_default @ /boot/kernel/kernel
 100.0%  [2]         dastart
  100.0%  [2]          xpt_run_allocq
   100.0%  [2]           dastrategy

00.69%  [2]        g_dev_strategy @ /boot/kernel/kernel
 100.0%  [2]         dev_strategy_csw
  100.0%  [2]          physio
   100.0%  [2]           devfs_read_f

00.69%  [2]        foffset_unlock @ /boot/kernel/kernel
 100.0%  [2]         devfs_write_f
  100.0%  [2]          dofilewrite
   100.0%  [2]           kern_writev

00.69%  [2]        ithread_loop @ /boot/kernel/kernel
 100.0%  [2]         fork_exit

00.69%  [2]        pcpu_find @ /boot/kernel/kernel
 100.0%  [2]         tdq_notify
  100.0%  [2]          sched_add
   50.00%  [1]           intr_event_schedule_thread
   50.00%  [1]           turnstile_unpend

00.69%  [2]        g_disk_start @ /boot/kernel/kernel
 100.0%  [2]         g_io_request
  100.0%  [2]          dev_strategy_csw
   100.0%  [2]           physio

00.69%  [2]        relpbuf @ /boot/kernel/kernel
 100.0%  [2]         physio
  100.0%  [2]          devfs_read_f
   100.0%  [2]           dofileread

00.69%  [2]        mrsas_get_mpt_cmd @ /boot/kernel/mrsas.ko
 100.0%  [2]         mrsas_action
  100.0%  [2]          xpt_run_devq @ /boot/kernel/kernel
   100.0%  [2]           xpt_action_default

00.69%  [2]        pmap_kextract @ /boot/kernel/kernel
 100.0%  [2]         bounce_bus_dmamap_load_buffer
  100.0%  [2]          bus_dmamap_load
   100.0%  [2]           mrsas_map_request @ /boot/kernel/mrsas.ko

00.69%  [2]        _callout_stop_safe @ /boot/kernel/kernel
 100.0%  [2]         mrsas_cmd_done @ /boot/kernel/mrsas.ko
  100.0%  [2]          mrsas_complete_cmd
   100.0%  [2]           intr_event_execute_handlers @ /boot/kernel/kernel

00.69%  [2]        bzero @ /boot/kernel/kernel
 50.00%  [1]         mrsas_action @ /boot/kernel/mrsas.ko
  100.0%  [1]          xpt_run_devq @ /boot/kernel/kernel
   100.0%  [1]           xpt_action_default
 50.00%  [1]         g_clone_bio
  100.0%  [1]          g_dev_strategy
   100.0%  [1]           dev_strategy_csw

00.69%  [2]        sched_pickcpu @ /boot/kernel/kernel
 100.0%  [2]         sched_add
  50.00%  [1]          setrunnable
   100.0%  [1]           sleepq_broadcast
  50.00%  [1]          intr_event_schedule_thread
   100.0%  [1]           intr_event_handle

00.69%  [2]        bt_freetrim @ /boot/kernel/kernel
 100.0%  [2]         biodone
  100.0%  [2]          dadone
   100.0%  [2]           xpt_done_process

00.69%  [2]        g_io_request @ /boot/kernel/kernel
 100.0%  [2]         dev_strategy_csw
  100.0%  [2]          physio
   100.0%  [2]           devfs_read_f

00.69%  [2]        dadone @ /boot/kernel/kernel
 100.0%  [2]         xpt_done_process
  100.0%  [2]          xpt_done_td
   100.0%  [2]           fork_exit

00.69%  [2]        sys_read @ /boot/kernel/kernel
 100.0%  [2]         amd64_syscall

00.69%  [2]        setrunnable @ /boot/kernel/kernel
 100.0%  [2]         sleepq_broadcast
  100.0%  [2]          wakeup
   50.00%  [1]           mrsas_cmd_done @ /boot/kernel/mrsas.ko
   50.00%  [1]           bdone @ /boot/kernel/kernel

00.69%  [2]        __lockmgr_args @ /boot/kernel/kernel
 100.0%  [2]         getpbuf
  100.0%  [2]          physio
   100.0%  [2]           devfs_read_f

00.34%  [1]        uma_zalloc_arg @ /boot/kernel/kernel
 100.0%  [1]         malloc
  100.0%  [1]          xpt_run_allocq
   100.0%  [1]           dastrategy

00.34%  [1]        g_is_geom_thread @ /boot/kernel/kernel
 100.0%  [1]         g_io_deliver
  100.0%  [1]          g_disk_done_single
   100.0%  [1]           dadone

00.34%  [1]        camq_remove @ /boot/kernel/kernel
 100.0%  [1]         xpt_run_devq
  100.0%  [1]          xpt_action_default
   100.0%  [1]           dastart

00.34%  [1]        sched_choose @ /boot/kernel/kernel
 100.0%  [1]         choosethread
  100.0%  [1]          sched_switch
   100.0%  [1]           mi_switch

00.34%  [1]        tdq_notify @ /boot/kernel/kernel
 100.0%  [1]         sched_add
  100.0%  [1]          setrunnable
   100.0%  [1]           sleepq_broadcast

00.34%  [1]        intr_execute_handlers @ /boot/kernel/kernel
 100.0%  [1]         lapic_handle_intr

00.34%  [1]        xpt_action @ /boot/kernel/kernel
 100.0%  [1]         dastart
  100.0%  [1]          xpt_run_allocq
   100.0%  [1]           dastrategy

00.34%  [1]        turnstile_trywait @ /boot/kernel/kernel
 100.0%  [1]         __mtx_lock_sleep
  100.0%  [1]          xpt_done_process
   100.0%  [1]           xpt_done_td

00.34%  [1]        kern_writev @ /boot/kernel/kernel
 100.0%  [1]         sys_write
  100.0%  [1]          amd64_syscall

00.34%  [1]        Xfast_syscall @ /boot/kernel/kernel

00.34%  [1]        wakeup @ /boot/kernel/kernel
 100.0%  [1]         bdone
  100.0%  [1]          bufdone
   100.0%  [1]           bufdonebio

00.34%  [1]        devstat_end_transaction @ /boot/kernel/kernel
 100.0%  [1]         devstat_end_transaction_bio_bt
  100.0%  [1]          g_disk_done_single
   100.0%  [1]           dadone

00.34%  [1]        sleepq_broadcast @ /boot/kernel/kernel
 100.0%  [1]         wakeup
  100.0%  [1]          bdone
   100.0%  [1]           bufdone

00.34%  [1]        xpt_run_allocq @ /boot/kernel/kernel
 100.0%  [1]         dastrategy
  100.0%  [1]          g_disk_start
   100.0%  [1]           g_io_request

00.34%  [1]        g_disk_done_single @ /boot/kernel/kernel
 100.0%  [1]         dadone
  100.0%  [1]          xpt_done_process
   100.0%  [1]           xpt_done_td

00.34%  [1]        lapic_ipi_vectored @ /boot/kernel/kernel
 100.0%  [1]         smp_tlb_shootdown
  100.0%  [1]          pmap_invalidate_range
   100.0%  [1]           biodone

00.34%  [1]        scsi_read_write @ /boot/kernel/kernel
 100.0%  [1]         dastart
  100.0%  [1]          xpt_run_allocq
   100.0%  [1]           dastrategy

00.34%  [1]        dastrategy @ /boot/kernel/kernel
 100.0%  [1]         g_disk_start
  100.0%  [1]          g_io_request
   100.0%  [1]           dev_strategy_csw

00.34%  [1]        kern_readv @ /boot/kernel/kernel
 100.0%  [1]         sys_read
  100.0%  [1]          amd64_syscall

00.34%  [1]        xpt_done_process @ /boot/kernel/kernel
 100.0%  [1]         xpt_done_td
  100.0%  [1]          fork_exit

00.34%  [1]        turnstile_wait @ /boot/kernel/kernel
 100.0%  [1]         __mtx_lock_sleep
  100.0%  [1]          xpt_action_default
   100.0%  [1]           dastart

00.34%  [1]        __sys_write @ /lib/libc.so.7

00.34%  [1]        mtx_pool_find @ /boot/kernel/kernel
 100.0%  [1]         bdone
  100.0%  [1]          bufdone
   100.0%  [1]           bufdonebio

00.34%  [1]        handleevents @ /boot/kernel/kernel
 100.0%  [1]         timercb
  100.0%  [1]          lapic_handle_timer

00.34%  [1]        malloc @ /boot/kernel/kernel
 100.0%  [1]         xpt_run_allocq
  100.0%  [1]          dastrategy
   100.0%  [1]           g_disk_start

00.34%  [1]        atomic_fetchadd_int @ /boot/kernel/kernel
 100.0%  [1]         mrsas_action @ /boot/kernel/mrsas.ko
  100.0%  [1]          xpt_run_devq @ /boot/kernel/kernel
   100.0%  [1]           xpt_action_default

00.34%  [1]        sched_runnable @ /boot/kernel/kernel
 100.0%  [1]         cpu_idle
  100.0%  [1]          sched_idletd
   100.0%  [1]           fork_exit

00.34%  [1]        __cap_rights_is_set @ /boot/kernel/kernel
 100.0%  [1]         fget_unlocked
  100.0%  [1]          _fget
   100.0%  [1]           kern_readv

00.34%  [1]        dofilewrite @ /boot/kernel/kernel
 100.0%  [1]         kern_writev
  100.0%  [1]          sys_write
   100.0%  [1]           amd64_syscall

00.34%  [1]        mrsas_complete_cmd @ /boot/kernel/mrsas.ko
 100.0%  [1]         intr_event_execute_handlers @ /boot/kernel/kernel
  100.0%  [1]          ithread_loop
   100.0%  [1]           fork_exit

00.34%  [1]        __mtx_lock_flags @ /boot/kernel/kernel
 100.0%  [1]         mrsas_get_mpt_cmd @ /boot/kernel/mrsas.ko
  100.0%  [1]          mrsas_action
   100.0%  [1]           xpt_run_devq @ /boot/kernel/kernel

00.34%  [1]        g_dev_done @ /boot/kernel/kernel
 100.0%  [1]         g_io_deliver
  100.0%  [1]          g_disk_done_single
   100.0%  [1]           dadone

00.34%  [1]        thread_lock_set @ /boot/kernel/kernel
 100.0%  [1]         sleepq_switch
  100.0%  [1]          sleepq_wait
   100.0%  [1]           _sleep

00.34%  [1]        sleepq_wait @ /boot/kernel/kernel
 100.0%  [1]         _sleep
  100.0%  [1]          bwait
   100.0%  [1]           physio

00.34%  [1]        sched_sleep @ /boot/kernel/kernel
 100.0%  [1]         sleepq_switch
  100.0%  [1]          sleepq_wait
   100.0%  [1]           _sleep

00.34%  [1]        lock_mtx @ /boot/kernel/kernel
 100.0%  [1]         _sleep
  100.0%  [1]          bwait
   100.0%  [1]           physio

00.34%  [1]        syscall_thread_exit @ /boot/kernel/kernel
 100.0%  [1]         amd64_syscall

00.34%  [1]        bwait @ /boot/kernel/kernel
 100.0%  [1]         physio
  100.0%  [1]          devfs_read_f
   100.0%  [1]           dofileread

00.34%  [1]        choosethread @ /boot/kernel/kernel
 100.0%  [1]         sched_switch
  100.0%  [1]          mi_switch
   100.0%  [1]           sleepq_wait

00.34%  [1]        vm_fault_quick_hold_pages @ /boot/kernel/kernel
 100.0%  [1]         vmapbuf
  100.0%  [1]          physio
   100.0%  [1]           devfs_read_f

00.34%  [1]        getpbuf @ /boot/kernel/kernel
 100.0%  [1]         physio
  100.0%  [1]          devfs_read_f
   100.0%  [1]           dofileread

00.34%  [1]        xpt_done @ /boot/kernel/kernel
 100.0%  [1]         mrsas_cmd_done @ /boot/kernel/mrsas.ko
  100.0%  [1]          mrsas_complete_cmd
   100.0%  [1]           intr_event_execute_handlers @ /boot/kernel/kernel

00.34%  [1]        PHYS_TO_VM_PAGE @ /boot/kernel/kernel
 100.0%  [1]         free
  100.0%  [1]          xpt_release_ccb
   100.0%  [1]           dadone

00.34%  [1]        uma_zfree_arg @ /boot/kernel/kernel
 100.0%  [1]         g_dev_done
  100.0%  [1]          g_io_deliver
   100.0%  [1]           g_disk_done_single

00.34%  [1]        devstat_start_transaction_bio @ /boot/kernel/kernel
 100.0%  [1]         g_disk_start
  100.0%  [1]          g_io_request
   100.0%  [1]           dev_strategy_csw

00.34%  [1]        devfs_write_f @ /boot/kernel/kernel
 100.0%  [1]         dofilewrite
  100.0%  [1]          kern_writev
   100.0%  [1]           sys_write

00.34%  [1]        bcopy @ /boot/kernel/kernel
 100.0%  [1]         amd64_syscall

@ CPU_CLK_UNHALTED_CORE [290 samples]

12.07%  [35]       cpu_idle @ /boot/kernel/kernel
 100.0%  [35]        sched_idletd
  100.0%  [35]         fork_exit

11.38%  [33]       Xinvlrng @ /boot/kernel/kernel

08.97%  [26]       invlrng_handler @ /boot/kernel/kernel

05.86%  [17]       smp_tlb_shootdown @ /boot/kernel/kernel
 100.0%  [17]        pmap_invalidate_range
  100.0%  [17]         biodone
   100.0%  [17]          dadone

03.79%  [11]       __mtx_lock_sleep @ /boot/kernel/kernel
 54.55%  [6]         __mtx_lock_flags
  100.0%  [6]          mrsas_cmd_done @ /boot/kernel/mrsas.ko
   100.0%  [6]           mrsas_complete_cmd
 18.18%  [2]         xpt_run_devq @ /boot/kernel/kernel
  100.0%  [2]          xpt_action_default
   100.0%  [2]           dastart
 18.18%  [2]         vmem_xalloc
  100.0%  [2]          vmem_alloc
   100.0%  [2]           g_io_check
 09.09%  [1]         dastart
  100.0%  [1]          xpt_run_allocq
   100.0%  [1]           dastrategy

03.45%  [10]       cpu_search_highest @ /boot/kernel/kernel
 90.00%  [9]         cpu_search_highest
  55.56%  [5]          sched_idletd
   100.0%  [5]           fork_exit
  44.44%  [4]          cpu_search_highest
   100.0%  [4]           sched_idletd
 10.00%  [1]         sched_idletd
  100.0%  [1]          fork_exit

03.10%  [9]        cpu_search_lowest @ /boot/kernel/kernel
 100.0%  [9]         cpu_search_lowest
  66.67%  [6]          cpu_search_lowest
   100.0%  [6]           sched_pickcpu
  33.33%  [3]          sched_pickcpu
   100.0%  [3]           sched_add

02.76%  [8]        cpu_switch @ /boot/kernel/kernel
 100.0%  [8]         mi_switch
  62.50%  [5]          sched_idletd
   100.0%  [5]           fork_exit
  37.50%  [3]          sleepq_wait
   100.0%  [3]           _sleep

02.07%  [6]        __mtx_unlock_flags @ /boot/kernel/kernel
 33.33%  [2]         mrsas_complete_cmd @ /boot/kernel/mrsas.ko
  100.0%  [2]          intr_event_execute_handlers @ /boot/kernel/kernel
   100.0%  [2]           ithread_loop
 33.33%  [2]         xpt_run_devq
  100.0%  [2]          xpt_action_default
   100.0%  [2]           dastart
 16.67%  [1]         mrsas_cmd_done @ /boot/kernel/mrsas.ko
  100.0%  [1]          mrsas_complete_cmd
   100.0%  [1]           intr_event_execute_handlers @ /boot/kernel/kernel
 16.67%  [1]         mrsas_get_mpt_cmd @ /boot/kernel/mrsas.ko
  100.0%  [1]          mrsas_action
   100.0%  [1]           xpt_run_devq @ /boot/kernel/kernel

02.07%  [6]        amd64_syscall @ /boot/kernel/kernel

02.07%  [6]        spinlock_exit @ /boot/kernel/kernel
 33.33%  [2]         sched_idletd
  100.0%  [2]          fork_exit
 16.67%  [1]         thread_lock_flags_
  100.0%  [1]          sched_idletd
   100.0%  [1]           fork_exit
 16.67%  [1]         sleepq_broadcast
  100.0%  [1]          wakeup
   100.0%  [1]           bdone
 16.67%  [1]         _sleep
  100.0%  [1]          bwait
   100.0%  [1]           physio
 16.67%  [1]         intr_event_handle
  100.0%  [1]          intr_execute_handlers
   100.0%  [1]           lapic_handle_intr

01.72%  [5]        doreti @ /boot/kernel/kernel

01.72%  [5]        thread_lock_flags_ @ /boot/kernel/kernel
 40.00%  [2]         sleepq_add
  100.0%  [2]          _sleep
   100.0%  [2]           bwait
 20.00%  [1]         sched_idletd
  100.0%  [1]          fork_exit
 20.00%  [1]         critical_exit
  100.0%  [1]          intr_event_handle
   100.0%  [1]           intr_execute_handlers
 20.00%  [1]         intr_event_schedule_thread
  100.0%  [1]          intr_event_handle
   100.0%  [1]           intr_execute_handlers

01.38%  [4]        vm_page_unhold_pages @ /boot/kernel/kernel
 100.0%  [4]         vunmapbuf
  100.0%  [4]          physio
   100.0%  [4]           devfs_read_f

01.38%  [4]        sched_idletd @ /boot/kernel/kernel
 100.0%  [4]         fork_exit

01.38%  [4]        spinlock_enter @ /boot/kernel/kernel
 75.00%  [3]         thread_lock_flags_
  66.67%  [2]          intr_event_schedule_thread
   100.0%  [2]           intr_event_handle
  33.33%  [1]          ithread_loop
   100.0%  [1]           fork_exit
 25.00%  [1]         sched_switch
  100.0%  [1]          mi_switch
   100.0%  [1]           sleepq_wait

01.38%  [4]        sched_switch @ /boot/kernel/kernel
 100.0%  [4]         mi_switch
  75.00%  [3]          sched_idletd
   100.0%  [3]           fork_exit
  25.00%  [1]          sleepq_wait
   100.0%  [1]           _sleep

01.38%  [4]        __lockmgr_args @ /boot/kernel/kernel
 50.00%  [2]         relpbuf
  100.0%  [2]          physio
   100.0%  [2]           devfs_read_f
 50.00%  [2]         getpbuf
  100.0%  [2]          physio
   100.0%  [2]           devfs_read_f

01.03%  [3]        Xapic_isr1 @ /boot/kernel/kernel

01.03%  [3]        bzero @ /boot/kernel/kernel
 33.33%  [1]         g_clone_bio
  100.0%  [1]          g_dev_strategy
   100.0%  [1]           dev_strategy_csw
 33.33%  [1]         kern_writev
  100.0%  [1]          sys_write
   100.0%  [1]           amd64_syscall
 33.33%  [1]         dev_strategy_csw
  100.0%  [1]          physio
   100.0%  [1]           devfs_read_f

01.03%  [3]        callout_lock @ /boot/kernel/kernel
 66.67%  [2]         _callout_stop_safe
  100.0%  [2]          mrsas_cmd_done @ /boot/kernel/mrsas.ko
   100.0%  [2]           mrsas_complete_cmd
 33.33%  [1]         callout_reset_sbt_on @ /boot/kernel/kernel
  100.0%  [1]          mrsas_action @ /boot/kernel/mrsas.ko
   100.0%  [1]           xpt_run_devq @ /boot/kernel/kernel

01.03%  [3]        mi_switch @ /boot/kernel/kernel
 66.67%  [2]         sleepq_wait
  100.0%  [2]          _sleep
   100.0%  [2]           bwait
 33.33%  [1]         sched_idletd
  100.0%  [1]          fork_exit

01.03%  [3]        dastart @ /boot/kernel/kernel
 100.0%  [3]         xpt_run_allocq
  100.0%  [3]          dastrategy
   100.0%  [3]           g_disk_start

01.03%  [3]        vmem_xalloc @ /boot/kernel/kernel
 100.0%  [3]         vmem_alloc
  100.0%  [3]          g_io_check
   100.0%  [3]           g_io_request

01.03%  [3]        getpbuf @ /boot/kernel/kernel
 100.0%  [3]         physio
  100.0%  [3]          devfs_read_f
   100.0%  [3]           dofileread

01.03%  [3]        ithread_loop @ /boot/kernel/kernel
 100.0%  [3]         fork_exit

01.03%  [3]        xpt_run_devq @ /boot/kernel/kernel
 100.0%  [3]         xpt_action_default
  100.0%  [3]          dastart
   100.0%  [3]           xpt_run_allocq

00.69%  [2]        handleevents @ /boot/kernel/kernel
 100.0%  [2]         timercb
  100.0%  [2]          lapic_handle_timer

00.69%  [2]        __sys_read @ /lib/libc.so.7

00.69%  [2]        dev_relthread @ /boot/kernel/kernel
 100.0%  [2]         devfs_read_f
  100.0%  [2]          dofileread
   100.0%  [2]           kern_readv

00.69%  [2]        critical_exit @ /boot/kernel/kernel
 50.00%  [1]         malloc
  100.0%  [1]          xpt_run_allocq
   100.0%  [1]           dastrategy
 50.00%  [1]         spinlock_exit
  100.0%  [1]          thread_lock_block
   100.0%  [1]           sched_add

00.69%  [2]        atomic_fetchadd_int @ /boot/kernel/kernel
 50.00%  [1]         mrsas_action @ /boot/kernel/mrsas.ko
  100.0%  [1]          xpt_run_devq @ /boot/kernel/kernel
   100.0%  [1]           xpt_action_default
 50.00%  [1]         mrsas_complete_cmd @ /boot/kernel/mrsas.ko
  100.0%  [1]          intr_event_execute_handlers @ /boot/kernel/kernel
   100.0%  [1]           ithread_loop

00.69%  [2]        vmem_xfree @ /boot/kernel/kernel
 100.0%  [2]         biodone
  100.0%  [2]          dadone
   100.0%  [2]           xpt_done_process

00.69%  [2]        g_disk_start @ /boot/kernel/kernel
 100.0%  [2]         g_io_request
  100.0%  [2]          dev_strategy_csw
   100.0%  [2]           physio

00.69%  [2]        pmap_invalidate_range @ /boot/kernel/kernel
 100.0%  [2]         biodone
  100.0%  [2]          dadone
   100.0%  [2]           xpt_done_process

00.69%  [2]        physio @ /boot/kernel/kernel
 100.0%  [2]         devfs_read_f
  100.0%  [2]          dofileread
   100.0%  [2]           kern_readv

00.69%  [2]        pcpu_find @ /boot/kernel/kernel
 100.0%  [2]         tdq_notify
  100.0%  [2]          sched_add
   50.00%  [1]           turnstile_unpend
   50.00%  [1]           setrunnable

00.69%  [2]        kern_writev @ /boot/kernel/kernel
 100.0%  [2]         sys_write
  100.0%  [2]          amd64_syscall

00.69%  [2]        fget_unlocked @ /boot/kernel/kernel
 100.0%  [2]         _fget
  50.00%  [1]          kern_writev
   100.0%  [1]           sys_write
  50.00%  [1]          kern_readv
   100.0%  [1]           sys_read

00.69%  [2]        g_disk_done_single @ /boot/kernel/kernel
 100.0%  [2]         dadone
  100.0%  [2]          xpt_done_process
   100.0%  [2]           xpt_done_td

00.34%  [1]        _mtx_lock_spin_cookie @ /boot/kernel/kernel
 100.0%  [1]         sched_idletd
  100.0%  [1]          fork_exit

00.34%  [1]        xpt_action_default @ /boot/kernel/kernel
 100.0%  [1]         dastart
  100.0%  [1]          xpt_run_allocq
   100.0%  [1]           dastrategy

00.34%  [1]        devvn_refthread @ /boot/kernel/kernel
 100.0%  [1]         devfs_read_f
  100.0%  [1]          dofileread
   100.0%  [1]           kern_readv

00.34%  [1]        pmap_extract_and_hold @ /boot/kernel/kernel
 100.0%  [1]         vm_fault_quick_hold_pages
  100.0%  [1]          vmapbuf
   100.0%  [1]           physio

00.34%  [1]        bt_freetrim @ /boot/kernel/kernel
 100.0%  [1]         biodone
  100.0%  [1]          dadone
   100.0%  [1]           xpt_done_process

00.34%  [1]        devstat_end_transaction @ /boot/kernel/kernel
 100.0%  [1]         devstat_end_transaction_bio_bt
  100.0%  [1]          g_disk_done_single
   100.0%  [1]           dadone

00.34%  [1]        xpt_run_allocq @ /boot/kernel/kernel
 100.0%  [1]         dastrategy
  100.0%  [1]          g_disk_start
   100.0%  [1]           g_io_request

00.34%  [1]        devfs_read_f @ /boot/kernel/kernel
 100.0%  [1]         dofileread
  100.0%  [1]          kern_readv
   100.0%  [1]           sys_read

00.34%  [1]        runq_choose_from @ /boot/kernel/kernel
 100.0%  [1]         sched_rem
  100.0%  [1]          tdq_move
   100.0%  [1]           sched_idletd

00.34%  [1]        scsi_read_write @ /boot/kernel/kernel
 100.0%  [1]         dastart
  100.0%  [1]          xpt_run_allocq
   100.0%  [1]           dastrategy

00.34%  [1]        dastrategy @ /boot/kernel/kernel
 100.0%  [1]         g_disk_start
  100.0%  [1]          g_io_request
   100.0%  [1]           dev_strategy_csw

00.34%  [1]        lapic_handle_timer @ /boot/kernel/kernel

00.34%  [1]        vfs_timestamp @ /boot/kernel/kernel
 100.0%  [1]         devfs_write_f
  100.0%  [1]          dofilewrite
   100.0%  [1]           kern_writev

00.34%  [1]        g_io_deliver @ /boot/kernel/kernel
 100.0%  [1]         g_disk_done_single
  100.0%  [1]          dadone
   100.0%  [1]           xpt_done_process

00.34%  [1]        mtx_pool_find @ /boot/kernel/kernel
 100.0%  [1]         bwait
  100.0%  [1]          physio
   100.0%  [1]           devfs_read_f

00.34%  [1]        g_dev_strategy @ /boot/kernel/kernel
 100.0%  [1]         dev_strategy_csw
  100.0%  [1]          physio
   100.0%  [1]           devfs_read_f

00.34%  [1]        intr_disable_src @ /boot/kernel/kernel
 100.0%  [1]         intr_event_handle
  100.0%  [1]          intr_execute_handlers
   100.0%  [1]           lapic_handle_intr

00.34%  [1]        critical_enter @ /boot/kernel/kernel
 100.0%  [1]         sched_add
  100.0%  [1]          intr_event_schedule_thread
   100.0%  [1]           intr_event_handle

00.34%  [1]        sched_rem @ /boot/kernel/kernel
 100.0%  [1]         tdq_move
  100.0%  [1]          sched_idletd
   100.0%  [1]           fork_exit

00.34%  [1]        sched_add @ /boot/kernel/kernel
 100.0%  [1]         turnstile_unpend
  100.0%  [1]          __mtx_unlock_sleep
   100.0%  [1]           xpt_run_devq

00.34%  [1]        lapic_handle_intr @ /boot/kernel/kernel

00.34%  [1]        _sleep @ /boot/kernel/kernel
 100.0%  [1]         bwait
  100.0%  [1]          physio
   100.0%  [1]           devfs_read_f

00.34%  [1]        sched_priority @ /boot/kernel/kernel
 100.0%  [1]         sched_add
  100.0%  [1]          setrunnable
   100.0%  [1]           sleepq_broadcast

00.34%  [1]        dofilewrite @ /boot/kernel/kernel
 100.0%  [1]         kern_writev
  100.0%  [1]          sys_write
   100.0%  [1]           amd64_syscall

00.34%  [1]        mrsas_complete_cmd @ /boot/kernel/mrsas.ko
 100.0%  [1]         intr_event_execute_handlers @ /boot/kernel/kernel
  100.0%  [1]          ithread_loop
   100.0%  [1]           fork_exit

00.34%  [1]        intr_event_handle @ /boot/kernel/kernel
 100.0%  [1]         intr_execute_handlers
  100.0%  [1]          lapic_handle_intr

00.34%  [1]        pmap_kextract @ /boot/kernel/kernel
 100.0%  [1]         free
  100.0%  [1]          xpt_release_ccb
   100.0%  [1]           dadone

00.34%  [1]        g_dev_done @ /boot/kernel/kernel
 100.0%  [1]         g_io_deliver
  100.0%  [1]          g_disk_done_single
   100.0%  [1]           dadone

00.34%  [1]        bioq_takefirst @ /boot/kernel/kernel
 100.0%  [1]         dastart
  100.0%  [1]          xpt_run_allocq
   100.0%  [1]           dastrategy

00.34%  [1]        bge_ape_lock @ /boot/kernel/kernel
 100.0%  [1]         bge_miibus_readreg
  100.0%  [1]          brgphy_status
   100.0%  [1]           brgphy_service

00.34%  [1]        mrsas_map_request @ /boot/kernel/mrsas.ko
 100.0%  [1]         mrsas_build_dcdb
  100.0%  [1]          mrsas_action
   100.0%  [1]           xpt_run_devq @ /boot/kernel/kernel

00.34%  [1]        bwait @ /boot/kernel/kernel
 100.0%  [1]         physio
  100.0%  [1]          devfs_read_f
   100.0%  [1]           dofileread

00.34%  [1]        dadone @ /boot/kernel/kernel
 100.0%  [1]         xpt_done_process
  100.0%  [1]          xpt_done_td
   100.0%  [1]           fork_exit

00.34%  [1]        xpt_done @ /boot/kernel/kernel
 100.0%  [1]         mrsas_cmd_done @ /boot/kernel/mrsas.ko
  100.0%  [1]          mrsas_complete_cmd
   100.0%  [1]           intr_event_execute_handlers @ /boot/kernel/kernel

00.34%  [1]        devstat_start_transaction_bio @ /boot/kernel/kernel
 100.0%  [1]         g_disk_start
  100.0%  [1]          g_io_request
   100.0%  [1]           dev_strategy_csw

00.34%  [1]        intr_execute_handlers @ /boot/kernel/kernel
 100.0%  [1]         lapic_handle_intr

00.34%  [1]        tdq_move @ /boot/kernel/kernel
 100.0%  [1]         sched_idletd
  100.0%  [1]          fork_exit

00.34%  [1]        _callout_stop_safe @ /boot/kernel/kernel
 100.0%  [1]         mrsas_cmd_done @ /boot/kernel/mrsas.ko
  100.0%  [1]          mrsas_complete_cmd
   100.0%  [1]           intr_event_execute_handlers @ /boot/kernel/kernel

00.34%  [1]        runq_remove @ /boot/kernel/kernel
 100.0%  [1]         sched_rem
  100.0%  [1]          tdq_move
   100.0%  [1]           sched_idletd

00.34%  [1]        g_io_request @ /boot/kernel/kernel
 100.0%  [1]         dev_strategy_csw
  100.0%  [1]          physio
   100.0%  [1]           devfs_read_f

@ CPU_CLK_UNHALTED_CORE [294 samples]

10.20%  [30]       cpu_idle @ /boot/kernel/kernel
 100.0%  [30]        sched_idletd
  100.0%  [30]         fork_exit

09.18%  [27]       smp_tlb_shootdown @ /boot/kernel/kernel
 100.0%  [27]        pmap_invalidate_range
  100.0%  [27]         biodone
   100.0%  [27]          dadone

07.48%  [22]       invlrng_handler @ /boot/kernel/kernel

06.46%  [19]       Xinvlrng @ /boot/kernel/kernel

04.76%  [14]       cpu_search_lowest @ /boot/kernel/kernel
 85.71%  [12]        cpu_search_lowest
  83.33%  [10]         cpu_search_lowest
   100.0%  [10]          sched_pickcpu
  16.67%  [2]          sched_pickcpu
   100.0%  [2]           sched_add
 14.29%  [2]         sched_pickcpu
  100.0%  [2]          sched_add
   100.0%  [2]           setrunnable

04.42%  [13]       __mtx_lock_sleep @ /boot/kernel/kernel
 76.92%  [10]        __mtx_lock_flags
  100.0%  [10]         mrsas_cmd_done @ /boot/kernel/mrsas.ko
   100.0%  [10]          mrsas_complete_cmd
 15.38%  [2]         vmem_xalloc @ /boot/kernel/kernel
  100.0%  [2]          vmem_alloc
   100.0%  [2]           g_io_check
 07.69%  [1]         vmem_xfree
  100.0%  [1]          biodone
   100.0%  [1]           dadone

03.74%  [11]       cpu_search_highest @ /boot/kernel/kernel
 72.73%  [8]         cpu_search_highest
  62.50%  [5]          sched_idletd
   100.0%  [5]           fork_exit
  37.50%  [3]          cpu_search_highest
   100.0%  [3]           sched_idletd
 27.27%  [3]         sched_idletd
  100.0%  [3]          fork_exit

03.06%  [9]        __mtx_lock_flags @ /boot/kernel/kernel
 33.33%  [3]         mrsas_map_request @ /boot/kernel/mrsas.ko
  100.0%  [3]          mrsas_build_dcdb
   100.0%  [3]           mrsas_action
 22.22%  [2]         mrsas_cmd_done
  100.0%  [2]          mrsas_complete_cmd
   100.0%  [2]           intr_event_execute_handlers @ /boot/kernel/kernel
 22.22%  [2]         mrsas_unmap_request @ /boot/kernel/mrsas.ko
  100.0%  [2]          mrsas_cmd_done
   100.0%  [2]           mrsas_complete_cmd
 11.11%  [1]         mrsas_get_mpt_cmd
  100.0%  [1]          mrsas_action
   100.0%  [1]           xpt_run_devq @ /boot/kernel/kernel
 11.11%  [1]         mrsas_action @ /boot/kernel/mrsas.ko
  100.0%  [1]          xpt_run_devq @ /boot/kernel/kernel
   100.0%  [1]           xpt_action_default

02.72%  [8]        sched_switch @ /boot/kernel/kernel
 100.0%  [8]         mi_switch
  37.50%  [3]          ithread_loop
   100.0%  [3]           fork_exit
  25.00%  [2]          sched_idletd
   100.0%  [2]           fork_exit
  12.50%  [1]          critical_exit
   100.0%  [1]           intr_event_handle
  12.50%  [1]          turnstile_wait
   100.0%  [1]           __mtx_lock_sleep
  12.50%  [1]          sleepq_wait
   100.0%  [1]           _sleep

02.38%  [7]        sched_idletd @ /boot/kernel/kernel
 100.0%  [7]         fork_exit

02.04%  [6]        cpu_switch @ /boot/kernel/kernel
 100.0%  [6]         mi_switch
  83.33%  [5]          sched_idletd
   100.0%  [5]           fork_exit
  16.67%  [1]          critical_exit
   100.0%  [1]           intr_event_handle

01.70%  [5]        __sys_read @ /lib/libc.so.7

01.70%  [5]        getpbuf @ /boot/kernel/kernel
 100.0%  [5]         physio
  100.0%  [5]          devfs_read_f
   100.0%  [5]           dofileread

01.36%  [4]        thread_lock_flags_ @ /boot/kernel/kernel
 25.00%  [1]         intr_event_schedule_thread
  100.0%  [1]          intr_event_handle
   100.0%  [1]           intr_execute_handlers
 25.00%  [1]         propagate_priority
  100.0%  [1]          turnstile_wait
   100.0%  [1]           __mtx_lock_sleep
 25.00%  [1]         sleepq_broadcast
  100.0%  [1]          wakeup
   100.0%  [1]           bdone
 25.00%  [1]         ithread_loop
  100.0%  [1]          fork_exit

01.36%  [4]        _mtx_lock_spin_cookie @ /boot/kernel/kernel
 75.00%  [3]         smp_tlb_shootdown
  100.0%  [3]          pmap_invalidate_range
   100.0%  [3]           biodone
 25.00%  [1]         sched_add
  100.0%  [1]          setrunnable
   100.0%  [1]           sleepq_broadcast

01.36%  [4]        __mtx_unlock_flags @ /boot/kernel/kernel
 25.00%  [1]         mrsas_complete_cmd @ /boot/kernel/mrsas.ko
  100.0%  [1]          intr_event_execute_handlers @ /boot/kernel/kernel
   100.0%  [1]           ithread_loop
 25.00%  [1]         mrsas_cmd_done @ /boot/kernel/mrsas.ko
  100.0%  [1]          mrsas_complete_cmd
   100.0%  [1]           intr_event_execute_handlers @ /boot/kernel/kernel
 25.00%  [1]         xpt_run_devq
  100.0%  [1]          xpt_action_default
   100.0%  [1]           dastart
 25.00%  [1]         mrsas_get_mpt_cmd @ /boot/kernel/mrsas.ko
  100.0%  [1]          mrsas_action
   100.0%  [1]           xpt_run_devq @ /boot/kernel/kernel

01.36%  [4]        pmap_invalidate_range @ /boot/kernel/kernel
 100.0%  [4]         biodone
  100.0%  [4]          dadone
   100.0%  [4]           xpt_done_process

01.36%  [4]        spinlock_exit @ /boot/kernel/kernel
 50.00%  [2]         sched_idletd
  100.0%  [2]          fork_exit
 25.00%  [1]         tdq_move
  100.0%  [1]          sched_idletd
   100.0%  [1]           fork_exit
 25.00%  [1]         wakeup
  100.0%  [1]          bdone
   100.0%  [1]           bufdone

01.02%  [3]        xpt_action_default @ /boot/kernel/kernel
 100.0%  [3]         dastart
  100.0%  [3]          xpt_run_allocq
   100.0%  [3]           dastrategy

01.02%  [3]        _fget @ /boot/kernel/kernel
 100.0%  [3]         kern_writev
  100.0%  [3]          sys_write
   100.0%  [3]           amd64_syscall

01.02%  [3]        vmem_xalloc @ /boot/kernel/kernel
 100.0%  [3]         vmem_alloc
  100.0%  [3]          g_io_check
   100.0%  [3]           g_io_request

01.02%  [3]        g_dev_strategy @ /boot/kernel/kernel
 100.0%  [3]         dev_strategy_csw
  100.0%  [3]          physio
   100.0%  [3]           devfs_read_f

01.02%  [3]        vmem_xfree @ /boot/kernel/kernel
 100.0%  [3]         biodone
  100.0%  [3]          dadone
   100.0%  [3]           xpt_done_process

00.68%  [2]        sched_choose @ /boot/kernel/kernel
 100.0%  [2]         choosethread
  100.0%  [2]          sched_switch
   100.0%  [2]           mi_switch

00.68%  [2]        ithread_loop @ /boot/kernel/kernel
 100.0%  [2]         fork_exit

00.68%  [2]        mrsas_get_mpt_cmd @ /boot/kernel/mrsas.ko
 100.0%  [2]         mrsas_action
  100.0%  [2]          xpt_run_devq @ /boot/kernel/kernel
   100.0%  [2]           xpt_action_default

00.68%  [2]        devstat_end_transaction @ /boot/kernel/kernel
 100.0%  [2]         devstat_end_transaction_bio_bt
  100.0%  [2]          g_disk_done_single
   100.0%  [2]           dadone

00.68%  [2]        xpt_run_allocq @ /boot/kernel/kernel
 100.0%  [2]         dastrategy
  100.0%  [2]          g_disk_start
   100.0%  [2]           g_io_request

00.68%  [2]        Xapic_isr1 @ /boot/kernel/kernel

00.68%  [2]        g_io_check @ /boot/kernel/kernel
 100.0%  [2]         g_io_request
  100.0%  [2]          dev_strategy_csw
   100.0%  [2]           physio

00.68%  [2]        mrsas_isr @ /boot/kernel/mrsas.ko
 100.0%  [2]         intr_event_execute_handlers @ /boot/kernel/kernel
  100.0%  [2]          ithread_loop
   100.0%  [2]           fork_exit

00.68%  [2]        xpt_done_process @ /boot/kernel/kernel
 100.0%  [2]         xpt_done_td
  100.0%  [2]          fork_exit

00.68%  [2]        g_io_deliver @ /boot/kernel/kernel
 100.0%  [2]         g_disk_done_single
  100.0%  [2]          dadone
   100.0%  [2]           xpt_done_process

00.68%  [2]        critical_enter @ /boot/kernel/kernel
 50.00%  [1]         uma_zfree_arg
  100.0%  [1]          g_dev_done
   100.0%  [1]           g_io_deliver
 50.00%  [1]         smp_tlb_shootdown
  100.0%  [1]          pmap_invalidate_range
   100.0%  [1]           biodone

00.68%  [2]        g_disk_start @ /boot/kernel/kernel
 100.0%  [2]         g_io_request
  100.0%  [2]          dev_strategy_csw
   100.0%  [2]           physio

00.68%  [2]        free @ /boot/kernel/kernel
 100.0%  [2]         xpt_release_ccb
  100.0%  [2]          dadone
   100.0%  [2]           xpt_done_process

00.68%  [2]        callout_lock @ /boot/kernel/kernel
 100.0%  [2]         _callout_stop_safe
  100.0%  [2]          mrsas_cmd_done @ /boot/kernel/mrsas.ko
   100.0%  [2]           mrsas_complete_cmd

00.68%  [2]        dev_relthread @ /boot/kernel/kernel
 100.0%  [2]         devfs_read_f
  100.0%  [2]          dofileread
   100.0%  [2]           kern_readv

00.68%  [2]        critical_exit @ /boot/kernel/kernel
 50.00%  [1]         spinlock_exit
  100.0%  [1]          sleepq_switch
   100.0%  [1]           sleepq_wait
 50.00%  [1]         free
  100.0%  [1]          xpt_release_ccb
   100.0%  [1]           dadone

00.68%  [2]        dadone @ /boot/kernel/kernel
 100.0%  [2]         xpt_done_process
  100.0%  [2]          xpt_done_td
   100.0%  [2]           fork_exit

00.68%  [2]        physio @ /boot/kernel/kernel
 100.0%  [2]         devfs_read_f
  100.0%  [2]          dofileread
   100.0%  [2]           kern_readv

00.68%  [2]        amd64_syscall @ /boot/kernel/kernel

00.68%  [2]        mrsas_action @ /boot/kernel/mrsas.ko
 100.0%  [2]         xpt_run_devq @ /boot/kernel/kernel
  100.0%  [2]          xpt_action_default
   100.0%  [2]           dastart

00.68%  [2]        bzero @ /boot/kernel/kernel
 100.0%  [2]         mrsas_action @ /boot/kernel/mrsas.ko
  100.0%  [2]          xpt_run_devq @ /boot/kernel/kernel
   100.0%  [2]           xpt_action_default

00.34%  [1]        setrunnable @ /boot/kernel/kernel
 100.0%  [1]         sleepq_broadcast
  100.0%  [1]          wakeup
   100.0%  [1]           bdone

00.34%  [1]        mrsas_complete_cmd @ /boot/kernel/mrsas.ko
 100.0%  [1]         intr_event_execute_handlers @ /boot/kernel/kernel
  100.0%  [1]          ithread_loop
   100.0%  [1]           fork_exit

00.34%  [1]        fget_unlocked @ /boot/kernel/kernel
 100.0%  [1]         _fget
  100.0%  [1]          kern_writev
   100.0%  [1]           sys_write

00.34%  [1]        dastart @ /boot/kernel/kernel
 100.0%  [1]         xpt_run_allocq
  100.0%  [1]          dastrategy
   100.0%  [1]           g_disk_start

00.34%  [1]        mrsas_cmd_done @ /boot/kernel/mrsas.ko
 100.0%  [1]         mrsas_complete_cmd
  100.0%  [1]          intr_event_execute_handlers @ /boot/kernel/kernel
   100.0%  [1]           ithread_loop

00.34%  [1]        biodone @ /boot/kernel/kernel
 100.0%  [1]         dadone
  100.0%  [1]          xpt_done_process
   100.0%  [1]           xpt_done_td

00.34%  [1]        Xxen_intr_upcall @ /boot/kernel/kernel

00.34%  [1]        mrsas_data_load_cb @ /boot/kernel/mrsas.ko
 100.0%  [1]         bus_dmamap_load @ /boot/kernel/kernel
  100.0%  [1]          mrsas_map_request @ /boot/kernel/mrsas.ko
   100.0%  [1]           mrsas_build_dcdb

00.34%  [1]        runq_remove @ /boot/kernel/kernel
 100.0%  [1]         sched_choose
  100.0%  [1]          choosethread
   100.0%  [1]           sched_switch

00.34%  [1]        lapic_handle_timer @ /boot/kernel/kernel

00.34%  [1]        devvn_refthread @ /boot/kernel/kernel
 100.0%  [1]         devfs_read_f
  100.0%  [1]          dofileread
   100.0%  [1]           kern_readv

00.34%  [1]        g_dev_done @ /boot/kernel/kernel
 100.0%  [1]         g_io_deliver
  100.0%  [1]          g_disk_done_single
   100.0%  [1]           dadone

00.34%  [1]        runq_add @ /boot/kernel/kernel
 100.0%  [1]         sched_add
  100.0%  [1]          setrunnable
   100.0%  [1]           sleepq_broadcast

00.34%  [1]        uma_zalloc_arg @ /boot/kernel/kernel
 100.0%  [1]         dev_strategy_csw
  100.0%  [1]          physio
   100.0%  [1]           devfs_read_f

00.34%  [1]        doreti @ /boot/kernel/kernel

00.34%  [1]        dastrategy @ /boot/kernel/kernel
 100.0%  [1]         g_disk_start
  100.0%  [1]          g_io_request
   100.0%  [1]           dev_strategy_csw

00.34%  [1]        g_clone_bio @ /boot/kernel/kernel
 100.0%  [1]         g_dev_strategy
  100.0%  [1]          dev_strategy_csw
   100.0%  [1]           physio

00.34%  [1]        pmap_qenter @ /boot/kernel/kernel
 100.0%  [1]         g_io_check
  100.0%  [1]          g_io_request
   100.0%  [1]           dev_strategy_csw

00.34%  [1]        spinlock_enter @ /boot/kernel/kernel
 100.0%  [1]         thread_lock_flags_
  100.0%  [1]          sleepq_add
   100.0%  [1]           _sleep

00.34%  [1]        g_disk_done_single @ /boot/kernel/kernel
 100.0%  [1]         dadone
  100.0%  [1]          xpt_done_process
   100.0%  [1]           xpt_done_td

00.34%  [1]        sleepq_lock @ /boot/kernel/kernel
 100.0%  [1]         _sleep
  100.0%  [1]          bwait
   100.0%  [1]           physio

00.34%  [1]        devfs_read_f @ /boot/kernel/kernel
 100.0%  [1]         dofileread
  100.0%  [1]          kern_readv
   100.0%  [1]           sys_read

00.34%  [1]        sleepq_broadcast @ /boot/kernel/kernel
 100.0%  [1]         wakeup
  100.0%  [1]          bdone
   100.0%  [1]           bufdone

00.34%  [1]        uma_zfree_arg @ /boot/kernel/kernel
 100.0%  [1]         g_dev_done
  100.0%  [1]          g_io_deliver
   100.0%  [1]           g_disk_done_single

00.34%  [1]        sleepq_add @ /boot/kernel/kernel
 100.0%  [1]         _sleep
  100.0%  [1]          bwait
   100.0%  [1]           physio

00.34%  [1]        g_io_request @ /boot/kernel/kernel
 100.0%  [1]         dev_strategy_csw
  100.0%  [1]          physio
   100.0%  [1]           devfs_read_f

00.34%  [1]        bt_freetrim @ /boot/kernel/kernel
 100.0%  [1]         biodone
  100.0%  [1]          dadone
   100.0%  [1]           xpt_done_process

00.34%  [1]        runq_choose @ /boot/kernel/kernel
 100.0%  [1]         sched_choose
  100.0%  [1]          choosethread
   100.0%  [1]           sched_switch

00.34%  [1]        relpbuf @ /boot/kernel/kernel
 100.0%  [1]         physio
  100.0%  [1]          devfs_read_f
   100.0%  [1]           dofileread

00.34%  [1]        pmap_extract_and_hold @ /boot/kernel/kernel
 100.0%  [1]         vm_fault_quick_hold_pages
  100.0%  [1]          vmapbuf
   100.0%  [1]           physio

00.34%  [1]        cpu_set_syscall_retval @ /boot/kernel/kernel
 100.0%  [1]         amd64_syscall

00.34%  [1]        atomic_fetchadd_int @ /boot/kernel/kernel
 100.0%  [1]         mrsas_complete_cmd @ /boot/kernel/mrsas.ko
  100.0%  [1]          intr_event_execute_handlers @ /boot/kernel/kernel
   100.0%  [1]           ithread_loop

00.34%  [1]        sched_add @ /boot/kernel/kernel
 100.0%  [1]         turnstile_unpend
  100.0%  [1]          __mtx_unlock_sleep
   100.0%  [1]           bufdone

00.34%  [1]        vm_page_unhold_pages @ /boot/kernel/kernel
 100.0%  [1]         vunmapbuf
  100.0%  [1]          physio
   100.0%  [1]           devfs_read_f

00.34%  [1]        malloc @ /boot/kernel/kernel
 100.0%  [1]         xpt_run_allocq
  100.0%  [1]          dastrategy
   100.0%  [1]           g_disk_start

00.34%  [1]        sched_priority @ /boot/kernel/kernel
 100.0%  [1]         sched_add
  100.0%  [1]          setrunnable
   100.0%  [1]           sleepq_broadcast

00.34%  [1]        _sleep @ /boot/kernel/kernel
 100.0%  [1]         bwait
  100.0%  [1]          physio
   100.0%  [1]           devfs_read_f

00.34%  [1]        bufdone @ /boot/kernel/kernel
 100.0%  [1]         bufdonebio
  100.0%  [1]          g_io_deliver
   100.0%  [1]           g_disk_done_single

00.34%  [1]        xpt_run_devq @ /boot/kernel/kernel
 100.0%  [1]         xpt_action_default
  100.0%  [1]          dastart
   100.0%  [1]           xpt_run_allocq

00.34%  [1]        sched_pickcpu @ /boot/kernel/kernel
 100.0%  [1]         sched_add
  100.0%  [1]          setrunnable
   100.0%  [1]           sleepq_broadcast

00.34%  [1]        _mtx_trylock_flags_ @ /boot/kernel/kernel
 100.0%  [1]         vm_page_pa_tryrelock
  100.0%  [1]          pmap_extract_and_hold
   100.0%  [1]           vm_fault_quick_hold_pages


Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?9f138f242e278476e5c542d695e58bc8>