From owner-freebsd-stable@FreeBSD.ORG Wed Jul 9 21:44:07 2014 Return-Path: Delivered-To: freebsd-stable@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 4355F22C for ; Wed, 9 Jul 2014 21:44:07 +0000 (UTC) Received: from mail-vc0-x236.google.com (mail-vc0-x236.google.com [IPv6:2607:f8b0:400c:c03::236]) (using TLSv1 with cipher ECDHE-RSA-RC4-SHA (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id ED6732DEC for ; Wed, 9 Jul 2014 21:44:06 +0000 (UTC) Received: by mail-vc0-f182.google.com with SMTP id il7so8894716vcb.27 for ; Wed, 09 Jul 2014 14:44:06 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=packetdam.com; s=google; h=mime-version:in-reply-to:references:from:date:message-id:subject:to :cc:content-type; bh=3TU2Isi8d1adr7mFUUZd0kTf4nVXM4RtigPHH5szADk=; b=QeqnVrP+6Sup7B5q6kI09n6yMlvRf9yW/uAnLVra7ecKACB0N06LcFqUB29RKFNwpI KSVy1Nt7wH03hfUH5xr22sKzSVB6oInY307puZE11uKkpjT42EBlhX3qRc96dQvw3aMG YQEWKRfpToJZy9koiGPlvd0VLmm5UEX9vCMtw= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:mime-version:in-reply-to:references:from:date :message-id:subject:to:cc:content-type; bh=3TU2Isi8d1adr7mFUUZd0kTf4nVXM4RtigPHH5szADk=; b=N7uoDsHXIFcaPgj4qOCM6ZR4roGA2/zASYLyW3FAzp5R4zsxh9iN6gJghfLKKyXm2g BLJoVh3L+48QBAbB1fzt63EhVZGuJJaiGUHSCeN7zP2XNxA3rWs64HwT3K/HLuzfYfYx +xZ4VkHCOlp4by55XILJiX4wQsiE83dFE1oDSFKg78xF+pbHnaqoIDMQR9MWHl1Q7sqR 30iRVm0jeJjdxM9CkSDDeK0Cf+YXoUZc0S7XM2yNEP/cqld8xcIfjth7dL0tYAA8Wamc oGnOv4RN8bW9vojvsXndHcWVRz3iOoUtV14AkEvg6GtI044Q8wnoDWKAJhCIFzR1oVlV dOYA== X-Gm-Message-State: ALoCoQn5X5t8+lv0JSoIlCW3sfDvXnIm6R2u1DtjvMRgsqlHq9gYnBXpoW0gYgNDOt8Mu8KXWhch X-Received: by 10.52.171.233 with SMTP id ax9mr334245vdc.76.1404942245798; Wed, 09 Jul 2014 14:44:05 -0700 (PDT) MIME-Version: 1.0 Received: by 10.229.185.6 with HTTP; Wed, 9 Jul 2014 14:43:25 -0700 (PDT) X-Originating-IP: [148.251.50.122] In-Reply-To: <201407091402.23537.jhb@freebsd.org> References: <201407091402.23537.jhb@freebsd.org> From: Vlad Galu Date: Wed, 9 Jul 2014 22:43:25 +0100 Message-ID: Subject: Re: Weird ISR accounting in 10-STABLE To: John Baldwin Content-Type: text/plain; charset=UTF-8 X-Content-Filtered-By: Mailman/MimeDel 2.1.18 Cc: freebsd-stable@freebsd.org X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 09 Jul 2014 21:44:07 -0000 I hope I'm doing this the right way. Invoking DTrace as dtrace -n 'profile-1000 /curthread->td_name == "swi1: netisr 0"/ { @[func(arg0)] = count(); }' outputs: -- cut here -- kernel`__mtx_lock_sleep 2 kernel`in6_rtalloc1 15 kernel`mb_dtor_mbuf 25 kernel`m_tag_delete_nonpersistent 26 kernel`m_tag_delete_chain 29 kernel`swi_net 29 kernel`netisr_select_cpuid 34 kernel`mac_mbuf_init 37 kernel`ipsec_in_reject 42 kernel`nd6_output 44 kernel`key_havesp 44 kernel`mac_policy_sunlock_nosleep 51 kernel`mac_mbuf_to_label 51 kernel`m_tag_copy_chain 52 kernel`ip6_ipsec_filtertunnel 53 kernel`critical_enter 58 kernel`critical_exit 58 kernel`pf_check6_in 58 kernel`netisr_queue 59 kernel`ipsec6_in_reject 59 kernel`m_freem 63 kernel`mac_policy_slock_nosleep 63 kernel`pf_check6_out 69 kernel`in6_lltable_lookup 87 kernel`in6_clearscope 102 kernel`looutput 108 kernel`if_simloop 121 kernel`ipsec_getpolicybyaddr 134 kernel`rtfree 160 kernel`uma_zalloc_arg 211 kernel`m_copym 262 kernel`mac_netinet6_nd6_send 268 kernel`key_allocsp_default 270 kernel`pfil_run_hooks 303 kernel`pf_test6 335 kernel`uma_zfree_arg 337 kernel`bcopy 340 kernel`mac_ifnet_create_mbuf 358 kernel`netisr_queue_internal 368 kernel`_rm_runlock 371 kernel`rtalloc1_fib 385 kernel`mb_ctor_mbuf 412 kernel`nd6_output_lle 465 kernel`_key_freesp 506 kernel`key_addref 521 kernel`_rw_runlock_cookie 602 kernel`__rw_rlock 606 kernel`in6_setscope 662 kernel`_rm_rlock 758 kernel`rn_match 797 kernel`mac_ifnet_check_transmit 1110 kernel`ip6_forward 1334 kernel`bzero 1492 kernel`ip6_input 1685 kernel`bcmp 3511 -- and here -- The swi thread is definitely the naughtiest, running dtrace -n 'profile-1000 { @[stringof(curthread->td_name)] = count(); }' shows: -- cut here -- aiod1 1 arc_reclaim_thread 1 trim tank 1 txg_thread_enter 1 vmdaemon 1 zio_write_issue_1 1 acpi_cooling1 2 acpi_thermal 2 irq265: re0 2 vnlru 2 bufdaemon 3 irq266: ahci0 3 l2arc_feed_thread 3 sshd 5 pagedaemon 6 syncer 13 dtrace 17 cut 18 sh 18 tmux 30 rand_harvestq 38 pf purge 51 swi4: clock 256 idle: cpu3 18803 idle: cpu5 18810 idle: cpu4 18826 idle: cpu2 18836 idle: cpu7 18839 idle: cpu6 18842 idle: cpu0 18882 swi1: netisr 0 19008 -- and here -- On Wed, Jul 9, 2014 at 7:02 PM, John Baldwin wrote: > On Tuesday, July 08, 2014 8:56:44 am Vlad Galu wrote: > > The symptom has reappeared, although interestingly enough I can't spot > > anything out of the ordinary in this DTrace output: > > Can you get dtrace to only display the functions called by that thread? > > -- > John Baldwin > -- Regards, Vlad