From owner-freebsd-dtrace@FreeBSD.ORG Wed Apr 15 19:06:18 2015 Return-Path: Delivered-To: freebsd-dtrace@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 98109ACF for ; Wed, 15 Apr 2015 19:06:18 +0000 (UTC) Received: from mail-pd0-x22d.google.com (mail-pd0-x22d.google.com [IPv6:2607:f8b0:400e:c02::22d]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 6CE0D3CD for ; Wed, 15 Apr 2015 19:06:15 +0000 (UTC) Received: by pdbqd1 with SMTP id qd1so62463007pdb.2 for ; Wed, 15 Apr 2015 12:06:15 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=message-id:date:from:user-agent:mime-version:to:subject :content-type:content-transfer-encoding; bh=cN6IPGYvYgGbHrfYKOPb63axHLJN4J/s2wvU0DyAdys=; b=YAWEg1OSaZ8+qlndPAmlBKdNHm7tQcDN7gZht9HJmTBdLVQkei+P5jJDcMOHQPoAmh DBysYWL5t1/NSOTnZMM7M+VflPs85I1M7B17omckxPyoW7cxyzsXced308CKFby615zi a1w3pbi+cnOlksANl8OmcUs0vDnrt7vnmrHCKKp7oH2S/0BaweeUmyz/OAGG9zMZxw3G jvUOApiRSoVvTH2namKiMpdA+hHgpbHDrvhbLJTLaytvKN4ZR5wrjiPgczR3W51VjPq8 FfUJ3/zFObxm8sdF71NJodVnVIxARyQIPdITJwUgitxzZxDGEk68KMLKlkMTpvFDAbLT I6tQ== X-Received: by 10.68.57.132 with SMTP id i4mr49483110pbq.138.1429124774934; Wed, 15 Apr 2015 12:06:14 -0700 (PDT) Received: from [192.168.0.3] (c-50-186-4-10.hsd1.or.comcast.net. [50.186.4.10]) by mx.google.com with ESMTPSA id gy1sm4837984pbc.55.2015.04.15.12.06.13 for (version=TLSv1.2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Wed, 15 Apr 2015 12:06:13 -0700 (PDT) Message-ID: <552EB6A4.4070406@gmail.com> Date: Wed, 15 Apr 2015 12:06:12 -0700 From: Lacey Powers User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:31.0) Gecko/20100101 Thunderbird/31.6.0 MIME-Version: 1.0 To: freebsd-dtrace@freebsd.org Subject: io:kernel::done invalid address (0x0) in predicate on FreeBSD 10.1 Content-Type: text/plain; charset=utf-8; format=flowed Content-Transfer-Encoding: 7bit X-BeenThere: freebsd-dtrace@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: "A discussion list for developers working on DTrace in FreeBSD." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 15 Apr 2015 19:06:18 -0000 Hello Everyone, I was trying to port a script from Illumos to FreeBSD 10.1, from this blog post, here: http://dtrace.org/blogs/ahl/2014/08/31/openzfs-tuning/ The rw.d script specifically. I wished to use it forsome ZFS tuning and testing I have been doing on my workstation. Here is the script I came up with: https://gist.github.com/kassandry/219596d7f17e5b128e68 The version I have mostly seems to work, aside from a single error, repeated a very large number of times during a run.=( Example: dtrace: error on enabled probe ID 3 (ID 58515: io:kernel::done): invalid address (0x0) in predicate at DIF offset 60 dtrace: error on enabled probe ID 3 (ID 58515: io:kernel::done): invalid address (0x0) in predicate at DIF offset 60 Despite adding args[0] != NULL to the predicate for io:::done, it still comes up. The output seems reasonable, and provides latency and distributions for each of the types of bio commands issued during the run. The only other references I can find to such an issue are here: https://lists.freebsd.org/pipermail/freebsd-dtrace/2015-February/000346.html https://github.com/brendangregg/DTrace-tools/blob/master/disk/dilt.d#L80 Which suggests it may be a problem with the io provider? =( The original script does not have these errors on OmniOS a708424. I'm not sure what else to do to address the specific errors that are coming up with this script. Have I done something incorrect in porting this particular script to FreeBSD 10.1? Suggestions and advice are most welcome. If further information is needed, please let me know. Thank you for your time. =) Regards, Lacey From owner-freebsd-dtrace@FreeBSD.ORG Wed Apr 15 20:18:56 2015 Return-Path: Delivered-To: freebsd-dtrace@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id EAA02693 for ; Wed, 15 Apr 2015 20:18:56 +0000 (UTC) Received: from aussmtpmrkpc120.us.dell.com (aussmtpmrkpc120.us.dell.com [143.166.82.159]) (using TLSv1 with cipher RC4-SHA (128/128 bits)) (Client CN "ausxipmktpc11.us.dell.com", Issuer "Verizon Public SureServer CA G14-SHA1" (not verified)) by mx1.freebsd.org (Postfix) with ESMTPS id A50B9D1C for ; Wed, 15 Apr 2015 20:18:56 +0000 (UTC) X-Loopcount0: from 64.238.244.148 X-IronPort-AV: E=Sophos;i="5.11,583,1422943200"; d="scan'208";a="242821117" Message-ID: <552EC7A3.7050508@compellent.com> Date: Wed, 15 Apr 2015 15:18:43 -0500 From: Eric Badger User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:31.0) Gecko/20100101 Thunderbird/31.3.0 MIME-Version: 1.0 To: Lacey Powers , "freebsd-dtrace@freebsd.org" Subject: Re: io:kernel::done invalid address (0x0) in predicate on FreeBSD 10.1 References: <552EB6A4.4070406@gmail.com> In-Reply-To: <552EB6A4.4070406@gmail.com> Content-Type: text/plain; charset="windows-1252"; format=flowed Content-Transfer-Encoding: 7bit X-BeenThere: freebsd-dtrace@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: "A discussion list for developers working on DTrace in FreeBSD." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 15 Apr 2015 20:18:57 -0000 On 04/15/15 14:06, Lacey Powers wrote: > Hello Everyone, > > I was trying to port a script from Illumos to FreeBSD 10.1, from this > blog post, here: > > http://dtrace.org/blogs/ahl/2014/08/31/openzfs-tuning/ > > The rw.d script specifically. I wished to use it forsome ZFS tuning and > testing I have been doing on my workstation. > > Here is the script I came up with: > > https://gist.github.com/kassandry/219596d7f17e5b128e68 > > The version I have mostly seems to work, aside from a single error, > repeated a very large number of times during a run.=( > > Example: > > dtrace: error on enabled probe ID 3 (ID 58515: io:kernel::done): invalid > address (0x0) in predicate at DIF offset 60 > dtrace: error on enabled probe ID 3 (ID 58515: io:kernel::done): invalid > address (0x0) in predicate at DIF offset 60 > > Despite adding > > args[0] != NULL > > to the predicate for io:::done, it still comes up. I can't claim expertise here, but it seems that the 'io:::done' probe can receive a 'struct bio' where 'bio_disk' is NULL. So a first thought would be to have a predicate like: /args[0] && args[0]->bio_disk && ts[...]/ I don't know under what circumstances bio_disk is NULL; would require some digging to find out. Someone more familiar with this area may have a better suggestion. Eric From owner-freebsd-dtrace@FreeBSD.ORG Thu Apr 16 01:31:04 2015 Return-Path: Delivered-To: freebsd-dtrace@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id C2B2559F for ; Thu, 16 Apr 2015 01:31:04 +0000 (UTC) Received: from mail-pa0-x235.google.com (mail-pa0-x235.google.com [IPv6:2607:f8b0:400e:c03::235]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 8B3841F4 for ; Thu, 16 Apr 2015 01:31:04 +0000 (UTC) Received: by pabtp1 with SMTP id tp1so70537104pab.2 for ; Wed, 15 Apr 2015 18:31:04 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=message-id:date:from:user-agent:mime-version:to:subject:references :in-reply-to:content-type:content-transfer-encoding; bh=P+0s7VnDKiSKTsm1UiryW3SWsFsSSwZiavC6p/ECpr0=; b=m0S27vHl5Qkm3CdVKYXd2MyYGQ7078qEDEHZu3JZkUMZFitdrqQjpHGtX4XzyLz6Gw r0mOixMbcgputdN/Bkxr4cTjQGiGWLOwn1pvfyxTr1EGtCMxPboi/QfCIvIJ0/QpbeSS 8R0YLQCI6t6idIayK5H7kdHqmZkTikt72rhpLfoWUPheJGgX2OtB+JhlT8VpuaWygj46 M5kjdmx0F1KMHz+NnruCWQ1RYuq8e+YgFCNWEOQD7cndinQ9gnQWeGIX6o9BlQ7aaMwr wRlFBb1WCA8MHS+MobgJZAnaakxIZHHoHvl2uk4Ea3Q6XRgxSldsFg1jMoPtNQRX9UfE uyXQ== X-Received: by 10.66.141.77 with SMTP id rm13mr51550270pab.14.1429147864041; Wed, 15 Apr 2015 18:31:04 -0700 (PDT) Received: from [192.168.0.3] (c-50-186-4-10.hsd1.or.comcast.net. [50.186.4.10]) by mx.google.com with ESMTPSA id f12sm5319988pat.43.2015.04.15.18.31.02 (version=TLSv1.2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Wed, 15 Apr 2015 18:31:02 -0700 (PDT) Message-ID: <552F10D5.3090204@gmail.com> Date: Wed, 15 Apr 2015 18:31:01 -0700 From: Lacey Powers User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:31.0) Gecko/20100101 Thunderbird/31.6.0 MIME-Version: 1.0 To: Eric Badger , "freebsd-dtrace@freebsd.org" Subject: Re: io:kernel::done invalid address (0x0) in predicate on FreeBSD 10.1 References: <552EB6A4.4070406@gmail.com> <552EC7A3.7050508@compellent.com> In-Reply-To: <552EC7A3.7050508@compellent.com> Content-Type: text/plain; charset=windows-1252; format=flowed Content-Transfer-Encoding: 7bit X-BeenThere: freebsd-dtrace@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: "A discussion list for developers working on DTrace in FreeBSD." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 16 Apr 2015 01:31:04 -0000 On 04/15/2015 01:18 PM, Eric Badger wrote: > > On 04/15/15 14:06, Lacey Powers wrote: >> Hello Everyone, >> >> I was trying to port a script from Illumos to FreeBSD 10.1, from this >> blog post, here: >> >> http://dtrace.org/blogs/ahl/2014/08/31/openzfs-tuning/ >> >> The rw.d script specifically. I wished to use it forsome ZFS tuning and >> testing I have been doing on my workstation. >> >> Here is the script I came up with: >> >> https://gist.github.com/kassandry/219596d7f17e5b128e68 >> >> The version I have mostly seems to work, aside from a single error, >> repeated a very large number of times during a run.=( >> >> Example: >> >> dtrace: error on enabled probe ID 3 (ID 58515: io:kernel::done): invalid >> address (0x0) in predicate at DIF offset 60 >> dtrace: error on enabled probe ID 3 (ID 58515: io:kernel::done): invalid >> address (0x0) in predicate at DIF offset 60 >> >> Despite adding >> >> args[0] != NULL >> >> to the predicate for io:::done, it still comes up. > > I can't claim expertise here, but it seems that the 'io:::done' probe > can receive a 'struct bio' where 'bio_disk' is NULL. So a first > thought would be to have a predicate like: > > /args[0] && args[0]->bio_disk && ts[...]/ > > I don't know under what circumstances bio_disk is NULL; would require > some digging to find out. Someone more familiar with this area may > have a better suggestion. > > Eric Hello Eric, Thank you for your reply. =) That certainly helped me figure out why the probe was giving me errors.Much appreciated. From the help I got on #freebsd and #bsddev on IRC, it seems that in sys/sys/bio.h struct disk *bio_disk; /* Valid below geom_disk.c only */ It is noted that the struct bio disk may only be populated below geom_disk.c It seems that each IO action can go through a long queue of functions, which may or may not actually add a valid struct disk to the struct bio in question, which is why it is occasionally NULL. At least, as far as I understood from the code and asking questions on IRC. Still lots of neat stuff to learn. =) Again, thank you for your help. Regards, Lacey