From owner-freebsd-dtrace@freebsd.org Sun Oct 28 10:56:15 2018 Return-Path: Delivered-To: freebsd-dtrace@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id ACECB10860A4 for ; Sun, 28 Oct 2018 10:56:15 +0000 (UTC) (envelope-from longwitz@incore.de) Received: from dss.incore.de (dss.incore.de [195.145.1.138]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 2164A7C31C; Sun, 28 Oct 2018 10:56:15 +0000 (UTC) (envelope-from longwitz@incore.de) Received: from inetmail.dmz (inetmail.dmz [10.3.0.3]) by dss.incore.de (Postfix) with ESMTP id D8AF9139B2; Sun, 28 Oct 2018 11:56:12 +0100 (CET) X-Virus-Scanned: amavisd-new at incore.de Received: from dss.incore.de ([10.3.0.3]) by inetmail.dmz (inetmail.dmz [10.3.0.3]) (amavisd-new, port 10024) with LMTP id PoqEy1OCnA0q; Sun, 28 Oct 2018 11:56:11 +0100 (CET) Received: from mail.local.incore (fwintern.dmz [10.0.0.253]) by dss.incore.de (Postfix) with ESMTP id 84DC4139B5; Sun, 28 Oct 2018 11:56:11 +0100 (CET) Received: from bsdmhs.longwitz (unknown [192.168.99.6]) by mail.local.incore (Postfix) with ESMTP id 615A9166; Sun, 28 Oct 2018 11:56:11 +0100 (CET) Message-ID: <5BD595CB.5030800@incore.de> Date: Sun, 28 Oct 2018 11:56:11 +0100 From: Andreas Longwitz User-Agent: Thunderbird 2.0.0.19 (X11/20090113) MIME-Version: 1.0 To: Mark Johnston CC: "freebsd-dtrace@freebsd.org" Subject: Re: Why my DTrace script does not work after installing a new kernel without reboot References: <5BD4E965.9040006@incore.de> <20181027225339.GA4191@spy> In-Reply-To: <20181027225339.GA4191@spy> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit X-BeenThere: freebsd-dtrace@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: "A discussion list for developers working on DTrace in FreeBSD." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 28 Oct 2018 10:56:15 -0000 Thans very much for quick answer ! >> I have a simple, probably stupid question. I run FreeBSD 10.4 Stable >> r338093 and use the script >> >> #!/usr/sbin/dtrace -s >> >> dtrace:::BEGIN >> { >> printf("pf_default_rule.timeout[2/16/17]=%d/%d/%d\n", >> kernel`pf_default_rule.timeout[2], kernel`pf_default_rule.timeout[16], >> kernel`pf_default_rule.timeout[17]); >> } >> >> The output normally is >> pf_default_rule.timeout[2/16/17]=86400/6000/12000 >> >> But when I change the sourcefile /sys/netpfil/pf/pf.c trying a patch and >> build and install the new kernel but without reboot, then the output of >> the script is wrong: >> >> pf_default_rule.timeout[2/16/17]=3237216124/0/0 >> >> After reboot everything works again. > > Does your patch change the layout of the timeout structure? If so, the > problem is that dtrace(1) is using the CTF from kern.bootfile, but that > doesn't match the layout of the structures used by the running kernel. My patch does not change data structures, I have added a debug statement for an actual pf problem: --- pf.c.orig 2018-08-14 10:17:41.000000000 +0200 +++ pf.c 2018-10-28 00:04:45.441327000 +0200 @@ -1543,6 +1543,12 @@ if (start) { end = state->rule.ptr->timeout[PFTM_ADAPTIVE_END]; states = counter_u64_fetch(state->rule.ptr->states_cur); + if (bootverbose > 5) { + if (states > 10000) { + bootverbose--; + printf("%s: counter value %u too big ", __func__, states); + } + } } else { start = V_pf_default_rule.timeout[PFTM_ADAPTIVE_START]; end = V_pf_default_rule.timeout[PFTM_ADAPTIVE_END]; If dtrace(1) uses the CTF of kern.bootfile, then everything should work as expected. Normally I have -> sysctl kern.bootfile kern.bootfile: /boot/kernel/kernel and after building and installing the patched kernel I have -> sysctl kern.bootfile kern.bootfile: /boot/kernel.old/kernel and that is correct the file of the actual running kernel. I do not understand the logic of dtrace(1) in handling the kern.bootfile variable. This is done in sourcefile dt_open.c: /* * On FreeBSD the kernel module name can't be hard-coded. The * 'kern.bootfile' sysctl value tells us exactly which file is being * used as the kernel. */ #ifndef illumos { char bootfile[MAXPATHLEN]; char *p; int i; size_t len = sizeof(bootfile); /* This call shouldn't fail, but use a default just in case. */ if (sysctlbyname("kern.bootfile", bootfile, &len, NULL, 0) != 0) strlcpy(bootfile, "kernel", sizeof(bootfile)); if ((p = strrchr(bootfile, '/')) != NULL) p++; else p = bootfile; If I am not comletely wrong this code snippet does not do what can be read in the comment. The function strrchr() returns a pointer to the last occurrence of '/' in bootfile, so we always end with *p="kernel". After cd /boot; mv kernel kernel.new; mv kernel.old kernel my DTrace script works correct again, that means dtrace(1) always uses /boot/kernel/kernel. So there may be a bug in handling the kern.bootfile variable in dtrace. Best regards, Andreas From owner-freebsd-dtrace@freebsd.org Sun Oct 28 19:00:19 2018 Return-Path: Delivered-To: freebsd-dtrace@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 16E9010DD707 for ; Sun, 28 Oct 2018 19:00:19 +0000 (UTC) (envelope-from jmg@gold.funkthat.com) Received: from gold.funkthat.com (gate2.funkthat.com [208.87.223.18]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "gate2.funkthat.com", Issuer "Let's Encrypt Authority X3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 7AFE16B109; Sun, 28 Oct 2018 19:00:18 +0000 (UTC) (envelope-from jmg@gold.funkthat.com) Received: from gold.funkthat.com (localhost [127.0.0.1]) by gold.funkthat.com (8.15.2/8.15.2) with ESMTPS id w9SJ0AY7063365 (version=TLSv1.2 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=NO); Sun, 28 Oct 2018 12:00:11 -0700 (PDT) (envelope-from jmg@gold.funkthat.com) Received: (from jmg@localhost) by gold.funkthat.com (8.15.2/8.15.2/Submit) id w9SJ0AaC063364; Sun, 28 Oct 2018 12:00:10 -0700 (PDT) (envelope-from jmg) Date: Sun, 28 Oct 2018 12:00:10 -0700 From: John-Mark Gurney To: Mark Johnston Cc: Andreas Longwitz , "freebsd-dtrace@freebsd.org" Subject: Re: Why my DTrace script does not work after installing a new kernel without reboot Message-ID: <20181028190010.GF75530@funkthat.com> References: <5BD4E965.9040006@incore.de> <20181027225339.GA4191@spy> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <20181027225339.GA4191@spy> X-Operating-System: FreeBSD 11.0-RELEASE-p7 amd64 X-PGP-Fingerprint: D87A 235F FB71 1F3F 55B7 ED9B D5FF 5A51 C0AC 3D65 X-Files: The truth is out there X-URL: https://www.funkthat.com/ X-Resume: https://www.funkthat.com/~jmg/resume.html X-TipJar: bitcoin:13Qmb6AeTgQecazTWph4XasEsP7nGRbAPE X-to-the-FBI-CIA-and-NSA: HI! HOW YA DOIN? can i haz chizburger? User-Agent: Mutt/1.6.1 (2016-04-27) X-Greylist: Sender IP whitelisted, not delayed by milter-greylist-4.4.3 (gold.funkthat.com [127.0.0.1]); Sun, 28 Oct 2018 12:00:11 -0700 (PDT) X-BeenThere: freebsd-dtrace@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: "A discussion list for developers working on DTrace in FreeBSD." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 28 Oct 2018 19:00:19 -0000 Mark Johnston wrote this message on Sat, Oct 27, 2018 at 18:53 -0400: > On Sun, Oct 28, 2018 at 12:40:37AM +0200, Andreas Longwitz wrote: > > I have a simple, probably stupid question. I run FreeBSD 10.4 Stable > > r338093 and use the script > > > > #!/usr/sbin/dtrace -s > > > > dtrace:::BEGIN > > { > > printf("pf_default_rule.timeout[2/16/17]=%d/%d/%d\n", > > kernel`pf_default_rule.timeout[2], kernel`pf_default_rule.timeout[16], > > kernel`pf_default_rule.timeout[17]); > > } > > > > The output normally is > > pf_default_rule.timeout[2/16/17]=86400/6000/12000 > > > > But when I change the sourcefile /sys/netpfil/pf/pf.c trying a patch and > > build and install the new kernel but without reboot, then the output of > > the script is wrong: > > > > pf_default_rule.timeout[2/16/17]=3237216124/0/0 > > > > After reboot everything works again. > > Does your patch change the layout of the timeout structure? If so, the > problem is that dtrace(1) is using the CTF from kern.bootfile, but that > doesn't match the layout of the structures used by the running kernel. Except that the default make installkernel will adjust kern.bootfile to point to the correct old location.. -- John-Mark Gurney Voice: +1 415 225 5579 "All that I will do, has been done, All that I have, has not." From owner-freebsd-dtrace@freebsd.org Wed Oct 31 18:10:19 2018 Return-Path: Delivered-To: freebsd-dtrace@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 92B9810E129F for ; Wed, 31 Oct 2018 18:10:19 +0000 (UTC) (envelope-from longwitz@incore.de) Received: from dss.incore.de (dss.incore.de [195.145.1.138]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 2F61D81840; Wed, 31 Oct 2018 18:10:18 +0000 (UTC) (envelope-from longwitz@incore.de) Received: from inetmail.dmz (inetmail.dmz [10.3.0.3]) by dss.incore.de (Postfix) with ESMTP id CEA32139A5; Wed, 31 Oct 2018 19:10:10 +0100 (CET) X-Virus-Scanned: amavisd-new at incore.de Received: from dss.incore.de ([10.3.0.3]) by inetmail.dmz (inetmail.dmz [10.3.0.3]) (amavisd-new, port 10024) with LMTP id AsTMrV7_v9Dn; Wed, 31 Oct 2018 19:10:10 +0100 (CET) Received: from mail.local.incore (fwintern.dmz [10.0.0.253]) by dss.incore.de (Postfix) with ESMTP id F250F1399F; Wed, 31 Oct 2018 19:10:09 +0100 (CET) Received: from bsdmhs.longwitz (unknown [192.168.99.6]) by mail.local.incore (Postfix) with ESMTP id D372817D; Wed, 31 Oct 2018 19:10:09 +0100 (CET) Message-ID: <5BD9F001.9090603@incore.de> Date: Wed, 31 Oct 2018 19:10:09 +0100 From: Andreas Longwitz User-Agent: Thunderbird 2.0.0.19 (X11/20090113) MIME-Version: 1.0 To: Mark Johnston CC: "freebsd-dtrace@freebsd.org" Subject: Re: Why my DTrace script does not work after installing a new kernel without reboot References: <5BD4E965.9040006@incore.de> <20181027225339.GA4191@spy> In-Reply-To: <20181027225339.GA4191@spy> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit X-BeenThere: freebsd-dtrace@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: "A discussion list for developers working on DTrace in FreeBSD." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 31 Oct 2018 18:10:19 -0000 In the meantime I found the answer to my question. > Does your patch change the layout of the timeout structure? If so, the > problem is that dtrace(1) is using the CTF from kern.bootfile, but that > doesn't match the layout of the structures used by the running kernel. dtrace() gets the CTF of the kernel and kernel modules from the pathbames returned by kldstat(2). In my case they all start with '/boot/kernel/' because the kernel was loaded from that place. This can be veryfied with 'kldstat -v'. The output does not change after buildung a new kernel and the kernel itself does nit use the variable kern.bootfile. -- Andreas Longwitz From owner-freebsd-dtrace@freebsd.org Wed Oct 31 18:47:45 2018 Return-Path: Delivered-To: freebsd-dtrace@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 86EBB10E1DA6 for ; Wed, 31 Oct 2018 18:47:45 +0000 (UTC) (envelope-from markjdb@gmail.com) Received: from mail-io1-xd41.google.com (mail-io1-xd41.google.com [IPv6:2607:f8b0:4864:20::d41]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 1B34B83262 for ; Wed, 31 Oct 2018 18:47:45 +0000 (UTC) (envelope-from markjdb@gmail.com) Received: by mail-io1-xd41.google.com with SMTP id a5-v6so1563324ioq.8 for ; Wed, 31 Oct 2018 11:47:45 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=sender:date:from:to:cc:subject:message-id:references:mime-version :content-disposition:in-reply-to:user-agent; bh=balDIYtAsyNHZjMNLC2LrT35c+BTesmjMwuN4sUg5Xw=; b=q6XA+JPx2smmZwDe98LUxiDqqDpWlwhvs8O6Vo7z4J+FP88YegCzeSXsnB5Vnb1rgu VySB2s+euNgoQ/qK74IRwMRGQcRgcEbAkYkeMQhcX8ygkVVq7M8/KKuTdn5JzIKDUyOx kiRniPesEk/L7xbz7nEb6bmCx7he/Si7d4SPoQ+5ORi0P6xkZHYatmNyJPm679RkF0pe 3fJNhQkq1ApkeiUIagH4Bya0TgG1KQ0AwACaVNgZxg0HDR7+7kvcdgROcjRO+e4b/KuP mta/HbdY7gUjveThHMKdyMUzmuJ+X92xiF5DhMvhN6myziegusj0JBc/b2RJ67qhMNcw 445w== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:sender:date:from:to:cc:subject:message-id :references:mime-version:content-disposition:in-reply-to:user-agent; bh=balDIYtAsyNHZjMNLC2LrT35c+BTesmjMwuN4sUg5Xw=; b=qNlUBTtsJGoAW6yMMShe6GzeS1yw331BkYlaM+KR6y0nxvyVnXOC0TlrW/GZcIr9wd rl0q25qAP++1aGrsVvxV1g81QvRurra8QW1zkHz9Cbf3nPQOffPS/4eOy6T69RwxHa42 ism5yRQo16ML5/2Xmg8K+UCWCG8jfnlJZuO4Id1/x5YaQLq5NamPuOrnG/7iQTtX/Veu ldJUpO3UYW6EC/xCTPuYlX1mSdDOZg+NfUiDqulM3LTb8QNd7JfA6k6g0sUr1xjQDFp1 Wua0XmfcjwJ3q3yy7NlRhLGMh91bQ1GjMIpVncavhOtjmSnXj+IwsNzjGtdmBXOFrYQE azXg== X-Gm-Message-State: AGRZ1gISbPwC5SY1zLJWeKdhR6gDeAuY+tT5sw9G0t/tzSRgC21UkhcB 6Tiwf/d4VgtDgffi9vijUqk= X-Google-Smtp-Source: AJdET5cRCbmqlBOWhLA25ljRZZq7Wo4sXgV6lwemHVnjMESLPz1tqMY+hzxOt01zSpE28ZuEPmy3vQ== X-Received: by 2002:a6b:f818:: with SMTP id o24-v6mr3114844ioh.203.1541011664341; Wed, 31 Oct 2018 11:47:44 -0700 (PDT) Received: from spy (toroon0812w-lp140-01-64-229-12-188.dsl.bell.ca. [64.229.12.188]) by smtp.gmail.com with ESMTPSA id m1-v6sm8974413iob.15.2018.10.31.11.47.42 (version=TLS1_2 cipher=ECDHE-RSA-CHACHA20-POLY1305 bits=256/256); Wed, 31 Oct 2018 11:47:43 -0700 (PDT) Sender: Mark Johnston Date: Wed, 31 Oct 2018 14:46:43 -0400 From: Mark Johnston To: Andreas Longwitz Cc: "freebsd-dtrace@freebsd.org" Subject: Re: Why my DTrace script does not work after installing a new kernel without reboot Message-ID: <20181031184643.GA1686@spy> References: <5BD4E965.9040006@incore.de> <20181027225339.GA4191@spy> <5BD9F001.9090603@incore.de> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <5BD9F001.9090603@incore.de> User-Agent: Mutt/1.10.1 (2018-07-13) X-BeenThere: freebsd-dtrace@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: "A discussion list for developers working on DTrace in FreeBSD." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 31 Oct 2018 18:47:45 -0000 On Wed, Oct 31, 2018 at 07:10:09PM +0100, Andreas Longwitz wrote: > In the meantime I found the answer to my question. > > > Does your patch change the layout of the timeout structure? If so, the > > problem is that dtrace(1) is using the CTF from kern.bootfile, but that > > doesn't match the layout of the structures used by the running kernel. > > dtrace() gets the CTF of the kernel and kernel modules from the > pathbames returned by kldstat(2). In my case they all start with > '/boot/kernel/' because the kernel was loaded from that place. This can > be veryfied with 'kldstat -v'. The output does not change after buildung > a new kernel and the kernel itself does nit use the variable > kern.bootfile. I see, thanks for digging into it. I suspect that the real solution is to ensure that CTF files are mapped into the kernel's address space when an object is loaded, and provide an interface for userland to fetch CTF files for the running kernel. We've talked about doing this before, but I'm a bit unhappy about the extra memory consumption; the compressed CTF file for the kernel's file alone on my laptop is slightly under 1MB in size. Maybe that's not worth worrying about. Note that the same bug exists today in the kernel: link_elf_ctf_get() fetches a CTF file using the path cached in the KLD structure, i.e., the path exported by kldstat -v. At present, the kernel only uses CTF to fill out type info when creating FBT probes, so it's not a major problem.