From owner-freebsd-dtrace@FreeBSD.ORG Sun Dec 21 23:22:29 2014 Return-Path: Delivered-To: freebsd-dtrace@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id DD453DD0 for ; Sun, 21 Dec 2014 23:22:29 +0000 (UTC) Received: from mail-pd0-x22a.google.com (mail-pd0-x22a.google.com [IPv6:2607:f8b0:400e:c02::22a]) (using TLSv1 with cipher ECDHE-RSA-RC4-SHA (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id AA8BA2E19 for ; Sun, 21 Dec 2014 23:22:29 +0000 (UTC) Received: by mail-pd0-f170.google.com with SMTP id v10so4697143pde.29 for ; Sun, 21 Dec 2014 15:22:29 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=message-id:date:from:user-agent:mime-version:to:subject :content-type:content-transfer-encoding; bh=2Iz7bl0HY47RCC5RNTyVtcgCvHl+qdxx39Ius9bTemE=; b=b99zUAbjRu6zZRf+lLq4mdPqD/b9qqrfEiSOEf5Hr3SU5Fi8z5cNiotV+grQp6HCCD NjwEiKR69gd7cVzi+l57oVOHpCYLypYAO86pLHtyAafoJy5N1trwcNQBkMkJx3gFllRn LOAYAe/itka3HLR35XO0wsmWbGrA7s0P9PQY+2kNP0c2TEdhIqEWTfc1SWdaTDkQSOLF e0QrHxFJKp0hGuoMbc++QwlIEnshZbXdXKXF4RZyomKUvFR2jcMhEL3H6tYcuXvxblr+ Hq6ReS8rXCG1f2VFNKyBuKOinWTnePCeidhdNCTKK4t81HoVDIrlgxe2RnoWOtVkuGl+ KFYQ== X-Received: by 10.68.134.134 with SMTP id pk6mr30849110pbb.146.1419204149275; Sun, 21 Dec 2014 15:22:29 -0800 (PST) Received: from [192.168.1.6] (c-71-237-219-238.hsd1.or.comcast.net. [71.237.219.238]) by mx.google.com with ESMTPSA id f12sm15537224pat.43.2014.12.21.15.22.26 for (version=TLSv1.2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Sun, 21 Dec 2014 15:22:27 -0800 (PST) Message-ID: <54975631.3040106@gmail.com> Date: Sun, 21 Dec 2014 15:22:25 -0800 From: Lacey Powers User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:31.0) Gecko/20100101 Thunderbird/31.3.0 MIME-Version: 1.0 To: freebsd-dtrace@freebsd.org Subject: elf_begin failed on postgresql 9.4 with dtrace enabled Content-Type: text/plain; charset=utf-8; format=flowed Content-Transfer-Encoding: 7bit X-BeenThere: freebsd-dtrace@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: "A discussion list for developers working on DTrace in FreeBSD." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 21 Dec 2014 23:22:30 -0000 Hello Everyone, I was attempting to build and use PostgreSQL with dtrace on FreeBSD 10.1 on my home workstation. It builds, and the binary works, but fails to register any probes. After double checking that I had the dtraceall kernel module loaded, changing permissions on /dev/dtrace/helper (0666), verifying that devel/libelf was loaded, and digging around source code and Google for debug flags, I'm at a loss for what else to do. Here is the error: [pgsql@talizorah ~]$ DTRACE_DOF_INIT_DEBUG=1 /usr/local/bin/postgres -D /usr/local/pgsql/9.4/data dtrace DOF postgres: elf_begin failed LOG: ending log output to stderr HINT: Future log output will go to log destination "syslog". dtrace DOF postgres: DTrace ioctl failed to remove DOF (-1) dtrace DOF postgres: DTrace ioctl failed to remove DOF (-1) dtrace DOF postgres: DTrace ioctl failed to remove DOF (-1) ^Cdtrace DOF postgres: dtrace DOF postgres: DTrace ioctl failed to remove DOF (-1) DTrace ioctl failed to remove DOF (-1) dtrace DOF postgres: DTrace ioctl failed to remove DOF (-1) dtrace DOF postgres: DTrace ioctl failed to remove DOF (-1) dtrace DOF postgres: DTrace ioctl failed to remove DOF (-1) dtrace DOF postgres: DTrace ioctl failed to remove DOF (-1) [pgsql@talizorah ~]$ And here is the output of pg_config, showing the enabled options. [0]lacey@talizorah:~$ pg_config BINDIR = /usr/local/bin DOCDIR = /usr/local/share/doc/postgresql HTMLDIR = /usr/local/share/doc/postgresql INCLUDEDIR = /usr/local/include PKGINCLUDEDIR = /usr/local/include/postgresql INCLUDEDIR-SERVER = /usr/local/include/postgresql/server LIBDIR = /usr/local/lib PKGLIBDIR = /usr/local/lib/postgresql LOCALEDIR = /usr/local/share/locale MANDIR = /usr/local/man SHAREDIR = /usr/local/share/postgresql SYSCONFDIR = /usr/local/etc/postgresql PGXS = /usr/local/lib/postgresql/pgxs/src/makefiles/pgxs.mk CONFIGURE = '--with-libraries=/usr/local/lib' '--with-includes=/usr/local/include' '--enable-thread-safety' '--with-openssl' '--enable-dtrace' '--with-pam' '--with-libxml' '--enable-nls' '--without-gssapi' '--with-krb5' '--prefix=/usr/local' '--mandir=/usr/local/man' '--infodir=/usr/local/info/' '--build=amd64-portbld-freebsd10.1' 'build_alias=amd64-portbld-freebsd10.1' 'CC=cc' 'CFLAGS=-O2 -pipe -march=native -O3 -funroll-loops -fstack-protector -fno-strict-aliasing' 'LDFLAGS= -L/usr/local/lib -pthread -lelf -Wl,-rpath,/usr/lib:/usr/local/lib -fstack-protector' 'LIBS=' 'CPPFLAGS=' 'CPP=cpp' 'LDFLAGS_SL=' CC = cc CPPFLAGS = -I/usr/local/include/libxml2 -I/usr/include -I/usr/local/include -I/usr/local/include CFLAGS = -O2 -pipe -march=native -O3 -funroll-loops -fstack-protector -fno-strict-aliasing -Wall -Wmissing-prototypes -Wpointer-arith -Wdeclaration-after-statement -Wendif-labels -Wmissing-format-attribute -Wformat-security -fno-strict-aliasing -fwrapv CFLAGS_SL = -fPIC -DPIC LDFLAGS = -L../../../src/common -L/usr/local/lib -pthread -lelf -Wl,-rpath,/usr/lib:/usr/local/lib -fstack-protector -L/usr/local/lib -L/usr/lib -L/usr/local/lib -Wl,--as-needed -Wl,-R'/usr/local/lib' LDFLAGS_EX = LDFLAGS_SL = LIBS = -lpgcommon -lpgport -lintl -lxml2 -lpam -lssl -lcrypto -lz -lreadline -lcrypt -lm VERSION = PostgreSQL 9.4.0 [0]lacey@talizorah:~$ Are there suggestions for how to fix this? I would be happy to provide further details if needed. Thank you for your time and assistance, Lacey From owner-freebsd-dtrace@FreeBSD.ORG Sun Dec 21 23:58:13 2014 Return-Path: Delivered-To: freebsd-dtrace@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id AE2CEA20 for ; Sun, 21 Dec 2014 23:58:13 +0000 (UTC) Received: from mail-ie0-x231.google.com (mail-ie0-x231.google.com [IPv6:2607:f8b0:4001:c03::231]) (using TLSv1 with cipher ECDHE-RSA-RC4-SHA (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 71E27321B for ; Sun, 21 Dec 2014 23:58:13 +0000 (UTC) Received: by mail-ie0-f177.google.com with SMTP id rd18so3483174iec.22 for ; Sun, 21 Dec 2014 15:58:12 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=sender:date:from:to:cc:subject:message-id:references:mime-version :content-type:content-disposition:in-reply-to:user-agent; bh=T6BKSuiCPXp9FKBF8sMQabnK0dq82eLktrVuf8NVVb0=; b=rauaQuib2EkRJqRLJTrc+hrRtzu1jueg5UYTHZlJDbGT4/izzTGsD+O/crecgqpf5R jKcj7P6wyUiEc/70n0d8hdaMMlAJhS2weqVOSqSNnObdMvzos0GuYiUFs7k1I+ZiElaZ ivX/ryCHlalK1QzZyN5jhSsDY73Zz8CDiqNsOBMFL6JmLYcaFtTueECojfzgW80+s1SE dKZlXhWJcGPrxrVo4Kb4X08OYo4QbvBVHUJEzctTWN8szzYd+Zbp3ob8ebvjHFHkHD2P Mtkh9iAtgB9R/Qb3cTC5N/zC7lh150LOaR4sfRpkkn62iwSByZlxgIb0P61YJocGSYXI knCQ== X-Received: by 10.107.7.142 with SMTP id g14mr17994855ioi.81.1419206292421; Sun, 21 Dec 2014 15:58:12 -0800 (PST) Received: from brule.home (TOROON0812W-LP130-04-1279423115.dsl.bell.ca. [76.66.114.139]) by mx.google.com with ESMTPSA id mv6sm4112722igb.10.2014.12.21.15.58.10 (version=TLSv1.2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Sun, 21 Dec 2014 15:58:11 -0800 (PST) Sender: Mark Johnston Date: Sun, 21 Dec 2014 18:58:06 -0500 From: Mark Johnston To: Lacey Powers Subject: Re: elf_begin failed on postgresql 9.4 with dtrace enabled Message-ID: <20141221235806.GA20561@brule.home> References: <54975631.3040106@gmail.com> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <54975631.3040106@gmail.com> User-Agent: Mutt/1.5.23 (2014-03-12) Cc: freebsd-dtrace@freebsd.org X-BeenThere: freebsd-dtrace@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: "A discussion list for developers working on DTrace in FreeBSD." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 21 Dec 2014 23:58:13 -0000 On Sun, Dec 21, 2014 at 03:22:25PM -0800, Lacey Powers wrote: > Hello Everyone, > > I was attempting to build and use PostgreSQL with dtrace on FreeBSD 10.1 on > my home workstation. > > It builds, and the binary works, but fails to register any probes. > > After double checking that I had the dtraceall kernel module loaded, > changing permissions on /dev/dtrace/helper (0666), verifying that > devel/libelf was loaded, and digging around source code and Google for debug > flags, I'm at a loss for what else to do. > > Here is the error: > > [pgsql@talizorah ~]$ DTRACE_DOF_INIT_DEBUG=1 /usr/local/bin/postgres -D > /usr/local/pgsql/9.4/data > dtrace DOF postgres: elf_begin failed > LOG: ending log output to stderr > HINT: Future log output will go to log destination "syslog". > dtrace DOF postgres: DTrace ioctl failed to remove DOF (-1) > dtrace DOF postgres: DTrace ioctl failed to remove DOF (-1) > dtrace DOF postgres: DTrace ioctl failed to remove DOF (-1) > ^Cdtrace DOF postgres: dtrace DOF postgres: DTrace ioctl failed to remove > DOF (-1) > DTrace ioctl failed to remove DOF (-1) > dtrace DOF postgres: DTrace ioctl failed to remove DOF (-1) > dtrace DOF postgres: DTrace ioctl failed to remove DOF (-1) > dtrace DOF postgres: DTrace ioctl failed to remove DOF (-1) > dtrace DOF postgres: DTrace ioctl failed to remove DOF (-1) > [pgsql@talizorah ~]$ Hi Lacey, I've never seen this particular problem before. Could you post the output of "ldd /usr/local/bin/postgres"? I've never tried using libelf from ports, so perhaps there's a problem with it, assuming it is indeed being used (as opposed to the base system's libelf). If the ports libelf is being used (/usr/local/lib/libelf.so will show up in the ldd ouput if so), could you try deleting devel/libelf, rebuilding postgres, and trying again? If not, or if other packages require devel/libelf on your system, could you apply the patch below, rebuild drti, rebuild postgres (in that order), and try again? To rebuild drti, cd to /usr/src/cddl/lib/drti and run "make && make install". Thanks! -Mark diff --git a/cddl/contrib/opensolaris/lib/libdtrace/common/drti.c b/cddl/contrib/opensolaris/lib/libdtrace/common/drti.c index ccd4f9b..eafc4c6 100644 --- a/cddl/contrib/opensolaris/lib/libdtrace/common/drti.c +++ b/cddl/contrib/opensolaris/lib/libdtrace/common/drti.c @@ -158,7 +158,7 @@ dtrace_dof_init(void) return; } if ((e = elf_begin(efd, ELF_C_READ, NULL)) == NULL) { - dprintf(1, "elf_begin failed\n"); + dprintf(1, "elf_begin failed: %s\n", elf_errmsg(elf_errno())); close(efd); return; } From owner-freebsd-dtrace@FreeBSD.ORG Mon Dec 22 00:44:18 2014 Return-Path: Delivered-To: freebsd-dtrace@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 83330BD; Mon, 22 Dec 2014 00:44:18 +0000 (UTC) Received: from mail-pa0-x229.google.com (mail-pa0-x229.google.com [IPv6:2607:f8b0:400e:c03::229]) (using TLSv1 with cipher ECDHE-RSA-RC4-SHA (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4E7E03EEB; Mon, 22 Dec 2014 00:44:18 +0000 (UTC) Received: by mail-pa0-f41.google.com with SMTP id rd3so4830837pab.28; Sun, 21 Dec 2014 16:44:18 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=message-id:date:from:user-agent:mime-version:to:cc:subject :references:in-reply-to:content-type:content-transfer-encoding; bh=E80c3HmvjtXqcmHbrQoIusU3bb2A8A7HF6xQkpZdYag=; b=q/LYEuKp/wl8WIYqHnExO3V4SqYH/HKxLCV/I0oGZjNG7/ZsSsjMeaamnCWYwLN/Lf bT9Kqis4FXWnBAPvdDZ4E0oTaYKLMRn6qACb1k+SPkrs5H/riz73YDEK4qXXCkeo+aN/ 3qIjaonPveGP+EKRdRad3kr1tUEFTj2Lfe8QLcVl9Ya2Y8UPhFEV0rGwozhSXGezjE/+ 4GEcv6gcUrYxXlVlmko46jVV8bpbvnqt2Ny0TCwaTGBl4BzXjnRhx62Rd9XB3IYsya3N uOrJ7muUmQkDkb9FbVaSOGjugoN9lEHTOdL5JQ+Isu5hwEgmTeu0/kaRnMcJivIbwsWu roYQ== X-Received: by 10.68.200.36 with SMTP id jp4mr18914878pbc.153.1419209057877; Sun, 21 Dec 2014 16:44:17 -0800 (PST) Received: from [192.168.1.6] (c-71-237-219-238.hsd1.or.comcast.net. [71.237.219.238]) by mx.google.com with ESMTPSA id rx1sm1281986pbc.86.2014.12.21.16.44.16 (version=TLSv1.2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Sun, 21 Dec 2014 16:44:16 -0800 (PST) Message-ID: <5497695F.9020802@gmail.com> Date: Sun, 21 Dec 2014 16:44:15 -0800 From: Lacey Powers User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:31.0) Gecko/20100101 Thunderbird/31.3.0 MIME-Version: 1.0 To: Mark Johnston Subject: Re: elf_begin failed on postgresql 9.4 with dtrace enabled References: <54975631.3040106@gmail.com> <20141221235806.GA20561@brule.home> In-Reply-To: <20141221235806.GA20561@brule.home> Content-Type: text/plain; charset=windows-1252; format=flowed Content-Transfer-Encoding: 7bit Cc: freebsd-dtrace@freebsd.org X-BeenThere: freebsd-dtrace@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: "A discussion list for developers working on DTrace in FreeBSD." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 22 Dec 2014 00:44:18 -0000 Hello Mark, Thank you! devel/libelf was linked to the postgres binary [0]lacey@talizorah:~$ ldd /usr/local/bin/postgres /usr/local/bin/postgres: libelf.so.0 => /usr/local/lib/libelf.so.0 (0x800d89000) libintl.so.8 => /usr/local/lib/libintl.so.8 (0x800fa2000) libxml2.so.2 => /usr/local/lib/libxml2.so.2 (0x8011ad000) libpam.so.5 => /usr/lib/libpam.so.5 (0x801541000) libicuuc.so.53 => /usr/local/lib/libicuuc.so.53 (0x80174d000) libicui18n.so.53 => /usr/local/lib/libicui18n.so.53 (0x801adf000) libssl.so.7 => /usr/lib/libssl.so.7 (0x801f51000) libcrypto.so.7 => /lib/libcrypto.so.7 (0x8021bc000) libm.so.5 => /lib/libm.so.5 (0x8025af000) libthr.so.3 => /lib/libthr.so.3 (0x8027d7000) libc.so.7 => /lib/libc.so.7 (0x8029fc000) libz.so.6 => /lib/libz.so.6 (0x802da5000) liblzma.so.5 => /usr/lib/liblzma.so.5 (0x802fbb000) libicudata.so.53 => /usr/local/lib/libicudata.so.53 (0x8031e0000) libc++.so.1 => /usr/lib/libc++.so.1 (0x8048c3000) libcxxrt.so.1 => /lib/libcxxrt.so.1 (0x804b83000) libgcc_s.so.1 => /lib/libgcc_s.so.1 (0x804d9f000) [0]lacey@talizorah:~$ Once I removed that, and rebuilt PostgreSQL, per your suggestions, I have 462 dtrace probes. =) Many thanks for your clear and prompt reply. =) Regards, Lacey On 12/21/2014 15:58, Mark Johnston wrote: > On Sun, Dec 21, 2014 at 03:22:25PM -0800, Lacey Powers wrote: >> Hello Everyone, >> >> I was attempting to build and use PostgreSQL with dtrace on FreeBSD 10.1 on >> my home workstation. >> >> It builds, and the binary works, but fails to register any probes. >> >> After double checking that I had the dtraceall kernel module loaded, >> changing permissions on /dev/dtrace/helper (0666), verifying that >> devel/libelf was loaded, and digging around source code and Google for debug >> flags, I'm at a loss for what else to do. >> >> Here is the error: >> >> [pgsql@talizorah ~]$ DTRACE_DOF_INIT_DEBUG=1 /usr/local/bin/postgres -D >> /usr/local/pgsql/9.4/data >> dtrace DOF postgres: elf_begin failed >> LOG: ending log output to stderr >> HINT: Future log output will go to log destination "syslog". >> dtrace DOF postgres: DTrace ioctl failed to remove DOF (-1) >> dtrace DOF postgres: DTrace ioctl failed to remove DOF (-1) >> dtrace DOF postgres: DTrace ioctl failed to remove DOF (-1) >> ^Cdtrace DOF postgres: dtrace DOF postgres: DTrace ioctl failed to remove >> DOF (-1) >> DTrace ioctl failed to remove DOF (-1) >> dtrace DOF postgres: DTrace ioctl failed to remove DOF (-1) >> dtrace DOF postgres: DTrace ioctl failed to remove DOF (-1) >> dtrace DOF postgres: DTrace ioctl failed to remove DOF (-1) >> dtrace DOF postgres: DTrace ioctl failed to remove DOF (-1) >> [pgsql@talizorah ~]$ > Hi Lacey, > > I've never seen this particular problem before. Could you post the > output of "ldd /usr/local/bin/postgres"? I've never tried using libelf > from ports, so perhaps there's a problem with it, assuming it is indeed > being used (as opposed to the base system's libelf). > > If the ports libelf is being used (/usr/local/lib/libelf.so will show up > in the ldd ouput if so), could you try deleting devel/libelf, rebuilding > postgres, and trying again? If not, or if other packages require > devel/libelf on your system, could you apply the patch below, rebuild drti, > rebuild postgres (in that order), and try again? > > To rebuild drti, cd to /usr/src/cddl/lib/drti and run > "make && make install". > > Thanks! > -Mark > > diff --git a/cddl/contrib/opensolaris/lib/libdtrace/common/drti.c b/cddl/contrib/opensolaris/lib/libdtrace/common/drti.c > index ccd4f9b..eafc4c6 100644 > --- a/cddl/contrib/opensolaris/lib/libdtrace/common/drti.c > +++ b/cddl/contrib/opensolaris/lib/libdtrace/common/drti.c > @@ -158,7 +158,7 @@ dtrace_dof_init(void) > return; > } > if ((e = elf_begin(efd, ELF_C_READ, NULL)) == NULL) { > - dprintf(1, "elf_begin failed\n"); > + dprintf(1, "elf_begin failed: %s\n", elf_errmsg(elf_errno())); > close(efd); > return; > }