From owner-freebsd-hackers@freebsd.org Tue Mar 10 16:28:15 2020 Return-Path: Delivered-To: freebsd-hackers@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 66FFA2696A2; Tue, 10 Mar 2020 16:28:15 +0000 (UTC) (envelope-from wjw@digiware.nl) Received: from smtp.digiware.nl (smtp.digiware.nl [176.74.240.9]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) server-signature RSA-PSS (4096 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 48cL9q0hljz41fQ; Tue, 10 Mar 2020 16:28:14 +0000 (UTC) (envelope-from wjw@digiware.nl) Received: from router.digiware.nl (localhost.digiware.nl [127.0.0.1]) by smtp.digiware.nl (Postfix) with ESMTP id 9660D1530F; Tue, 10 Mar 2020 17:28:12 +0100 (CET) X-Virus-Scanned: amavisd-new at digiware.com Received: from smtp.digiware.nl ([127.0.0.1]) by router.digiware.nl (router.digiware.nl [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id 7U7cJZp7et5f; Tue, 10 Mar 2020 17:28:11 +0100 (CET) Received: from [192.168.10.67] (opteron [192.168.10.67]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits)) (No client certificate requested) by smtp.digiware.nl (Postfix) with ESMTPSA id 9A8381530D; Tue, 10 Mar 2020 17:28:11 +0100 (CET) Subject: Re: [RFC] Adding a Rados block driver to bhyve To: Alan Somers Cc: "freebsd-virtualization@freebsd.org" , FreeBSD Hackers References: <9c7a8dea-ac8a-4d17-ed33-b6c4e882add8@digiware.nl> <936ed7c2-99d2-5df8-de3f-f64f28d2ba6f@digiware.nl> From: Willem Jan Withagen Message-ID: Date: Tue, 10 Mar 2020 17:28:10 +0100 User-Agent: Mozilla/5.0 (Windows NT 10.0; WOW64; rv:68.0) Gecko/20100101 Firefox/68.0 Thunderbird/68.5.0 MIME-Version: 1.0 In-Reply-To: Content-Language: nl X-Rspamd-Queue-Id: 48cL9q0hljz41fQ X-Spamd-Bar: ---- Authentication-Results: mx1.freebsd.org; none X-Spamd-Result: default: False [-4.06 / 15.00]; NEURAL_SPAM_MEDIUM(0.93)[0.934,0]; REPLY(-4.00)[]; NEURAL_HAM_LONG(-0.99)[-0.995,0] Content-Type: text/plain; charset=utf-8; format=flowed Content-Transfer-Encoding: 8bit X-Content-Filtered-By: Mailman/MimeDel 2.1.29 X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 10 Mar 2020 16:28:15 -0000 On 10-3-2020 17:21, Alan Somers wrote: > On Tue, Mar 10, 2020 at 9:41 AM Willem Jan Withagen > wrote: > > On 10-3-2020 16:15, Alan Somers wrote: >> On Tue, Mar 10, 2020 at 3:59 AM Willem Jan Withagen >> > wrote: >> >> On 9-3-2020 14:46, Alan Somers wrote: >>> On Mon, Mar 9, 2020 at 4:32 AM Willem Jan Withagen >>> > wrote: >>> >>> Hi all, >>> >>> And sorry for crosspoing three groups, but the answer >>> can/could be a mix >>> of things to do in these three areas. >>> >>> I have a prototype of bhyve running on Rados/Ceph working: >>> https://github.com/freebsd/freebsd/pull/426 >>> >> .......... >>> >>> 4) Create a bhyve-blockrbd port. >>>      This is much like 3) but instead of building a >>> bhyve-rbd executable, >>>      it delivers a libblockrbd.so that is dynamically >>> loadable by the >>>      standaard bhyve that comes with base. >>> >> ............ >> >> > Great work!  I also agree that option 4 sounds like the >> best.  There's precedent for ports that >> > require the FreeBSD Sources.  For example, see >> devel/py-libzfs or emulators/virtualbox-ose. >> > You just need to define the SRC_BASE variable. >> Hi Alan, >> >> Thanx for the hint, and it made me check what is actually >> available within the poudriere jail >> And that does have full source, so the Makefile code is >> mainly for those that build in a different way. >> >> I've got a proto version working when compiling stuff with >> `make buildworld`, but run in the >> problem that libblock_rbd.so is stripped in such a way that >> the symbol I need is removed. >> Using the unstripped version does work. >> >> Is there an incantation for the SRC Makefiles that builds a >> dynamical loadable lib?? >> And I'm still looking for a PORTS example of building a >> dynamical loadable lib. >> Or is there no generic code for that in the PORTS Mk files? >> >> --WjW >> >> BTW: Still haven't worked in your AIO code :( >> >> >> There are plenty of dynamic libraries built with the SRC >> makefiles.  For example, >> https://svnweb.freebsd.org/base/head/lib/libbsdstat/Makefile?view=markup >> . > > That looks dangerously close to what I have for libblock_rbd. > === > > cat Makefile-librbd > # > # $FreeBSD$ > # > > PACKAGE=lib${LIB} > > .include > > > LIB=            block_rbd > SHLIB_MAJOR=    1 > > SRCS=   block_rbd.c > > CFLAGS+=-I${SRCTOP}/sys > CFLAGS+=-g -O0 -fPIC -rdynamic > LDFLAGS+=-Wl,-export-dynamic,-Bdynamic > CFLAGS+=-DWITHOUT_CAPSICUM > > LOCALBASE?=     /usr/local > CFLAGS+=        -I${LOCALBASE}/include > LDFLAGS+=       -L${LOCALBASE}/lib -lrados -lrbd > > WARNS?= 2 > > === > > This is the code that mk.lib.bsd runs: > objcopy --only-keep-debug libblock_rbd.so.1.full > libblock_rbd.so.1.debug > objcopy --strip-debug --add-gnu-debuglink=libblock_rbd.so.1.debug > libblock_rbd.so.1.full libblock_rbd.so.1 > > So still I get a stripped lib in /usr/lib. And then the one and > only symbol I need to load > is not found. Copying libblock_rbd.so.1.full actually works for me. > > So either I'm doing it the wrong way, like special options on the > symbols oid. > Or mk.lib.bsd cannot deliver dlopen/dlsym-able files? > > And there are plenty of ports that build shared libraries too, > just look at /usr/local/lib/*.so. However, the ports framework > doesn't have much special code just to support building > libraries.  Instead the hard work is always done by the ports > themselves. Some use autotools, some cmake, etc etc.  The simplest > port I can find that uses both SRC_BASE and INSTALL_LIB is this > one: https://svnweb.freebsd.org/ports/head/devel/linux_libusb/ . > > Oke thanx, I'll have a look at it, and given that I can see most > of the compile build stuff > in the SRC_BASE version I'll get it to work. > > --WjW > > > Try setting "STRIP=    " in your makefile.  That should prevent the > stripping.  However, I think there's something wrong with your > library, too.  The library should be usable even if it's stripped. I checked with objdump, and the symbol that I need is definitly not present in the stripped version. And it does not really matter if I declare it static or not. But I'll give it a few more itterations to try it out. Including 'STRIP= ' Thanx, --WjW