From owner-freebsd-hackers@freebsd.org Tue Mar 10 15:41:47 2020 Return-Path: Delivered-To: freebsd-hackers@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 29C022678B7; Tue, 10 Mar 2020 15:41:47 +0000 (UTC) (envelope-from wjw@digiware.nl) Received: from smtp.digiware.nl (smtp.digiware.nl [176.74.240.9]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) server-signature RSA-PSS (4096 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 48cK8B470Qz4bgt; Tue, 10 Mar 2020 15:41:46 +0000 (UTC) (envelope-from wjw@digiware.nl) Received: from router.digiware.nl (localhost.digiware.nl [127.0.0.1]) by smtp.digiware.nl (Postfix) with ESMTP id D02A713C81; Tue, 10 Mar 2020 16:41:43 +0100 (CET) X-Virus-Scanned: amavisd-new at digiware.com Received: from smtp.digiware.nl ([127.0.0.1]) by router.digiware.nl (router.digiware.nl [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id vYtHBxsbLW_x; Tue, 10 Mar 2020 16:41:43 +0100 (CET) Received: from [192.168.10.67] (opteron [192.168.10.67]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by smtp.digiware.nl (Postfix) with ESMTPSA id CA01113C75; Tue, 10 Mar 2020 16:41:42 +0100 (CET) Subject: Re: [RFC] Adding a Rados block driver to bhyve To: Alan Somers Cc: "freebsd-virtualization@freebsd.org" , FreeBSD Hackers , "ports@freebsd.org" References: <9c7a8dea-ac8a-4d17-ed33-b6c4e882add8@digiware.nl> From: Willem Jan Withagen Message-ID: <936ed7c2-99d2-5df8-de3f-f64f28d2ba6f@digiware.nl> Date: Tue, 10 Mar 2020 16:41:41 +0100 User-Agent: Mozilla/5.0 (Windows NT 10.0; WOW64; rv:68.0) Gecko/20100101 Firefox/68.0 Thunderbird/68.5.0 MIME-Version: 1.0 In-Reply-To: Content-Language: nl X-Rspamd-Queue-Id: 48cK8B470Qz4bgt X-Spamd-Bar: ---- Authentication-Results: mx1.freebsd.org; none X-Spamd-Result: default: False [-4.10 / 15.00]; NEURAL_SPAM_MEDIUM(0.90)[0.897,0]; REPLY(-4.00)[]; NEURAL_HAM_LONG(-0.99)[-0.994,0] Content-Type: text/plain; charset=utf-8; format=flowed Content-Transfer-Encoding: 8bit X-Content-Filtered-By: Mailman/MimeDel 2.1.29 X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 10 Mar 2020 15:41:47 -0000 On 10-3-2020 16:15, Alan Somers wrote: > On Tue, Mar 10, 2020 at 3:59 AM Willem Jan Withagen > wrote: > > On 9-3-2020 14:46, Alan Somers wrote: >> On Mon, Mar 9, 2020 at 4:32 AM Willem Jan Withagen >> > wrote: >> >> Hi all, >> >> And sorry for crosspoing three groups, but the answer >> can/could be a mix >> of things to do in these three areas. >> >> I have a prototype of bhyve running on Rados/Ceph working: >> https://github.com/freebsd/freebsd/pull/426 >> > .......... >> >> 4) Create a bhyve-blockrbd port. >>      This is much like 3) but instead of building a bhyve-rbd >> executable, >>      it delivers a libblockrbd.so that is dynamically >> loadable by the >>      standaard bhyve that comes with base. >> > ............ > > > Great work!  I also agree that option 4 sounds like the best.  > There's precedent for ports that > > require the FreeBSD Sources.  For example, see devel/py-libzfs > or emulators/virtualbox-ose. > > You just need to define the SRC_BASE variable. > Hi Alan, > > Thanx for the hint, and it made me check what is actually > available within the poudriere jail > And that does have full source, so the Makefile code is mainly for > those that build in a different way. > > I've got a proto version working when compiling stuff with `make > buildworld`, but run in the > problem that libblock_rbd.so is stripped in such a way that the > symbol I need is removed. > Using the unstripped version does work. > > Is there an incantation for the SRC Makefiles that builds a > dynamical loadable lib?? > And I'm still looking for a PORTS example of building a dynamical > loadable lib. > Or is there no generic code for that in the PORTS Mk files? > > --WjW > > BTW: Still haven't worked in your AIO code :( > > > There are plenty of dynamic libraries built with the SRC makefiles.  > For example, > https://svnweb.freebsd.org/base/head/lib/libbsdstat/Makefile?view=markup > . That looks dangerously close to what I have for libblock_rbd. === > cat Makefile-librbd # # $FreeBSD$ # PACKAGE=lib${LIB} .include LIB=            block_rbd SHLIB_MAJOR=    1 SRCS=   block_rbd.c CFLAGS+=-I${SRCTOP}/sys CFLAGS+=-g -O0 -fPIC -rdynamic LDFLAGS+=-Wl,-export-dynamic,-Bdynamic CFLAGS+=-DWITHOUT_CAPSICUM LOCALBASE?=     /usr/local CFLAGS+=        -I${LOCALBASE}/include LDFLAGS+=       -L${LOCALBASE}/lib -lrados -lrbd WARNS?= 2 === This is the code that mk.lib.bsd runs: objcopy --only-keep-debug libblock_rbd.so.1.full libblock_rbd.so.1.debug objcopy --strip-debug --add-gnu-debuglink=libblock_rbd.so.1.debug libblock_rbd.so.1.full libblock_rbd.so.1 So still I get a stripped lib in /usr/lib. And then the one and only symbol I need to load is not found. Copying libblock_rbd.so.1.full actually works for me. So either I'm doing it the wrong way, like special options on the symbols oid. Or mk.lib.bsd cannot deliver dlopen/dlsym-able files? And there are plenty of ports that build shared libraries too, just look at /usr/local/lib/*.so.  However, the ports framework doesn't have much special code just to support building libraries.  Instead the hard work is always done by the ports themselves.  Some use autotools, some cmake, etc etc.  The simplest port I can find that uses both SRC_BASE and INSTALL_LIB is this one: https://svnweb.freebsd.org/ports/head/devel/linux_libusb/ . Oke thanx, I'll have a look at it, and given that I can see most of the compile build stuff in the SRC_BASE version I'll get it to work. --WjW