From owner-freebsd-hackers@freebsd.org Sun Mar 15 15:57:06 2020 Return-Path: Delivered-To: freebsd-hackers@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id B841D26F636; Sun, 15 Mar 2020 15:57:06 +0000 (UTC) (envelope-from wjw@digiware.nl) Received: from smtp.digiware.nl (smtp.digiware.nl [IPv6:2001:4cb8:90:ffff::3]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) server-signature RSA-PSS (4096 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 48gPFX6Lc9z3yHg; Sun, 15 Mar 2020 15:57:04 +0000 (UTC) (envelope-from wjw@digiware.nl) Received: from router.digiware.nl (localhost.digiware.nl [127.0.0.1]) by smtp.digiware.nl (Postfix) with ESMTP id 7195B14F9B; Sun, 15 Mar 2020 16:56:55 +0100 (CET) X-Virus-Scanned: amavisd-new at digiware.com Received: from smtp.digiware.nl ([127.0.0.1]) by router.digiware.nl (router.digiware.nl [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id 8oDTxIEBuVSh; Sun, 15 Mar 2020 16:56:54 +0100 (CET) Received: from [192.168.10.9] (vaio [192.168.10.9]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by smtp.digiware.nl (Postfix) with ESMTPSA id C2A1014F8C; Sun, 15 Mar 2020 16:56:54 +0100 (CET) Subject: Re: [RFC] Adding a Rados block driver to bhyve To: Alan Somers Cc: "freebsd-virtualization@freebsd.org" , FreeBSD Hackers References: <9c7a8dea-ac8a-4d17-ed33-b6c4e882add8@digiware.nl> <936ed7c2-99d2-5df8-de3f-f64f28d2ba6f@digiware.nl> From: Willem Jan Withagen Message-ID: <64ff960a-0ea9-60aa-a3b9-bd7f3e1a9d35@digiware.nl> Date: Sun, 15 Mar 2020 16:56:54 +0100 User-Agent: Mozilla/5.0 (Windows NT 10.0; WOW64; rv:68.0) Gecko/20100101 Firefox/68.0 Thunderbird/68.6.0 MIME-Version: 1.0 In-Reply-To: Content-Type: text/plain; charset=windows-1252; format=flowed Content-Transfer-Encoding: 8bit Content-Language: nl X-Rspamd-Queue-Id: 48gPFX6Lc9z3yHg X-Spamd-Bar: ----- Authentication-Results: mx1.freebsd.org; dkim=none; dmarc=none; spf=pass (mx1.freebsd.org: domain of wjw@digiware.nl designates 2001:4cb8:90:ffff::3 as permitted sender) smtp.mailfrom=wjw@digiware.nl X-Spamd-Result: default: False [-5.25 / 15.00]; ARC_NA(0.00)[]; RCVD_VIA_SMTP_AUTH(0.00)[]; NEURAL_HAM_MEDIUM(-0.85)[-0.848,0]; FROM_HAS_DN(0.00)[]; RCPT_COUNT_THREE(0.00)[3]; R_SPF_ALLOW(-0.20)[+mx]; TO_MATCH_ENVRCPT_ALL(0.00)[]; MIME_GOOD(-0.10)[text/plain]; RCVD_TLS_LAST(0.00)[]; DMARC_NA(0.00)[digiware.nl]; NEURAL_HAM_LONG(-1.00)[-1.000,0]; RCVD_COUNT_THREE(0.00)[4]; IP_SCORE(-3.10)[ip: (-9.51), ipnet: 2001:4cb8::/29(-4.68), asn: 28878(-1.35), country: NL(0.03)]; TO_DN_ALL(0.00)[]; FROM_EQ_ENVFROM(0.00)[]; R_DKIM_NA(0.00)[]; MIME_TRACE(0.00)[0:+]; ASN(0.00)[asn:28878, ipnet:2001:4cb8::/29, country:NL]; MID_RHS_MATCH_FROM(0.00)[] X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 15 Mar 2020 15:57:06 -0000 On 10-3-2020 17:48, Alan Somers wrote: > On Tue, Mar 10, 2020 at 10:28 AM Willem Jan Withagen > wrote: > >> On 10-3-2020 17:21, Alan Somers wrote: >> >> On Tue, Mar 10, 2020 at 9:41 AM Willem Jan Withagen >> wrote: >> >>> On 10-3-2020 16:15, Alan Somers wrote: >>> >>> On Tue, Mar 10, 2020 at 3:59 AM Willem Jan Withagen >>> wrote: >>> >>>> On 9-3-2020 14:46, Alan Somers wrote: >>>> >>>> On Mon, Mar 9, 2020 at 4:32 AM Willem Jan Withagen >>>> wrote: >>>> >>>>> Hi all, >>>>> >>>>> And sorry for crosspoing three groups, but the answer can/could be a mix >>>>> of things to do in these three areas. >>>>> >>>>> I have a prototype of bhyve running on Rados/Ceph working: >>>>> https://github.com/freebsd/freebsd/pull/426 >>>>> >>>>> .......... >>>> 4) Create a bhyve-blockrbd port. >>>>> This is much like 3) but instead of building a bhyve-rbd >>>>> executable, >>>>> it delivers a libblockrbd.so that is dynamically loadable by the >>>>> standaard bhyve that comes with base. >>>>> >>>> ............ >>>> >>>>> Great work! I also agree that option 4 sounds like the best. There's >>>> precedent for ports that >>>>> require the FreeBSD Sources. For example, see devel/py-libzfs or >>>> emulators/virtualbox-ose. >>>>> You just need to define the SRC_BASE variable. >>>> Hi Alan, >>>> >>>> Thanx for the hint, and it made me check what is actually available >>>> within the poudriere jail >>>> And that does have full source, so the Makefile code is mainly for those >>>> that build in a different way. >>>> >>>> I've got a proto version working when compiling stuff with `make >>>> buildworld`, but run in the >>>> problem that libblock_rbd.so is stripped in such a way that the symbol I >>>> need is removed. >>>> Using the unstripped version does work. >>>> >>>> Is there an incantation for the SRC Makefiles that builds a dynamical >>>> loadable lib?? >>>> And I'm still looking for a PORTS example of building a dynamical >>>> loadable lib. >>>> Or is there no generic code for that in the PORTS Mk files? >>>> >>>> --WjW >>>> >>>> BTW: Still haven't worked in your AIO code :( >>>> >>> There are plenty of dynamic libraries built with the SRC makefiles. For >>> example, >>> https://svnweb.freebsd.org/base/head/lib/libbsdstat/Makefile?view=markup >>> . >>> >>> >>> That looks dangerously close to what I have for libblock_rbd. >>> === >>>> cat Makefile-librbd >>> # >>> # $FreeBSD$ >>> # >>> >>> PACKAGE=lib${LIB} >>> >>> .include >>> >>> LIB= block_rbd >>> SHLIB_MAJOR= 1 >>> >>> SRCS= block_rbd.c >>> >>> CFLAGS+=-I${SRCTOP}/sys >>> CFLAGS+=-g -O0 -fPIC -rdynamic >>> LDFLAGS+=-Wl,-export-dynamic,-Bdynamic >>> CFLAGS+=-DWITHOUT_CAPSICUM >>> >>> LOCALBASE?= /usr/local >>> CFLAGS+= -I${LOCALBASE}/include >>> LDFLAGS+= -L${LOCALBASE}/lib -lrados -lrbd >>> >>> WARNS?= 2 >>> >>> === >>> >>> This is the code that mk.lib.bsd runs: >>> objcopy --only-keep-debug libblock_rbd.so.1.full libblock_rbd.so.1.debug >>> objcopy --strip-debug --add-gnu-debuglink=libblock_rbd.so.1.debug >>> libblock_rbd.so.1.full libblock_rbd.so.1 >>> >>> So still I get a stripped lib in /usr/lib. And then the one and only >>> symbol I need to load >>> is not found. Copying libblock_rbd.so.1.full actually works for me. >>> >>> So either I'm doing it the wrong way, like special options on the symbols >>> oid. >>> Or mk.lib.bsd cannot deliver dlopen/dlsym-able files? >>> >>> And there are plenty of ports that build shared libraries too, just look >>> at /usr/local/lib/*.so. However, the ports framework doesn't have much >>> special code just to support building libraries. Instead the hard work is >>> always done by the ports themselves. Some use autotools, some cmake, etc >>> etc. The simplest port I can find that uses both SRC_BASE and INSTALL_LIB >>> is this one: https://svnweb.freebsd.org/ports/head/devel/linux_libusb/ . >>> >>> Oke thanx, I'll have a look at it, and given that I can see most of the >>> compile build stuff >>> in the SRC_BASE version I'll get it to work. >>> >>> --WjW >>> >> Try setting "STRIP= " in your makefile. That should prevent the >> stripping. However, I think there's something wrong with your library, >> too. The library should be usable even if it's stripped. >> >> I checked with objdump, and the symbol that I need is definitly not >> present in the stripped version. >> And it does not really matter if I declare it static or not. >> But I'll give it a few more itterations to try it out. >> Including 'STRIP= ' >> >> Thanx, >> --WjW >> > What does "nm --dynamic libblock_rbd.so.1" show? Don't know what has changed, but I just rebuild everything. And now it seems to work.... ;-) One thing I'm wondering if th DATA_SET(block_backend_set, ....) gets updated when dyn-loading the library it is in? Or if not, how do I add an entry to this DATA_SET, once we are running the program? I'm dlsym() looking for block_backend_rbd:                  w _Jv_RegisterClasses                  U __assert                  w __cxa_finalize                  U __error                  U __stack_chk_fail                  U __stack_chk_guard 0000000000006098 D __start_set_block_backend_set                  U __stderrp 00000000000060a0 D __stop_set_block_backend_set 000000000000475c T _fini 000000000000474c T _init 0000000000006008 D block_backend_rbd 0000000000006270 B blocklocal_backend 0000000000002170 T blockrbd_cleanup 0000000000006250 B blockrbd_debug 0000000000002080 T blockrbd_init Which is the variable holding the block_if:jumptable, and gives me access to all that I need to work with the driver. ============== block_backend_t block_backend_rbd = {     .bb_name      = "rbd",     .bb_init      = blockrbd_init,     .bb_cleanup   = blockrbd_cleanup,     .bb_open      = blockrbd_open,     .bb_size      = blockrbd_size,     .bb_chs       = blockrbd_chs,     .bb_sectsz    = blockrbd_sectsz,     .bb_psectsz   = blockrbd_psectsz,     .bb_queuesz   = blockrbd_queuesz,     .bb_is_ro     = blockrbd_is_ro,     .bb_candelete = blockrbd_candelete,     .bb_read      = blockrbd_read,     .bb_write     = blockrbd_write,     .bb_flush     = blockrbd_flush,     .bb_delete    = blockrbd_delete,     .bb_cancel    = blockrbd_cancel,     .bb_close     = blockrbd_close,     .bb_opaque    = (void*)&block_backend_rbd, }; DATA_SET(block_backend_set, block_backend_rbd); ============== This is the Makefile: # # $FreeBSD$ # PACKAGE=        lib${LIB} LIB=    block_rbd .include SHLIB_MAJOR=    1 SRCS=   block_rbd.c CFLAGS+=        -I${SRCTOP}/sys CFLAGS+=        -g -O0 -fPIC -rdynamic LDFLAGS+=       -Wl,-export-dynamic,-Bdynamic CFLAGS+=        -DWITHOUT_CAPSICUM LOCALBASE?=     /usr/local CFLAGS+=        -I${LOCALBASE}/include LDFLAGS+=       -L${LOCALBASE}/lib -lrados -lrbd WARNS?= 2 .include --WjW