Date: Sun, 15 Mar 2020 16:56:54 +0100 From: Willem Jan Withagen <wjw@digiware.nl> To: Alan Somers <asomers@freebsd.org> Cc: "freebsd-virtualization@freebsd.org" <virtualization@freebsd.org>, FreeBSD Hackers <freebsd-hackers@freebsd.org> Subject: Re: [RFC] Adding a Rados block driver to bhyve Message-ID: <64ff960a-0ea9-60aa-a3b9-bd7f3e1a9d35@digiware.nl> In-Reply-To: <CAOtMX2iYx7CAd2Axo_reeZopWYEd_i%2ByHk5NeKt4bt88ecZTAg@mail.gmail.com> References: <9c7a8dea-ac8a-4d17-ed33-b6c4e882add8@digiware.nl> <CAOtMX2iyhS230oCysYx3YC72B8TwFrrkcXtCoMCYK85KbFORaQ@mail.gmail.com> <c2090191-920b-3d2f-f797-75a448b18ad6@digiware.nl> <CAOtMX2iwJ1_VVDXOnSP6nui-JcYMwndXMex8fxtyi6qEikUX8A@mail.gmail.com> <936ed7c2-99d2-5df8-de3f-f64f28d2ba6f@digiware.nl> <CAOtMX2gKkh=Fik9wjHoAx9Gt9zwePMjmRaGP63eE2quNTmyDGw@mail.gmail.com> <ba937caf-ce4b-eeb9-8915-f9b9910aa711@digiware.nl> <CAOtMX2iYx7CAd2Axo_reeZopWYEd_i%2ByHk5NeKt4bt88ecZTAg@mail.gmail.com>
next in thread | previous in thread | raw e-mail | index | archive | help
On 10-3-2020 17:48, Alan Somers wrote: > On Tue, Mar 10, 2020 at 10:28 AM Willem Jan Withagen <wjw@digiware.nl> > wrote: > >> On 10-3-2020 17:21, Alan Somers wrote: >> >> On Tue, Mar 10, 2020 at 9:41 AM Willem Jan Withagen <wjw@digiware.nl> >> wrote: >> >>> On 10-3-2020 16:15, Alan Somers wrote: >>> >>> On Tue, Mar 10, 2020 at 3:59 AM Willem Jan Withagen <wjw@digiware.nl> >>> wrote: >>> >>>> On 9-3-2020 14:46, Alan Somers wrote: >>>> >>>> On Mon, Mar 9, 2020 at 4:32 AM Willem Jan Withagen <wjw@digiware.nl> >>>> wrote: >>>> >>>>> Hi all, >>>>> >>>>> And sorry for crosspoing three groups, but the answer can/could be a mix >>>>> of things to do in these three areas. >>>>> >>>>> I have a prototype of bhyve running on Rados/Ceph working: >>>>> https://github.com/freebsd/freebsd/pull/426 >>>>> >>>>> .......... >>>> 4) Create a bhyve-blockrbd port. >>>>> This is much like 3) but instead of building a bhyve-rbd >>>>> executable, >>>>> it delivers a libblockrbd.so that is dynamically loadable by the >>>>> standaard bhyve that comes with base. >>>>> >>>> ............ >>>> >>>>> Great work! I also agree that option 4 sounds like the best. There's >>>> precedent for ports that >>>>> require the FreeBSD Sources. For example, see devel/py-libzfs or >>>> emulators/virtualbox-ose. >>>>> You just need to define the SRC_BASE variable. >>>> Hi Alan, >>>> >>>> Thanx for the hint, and it made me check what is actually available >>>> within the poudriere jail >>>> And that does have full source, so the Makefile code is mainly for those >>>> that build in a different way. >>>> >>>> I've got a proto version working when compiling stuff with `make >>>> buildworld`, but run in the >>>> problem that libblock_rbd.so is stripped in such a way that the symbol I >>>> need is removed. >>>> Using the unstripped version does work. >>>> >>>> Is there an incantation for the SRC Makefiles that builds a dynamical >>>> loadable lib?? >>>> And I'm still looking for a PORTS example of building a dynamical >>>> loadable lib. >>>> Or is there no generic code for that in the PORTS Mk files? >>>> >>>> --WjW >>>> >>>> BTW: Still haven't worked in your AIO code :( >>>> >>> There are plenty of dynamic libraries built with the SRC makefiles. For >>> example, >>> https://svnweb.freebsd.org/base/head/lib/libbsdstat/Makefile?view=markup >>> . >>> >>> >>> That looks dangerously close to what I have for libblock_rbd. >>> === >>>> cat Makefile-librbd >>> # >>> # $FreeBSD$ >>> # >>> >>> PACKAGE=lib${LIB} >>> >>> .include <src.opts.mk> >>> >>> LIB= block_rbd >>> SHLIB_MAJOR= 1 >>> >>> SRCS= block_rbd.c >>> >>> CFLAGS+=-I${SRCTOP}/sys >>> CFLAGS+=-g -O0 -fPIC -rdynamic >>> LDFLAGS+=-Wl,-export-dynamic,-Bdynamic >>> CFLAGS+=-DWITHOUT_CAPSICUM >>> >>> LOCALBASE?= /usr/local >>> CFLAGS+= -I${LOCALBASE}/include >>> LDFLAGS+= -L${LOCALBASE}/lib -lrados -lrbd >>> >>> WARNS?= 2 >>> >>> === >>> >>> This is the code that mk.lib.bsd runs: >>> objcopy --only-keep-debug libblock_rbd.so.1.full libblock_rbd.so.1.debug >>> objcopy --strip-debug --add-gnu-debuglink=libblock_rbd.so.1.debug >>> libblock_rbd.so.1.full libblock_rbd.so.1 >>> >>> So still I get a stripped lib in /usr/lib. And then the one and only >>> symbol I need to load >>> is not found. Copying libblock_rbd.so.1.full actually works for me. >>> >>> So either I'm doing it the wrong way, like special options on the symbols >>> oid. >>> Or mk.lib.bsd cannot deliver dlopen/dlsym-able files? >>> >>> And there are plenty of ports that build shared libraries too, just look >>> at /usr/local/lib/*.so. However, the ports framework doesn't have much >>> special code just to support building libraries. Instead the hard work is >>> always done by the ports themselves. Some use autotools, some cmake, etc >>> etc. The simplest port I can find that uses both SRC_BASE and INSTALL_LIB >>> is this one: https://svnweb.freebsd.org/ports/head/devel/linux_libusb/ . >>> >>> Oke thanx, I'll have a look at it, and given that I can see most of the >>> compile build stuff >>> in the SRC_BASE version I'll get it to work. >>> >>> --WjW >>> >> Try setting "STRIP= " in your makefile. That should prevent the >> stripping. However, I think there's something wrong with your library, >> too. The library should be usable even if it's stripped. >> >> I checked with objdump, and the symbol that I need is definitly not >> present in the stripped version. >> And it does not really matter if I declare it static or not. >> But I'll give it a few more itterations to try it out. >> Including 'STRIP= ' >> >> Thanx, >> --WjW >> > What does "nm --dynamic libblock_rbd.so.1" show? Don't know what has changed, but I just rebuild everything. And now it seems to work.... ;-) One thing I'm wondering if th DATA_SET(block_backend_set, ....) gets updated when dyn-loading the library it is in? Or if not, how do I add an entry to this DATA_SET, once we are running the program? I'm dlsym() looking for block_backend_rbd: w _Jv_RegisterClasses U __assert w __cxa_finalize U __error U __stack_chk_fail U __stack_chk_guard 0000000000006098 D __start_set_block_backend_set U __stderrp 00000000000060a0 D __stop_set_block_backend_set 000000000000475c T _fini 000000000000474c T _init 0000000000006008 D block_backend_rbd 0000000000006270 B blocklocal_backend 0000000000002170 T blockrbd_cleanup 0000000000006250 B blockrbd_debug 0000000000002080 T blockrbd_init Which is the variable holding the block_if:jumptable, and gives me access to all that I need to work with the driver. ============== block_backend_t block_backend_rbd = { .bb_name = "rbd", .bb_init = blockrbd_init, .bb_cleanup = blockrbd_cleanup, .bb_open = blockrbd_open, .bb_size = blockrbd_size, .bb_chs = blockrbd_chs, .bb_sectsz = blockrbd_sectsz, .bb_psectsz = blockrbd_psectsz, .bb_queuesz = blockrbd_queuesz, .bb_is_ro = blockrbd_is_ro, .bb_candelete = blockrbd_candelete, .bb_read = blockrbd_read, .bb_write = blockrbd_write, .bb_flush = blockrbd_flush, .bb_delete = blockrbd_delete, .bb_cancel = blockrbd_cancel, .bb_close = blockrbd_close, .bb_opaque = (void*)&block_backend_rbd, }; DATA_SET(block_backend_set, block_backend_rbd); ============== This is the Makefile: # # $FreeBSD$ # PACKAGE= lib${LIB} LIB= block_rbd .include <src.opts.mk> SHLIB_MAJOR= 1 SRCS= block_rbd.c CFLAGS+= -I${SRCTOP}/sys CFLAGS+= -g -O0 -fPIC -rdynamic LDFLAGS+= -Wl,-export-dynamic,-Bdynamic CFLAGS+= -DWITHOUT_CAPSICUM LOCALBASE?= /usr/local CFLAGS+= -I${LOCALBASE}/include LDFLAGS+= -L${LOCALBASE}/lib -lrados -lrbd WARNS?= 2 .include <bsd.lib.mk> --WjW
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?64ff960a-0ea9-60aa-a3b9-bd7f3e1a9d35>