From owner-freebsd-toolchain@freebsd.org Sat Mar 2 13:22:07 2019 Return-Path: Delivered-To: freebsd-toolchain@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 2DF3A1522C0D for ; Sat, 2 Mar 2019 13:22:07 +0000 (UTC) (envelope-from wjw@digiware.nl) Received: from smtp.digiware.nl (smtp.digiware.nl [176.74.240.9]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id E51998E01E; Sat, 2 Mar 2019 13:22:05 +0000 (UTC) (envelope-from wjw@digiware.nl) Received: from router.digiware.nl (localhost.digiware.nl [127.0.0.1]) by smtp.digiware.nl (Postfix) with ESMTP id DE2DD6EE50; Sat, 2 Mar 2019 14:21:59 +0100 (CET) X-Virus-Scanned: amavisd-new at digiware.com Received: from smtp.digiware.nl ([127.0.0.1]) by router.digiware.nl (router.digiware.nl [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id u3ZqV6UlL0Yr; Sat, 2 Mar 2019 14:21:58 +0100 (CET) Received: from [192.168.10.9] (vaio [192.168.10.9]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.digiware.nl (Postfix) with ESMTPSA id 5D4E86EE4F; Sat, 2 Mar 2019 14:21:58 +0100 (CET) Subject: Re: Linking problem with lld From: Willem Jan Withagen To: =?UTF-8?Q?T=c4=b3l_Coosemans?= Cc: FreeBSD Toolchain , Ed Maste References: <71079fbc-c3c0-9b93-0ae4-8ceda5d1f751@digiware.nl> <20190222223456.5d7840b7@kalimero.tijl.coosemans.org> <29a75c78-f2df-e80b-d013-631584e0fc71@digiware.nl> Message-ID: Date: Sat, 2 Mar 2019 14:21:57 +0100 User-Agent: Mozilla/5.0 (Windows NT 10.0; WOW64; rv:60.0) Gecko/20100101 Thunderbird/60.5.1 MIME-Version: 1.0 In-Reply-To: <29a75c78-f2df-e80b-d013-631584e0fc71@digiware.nl> Content-Type: text/plain; charset=utf-8; format=flowed Content-Transfer-Encoding: 8bit Content-Language: nl X-Rspamd-Queue-Id: E51998E01E X-Spamd-Bar: ----- Authentication-Results: mx1.freebsd.org; spf=pass (mx1.freebsd.org: domain of wjw@digiware.nl designates 176.74.240.9 as permitted sender) smtp.mailfrom=wjw@digiware.nl X-Spamd-Result: default: False [-5.05 / 15.00]; ARC_NA(0.00)[]; RCVD_VIA_SMTP_AUTH(0.00)[]; NEURAL_HAM_MEDIUM(-1.00)[-1.000,0]; FROM_HAS_DN(0.00)[]; RCPT_COUNT_THREE(0.00)[3]; R_SPF_ALLOW(-0.20)[+mx]; TO_MATCH_ENVRCPT_ALL(0.00)[]; MIME_GOOD(-0.10)[text/plain]; RCVD_TLS_LAST(0.00)[]; DMARC_NA(0.00)[digiware.nl]; NEURAL_HAM_LONG(-1.00)[-1.000,0]; RCVD_COUNT_THREE(0.00)[4]; IP_SCORE(-1.67)[ipnet: 176.74.224.0/19(-4.86), asn: 28878(-3.51), country: NL(0.02)]; TO_DN_ALL(0.00)[]; RCVD_IN_DNSWL_MED(-0.20)[9.240.74.176.list.dnswl.org : 127.0.9.2]; MX_GOOD(-0.01)[smtp.digiware.nl,www.digiware.nl]; NEURAL_HAM_SHORT(-0.87)[-0.867,0]; FROM_EQ_ENVFROM(0.00)[]; R_DKIM_NA(0.00)[]; MIME_TRACE(0.00)[0:+]; ASN(0.00)[asn:28878, ipnet:176.74.224.0/19, country:NL]; MID_RHS_MATCH_FROM(0.00)[] X-BeenThere: freebsd-toolchain@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Maintenance of FreeBSD's integrated toolchain List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 02 Mar 2019 13:22:07 -0000 On 22-2-2019 23:47, Willem Jan Withagen wrote: > On 22-2-2019 22:34, Tijl Coosemans wrote: >> On Fri, 22 Feb 2019 16:09:09 +0100 Willem Jan Withagen >> wrote: >>> Hi Ed ea, >>> >>> In het ceph project, they started to use library versioning, so it >>> seems. >>> >>> But compiling/linking that now with Clang and lld I run into trouble >>> building the ceph-tools. >>> >>> This is what I get back of one of the Devs: >>> -------------- >>> My guess is that your linker doesn't support the new symbol versioning >>> exports and since the symbols are hidden by default they aren't visible >>> in the shared library. Previously there was a bug (since Luminous and >>> the switch the cmake) where every public and private symbol was >>> exported >>> by librados. >>> >>> We just need to know the magic words to use for your compiler/linker to >>> change the symbol visibility to public for the API methods. >>> >>> -------------- >>> >>> Now I looked thru the options on both Clang and lld, but nothing >>> obviously springs into focus, as to add to change.... >>> >>> This is how librados is build: >>>    /usr/bin/c++ -fPIC  -Wall -Wtype-limits -Wignored-qualifiers >>> -Winit-self -Wpointer-arith -Werror=format-security >>> -fno-strict-aliasing >>> -fsigned-char -Wno-unknown-pragmas -Wno-unused-function >>> -Wno-unused-local-typedef -Wno-varargs -Wno-gnu-designator >>> -Wno-missing-braces -Wno-parentheses -Wno-deprecated-register >>> -ftemplate-depth-1024 -Wnon-virtual-dtor -Wno-unknown-pragmas >>> -Wno-ignored-qualifiers -Wno-inconsistent-missing-override >>> -Wno-mismatched-tags -Wno-unused-private-field >>> -Wno-address-of-packed-member -DCEPH_DEBUG_MUTEX >>> -fdiagnostics-color=auto -fno-builtin-malloc -fno-builtin-calloc >>> -fno-builtin-realloc -fno-builtin-free  -O0 -g >>> -Wno-unused-command-line-argument -fuse-ld=/usr/bin/ld.lld >>> -Wl,--exclude-libs,ALL >>> -Wl,--version-script=/home/jenkins/workspace/ceph-master/src/librados/librados.map >>> >> Can you provide this librados.map file? > https://github.com/ceph/ceph/blob/master/src/librados/librados.map: > ------ > LIBRADOS_PRIVATE { >         local:  *; > }; > > LIBRADOS_14.2.0 { >         global: >                 extern "C++" { >                         ceph::buffer::v14_2_0::*; >                         librados::v14_2_0::*; > >                         "typeinfo for > librados::v14_2_0::ObjectOperation"; >                         "typeinfo name for > librados::v14_2_0::ObjectOperation"; >                         "vtable for librados::v14_2_0::ObjectOperation"; > >                         "typeinfo for > librados::v14_2_0::ObjectReadOperation"; >                         "typeinfo name for > librados::v14_2_0::ObjectReadOperation"; >                         "vtable for > librados::v14_2_0::ObjectReadOperation"; > >                         "typeinfo for > librados::v14_2_0::ObjectWriteOperation"; >                         "typeinfo name for > librados::v14_2_0::ObjectWriteOperation"; >                         "vtable for > librados::v14_2_0::ObjectWriteOperation"; > >                         "typeinfo for librados::v14_2_0::WatchCtx"; >                         "typeinfo name for librados::v14_2_0::WatchCtx"; >                         "vtable for librados::v14_2_0::WatchCtx"; > >                         "typeinfo for librados::v14_2_0::WatchCtx2"; >                         "typeinfo name for librados::v14_2_0::WatchCtx2"; >                         "vtable for librados::v14_2_0::WatchCtx2"; >                 }; > } LIBRADOS_PRIVATE; > ------ > --WjW > Still haven't been able to work this out, I know a bit more about versioning but I'n not sure that the way selected in Ceph is working for FreeBSD. I've collected most of the essentials below I hope. --WjW So the test program is simple enough: ==== #include "include/rados/librados.h" #include "gtest/gtest.h" TEST(Librados, CreateShutdown) {   rados_t cluster;   int err;   err = rados_create(&cluster, "someid");   EXPECT_EQ(err, 0);   rados_shutdown(cluster); } ==== librados is big, but in summary the releavant parts are: ==== #ifdef __cplusplus extern "C" { #endif ..... #define CEPH_RADOS_API ..... CEPH_RADOS_API int rados_create(rados_t *cluster, const char * const id); CEPH_RADOS_API void rados_shutdown(rados_t cluster); #ifdef __cplusplus } #endif ==== And in the implementation of librados in `librados_c.cc` it looks like: ==== .... #define LIBRADOS_C_API_BASE(fn)               \   asm(".symver _" #fn "_base, " #fn "@") #define LIBRADOS_C_API_BASE_DEFAULT(fn)       \   asm(".symver _" #fn ", " #fn "@@") #define LIBRADOS_C_API_DEFAULT(fn, ver)       \   asm(".symver _" #fn ", " #fn "@@LIBRADOS_" #ver) extern "C" int _rados_create(rados_t *pcluster, const char * const id) { ....... } LIBRADOS_C_API_BASE_DEFAULT(rados_create); extern "C" void _rados_shutdown(rados_t cluster) { ....... } LIBRADOS_C_API_BASE_DEFAULT(rados_shutdown); ==== Then this is the Cmake receipe to bolt it all together into librados.so.2.0.0: ==== # C/C++ API add_library(librados ${CEPH_SHARED}   librados_c.cc   librados_cxx.cc   $) if(ENABLE_SHARED)   set_target_properties(librados PROPERTIES     OUTPUT_NAME rados     VERSION 2.0.0     SOVERSION 2     VISIBILITY_INLINES_HIDDEN ON)   if(NOT APPLE)     set_property(TARGET librados APPEND_STRING PROPERTY       LINK_FLAGS " -Wl,--exclude-libs,ALL")     set_property(TARGET librados APPEND_STRING PROPERTY       LINK_FLAGS " -Wl,--version-script=${CMAKE_CURRENT_SOURCE_DIR}/librados.map")   endif()   if(WITH_STATIC_LIBSTDCXX)     set_property(TARGET librados APPEND_STRING PROPERTY       LINK_FLAGS " -static-libstdc++ -static-libgcc")   endif() endif() target_link_libraries(librados PRIVATE   librados_impl osdc ceph-common cls_lock_client   ${BLKID_LIBRARIES} ${CRYPTO_LIBS} ${EXTRALIBS} ${GSSAPI_LIBRARIES}) install(TARGETS librados DESTINATION ${CMAKE_INSTALL_LIBDIR}) ===== Now if I look in librados.so for rados_create that gives: ==== > objdump -t librados.so.2.0.0 | grep rados_create 000000000008a430 l     F .text  00000000000000a6 _ZL16rados_create_cctPKcP18CephInitParameters 000000000008a250 l     F .text  00000000000001df _rados_create 000000000008a4e0 l     F .text  000000000000028d _rados_create2 000000000009f490 l     F .text  00000000000000a4 _rados_create_read_op 000000000008a770 l     F .text  0000000000000085 _rados_create_with_context 000000000009dab0 l     F .text  00000000000000a4 _rados_create_write_op 000000000008a4e0 l     F .text  000000000000028d rados_create2 000000000008a250 l     F .text  00000000000001df rados_create 000000000009f490 l     F .text  00000000000000a4 rados_create_read_op 000000000008a770 l     F .text  0000000000000085 rados_create_with_context 000000000009dab0 l     F .text  00000000000000a4 rados_create_write_op ==== So now I'm sort of baffled why linking does not work: ==== /usr/bin/c++   -Wall -Wtype-limits -Wignored-qualifiers -Winit-self -Wpointer-arith -Werror=format-security -fno-strict-aliasing -fsigned-char -Wno-unknown-pragmas -Wno-unused-function -Wno-unused-local-typedef -Wno-varargs -Wno-gnu-designator -Wno-missing-braces -Wno-parentheses -Wno-deprecated-register  -ftemplate-depth-1024 -Wnon-virtual-dtor -Wno-unknown-pragmas -Wno-ignored-qualifiers -Wno-inconsistent-missing-override -Wno-mismatched-tags -Wno-unused-private-field -Wno-address-of-packed-member -DCEPH_DEBUG_MUTEX -fdiagnostics-color=auto -fno-builtin-malloc -fno-builtin-calloc -fno-builtin-realloc -fno-builtin-free  -O0 -g -Wno-unused-command-line-argument  -Wl,--export-dynamic CMakeFiles/unittest_librados.dir/librados.cc.o  -o ../../../bin/unittest_librados  -L/usr/local/lib -Wl,-rpath,/usr/local/lib:/home/jenkins/workspace/ceph-master/build/lib ../../../lib/libgmock_main.a ../../../lib/libgmock.a ../../../lib/libgtest.a -lpthread /usr/local/lib/libldap.so /usr/local/lib/liblber.so ../../../lib/librados.so.2.0.0 /usr/local/lib/libldap.so /usr/local/lib/liblber.so ld: error: undefined symbol: rados_create >>> referenced by librados.cc:9 (/home/jenkins/workspace/ceph-master/src/test/librados/librados.cc:9) >>> CMakeFiles/unittest_librados.dir/librados.cc.o:(Librados_CreateShutdown_Test::TestBody()) ld: error: undefined symbol: rados_shutdown >>> referenced by librados.cc:12 (/home/jenkins/workspace/ceph-master/src/test/librados/librados.cc:12) >>> CMakeFiles/unittest_librados.dir/librados.cc.o:(Librados_CreateShutdown_Test::TestBody()) c++: error: linker command failed with exit code 1 (use -v to see invocation) ====