From owner-svn-ports-all@freebsd.org Tue Jan 21 07:31:47 2020 Return-Path: Delivered-To: svn-ports-all@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id A328123411E; Tue, 21 Jan 2020 07:31:47 +0000 (UTC) (envelope-from kbowling@FreeBSD.org) Received: from mxrelay.nyi.freebsd.org (mxrelay.nyi.freebsd.org [IPv6:2610:1c1:1:606c::19:3]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) server-signature RSA-PSS (4096 bits) client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "mxrelay.nyi.freebsd.org", Issuer "Let's Encrypt Authority X3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4820bR3y2qz4HDS; Tue, 21 Jan 2020 07:31:47 +0000 (UTC) (envelope-from kbowling@FreeBSD.org) Received: from repo.freebsd.org (repo.freebsd.org [IPv6:2610:1c1:1:6068::e6a:0]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mxrelay.nyi.freebsd.org (Postfix) with ESMTPS id 82F74920E; Tue, 21 Jan 2020 07:31:47 +0000 (UTC) (envelope-from kbowling@FreeBSD.org) Received: from repo.freebsd.org ([127.0.1.37]) by repo.freebsd.org (8.15.2/8.15.2) with ESMTP id 00L7Vl1c005669; Tue, 21 Jan 2020 07:31:47 GMT (envelope-from kbowling@FreeBSD.org) Received: (from kbowling@localhost) by repo.freebsd.org (8.15.2/8.15.2/Submit) id 00L7VlNp005667; Tue, 21 Jan 2020 07:31:47 GMT (envelope-from kbowling@FreeBSD.org) Message-Id: <202001210731.00L7VlNp005667@repo.freebsd.org> X-Authentication-Warning: repo.freebsd.org: kbowling set sender to kbowling@FreeBSD.org using -f From: Kevin Bowling Date: Tue, 21 Jan 2020 07:31:47 +0000 (UTC) To: ports-committers@freebsd.org, svn-ports-all@freebsd.org, svn-ports-head@freebsd.org Subject: svn commit: r523702 - in head/devel/hadoop2: . files X-SVN-Group: ports-head X-SVN-Commit-Author: kbowling X-SVN-Commit-Paths: in head/devel/hadoop2: . files X-SVN-Commit-Revision: 523702 X-SVN-Commit-Repository: ports MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit X-BeenThere: svn-ports-all@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: SVN commit messages for the ports tree List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 21 Jan 2020 07:31:47 -0000 Author: kbowling Date: Tue Jan 21 07:31:46 2020 New Revision: 523702 URL: https://svnweb.freebsd.org/changeset/ports/523702 Log: devel/hadoop2: patch to build with OpenSSL 1.1 PR: 228946 Submitted by: Walter Schwarzenfeld Reported by: brnrd Approved by: krion (mentor) Obtained from: upstream Sponsored by: BBOX.io Differential Revision: https://reviews.freebsd.org/D23273 Added: head/devel/hadoop2/files/patch-openssl111 (contents, props changed) Modified: head/devel/hadoop2/Makefile Modified: head/devel/hadoop2/Makefile ============================================================================== --- head/devel/hadoop2/Makefile Tue Jan 21 06:38:12 2020 (r523701) +++ head/devel/hadoop2/Makefile Tue Jan 21 07:31:46 2020 (r523702) @@ -7,7 +7,7 @@ # no problems encountered so far. Thanks. PORTNAME= hadoop PORTVERSION= 2.7.2 -PORTREVISION= 2 +PORTREVISION= 3 CATEGORIES= devel java MASTER_SITES= APACHE/${PORTNAME}/common/hadoop-${PORTVERSION} \ LOCAL/demon/:maven \ @@ -24,9 +24,6 @@ COMMENT= Apache Map/Reduce framework LICENSE= APACHE20 -BROKEN_SSL= openssl -BROKEN_SSL_REASON_openssl= incomplete definition of type 'struct evp_cipher_ctx_st' - BUILD_DEPENDS= ${LOCALBASE}/share/java/maven3/bin/mvn:devel/maven3 \ cmake:devel/cmake \ ${LOCALBASE}/protobuf25/bin/protoc:devel/protobuf25 @@ -74,11 +71,6 @@ SUB_LIST= HDFS_USER="${HDFS_USER}" \ HADOOP_RUNDIR="${HADOOP_RUNDIR}" .include - -.if ${SSL_DEFAULT} == base -BROKEN_FreeBSD_12= incomplete definition of type 'struct evp_cipher_ctx_st' -BROKEN_FreeBSD_13= incomplete definition of type 'struct evp_cipher_ctx_st' -.endif post-patch: ${REINPLACE_CMD} -e "s#/bin/bash#${LOCALBASE}/bin/bash#" ${WRKSRC}/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/DefaultContainerExecutor.java ${WRKSRC}/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/MRJobConfig.java ${WRKSRC}/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/launcher/ContainerLaunch.java ${WRKSRC}/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/util/Shell.java ${WRKSRC}/hadoop-common-project/hadoop-common/src/main/bin/hadoop-daemon.sh Added: head/devel/hadoop2/files/patch-openssl111 ============================================================================== --- /dev/null 00:00:00 1970 (empty, because file is newly added) +++ head/devel/hadoop2/files/patch-openssl111 Tue Jan 21 07:31:46 2020 (r523702) @@ -0,0 +1,134 @@ +--- hadoop-common-project/hadoop-common/src/main/native/src/org/apache/hadoop/crypto/OpensslCipher.c ++++ hadoop-common-project/hadoop-common/src/main/native/src/org/apache/hadoop/crypto/OpensslCipher.c +@@ -30,6 +30,11 @@ static void (*dlsym_EVP_CIPHER_CTX_free)(EVP_CIPHER_CTX *); + static int (*dlsym_EVP_CIPHER_CTX_cleanup)(EVP_CIPHER_CTX *); + static void (*dlsym_EVP_CIPHER_CTX_init)(EVP_CIPHER_CTX *); + static int (*dlsym_EVP_CIPHER_CTX_set_padding)(EVP_CIPHER_CTX *, int); ++static int (*dlsym_EVP_CIPHER_CTX_test_flags)(const EVP_CIPHER_CTX *, int); ++static int (*dlsym_EVP_CIPHER_CTX_block_size)(const EVP_CIPHER_CTX *); ++#if OPENSSL_VERSION_NUMBER >= 0x10100000L ++static int (*dlsym_EVP_CIPHER_CTX_encrypting)(const EVP_CIPHER_CTX *); ++#endif + static int (*dlsym_EVP_CipherInit_ex)(EVP_CIPHER_CTX *, const EVP_CIPHER *, \ + ENGINE *, const unsigned char *, const unsigned char *, int); + static int (*dlsym_EVP_CipherUpdate)(EVP_CIPHER_CTX *, unsigned char *, \ +@@ -46,6 +51,11 @@ typedef void (__cdecl *__dlsym_EVP_CIPHER_CTX_free)(EVP_CIPHER_CTX *); + typedef int (__cdecl *__dlsym_EVP_CIPHER_CTX_cleanup)(EVP_CIPHER_CTX *); + typedef void (__cdecl *__dlsym_EVP_CIPHER_CTX_init)(EVP_CIPHER_CTX *); + typedef int (__cdecl *__dlsym_EVP_CIPHER_CTX_set_padding)(EVP_CIPHER_CTX *, int); ++typedef int (__cdecl *__dlsym_EVP_CIPHER_CTX_test_flags)(const EVP_CIPHER_CTX *, int); ++typedef int (__cdecl *__dlsym_EVP_CIPHER_CTX_block_size)(const EVP_CIPHER_CTX *); ++#if OPENSSL_VERSION_NUMBER >= 0x10100000L ++typedef int (__cdecl *__dlsym_EVP_CIPHER_CTX_encrypting)(const EVP_CIPHER_CTX *); ++#endif + typedef int (__cdecl *__dlsym_EVP_CipherInit_ex)(EVP_CIPHER_CTX *, \ + const EVP_CIPHER *, ENGINE *, const unsigned char *, \ + const unsigned char *, int); +@@ -60,6 +70,11 @@ static __dlsym_EVP_CIPHER_CTX_free dlsym_EVP_CIPHER_CTX_free; + static __dlsym_EVP_CIPHER_CTX_cleanup dlsym_EVP_CIPHER_CTX_cleanup; + static __dlsym_EVP_CIPHER_CTX_init dlsym_EVP_CIPHER_CTX_init; + static __dlsym_EVP_CIPHER_CTX_set_padding dlsym_EVP_CIPHER_CTX_set_padding; ++static __dlsym_EVP_CIPHER_CTX_test_flags dlsym_EVP_CIPHER_CTX_test_flags; ++static __dlsym_EVP_CIPHER_CTX_block_size dlsym_EVP_CIPHER_CTX_block_size; ++#if OPENSSL_VERSION_NUMBER >= 0x10100000L ++static __dlsym_EVP_CIPHER_CTX_encrypting dlsym_EVP_CIPHER_CTX_encrypting; ++#endif + static __dlsym_EVP_CipherInit_ex dlsym_EVP_CipherInit_ex; + static __dlsym_EVP_CipherUpdate dlsym_EVP_CipherUpdate; + static __dlsym_EVP_CipherFinal_ex dlsym_EVP_CipherFinal_ex; +@@ -114,6 +129,14 @@ JNIEXPORT void JNICALL Java_org_apache_hadoop_crypto_OpensslCipher_initIDs + "EVP_CIPHER_CTX_init"); + LOAD_DYNAMIC_SYMBOL(dlsym_EVP_CIPHER_CTX_set_padding, env, openssl, \ + "EVP_CIPHER_CTX_set_padding"); ++ LOAD_DYNAMIC_SYMBOL(dlsym_EVP_CIPHER_CTX_test_flags, env, openssl, \ ++ "EVP_CIPHER_CTX_test_flags"); ++ LOAD_DYNAMIC_SYMBOL(dlsym_EVP_CIPHER_CTX_block_size, env, openssl, \ ++ "EVP_CIPHER_CTX_block_size"); ++#if OPENSSL_VERSION_NUMBER >= 0x10100000L ++ LOAD_DYNAMIC_SYMBOL(dlsym_EVP_CIPHER_CTX_encrypting, env, openssl, \ ++ "EVP_CIPHER_CTX_encrypting"); ++#endif + LOAD_DYNAMIC_SYMBOL(dlsym_EVP_CipherInit_ex, env, openssl, \ + "EVP_CipherInit_ex"); + LOAD_DYNAMIC_SYMBOL(dlsym_EVP_CipherUpdate, env, openssl, \ +@@ -135,6 +158,17 @@ JNIEXPORT void JNICALL Java_org_apache_hadoop_crypto_OpensslCipher_initIDs + LOAD_DYNAMIC_SYMBOL(__dlsym_EVP_CIPHER_CTX_set_padding, \ + dlsym_EVP_CIPHER_CTX_set_padding, env, \ + openssl, "EVP_CIPHER_CTX_set_padding"); ++ LOAD_DYNAMIC_SYMBOL(__dlsym_EVP_CIPHER_CTX_test_flags, \ ++ dlsym_EVP_CIPHER_CTX_test_flags, env, \ ++ openssl, "EVP_CIPHER_CTX_test_flags"); ++ LOAD_DYNAMIC_SYMBOL(__dlsym_EVP_CIPHER_CTX_block_size, \ ++ dlsym_EVP_CIPHER_CTX_block_size, env, \ ++ openssl, "EVP_CIPHER_CTX_block_size"); ++#if OPENSSL_VERSION_NUMBER >= 0x10100000L ++ LOAD_DYNAMIC_SYMBOL(__dlsym_EVP_CIPHER_CTX_encrypting, \ ++ dlsym_EVP_CIPHER_CTX_encrypting, env, \ ++ openssl, "EVP_CIPHER_CTX_encrypting"); ++#endif + LOAD_DYNAMIC_SYMBOL(__dlsym_EVP_CipherInit_ex, dlsym_EVP_CipherInit_ex, \ + env, openssl, "EVP_CipherInit_ex"); + LOAD_DYNAMIC_SYMBOL(__dlsym_EVP_CipherUpdate, dlsym_EVP_CipherUpdate, \ +@@ -253,14 +287,18 @@ JNIEXPORT jlong JNICALL Java_org_apache_hadoop_crypto_OpensslCipher_init + static int check_update_max_output_len(EVP_CIPHER_CTX *context, int input_len, + int max_output_len) + { +- if (context->flags & EVP_CIPH_NO_PADDING) { ++ if ( dlsym_EVP_CIPHER_CTX_test_flags(context, EVP_CIPH_NO_PADDING) ) { + if (max_output_len >= input_len) { + return 1; + } + return 0; + } else { +- int b = context->cipher->block_size; ++ int b = dlsym_EVP_CIPHER_CTX_block_size(context); ++#if OPENSSL_VERSION_NUMBER < 0x10100000L + if (context->encrypt) { ++#else ++ if (dlsym_EVP_CIPHER_CTX_encrypting(context)) { ++#endif + if (max_output_len >= input_len + b - 1) { + return 1; + } +@@ -307,10 +345,10 @@ JNIEXPORT jint JNICALL Java_org_apache_hadoop_crypto_OpensslCipher_update + static int check_doFinal_max_output_len(EVP_CIPHER_CTX *context, + int max_output_len) + { +- if (context->flags & EVP_CIPH_NO_PADDING) { ++ if ( dlsym_EVP_CIPHER_CTX_test_flags(context, EVP_CIPH_NO_PADDING) ) { + return 1; + } else { +- int b = context->cipher->block_size; ++ int b = dlsym_EVP_CIPHER_CTX_block_size(context); + if (max_output_len >= b) { + return 1; + } +--- hadoop-tools/hadoop-pipes/src/main/native/pipes/impl/HadoopPipes.cc ++++ hadoop-tools/hadoop-pipes/src/main/native/pipes/impl/HadoopPipes.cc +@@ -420,6 +420,7 @@ namespace HadoopPipes { + } + + string createDigest(string &password, string& msg) { ++#if OPENSSL_VERSION_NUMBER < 0x10100000L + HMAC_CTX ctx; + unsigned char digest[EVP_MAX_MD_SIZE]; + HMAC_Init(&ctx, (const unsigned char *)password.c_str(), +@@ -428,7 +429,16 @@ namespace HadoopPipes { + unsigned int digestLen; + HMAC_Final(&ctx, digest, &digestLen); + HMAC_cleanup(&ctx); +- ++#else ++ HMAC_CTX *ctx = HMAC_CTX_new(); ++ unsigned char digest[EVP_MAX_MD_SIZE]; ++ HMAC_Init_ex(ctx, (const unsigned char *)password.c_str(), ++ password.length(), EVP_sha1(), NULL); ++ HMAC_Update(ctx, (const unsigned char *)msg.c_str(), msg.length()); ++ unsigned int digestLen; ++ HMAC_Final(ctx, digest, &digestLen); ++ HMAC_CTX_free(ctx); ++#endif + //now apply base64 encoding + BIO *bmem, *b64; + BUF_MEM *bptr; +