From owner-freebsd-ports-bugs@freebsd.org Tue Sep 29 03:07:05 2015 Return-Path: Delivered-To: freebsd-ports-bugs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 09D63A0BF2B for ; Tue, 29 Sep 2015 03:07:05 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2001:1900:2254:206a::16:76]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id E9B561AFA for ; Tue, 29 Sep 2015 03:07:04 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from bugs.freebsd.org ([127.0.1.118]) by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id t8T374w2086486 for ; Tue, 29 Sep 2015 03:07:04 GMT (envelope-from bugzilla-noreply@freebsd.org) From: bugzilla-noreply@freebsd.org To: freebsd-ports-bugs@FreeBSD.org Subject: [Bug 203432] apache-spark cannot find the snappy native library Date: Tue, 29 Sep 2015 03:07:04 +0000 X-Bugzilla-Reason: AssignedTo X-Bugzilla-Type: new X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: Ports & Packages X-Bugzilla-Component: Individual Port(s) X-Bugzilla-Version: Latest X-Bugzilla-Keywords: X-Bugzilla-Severity: Affects Only Me X-Bugzilla-Who: tntc.tig@gmail.com X-Bugzilla-Status: New X-Bugzilla-Priority: --- X-Bugzilla-Assigned-To: freebsd-ports-bugs@FreeBSD.org X-Bugzilla-Target-Milestone: --- X-Bugzilla-Flags: X-Bugzilla-Changed-Fields: bug_id short_desc product version rep_platform op_sys bug_status bug_severity priority component assigned_to reporter Message-ID: Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: 7bit X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated MIME-Version: 1.0 X-BeenThere: freebsd-ports-bugs@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Ports bug reports List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 29 Sep 2015 03:07:05 -0000 https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=203432 Bug ID: 203432 Summary: apache-spark cannot find the snappy native library Product: Ports & Packages Version: Latest Hardware: amd64 OS: Any Status: New Severity: Affects Only Me Priority: --- Component: Individual Port(s) Assignee: freebsd-ports-bugs@FreeBSD.org Reporter: tntc.tig@gmail.com Apache Spark is unable to find the native snappy library, even when the snappy and snappyjava port or package is installed. >>> input=sc.textFile('/some/folder/*.bz2') 15/09/28 22:58:25 INFO MemoryStore: ensureFreeSpace(163705) called with curMem=0, maxMem=278302556 15/09/28 22:58:25 INFO MemoryStore: Block broadcast_0 stored as values in memory (estimated size 159.9 KB, free 265.3 MB) Traceback (most recent call last): File "", line 1, in File "/usr/local/share/spark/python/pyspark/context.py", line 370, in textFile return RDD(self._jsc.textFile(name, minPartitions), self, File "/usr/local/share/spark/python/lib/py4j-0.8.2.1-src.zip/py4j/java_gateway.py", line 538, in __call__ File "/usr/local/share/spark/python/lib/py4j-0.8.2.1-src.zip/py4j/protocol.py", line 300, in get_return_value py4j.protocol.Py4JJavaError: An error occurred while calling o9.textFile. : org.xerial.snappy.SnappyError: [FAILED_TO_LOAD_NATIVE_LIBRARY] no native library is found for os.name=FreeBSD and os.arch=x86_64 at org.xerial.snappy.SnappyLoader.findNativeLibrary(SnappyLoader.java:299) at org.xerial.snappy.SnappyLoader.loadNativeLibrary(SnappyLoader.java:163) at org.xerial.snappy.SnappyLoader.load(SnappyLoader.java:145) at org.xerial.snappy.Snappy.(Snappy.java:47) at org.xerial.snappy.SnappyOutputStream.(SnappyOutputStream.java:90) at org.xerial.snappy.SnappyOutputStream.(SnappyOutputStream.java:83) at org.apache.spark.io.SnappyCompressionCodec.compressedOutputStream(CompressionCodec.scala:125) at org.apache.spark.broadcast.TorrentBroadcast$$anonfun$4.apply(TorrentBroadcast.scala:199) at org.apache.spark.broadcast.TorrentBroadcast$$anonfun$4.apply(TorrentBroadcast.scala:199) at scala.Option.map(Option.scala:145) at org.apache.spark.broadcast.TorrentBroadcast$.blockifyObject(TorrentBroadcast.scala:199) at org.apache.spark.broadcast.TorrentBroadcast.writeBlocks(TorrentBroadcast.scala:101) at org.apache.spark.broadcast.TorrentBroadcast.(TorrentBroadcast.scala:84) at org.apache.spark.broadcast.TorrentBroadcastFactory.newBroadcast(TorrentBroadcastFactory.scala:34) at org.apache.spark.broadcast.TorrentBroadcastFactory.newBroadcast(TorrentBroadcastFactory.scala:29) at org.apache.spark.broadcast.BroadcastManager.newBroadcast(BroadcastManager.scala:62) at org.apache.spark.SparkContext.broadcast(SparkContext.scala:980) at org.apache.spark.SparkContext.hadoopFile(SparkContext.scala:717) at org.apache.spark.SparkContext.textFile(SparkContext.scala:557) at org.apache.spark.api.java.JavaSparkContext.textFile(JavaSparkContext.scala:191) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:606) at py4j.reflection.MethodInvoker.invoke(MethodInvoker.java:231) at py4j.reflection.ReflectionEngine.invoke(ReflectionEngine.java:379) at py4j.Gateway.invoke(Gateway.java:259) at py4j.commands.AbstractCommand.invokeMethod(AbstractCommand.java:133) at py4j.commands.CallCommand.execute(CallCommand.java:79) at py4j.GatewayConnection.run(GatewayConnection.java:207) at java.lang.Thread.run(Thread.java:745) Setting "spark.broadcast.default false" or "spark.io.compression.codec lzf" in /usr/local/share/spark/conf/spark-defaults.conf appears to allow the spark-shell to load successfully. (workaround copied from https://issues.apache.org/jira/browse/SPARK-3532) -- You are receiving this mail because: You are the assignee for the bug.