Skip site navigation (1)Skip section navigation (2) 
Date:      Tue, 29 Sep 2015 03:07:04 +0000
From:      bugzilla-noreply@freebsd.org
To:        freebsd-ports-bugs@FreeBSD.org
Subject:   [Bug 203432] apache-spark cannot find the snappy native library
Message-ID:  <bug-203432-13@https.bugs.freebsd.org/bugzilla/>
next in thread | raw e-mail | index | archive | help 
https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=203432

            Bug ID: 203432
           Summary: apache-spark cannot find the snappy native library
           Product: Ports & Packages
           Version: Latest
          Hardware: amd64
                OS: Any
            Status: New
          Severity: Affects Only Me
          Priority: ---
         Component: Individual Port(s)
          Assignee: freebsd-ports-bugs@FreeBSD.org
          Reporter: tntc.tig@gmail.com

Apache Spark is unable to find the native snappy library, even when the snappy
and snappyjava port or package is installed.


>>> input=sc.textFile('/some/folder/*.bz2')
15/09/28 22:58:25 INFO MemoryStore: ensureFreeSpace(163705) called with
curMem=0, maxMem=278302556
15/09/28 22:58:25 INFO MemoryStore: Block broadcast_0 stored as values in
memory (estimated size 159.9 KB, free 265.3 MB)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/local/share/spark/python/pyspark/context.py", line 370, in
textFile
    return RDD(self._jsc.textFile(name, minPartitions), self,
  File
"/usr/local/share/spark/python/lib/py4j-0.8.2.1-src.zip/py4j/java_gateway.py",
line 538, in __call__
  File
"/usr/local/share/spark/python/lib/py4j-0.8.2.1-src.zip/py4j/protocol.py", line
300, in get_return_value
py4j.protocol.Py4JJavaError: An error occurred while calling o9.textFile.
: org.xerial.snappy.SnappyError: [FAILED_TO_LOAD_NATIVE_LIBRARY] no native
library is found for os.name=FreeBSD and os.arch=x86_64
    at org.xerial.snappy.SnappyLoader.findNativeLibrary(SnappyLoader.java:299)
    at org.xerial.snappy.SnappyLoader.loadNativeLibrary(SnappyLoader.java:163)
    at org.xerial.snappy.SnappyLoader.load(SnappyLoader.java:145)
    at org.xerial.snappy.Snappy.<clinit>(Snappy.java:47)
    at org.xerial.snappy.SnappyOutputStream.<init>(SnappyOutputStream.java:90)
    at org.xerial.snappy.SnappyOutputStream.<init>(SnappyOutputStream.java:83)
    at
org.apache.spark.io.SnappyCompressionCodec.compressedOutputStream(CompressionCodec.scala:125)
    at
org.apache.spark.broadcast.TorrentBroadcast$$anonfun$4.apply(TorrentBroadcast.scala:199)
    at
org.apache.spark.broadcast.TorrentBroadcast$$anonfun$4.apply(TorrentBroadcast.scala:199)
    at scala.Option.map(Option.scala:145)
    at
org.apache.spark.broadcast.TorrentBroadcast$.blockifyObject(TorrentBroadcast.scala:199)
    at
org.apache.spark.broadcast.TorrentBroadcast.writeBlocks(TorrentBroadcast.scala:101)
    at
org.apache.spark.broadcast.TorrentBroadcast.<init>(TorrentBroadcast.scala:84)
    at
org.apache.spark.broadcast.TorrentBroadcastFactory.newBroadcast(TorrentBroadcastFactory.scala:34)
    at
org.apache.spark.broadcast.TorrentBroadcastFactory.newBroadcast(TorrentBroadcastFactory.scala:29)
    at
org.apache.spark.broadcast.BroadcastManager.newBroadcast(BroadcastManager.scala:62)
    at org.apache.spark.SparkContext.broadcast(SparkContext.scala:980)
    at org.apache.spark.SparkContext.hadoopFile(SparkContext.scala:717)
    at org.apache.spark.SparkContext.textFile(SparkContext.scala:557)
    at
org.apache.spark.api.java.JavaSparkContext.textFile(JavaSparkContext.scala:191)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
    at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:606)
    at py4j.reflection.MethodInvoker.invoke(MethodInvoker.java:231)
    at py4j.reflection.ReflectionEngine.invoke(ReflectionEngine.java:379)
    at py4j.Gateway.invoke(Gateway.java:259)
    at py4j.commands.AbstractCommand.invokeMethod(AbstractCommand.java:133)
    at py4j.commands.CallCommand.execute(CallCommand.java:79)
    at py4j.GatewayConnection.run(GatewayConnection.java:207)
    at java.lang.Thread.run(Thread.java:745)

Setting "spark.broadcast.default false" or "spark.io.compression.codec lzf" in
/usr/local/share/spark/conf/spark-defaults.conf appears to allow the
spark-shell to load successfully.

(workaround copied from https://issues.apache.org/jira/browse/SPARK-3532)

-- 
You are receiving this mail because:
You are the assignee for the bug.

Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?bug-203432-13>