Repository navigation
Closed
Labels
Description
I'm trying to publish my local natives inside a jar, is not my first time in doing that, but this is going to be difficult.
I've found some problems:
./sbtis no more a valid command. [Solution] Now mac users have to runbrew install sbtand then for examplesbt native/nativeCompile- Running
scripts/publish-local-native.shwill incur into./scripts/crosscompile-linux.sh: Permission denied[Solution]chmod +x ./scripts/crosscompile-linux.sh - Now running the file
publish-local-native.shit goes into
WARNING: The requested image's platform (linux/amd64) does not match the detected host platform (linux/arm64/v8) and no specific platform was requested
bash: ./sbt: No such file or directory
I found two solutions for the ./sbt no such file error. The first is to take back in the repo the old sbt file from old versions. The second one is to modify the crosscompile-linux.sh in order to install sbt and execute sbt instead of ./sbt
docker run -it --rm \
-v $PWD:/pdal-java \
-v $HOME/.ivy2:/root/.ivy2 \
-v $HOME/.sbt:/root/.sbt \
-v $HOME/.coursier/cache:/root/.cache/coursier \
daunnc/pdal-ubuntu:2.6.0 bash -c "\
apt-get update && \
apt-get install -y curl gnupg && \
echo 'deb https://repo.scala-sbt.org/scalasbt/debian all main' | tee /etc/apt/sources.list.d/sbt.list && \
echo 'deb https://repo.scala-sbt.org/scalasbt/debian /' | tee /etc/apt/sources.list.d/sbt_old.list && \
curl -sL 'https://keyserver.ubuntu.com/pks/lookup?op=get&search=0x2EE0EA64E40A89B84B2DF73499E82A75642AC823' | apt-key add - && \
apt-get update && \
apt-get install -y sbt && \
cd ./pdal-java; sbt native/compile"
Both solution goes into the next error
- When executing the file crosscompile-linux.sh it stop at this error:
curl: /lib/x86_64-linux-gnu/libcurl.so.4: no version information available (required by curl)
java.lang.UnsatisfiedLinkError: Error looking up function 'stat': java: undefined symbol: stat
at com.sun.jna.Function.<init>(Function.java:252)
at com.sun.jna.NativeLibrary.getFunction(NativeLibrary.java:620)
at com.sun.jna.NativeLibrary.getFunction(NativeLibrary.java:596)
at com.sun.jna.NativeLibrary.getFunction(NativeLibrary.java:582)
at com.sun.jna.Library$Handler.invoke(Library.java:248)
at com.sun.proxy.$Proxy2.stat(Unknown Source)
at sbt.internal.io.Linux64Milli$.$anonfun$getModifiedTimeNative$1(Milli.scala:215)
at sbt.internal.io.MilliPosixBase.checkedIO(Milli.scala:100)
at sbt.internal.io.Linux64Milli$.getModifiedTimeNative(Milli.scala:215)
at sbt.internal.io.Linux64Milli$.getModifiedTimeNative(Milli.scala:210)
at sbt.internal.io.MilliNative.getModifiedTime(Milli.scala:65)
at sbt.internal.io.Milli$.getModifiedTime(Milli.scala:387)
at sbt.io.IO$.$anonfun$getModifiedTimeOrZero$1(IO.scala:1436)
at sbt.internal.io.Retry$.apply$mJc$sp(Retry.scala:47)
at sbt.internal.io.Retry$.apply$mJc$sp(Retry.scala:29)
at sbt.internal.io.Retry$.apply$mJc$sp(Retry.scala:24)
at sbt.io.IO$.getModifiedTimeOrZero(IO.scala:1436)
at sbt.internal.classpath.ClassLoaderCache$Key$$anonfun$$lessinit$greater$3.apply(ClassLoaderCache.scala:59)
at sbt.internal.classpath.ClassLoaderCache$Key$$anonfun$$lessinit$greater$3.apply(ClassLoaderCache.scala:59)
at scala.collection.immutable.List.map(List.scala:293)
at sbt.internal.classpath.ClassLoaderCache$Key.<init>(ClassLoaderCache.scala:59)
at sbt.internal.classpath.ClassLoaderCache$Key.<init>(ClassLoaderCache.scala:60)
at sbt.internal.classpath.ClassLoaderCache.cachedCustomClassloader(ClassLoaderCache.scala:195)
at sbt.State$StateOpsImpl$.initializeClassLoaderCache$extension(State.scala:417)
at sbt.StandardMain$.initialState(Main.scala:286)
at sbt.xMain$.$anonfun$run$11(Main.scala:128)
at scala.util.DynamicVariable.withValue(DynamicVariable.scala:62)
at scala.Console$.withIn(Console.scala:230)
at sbt.internal.util.Terminal$.withIn(Terminal.scala:610)
at sbt.internal.util.Terminal$.$anonfun$withStreams$1(Terminal.scala:390)
at scala.util.DynamicVariable.withValue(DynamicVariable.scala:62)
at scala.Console$.withOut(Console.scala:167)
at sbt.internal.util.Terminal$.$anonfun$withOut$2(Terminal.scala:600)
at scala.util.DynamicVariable.withValue(DynamicVariable.scala:62)
at scala.Console$.withErr(Console.scala:196)
at sbt.internal.util.Terminal$.withOut(Terminal.scala:600)
at sbt.internal.util.Terminal$.withStreams(Terminal.scala:390)
at sbt.xMain$.withStreams$1(Main.scala:88)
at sbt.xMain$.run(Main.scala:123)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at sbt.internal.XMainConfiguration.run(XMainConfiguration.java:59)
at sbt.xMain.run(Main.scala:47)
at xsbt.boot.Launch$.$anonfun$run$1(Launch.scala:149)
at xsbt.boot.Launch$.withContextLoader(Launch.scala:176)
at xsbt.boot.Launch$.run(Launch.scala:149)
at xsbt.boot.Launch$.$anonfun$apply$1(Launch.scala:44)
at xsbt.boot.Launch$.launch(Launch.scala:159)
at xsbt.boot.Launch$.apply(Launch.scala:44)
at xsbt.boot.Launch$.apply(Launch.scala:21)
at xsbt.boot.Boot$.runImpl(Boot.scala:78)
at xsbt.boot.Boot$.run(Boot.scala:73)
at xsbt.boot.Boot$.main(Boot.scala:21)
at xsbt.boot.Boot.main(Boot.scala)
[error] [launcher] error during sbt launcher: java.lang.UnsatisfiedLinkError: Error looking up function 'stat': java: undefined symbol: stat
```
`
I'm currently investigating for a solution. Any advice is appreciated
Activity
pomadchin commentedon Jan 19, 2024
Hi @WhiteSte I’ll update these scripts, thanks for noticing it!
Usually there is no need in cross compilation for local needs, and if that’s the case just run
sbt publishLocalcommand (SBT should be installed on the host machine)pomadchin commentedon Jan 19, 2024
I used to use these scripts and publish PDAL from the laptop, but since releases moved to CI it got simplified in some sense.
WhiteSte commentedon Jan 19, 2024
is there something like a
sbt publishLocalNative? I only need the jar containing the mac os arm nativeRunning sbt publishLocal goes into
pomadchin commentedon Jan 19, 2024
@WhiteSte what commands you're experiencing troubles with exactly?
The intended way to publishLocal it is via
+is not a typo, it takes care of the cross Scala versions build.My output:
pomadchin commentedon Jan 19, 2024
If you want to publish locally native binaries only:
sbt native/publishLocalAlso more help around SBT can be found in their docs, and questions about the sbt-jni plugin feel free to ask here / or under the https://github.com/sbt/sbt-jni repo (I'm maintainer there as well)
WhiteSte commentedon Jan 19, 2024
I could make it work using
sbt native/publishLocal, but now the java virtual machine is going into a fatal error whenever it tries to access pdal ..What i did is to switch inside /.m2/io/pdal/pdal-native/2.6.0 the pdal-native.jar published by you with the one i have just built.. Actually i did the same the last year to make it work the version 2.4.3 (also there arm was not supported)
Going from v2.4.3 to v2.6.2 is going to be a pain for me
Thats the fatal error
WhiteSte commentedon Jan 19, 2024
ps: I'm doing everything in java 17 . Shouldn't be a problem i hope
Furthermore, my installed version is pdal v2.6.2, the only one available in nix after 2.6.0
btw, my log when building the native is quite different and contains few errors, even if it finishes
pomadchin commentedon Jan 19, 2024
@WhiteSte what is the
sbt core/testresult?WhiteSte commentedon Jan 19, 2024
the sbt core.test result is the same fatal error
pomadchin commentedon Jan 19, 2024
Gotcha, I had the same behavior with the "incorrectly" installed PDAL.
What works for me now is PDAL installed via conda, and build (+ test) with the active conda env.
What's your PDAL installation?
WhiteSte commentedon Jan 19, 2024
my pdal installation is done through nix env, I've installed pdal 2.6.2.
I can try with conda, will let you know
WhiteSte commentedon Jan 19, 2024
I've created a separate env in coda and installed pdal in there with
conda install -c conda-forge pdalNow i've built the natives and executed the
sbt core/testthe result of the test is success
Then i took the created jar and substituted into the maven directories, i ran my java app and that's the result, where at least it is saying something about C
WhiteSte commentedon Jan 19, 2024
Big Update:
if i remove
filters.relaxationdartthrowingfrom the pipeline then it works.....Of course i'm using 5 filters and the bugged one is from my list
hobu commentedon Jan 19, 2024
There was a bug with
filters.relaxationdartthrowingPDAL/PDAL#4184 but I don't know if it is relevant here or you've found a new bug with it.7 remaining items
pomadchin commentedon Jan 19, 2024
Ok, I was able to reproduce; crashes only with log level >= 3! I'll see what's going on there, could be a non existent log level anymore.
Try setting it to 0, 1, 2 https://github.com/PDAL/java/blob/main/core/src/main/scala/io/pdal/LogLevel.scala#L28-L30
WhiteSte commentedon Jan 19, 2024
That works! How did you found out that?
pomadchin commentedon Jan 19, 2024
Just played around with it for a bit;
It looks like it's a PDAL
filters.relaxationdartthrowingissue - crashes on a log level >= 3; worth documenting and filing a PDAL issue!The best is to have a reproducible example in C++ without any jni / py trickery around + a ticket in the PDAL repo.
WhiteSte commentedon Jan 20, 2024
how did we assumed that is a c++ pdal issue?
If you do
pdal pipeline pipeline.json --verbose 3it worksI suppose is a jni related problem
pomadchin commentedon Jan 20, 2024
I did not assume it's a C++ PDAL issue, I assumed that if it's reproducible in bare PDAL than it's not a bindings issue.
Could be totally a bindings issue (:
pomadchin commentedon Jan 20, 2024
But TBH I have hard times linking the log level setting affecting filter execution may be a bindings issue. Unless some pointer get lost somehow 🤔
pomadchin commentedon Jan 20, 2024
Will make a PR shortly, that's indeed a bindings issue. Logger output stream is not properly initialized (?) and it's basically an NPE (??) 100% of times. It refuses to write into
std::stringstreamwhich is peculiar.The only reason it never crashed before is that we were lucky enough to not have any logging statements for the specified levels (?)
It works with stdout / stderr / files.
WhiteSte commentedon Jan 20, 2024
That's strange because in v2.4.3 i was using log level 3, but i got the crash just after updating to v2.6.2
I'm trying to take a look but sincerly i'm not really sure where to put my hands on
pomadchin commentedon Jan 20, 2024
So the way it should work is the following:
The idea is to have logs written into that stream and later on convert it to a string to access.
I tried replacing this constructor with the other one and it worked.
Dropped investigating yesterday in the mids of the code cleanup and figuring out the nature of a crash.
pomadchin commentedon Jan 20, 2024
Also since 2.4.x there’s been a major refactor of the classes to be compatible with the modern PDAL versions, prior to the change logs collection was done via executor.
WhiteSte commentedon Jan 20, 2024
good you just solved it so. I've added a little guide in #81 that resume this issue
pomadchin commentedon Jan 20, 2024
Well it’s not solved yet, any help with debugging is really appreciated! Need still to figure out why it’s unhappy about the stringstream.
WhiteSte commentedon Jan 20, 2024
never coded in c unfortunately. I'm trying to see but it's hard
Idk if it can help but running it inside a docker went though this
in particular
C [libpdaljni.2.6.so+0xe7f5] libpdaljava::PipelineExecutor::setLogStream(std::ostream&)+0xa5