NoClassDefFoundError when trying to use CLBlast

[blueberry]

I include JOCL-0.2.0-RC1-SNAPSHOT and JOCLBlast-0.0.1-SNAPSHOT in my Cloajure leiningen project (basically the same as maven, an uses maven repositories and poms etc.)

I start the REPL and:

(import 'org.jocl.CL)
;;org.jocl.CL
(CL/setExceptionsEnabled false)
;;nil

So, JOCL loads and works as always.

But, then:

(import 'org.jocl.blast.CLBlast)
;;org.jocl.blast.CLBlast
(CLBlast/setExceptionsEnabled false)
;;NoClassDefFoundError Could not initialize class org.jocl.blast.CLBlast  uncomplicate.neanderthal.examples.guides.tutorial-opencl-test/eval51311 (form-init3160172747902287401.clj:416)

CLBlast itself is available on the system and can be loaded from Java:

(System/loadLibrary "clblast")
;;nil

As JOCL works well, I suspect there is some subtle error with JOCLBlast's library loading in exotic class loader setups (such as in Clojure). This is the complete stack trace:

                      REPL:  432  uncomplicate.neanderthal.examples.guides.tutorial-opencl-test/eval51325
                      REPL:  432  uncomplicate.neanderthal.examples.guides.tutorial-opencl-test/eval51325
             Compiler.java: 6927  clojure.lang.Compiler/eval
             Compiler.java: 6890  clojure.lang.Compiler/eval
                  core.clj: 3105  clojure.core/eval
                  core.clj: 3101  clojure.core/eval
                  main.clj:  240  clojure.main/repl/read-eval-print/fn
                  main.clj:  240  clojure.main/repl/read-eval-print
                  main.clj:  258  clojure.main/repl/fn
                  main.clj:  258  clojure.main/repl
    interruptible_eval.clj:  100  clojure.tools.nrepl.middleware.interruptible-eval/evaluate/fn
                  AFn.java:  152  clojure.lang.AFn/applyToHelper
                  AFn.java:  144  clojure.lang.AFn/applyTo
                  core.clj:  646  clojure.core/apply
                  core.clj: 1881  clojure.core/with-bindings*
                  core.clj: 1881  clojure.core/with-bindings*
               RestFn.java:  425  clojure.lang.RestFn/invoke
    interruptible_eval.clj:   85  clojure.tools.nrepl.middleware.interruptible-eval/evaluate
    interruptible_eval.clj:  219  clojure.tools.nrepl.middleware.interruptible-eval/interruptible-eval/fn/fn
    interruptible_eval.clj:  190  clojure.tools.nrepl.middleware.interruptible-eval/run-next/fn
                  AFn.java:   22  clojure.lang.AFn/run
   ThreadPoolExecutor.java: 1142  java.util.concurrent.ThreadPoolExecutor/runWorker
   ThreadPoolExecutor.java:  617  java.util.concurrent.ThreadPoolExecutor$Worker/run
               Thread.java:  745  java.lang.Thread/run

The tooling (I use emacs and emacs-cider for Clojure) recognizes CLBlast class and offers code completion, so the class itself is obviously available in the project, but the problem is in the library loading phase.

[gpu]

How much of potential error messages is "hidden" there? I only see

NoClassDefFoundError Could not initialize class org.jocl.blast.CLBlast

as the message, but it does not seem to say which class is not found (and, if it is CLBlast itself, why it could not initialize it).

So first things first: If the initialization throws the ubiquitous UnsatisfiedLinkError, then it would appear there, right?

[blueberry]

The complete stack trace is in my initial message. I can try with some debug printlns in JOCLBlast...

[gpu]

Sorry, I can't figure out what

 uncomplicate.neanderthal.examples.guides.tutorial-opencl-test/eval51325

exactly is (some lambda, obviously...), and what causes the actual error. To understand this better: When you (temporarily) provoke an error, by (temporarily) moving clblast.so to a directory where it can not be found, does this cause the same error?

[blueberry]

I doctored CLBlast class with printlns like this:

static
    {
        System.out.println("============= 1:");
        String libraryBaseName = "JOCLBlast_0_0_1";
        System.out.println("============= 2:");
        String libraryName =
            LibUtils.createPlatformLibraryName(libraryBaseName);
        System.out.println("============= 3:");
        String dependentLibraryNames[] = { "clblast" };
        System.out.println("============= 4:");
        try
        {
            LibUtils.loadLibrary(libraryName, dependentLibraryNames);
        }
        catch (UnsatisfiedLinkError e)
        {
            System.out.println("============= 5:");
            e.printStackTrace();
            throw e;
        }
    }

But nothing gets printed before the exception. It seems to me that java tries to load some class that is needed for CLBlast to even get to CLBlast initialization, but can not figure out which class that would be. As I've mentioned, org.jocl.CL works fine in the same setting, so I would look at potential problems in the differences between these two classes.

[blueberry]

That eval is this: (CLBlast/setExceptionsEnabled false)
It is evaluated dynamically in the repl, that's why it gives the line you mentioned.

When I remove libclblast.so, the error is the same. I doubt that the problem is related to libclblast, because (System/loadLibrary "clblast") works fine.

[gpu]

(This aimed at finding out whether there might be a problem with loading the JOCLBlast library because it did not "match" the CLBlast library in some way - but I assume that you compiled both from the current code state, so if there would have been differences, they would have caused linker errors and such).

Regarding the possible reasons for the NoClassDefFoundError ... this is unusual, to say the least. If at all, then this error may be caused by some outdated JAR reference or so. There are only few other classes in the clblast package, and I assume that they can be loaded.

Sorry, the only thing that I could suggest now is some tedious "binary search": Starting with an empty CLBlast class, only containing the setExceptionsEnabled method, which sets a flag or prints some debug output, and see whether it can be loaded.Then adding the static initializer block. Then adding a single method and its native counterpart. (Each step will require more imports - maybe the culprit can be identified then...)

[blueberry]

@gpu Believe or not, I identified the troublemaker:
I changed the CLBlast static initializer to this:

static
    {
        String libraryBaseName = "JOCLBlast_0_0_1";
        String libraryName =
            LibUtils.createPlatformLibraryName(libraryBaseName);
        //String dependentLibraryNames[] = { "clblast" };
        try
        {
            LibUtils.loadLibrary(libraryName);//, dependentLibraryNames);
        }
        catch (UnsatisfiedLinkError e)
        {
            throw e;
        }
    }

And now JOCLBlast loads, and does call appropriate methods.

However, when I call CLBlastSscal, i get this OpenCL error through OpenCL's error reporting mechanism:

(with-default
  (with-default-engine
    (with-release [x (sv-cl (range 3))]
      (CLBlast/CLBlastSscal 3 6.0 (uncomplicate.clojurecl.core/cl-mem (.buffer x)) 0 1 *command-queue* nil)
      (asum x)))))
;;CLException INVALID error code: -2048  org.jocl.blast.CLBlast.checkResult (CLBlast.java:96)

It seems that JOCLBlast (or CLBlast) does not like the (valid) command queue that i gave it as an argument, but I will investigate that tomorrow.

So, it seems that explicitly loading clblast causes problems. Why are you calling that explicitly? Shouldn't it be handled by the OS, since I suppose that you dynamically link clblast when you build joclblast through make? In neanderthal-atlas, the lib loader only loads libneanderthal, and atlas is provided automatically I think...

I hope that what I discovered helps you in identifying what could be wrong. If you have any other ideas that I should try, please tell.

[blueberry]

This error code is from CLBlast:

kKernelLaunchError         = -2048, // Problem occurred when enqueuing the kernel

[blueberry]

I am almost sure about the problem cause: I have 3 GPUs (the 1st supports OpenCL 1.2, while the 2nd and the 3rd support OpenCL 2.0). I use the 2nd to create the context and command queue, but CLBlast might use the 1st (OS's default) to build the program. That's why the command queue that I supply is invalid.

Do you think this could be the issue?
How to set the device and context that CLBlast uses from JOCLBlast?

[gpu]

The error code is something that I'd first investigate in native CLBlast (maybe I can try to see whether there is a test case for (the C-version of) CLBlastSscal tomorrow or early next week.

---

Now, that the problem is caused by manually loading the CLBlast library is once again irritating. Although that's what I was aiming at when I said to remove the CLBlast lib temporarily: Shouldn't it throw an UnsatisfiedLinkError when there is a problem with loading the library? I wonder how this can cause a NoClassDefFoundError...

The reason of why I'm loading the library manually is that the goal was to have one deployable JAR, with the native JOCLBlast library and the matching native CLBlas library. One can not say to Java developer: "Here are the Maven coordinates of JOCLBlast .... but you have to build the CLBLast library on your own and put it into the LD_LIBRARY_PATH". It should be possible to just use the JAR. And in order to load the native JOCLBlast library, the native CLBlast library has to be loaded first.

Of course, I tried this locally, on Windows, but obviously, this was once again not sufficient. Did you have the chance to test JOCLBlast on Linux with plain Java? (Maybe using the sample from this forum post: https://forum.byte-welt.net/byte-welt-projekte-projects/jocl/18180-joclblas-java-bindings-clblas-2.html#post131422 ).

(I just wonder whether this is also related to the Clojure classloading black-box, or wehther it's a general issue...)

[gpu]

Sorry, these overlapped:

Do you think this could be the issue?
How to set the device and context that CLBlast uses from JOCLBlast?

This can certainly be a reason for a command queue being invalid. I'll have to take a closer look at the inner workings of CLBlast to see how this could be solved, I'll try to do this tomorrow (it's already late here)

[blueberry]

We are in the same GMT+1 time zone I suppose :)

Regarding the libclblas availability: If you would like to include CLBlast in the jar, would'n it be easier to compile JOCLBlast native library statically?
I thought that you chose dynamic loading to enable the user to tune CLBlast (because the tuning database it uses only contains a dozen cards, otherwise it uses non-optimal defaults).

[blueberry]

CLBlast reference: CNugteren/CLBlast#43

[blueberry]

After restarting the JVM, CLBlast function calls work well with all queues. The problem was due to some initial fail in interaction with CLBlast's internal resource caching mechanism.

The problem with System.loadLibrary stays, that is, JOCLBLast works on my system with explicit loading disabled.

[gpu]

Statically linking might be an option for this particular setup - or so to say, for each particular setup. I actually wanted to cover the generic case, of one JNI library depending on (possibly multiple, possibly precompiled) other dynamic libraries.

I still wonder whether this is a general problem with the dependency loading mechanism (which was introduced only recently), or whether the "root" cause is the same that caused the problems with Sizeof in gpu/JOCL#5 .

I'll probably have to take a closer look at Clojure and its native library handling. A quick websearch brings some results (one (of many!) on stackoverflow). But also one that might be strongly related to our case: https://groups.google.com/forum/#!topic/clojure-dev/awe7-yeieIM (and the lower part of the thread that is linked from there: https://groups.google.com/forum/#!topic/clojure/br_sTSuWBJ8/discussion ). I'll try to read through this and related pages during the weekend.