# Can I use Tf native core platform in JAVA for ScaNN based model?

**URL:** <https://discuss.ai.google.dev/t/can-i-use-tf-native-core-platform-in-java-for-scann-based-model/31740>\
**Category:** General Discussion\
**Tags:** recommenders, models, java\
**Created:** [January 6, 2023, 5:20am UTC](https://discuss.ai.google.dev/t/can-i-use-tf-native-core-platform-in-java-for-scann-based-model/31740 "2023-01-06T05:20:29Z")\
**Posts on this page:** 13\
**Page:** 1

<div class="post-metadata">

**Author:** ![saurabh\_tripathy](https://d1dlmcr85iqnpo.cloudfront.net/user_avatar/discuss.ai.google.dev/saurabh_tripathy/32/14738_2.png) [@saurabh\_tripathy](https://discuss.ai.google.dev/u/saurabh_tripathy)\
**Post date:** [January 6, 2023, 5:20am UTC](https://discuss.ai.google.dev/t/can-i-use-tf-native-core-platform-in-java-for-scann-based-model/31740/1 "2023-01-06T05:20:29Z")

</div>

Hi All , I trained a ScaNN powered model described [here](https://www.tensorflow.org/recommenders/examples/efficient_serving) in Python.

Tf version: **2.7.0**  
Environment: **Mac** and **Ubuntu**

I am trying to infer it in **JAVA** using native **tensorflow-core-platform version 0.4.0**

Java code:

```auto
        URL modelURL = Main.class.getClassLoader().getResource("model/1");
        String modelPath = Paths.get(modelURL.toURI()).toString();

        SavedModelBundle model = SavedModelBundle.load(modelPath, "serve");

```

I am getting error `Op type not registered 'Scann>ScannSearchBatched' in binary running on xx. Make sure the Op and Kernel are registered in the binary running in this process. Note that if you are loading a saved graph which used ops from tf.contrib, accessing (e.g.) `tf.contrib.resampler` should be done before importing the graph, as contrib ops are lazily registered when the module is first accessed.`

Sometimes I get TensorFlowException: Could not find SavedModel .pb or .pbtxt at supplied export directory path

Please let me know what’s going on?  
I know Mac is not supported by ScaNN but its not working on Ubuntu too.  
So **Can I use Tf native core platform in JAVA for Scann based model?**

---

<div class="post-metadata">

**Author:** ![karllessard](https://d1dlmcr85iqnpo.cloudfront.net/user_avatar/discuss.ai.google.dev/karllessard/32/9185_2.png) [@karllessard](https://discuss.ai.google.dev/u/karllessard)\
**Post date:** [January 8, 2023, 2:31am UTC](https://discuss.ai.google.dev/t/can-i-use-tf-native-core-platform-in-java-for-scann-based-model/31740/3 "2023-01-08T02:31:00Z")

</div>

I’ve never played with Scann myself but the error message says that this library is providing custom ops that first need to be registered in TensorFlow.

To do this in Java, you must first load explicitly the native library that are providing these custom ops by calling [this method](https://github.com/tensorflow/java/blob/455fc731ece7aba751e7ec7ee1cd06dff3a1bfe0/tensorflow-core/tensorflow-core-api/src/main/java/org/tensorflow/TensorFlow.java#L98). Normally the library can be extracted from the Python wheel, I’ve looked at its content and I guess it is probably the `scann/scann_ops/cc/_scann_ops.so` file.

So to summarize:

```auto
TensorFlow.loadLibrary("/path/to/python/wheel/scann/scann_ops/cc/_scann_ops.so");

try (SavedModelBundle model = SavedModelBundle.load(modelPath, "serve")) {
   ...
}

```

Please share with us if that worked for you

---

<div class="post-metadata">

**Author:** ![saurabh\_tripathy](https://d1dlmcr85iqnpo.cloudfront.net/user_avatar/discuss.ai.google.dev/saurabh_tripathy/32/14738_2.png) [@saurabh\_tripathy](https://discuss.ai.google.dev/u/saurabh_tripathy)\
**Post date:** [January 9, 2023, 5:16am UTC](https://discuss.ai.google.dev/t/can-i-use-tf-native-core-platform-in-java-for-scann-based-model/31740/4 "2023-01-09T05:16:16Z")

</div>

@karllessard : I am now getting

```auto
2023-01-09 05:12:35.533897: I external/org_tensorflow/tensorflow/cc/saved_model/loader.cc:301] SavedModel load for tags { serve }; Status: fail: NOT_FOUND: Could not find SavedModel .pb or .pbtxt at supplied export directory path: model/1. Took 1004 microseconds.
Exception in thread "main" org.tensorflow.exceptions.TensorFlowException: Could not find SavedModel .pb or .pbtxt at supplied export directory path: model/1

```

This is my Docker file:

```auto
# Start with a base image containing Java runtime
FROM adoptopenjdk/openjdk11

# Install required compiler tools
RUN apt-get update && apt-get install -y software-properties-common

# Install python 3.7 and pip
RUN apt-get update && apt-get install -y python3.8 python3-pip && apt-get update

# Install scann
RUN python3.8 -m pip install python-dev-tools
RUN python3.8 -m pip install tensorflow==2.7.0 && pip install scann==1.2.3
# Add a volume pointing to /tmp
VOLUME /tmp

# Make port 8080 available to the world outside this container
EXPOSE 8080

RUN mkdir /app
WORKDIR /app
COPY . /app
RUN ./gradlew clean build

COPY build/libs/core-tf-1.0-SNAPSHOT.jar /opt/core_tf.jar
CMD ["java","-jar","/opt/core_tf.jar"]

```

This is my main code:

````auto
        TensorFlow.loadLibrary(" /usr/local/lib/python3.8/dist-packages/scann/scann_ops/cc/_scann_ops.so");
        String filePath = "model/1";
        try (SavedModelBundle model = SavedModelBundle.load(filePath, "serve")) {
            // Set up input tensors
            TString inputArray = TString.vectorOf("42");
            Tensor inputTensor = TString.tensorOf(inputArray);
            System.out.println(inputTensor.shape());

            Map<String, Tensor> feed_dict = new HashMap<>();
            feed_dict.put("input_1", inputTensor);

// Run the model and get the output
            System.out.println(model.function("predict").call(feed_dict));
        }
        catch (Exception ex){
            ex.printStackTrace();
        }```
````

---

<div class="post-metadata">

**Author:** ![karllessard](https://d1dlmcr85iqnpo.cloudfront.net/user_avatar/discuss.ai.google.dev/karllessard/32/9185_2.png) [@karllessard](https://discuss.ai.google.dev/u/karllessard)\
**Post date:** [January 9, 2023, 3:00pm UTC](https://discuss.ai.google.dev/t/can-i-use-tf-native-core-platform-in-java-for-scann-based-model/31740/6 "2023-01-09T15:00:25Z")

</div>

That error seems unrelated to loading `scann`, it cannot find the model at `model/1` and I don’t see neither where you copy this file in you docker container. Make sure that container has access to your model and try to use absolute paths whenever possible. Or is the model located in the class/resource path of your JAR?

---

<div class="post-metadata">

**Author:** ![saurabh\_tripathy](https://d1dlmcr85iqnpo.cloudfront.net/user_avatar/discuss.ai.google.dev/saurabh_tripathy/32/14738_2.png) [@saurabh\_tripathy](https://discuss.ai.google.dev/u/saurabh_tripathy)\
**Post date:** [January 9, 2023, 3:19pm UTC](https://discuss.ai.google.dev/t/can-i-use-tf-native-core-platform-in-java-for-scann-based-model/31740/7 "2023-01-09T15:19:24Z")

</div>

@karllessard : Yes the model is located in the resource path. (src/main/resources/model/1)

and in jar I am copying that. If I open my jar, I can see the model in root model directory.

Build.gradle

```auto
jar {duplicatesStrategy(DuplicatesStrategy.EXCLUDE)
    manifest {
        attributes(
                'Main-Class': 'org.example.Main'
        )
    }
    from {
        configurations.runtimeClasspath.filter{ it.exists() }.collect { it.isDirectory() ? it : zipTree(it) }
    }
    from 'resources', {
        into 'resources'
    }
}

```

---

<div class="post-metadata">

**Author:** ![karllessard](https://d1dlmcr85iqnpo.cloudfront.net/user_avatar/discuss.ai.google.dev/karllessard/32/9185_2.png) [@karllessard](https://discuss.ai.google.dev/u/karllessard)\
**Post date:** [January 9, 2023, 3:30pm UTC](https://discuss.ai.google.dev/t/can-i-use-tf-native-core-platform-in-java-for-scann-based-model/31740/8 "2023-01-09T15:30:45Z")

</div>

Yes that won’t work like that because the `SavedModelBundle` expect the model path to be a directory in your file system. You need to convert the resource path to a file path. Something like that should do:

```auto
var modelPath = Main.class.getResource("/model/1").toURI().toString();
try (var model = SavedModelBundle.load(modelPath, "serve")) {
   ...
}

```

---

<div class="post-metadata">

**Author:** ![saurabh\_tripathy](https://d1dlmcr85iqnpo.cloudfront.net/user_avatar/discuss.ai.google.dev/saurabh_tripathy/32/14738_2.png) [@saurabh\_tripathy](https://discuss.ai.google.dev/u/saurabh_tripathy)\
**Post date:** [January 9, 2023, 4:33pm UTC](https://discuss.ai.google.dev/t/can-i-use-tf-native-core-platform-in-java-for-scann-based-model/31740/9 "2023-01-09T16:33:23Z")

</div>

@karllessard : Ok Thanks!! Exec’ing into and running the jar gives me now:

```auto
Exception in thread "main" java.lang.UnsatisfiedLinkError: /usr/local/lib/python3.8/dist-packages/scann/scann_ops/cc/_scann_ops.so: undefined symbol: _ZN10tensorflow11ResourceMgr8DoCreateERKSsNS_9TypeIndexES2_PNS_12ResourceBaseE
        at org.tensorflow.TensorFlow.loadLibrary(TensorFlow.java:102)
        at org.example.Main.main(Main.java:33)

```

---

<div class="post-metadata">

**Author:** ![karllessard](https://d1dlmcr85iqnpo.cloudfront.net/user_avatar/discuss.ai.google.dev/karllessard/32/9185_2.png) [@karllessard](https://discuss.ai.google.dev/u/karllessard)\
**Post date:** [January 9, 2023, 8:54pm UTC](https://discuss.ai.google.dev/t/can-i-use-tf-native-core-platform-in-java-for-scann-based-model/31740/10 "2023-01-09T20:54:55Z")

</div>

mmh, looks like the version of TensorFlow the scann library has been compiled with (2.6.0) is not compatible with the one TensorFlow Java 0.4.x is using (2.7.0)? That’s surprising but can you try with `scann==1.2.5` instead (which has been compiled with TF 2.7.0)?

---

<div class="post-metadata">

**Author:** ![saurabh\_tripathy](https://d1dlmcr85iqnpo.cloudfront.net/user_avatar/discuss.ai.google.dev/saurabh_tripathy/32/14738_2.png) [@saurabh\_tripathy](https://discuss.ai.google.dev/u/saurabh_tripathy)\
**Post date:** [January 10, 2023, 3:48am UTC](https://discuss.ai.google.dev/t/can-i-use-tf-native-core-platform-in-java-for-scann-based-model/31740/11 "2023-01-10T03:48:22Z")

</div>

@karllessard : Ok the path thing still doesn’t work for me.

When passing absolute path exec’ing into the container, Now I am getting this exception:

```auto
java.lang.IllegalArgumentException: Function with signature [predict] not found
        at org.tensorflow.SavedModelBundle.function(SavedModelBundle.java:437)
        at org.example.Main.main(Main.java:45)

```

Do you see an obvious issue with predict call? I assume predict signature is not being exported. This is how I am exporting the savedModel: [Google Colab](https://colab.research.google.com/drive/1Repk2g-gunf8Dgp0uoKFTKDWjtjPwSLo#scrollTo=KyQcX7P5Gguw)

I tried session

```auto
try (SavedModelBundle model = SavedModelBundle.load(filePath, "serve")) {
            // Set up input tensors
            Session session = model.session();

            TString inputArray = TString.vectorOf("42");
            Tensor inputTensor = TString.tensorOf(inputArray);
            System.out.println(inputTensor.shape());

            Tensor output_tensor = session.runner().feed("input_1", inputTensor).fetch("output_1").run().get(0);

// Run the model and get the output
            System.out.println(output_tensor.toString());
        }

```

But I am getting java.lang.IllegalArgumentException: No Operation named [input\_1] in the Graph

Here are the model specifics

```auto
signature_def['serving_default']:
  The given SavedModel SignatureDef contains the following input(s):
    inputs['input_1'] tensor_info:
        dtype: DT_STRING
        shape: (-1)
        name: serving_default_input_1:0
  The given SavedModel SignatureDef contains the following output(s):
    outputs['output_1'] tensor_info:
        dtype: DT_FLOAT
        shape: unknown_rank
        name: StatefulPartitionedCall_1:0
    outputs['output_2'] tensor_info:
        dtype: DT_STRING
        shape: unknown_rank
        name: StatefulPartitionedCall_1:1
  Method name is: tensorflow/serving/predict

```

Please help. I think we are close.

---

<div class="post-metadata">

**Author:** ![karllessard](https://d1dlmcr85iqnpo.cloudfront.net/user_avatar/discuss.ai.google.dev/karllessard/32/9185_2.png) [@karllessard](https://discuss.ai.google.dev/u/karllessard)\
**Post date:** [January 10, 2023, 1:21pm UTC](https://discuss.ai.google.dev/t/can-i-use-tf-native-core-platform-in-java-for-scann-based-model/31740/14 "2023-01-10T13:21:11Z")

</div>

You’ll probably get better results if you call your model via a function instead of calling the session runner directly. In TF 0.4.0, you can try something like this:

```auto
try (SavedModelBundle model = SavedModelBundle.load(filePath, "serve")) {
    try (TString inputTensor = TString.vectorOf("42")) {
          System.out.println(inputTensor.shape());

          try (Tensor outputTensor = model.function(Signature.DEFAULT_KEY).call(Map.of("input_1", inputTensor)).get("output_1")) {
            System.out.println(((TFloat)outputTensor).getFloat());
          }

```

If you want to continue to use the session directly, then try to pass/retrieve the tensors by their name instead of their signature key, i.e. `serving_default_input_1:0` and `StatefulPartitionedCall_1:0`

---

<div class="post-metadata">

**Author:** ![saurabh\_tripathy](https://d1dlmcr85iqnpo.cloudfront.net/user_avatar/discuss.ai.google.dev/saurabh_tripathy/32/14738_2.png) [@saurabh\_tripathy](https://discuss.ai.google.dev/u/saurabh_tripathy)\
**Post date:** [January 16, 2023, 3:30pm UTC](https://discuss.ai.google.dev/t/can-i-use-tf-native-core-platform-in-java-for-scann-based-model/31740/16 "2023-01-16T15:30:12Z")

</div>

Thanks!! It works now.

---

<div class="post-metadata">

**Author:** ![karllessard](https://d1dlmcr85iqnpo.cloudfront.net/user_avatar/discuss.ai.google.dev/karllessard/32/9185_2.png) [@karllessard](https://discuss.ai.google.dev/u/karllessard)\
**Post date:** [January 17, 2023, 11:42am UTC](https://discuss.ai.google.dev/t/can-i-use-tf-native-core-platform-in-java-for-scann-based-model/31740/17 "2023-01-17T11:42:04Z")

</div>

Wonderful 🎉 Please do not hesitate if you have any more questions!

---

<div class="post-metadata">

**Author:** ![saurabh\_tripathy](https://d1dlmcr85iqnpo.cloudfront.net/user_avatar/discuss.ai.google.dev/saurabh_tripathy/32/14738_2.png) [@saurabh\_tripathy](https://discuss.ai.google.dev/u/saurabh_tripathy)\
**Post date:** [January 17, 2023, 6:30pm UTC](https://discuss.ai.google.dev/t/can-i-use-tf-native-core-platform-in-java-for-scann-based-model/31740/19 "2023-01-17T18:30:20Z")

</div>

@karllessard : Thanks 🙂 Sure!! It would be great if I can get some help on this other thread. [http://discuss.ai.google.dev/t/how-to-build-a-preprocessing-layer-with-different-preprocessing-for-each-feature/14248](http://discuss.ai.google.dev/t/how-to-build-a-preprocessing-layer-with-different-preprocessing-for-each-feature/14248)