I want to set up a Tensorflow decision forests model for inference using Google Earth Engine. There is already a Colab notebook example on how to do that using a Neural Network, I’m basically trying to do the same thing but with a random forest model.
However, when I try and prepare the model for “EEification” using the earth engine command line tool, it fails. I’m pretty sure the error is related to the way the earth engine command line tool calls the Tensorflow library but not sure what the solution is.
Example colab notebook with very simple example: Google Colab
The notebook requires that you have set up Earth Engine with the Google account that you authenticate with.
Sorry for my complete ignorance on earchengine but when you call the cmd line (with the error) what happens? does it creates a VM (or container) and uses that to run the model?
My gut guess says that if it’s that, maybe that VM is missing the TFDF dependency and so lacks the missing ops.
No worries, I am by no means an expert either! Based upon the (not very detailed) documentation, that command prepares the model somehow. From the docs:
the Earth Engine CLI has the model prepare command that wraps an existing SavedModel in the required operations to convert input/output formats.
So I think probably all this step is doing is adding a preprocessing step to the model and saving it again as a new model. The deployment step comes later, I didn’t include it in the notebook.
Based upon the stack trace it looks like the earth engine command line tool (which is just a Python library) imports tensorflow and tries to add this preprocessing step, but fails. There are a few issues (e.g. NotFoundError when load model from different device · Issue #14 · tensorflow/decision-forests · GitHub) with similar error messages which makes me think that there are custom Tensorflow ops required that are not being imported, which causes it to fail.
I can have a good guess on one of the problems: TF-DF defines what TensorFlow calls “TF Custom ops”: new algorithms (training/evaluating decision forests) implemented in c++ to be used in tensorflow.
When one does import tensorflow_decision_forests these “TF Custom Ops” get imported and it’s all fine.
Now when you run !earthengine you are likely starting another (python?) program, and likely it doesn’t import tensorflow_decision_forests, and hence cannot find the custom op …
Internally we are discussing how to provide a standard way of handling these situations: your SavedModel should somehow inform any other program what libraries (Custom Ops) it should import. But unfortunately this is not automatic yet :\ … it’s in our list of todo’s.
In the short term, this would be easier to solve by modifying earthengine to import the library.
earthengine is written in Python so was hoping I could just import it, however it’s designed as a command line tool and there’s no documentation on calling from Python, so sounds like it may be difficult…
Just wanted to add here that actually the issue is not solved - earth engine is using Tensorflow 1 which I guess is why it’s not working with TF-DF (which I assume is not backwards compatible?) Anyway, I raised an issue here: Google Issue Tracker, so anyone interested can see progress there
Let me try to summarize the situation (please tell me if I am wrong or if I am missing something), what works and what does not, and the available solutions.
Summary
The Earth Engine is a pure python program that can run TF1 SavedModel.
When importing TF-DF in Earth Engine here, you can use TF-DF models.
If you don’t import TF-DF in Earth Engine, you get the error: “NotFoundError: Op type not registered ‘SimpleMLCreateModelResource’”.
Analysis and solution
TensorFlow (TF) does not currently have support to automatically import custom tf ops. Instead, to use custom ops, the library containing the custom ops needs to be imported manually. This is true for all TF ports e.g. python, c++, java. In python, this import is done with “import {name of the library}”.
If this is true, I can see 3 solutions:
Have Earth Engine always importing TF-DF. This is probably a big decision for ee.
Have TF-DF + Earth Engine users recompile Earth Engine with TF-DF. This is not a user-friendly solution.
In Earth Engine CLI, add a flag for the user to specify a list of py libraries to import (using “import()”). I think this is the more elegant solution (small amount of code, support for all custom ops, no dependency from ee to tf-df).
Note
While TF-DF training is only for TF2 (we did not open source the TF1 version), the inference of TF-DF model works well with TF1 (here are some low lever examples).
Thanks for the summary @Mathieu - unfortunately I’ve since done some more digging and the situation is more complicated than that. importing TF-DF solves the initial “NotFoundError: Op type not registered ‘SimpleMLCreateModelResource’” but another problem becomes apparent later on.
As I understand it, the TF1 way of changing a model’s inputs is very different to the TF2 way (where you could add a new keras Layer to handle the preprocessing). I think something in that process is making the model unusable, probably some sort of backwards incompatibility TF-DF has with those TF1 commmands.
The command actually claims that it completes successfully, so no error is thrown. Only when I try to load the model with Tensorflow serving does it throw the error :
INVALID_ARGUMENT: You must feed a value for placeholder tensor 'asset_path_initializer_1' with dtype string [[{{node asset_path_initializer_1}}]]
I can confirm the model works fine before this command, and also that the command successfully creates a functional Neural network model.
The EE team are aware of it but I guess it’s not very high priority for them. I’ve started to try and update the EE code to prepare the model inputs in the TF2 way and I’ll open a PR if I get it working. In the meantime, it would be awesome if you could confirm that the function used by earth engine will render a TF-DF model unusable? Just want to make sure I’m not barking up the wrong tree… although I can’t see what else it could be!
Has there been any progress made on this that you know of? @Matt_Miller, were you able to update the EE code to prepare the model inputs as TF2?
I would like to use a TFDF model for image prediction in GEE and am still experiencing the same issues. Crossing my fingers that there have been positive developments in the last year!
I don’t know if the engine was updated, I’d need to check it but for your task, image prediction with TFDF. Are you going to use raw image as input? or are you doing something else?
@ktfank Yes there has been progress but it’s not completely working yet.
I managed to reverse engineer the earthengine model prepare command by creating some TF2 custom layers, so that part appears to be working. However, there are some artefacts that are being introduced during inference at scale. It’s not clear exactly where the issue is arising.
Earth Engine are in the process of migrating to Vertex AI and have given me assurances that they are actively working on the issue, but have not given a timeline. You can follow along here: Google Issue Tracker
@lgusm I am doing pixel-wise classification of crops with sparse training data on seasonal composite images, instead of raw time series images.
I opened a thread on Stack Exchange to this end. It shows me (naively) trying to do something similar with XGBoost, but may give you a better idea of my application.
…