How to Run a TFX Orchestration Pipeline Outside Jupyter?

TensorFlow General Discussion
,
1

Learning data ingestion and prediction orchestration with tensorflow from this notebook on github. It is using an InteractiveContext on the ipython environment. But obviously a production system does not run ipython. When I try to run the code in REPL python shell, I get

WARNING:absl:Method "run" is a no-op when invoked outside of IPython.

So how to run the orchestration outside a jupyter notebook, as a simple python script. Also, what other modification necessary to run the pipeline in real production environment?

3

This tutorial shows how to run a pipeline in a simple .py

This tutorial shows how to run a pipeline on Vertex, which is almost the same as running it in Kubernetes

This tutorial shows how to run it in Airflow. Warning, this is an older version and probably needs some work to run in the latest version.