What is the current dev status for Model parallel in tf.distribute.strategy

TensorFlow General Discussion
1

I’ve seen the talk called Inside TensorFlow: tf.distribute.Strategy, it’s in 2019.

Back to that time, TF teams has added the model parallelism ability in their TODO list, and now, it’s at the late Q3 of 2021, what is the dev progress? Is their any general progress of how to easily design a model parallel strategy using TF?

1 Like
3

Do you want an udpate on

1 Like
5

We are working on model parallelism APIs now, but we don’t have a date for when those will be released.

2 Likes
6

Thanks for the reply!

May I know if there is a RFC or plans for what kind of model parallelism will you support?
For example, https://huggingface.co/transformers/master/parallelism.html has described several kinds of model parallelism strategy, would you plan to support these kind of parallelism?

1 Like