Serving using ExportStrategy

[dennybritz]

Should figure out how to export models for serving, I think Tensorflow does provide something like an ExportStrategy that can be passed to the estimator and it will occasionally export the model.

[gidim]

It might be easier to add support for using feed_dict as an alternative for InputPipeline?
Many TF models use that paradigm rather than reading the input data directly from the graph.
I tried doing this myself but the current architecture is dependent on tf.slim.Dataset which cannot be used for online inference.

Usually you would declare two places holders, one for features and one for targets and then
feed them as the model inputs. Currently the model in basic_seq2seq expects labels and features as a dictionary. I could not find the structure for that dictionary but i suspect it's coming from Dataset

[skyw]

I think the standard way is freeze the model, then use feed_dict to feed input and get the output tensor.

[gidim]

@skyw that's correct but since the models in this repo use Readers you first have to declare a new input tensor to the model (Placeholder).

[xueyouluo]

Any updates of this feature? I agree with @gidim , currently the model depend of tf.Slim.Dataset which is hard to be used for online inference.

[amirj]

I think there is two options:

1. Load the model, add an input placeholder to the graph, remove the input pipeline from the graph and freeze it. As mentioned by @skyw

2. [better option] leverage TFserving component to plug a model contains input pipeline with queues to a web server.

I think, the serving feature is very important feature for this project referred in other issues (#114).

[gidim]

@amirj does tfserving supports option two? If so could link to the documentation/code ? I couldn't find anything about it.

[amirj]

Input pipelines are the standard way to feeding TF models. So, I think, its indeed possible. But the current documentation in TF serving module is not clear.

[gidim]

@amirj - the standard way when performing training. for inference most models use feed_dict.
Usually it's easy to rebuild the graph with two placeholders but the code here is a a little complex and has quite a few dependencies which makes it harder.

[amirj]

It's strange if TF serving is not compatible with input queues.
It's not a good design to leverage different feeding in the training and inferencing. For example, one application of ExportStrategy is online inference during the training phase #114 . Rebuilding the graph in this situation is not good.

[sathyarr]

@amirj @gidim

do you think the approach followed here to serve the graph is recommended?

[sathyarr]

Should figure out how to export models for serving, I think Tensorflow does provide something like an ExportStrategy that can be passed to the estimator and it will occasionally export the model.

@dennybritz
I have managed to export through export_strategies in Experiment.
But, facing problem with preprocessing and embedding.
Kindly have a look here and here

Are things performed in _preprocess exported to Graph?