Prediction Error when doing Textual Entailment using roberta

[drocobeth]

Describe the bug

I tried to run textual entailment using the demo code given on allennlp:
from allennlp.predictors.predictor import Predictor import allennlp_models.nli predictor = Predictor.from_path("https://storage.googleapis.com/allennlp-public-models/snli-roberta-large-2020.02.27.tar.gz", predictor_name="textual-entailment") predictor.predict(hypothesis="Two women are sitting on a blanket near some rocks talking about politics.",premise="Two women are wandering along the shore drinking iced tea.")

First, I tried to run it on my local jupyter notebook and got this error:

---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)
<ipython-input-52-f3796018f3e0> in <module>
      1 predictor.predict(
      2   hypothesis="Two women are sitting on a blanket near some rocks talking about politics.",
----> 3   premise="Two women are wandering along the shore drinking iced tea."
      4 )

TypeError: predict() got an unexpected keyword argument 'hypothesis'

Then I tried to run the same code on the google-colab and got this error:

---------------------------------------------------------------------------
KeyError                                  Traceback (most recent call last)
<ipython-input-3-f3796018f3e0> in <module>()
      1 predictor.predict(
      2   hypothesis="Two women are sitting on a blanket near some rocks talking about politics.",
----> 3   premise="Two women are wandering along the shore drinking iced tea."
      4 )

7 frames
/usr/local/lib/python3.6/dist-packages/allennlp/predictors/decomposable_attention.py in predict(self, premise, hypothesis)
     37             [entailment, contradiction, neutral].
     38         """
---> 39         return self.predict_json({"premise": premise, "hypothesis": hypothesis})
     40 
     41     @overrides

/usr/local/lib/python3.6/dist-packages/allennlp/predictors/predictor.py in predict_json(self, inputs)
     64     def predict_json(self, inputs: JsonDict) -> JsonDict:
     65         instance = self._json_to_instance(inputs)
---> 66         return self.predict_instance(instance)
     67 
     68     def json_to_labeled_instances(self, inputs: JsonDict) -> List[Instance]:

/usr/local/lib/python3.6/dist-packages/allennlp/predictors/predictor.py in predict_instance(self, instance)
    187 
    188     def predict_instance(self, instance: Instance) -> JsonDict:
--> 189         outputs = self._model.forward_on_instance(instance)
    190         return sanitize(outputs)
    191 

/usr/local/lib/python3.6/dist-packages/allennlp/models/model.py in forward_on_instance(self, instance)
    139         `torch.Tensors` into numpy arrays and remove the batch dimension.
    140         """
--> 141         return self.forward_on_instances([instance])[0]
    142 
    143     def forward_on_instances(self, instances: List[Instance]) -> List[Dict[str, numpy.ndarray]]:

/usr/local/lib/python3.6/dist-packages/allennlp/models/model.py in forward_on_instances(self, instances)
    165             dataset.index_instances(self.vocab)
    166             model_input = util.move_to_device(dataset.as_tensor_dict(), cuda_device)
--> 167             outputs = self.make_output_human_readable(self(**model_input))
    168 
    169             instance_separated_output: List[Dict[str, numpy.ndarray]] = [

/usr/local/lib/python3.6/dist-packages/allennlp/models/basic_classifier.py in make_output_human_readable(self, output_dict)
    171                 [
    172                     self.vocab.get_token_from_index(token_id.item(), namespace=self._namespace)
--> 173                     for token_id in instance_tokens
    174                 ]
    175             )

/usr/local/lib/python3.6/dist-packages/allennlp/models/basic_classifier.py in <listcomp>(.0)
    171                 [
    172                     self.vocab.get_token_from_index(token_id.item(), namespace=self._namespace)
--> 173                     for token_id in instance_tokens
    174                 ]
    175             )

/usr/local/lib/python3.6/dist-packages/allennlp/data/vocabulary.py in get_token_from_index(self, index, namespace)
    663 
    664     def get_token_from_index(self, index: int, namespace: str = "tokens") -> str:
--> 665         return self._index_to_token[namespace][index]
    666 
    667     def get_vocab_size(self, namespace: str = "tokens") -> int:

KeyError: 1596

To Reproduce
Steps to reproduce the behavior

1. pip install allennlp==1.0.0rc3 allennlp-models==1.0.0rc3
2. from allennlp.predictors.predictor import Predictor
3. import allennlp_models.nli
4. predictor = Predictor.from_path("https://storage.googleapis.com/allennlp-public-models/snli-roberta-large-2020.02.27.tar.gz", predictor_name="textual-entailment")
5. predictor.predict( hypothesis="Two women are sitting on a blanket near some rocks talking about politics.", premise="Two women are wandering along the shore drinking iced tea." )
   Expected behavior
   Should return values describing entailment, contradiction and neutrality.

System (please complete the following information):

• OS: [Microsoft, GoogleColab(Unix)]
• Python version: [3.6.10]
• AllenNLP version: [ v1.0.0.rc3]
• PyTorch version: (1.3.1)

[matt-gardner]

Looks like this was fixed here: allenai/allennlp-models@0b040af. @dirkgr, did you upload an updated model anywhere after fixing that?

I'm currently looking into what the underlying cause of this problem was, but if there's an updated model already, we can point people to that for now.

[matt-gardner]

Ok, the underlying issue is that there is no vocabulary saved with this model. So, when it gets loaded, it has only an empty vocabulary, and so make_output_human_readable fails when trying to translate from ids to strings. This is related to #3097 and #4034. I thought we had solved the underlying issue already for simple cases like this, and a more recently trained model wouldn't have this vocabulary problem, because it would have been saved with a correct vocabulary. @dirkgr, do you know if that's true?

[dirkgr]

I have a solution for this. Use the model archive at https://storage.googleapis.com/allennlp-public-models/snli-roberta-large-2020.04.30.tar.gz.

I didn't solve it by addressing the no-vocab issue though. That's not what I saw. The problem is that it was using the wrong vocab namespace.

[matt-gardner]

Ah, ok, thanks. I missed that the transformer vocabulary gets added to the model's vocabulary at the call of instance.index(vocab), which happens during prediction. Those issues I linked above are still relevant, because how this works is a bit of a mess, but I downloaded the model you linked and confirmed that it solved this issue for me.

The commit that introduced this particular failure is this one: d709e59.

We should update the usage info in the demo, and I'll open an issue there if there isn't one already. I'm closing this one, as as far as the library is concerned it's fixed.

[drocobeth]

Thanks for the help.
I used the same described model by @dirkgr. Also, predictor.get_path() method does not take the model downloaded in the local server. Instead, when given it starts downloading the model that has bug from the cloud.
Therefore, predictor.get_archive() method has to be used in order to take the model from the local directory by giving the path in load_archive() method.