DIGITS image classification view does not correctly handle output of FCNs

[AliaMYH]

I've trained a network and the validation accuracy is very good for it, almost 100 percent. However when I come to 'classify many', using the val.txt for the dataset, the accuracy returned is terrible. Only one of the classes is predicted every time. I'm not exactly sure what the problem is.
My validation set is 20% of my training set, I've used the split option in digits.

[AliaMYH]

I tried using mean pixel as recommended by #625 , but the same thing is happening.

Classify One gives correct results.

[AliaMYH]

@samansarraf According to your comment on #625, am I meant to preprocess all my data outside of digits before training?

[samansarraf]

@AliaMYH your understanding is correct. To solve the issue at that time , I came up with the idea to preprocess my data and made them totally independent from classify many preprocessing module . There should be something on their codes that is not working in some cases. After I preprocessed data outside (including image resizing ...) the classify many exactly produced the same results as I was getting during training and validation . Don't give up ! you are pretty close to it .

[AliaMYH]

A curious thing though is that I use the exact same dataset and cropping with two different networks and one gave the above results, while the other one worked perfectly. The one with the above results is a fine-tuned Squeezenet, and the one that gave proper results was a fine-tuned AlexNet which I trained for comparison. I'm not exactly sure why this is.

Also, does this mean that it's only the displayed results that are incorrect or the entire training of the network.. considering that the 'Classify One' results are correct. Is the deploy.prototxt what is used during the inference?

@samansarraf @gheinrich

[samansarraf]

@AliaMYH No , the entire training (including validation) process is correct. Only classify many has some preprocessing problem in my opinion. I had the same kind of problem with GoogleNet so I what I did I cropped the data before training and used the cropped images for both training and classify many module then it worked.

[AliaMYH]

I agree as well. @gheinrich Can you confirm this?

[gheinrich]

@AliaMYH are you saying that the standard Alexnet provided in DIGITS gives proper result whether the implement of SqueezeNet you're using doesn't? It might be a case of DIGITS being confused by the output of the network. Are you using a single SoftMax output in SqueezeNet?

As a side note, if small differences in pre-processing lead to wildly different accuracy, that's a sign of overfit. You'd have to see how well the model generalized to unseen samples.

[gheinrich]

@AliaMYH you might want to verify the shape of the network output. It is probably a case of the network outputting one more dimension than DIGITS expect, which would confuse the classify many path.

[gheinrich]

Any feedback please @AliaMYH ?

[SlipknotTN]

Me and my colleagues have noticed the same problem @gheinrich , SqueezeNet model (we have tested 1.1) outputs one additional dimension causing the problem on the confusion matrix.
You can fix the problem at inference.py level b2062c6 or in classification/views.py dbcb9ed but the first solution should be better, probably it is possible find other solutions if we understand why SqueezeNet has this behaviour.
We could review the problem and make a PR if you like.

[gheinrich]

I had never looked at SqueezeNet before. The reason you get more dimensions is because this is a fully convolutional network, unlike the typical classification CNN (e.g. Alexnet) which has a fully convolutional feature extractor and a classifier that is made of fully-connected layers. Therefore, SqueezeNet produces a spatial output while DIGITS expects a flattened probability distribution of classes. I think it's better to fix DIGITS in classification/views.py. We could for example collapse the dimensions that have a cardinality of one with np.squeeze. If you are willing to take this up @SlipknotTN it would be very much appreciated, thanks in advance!