Multiple Object Recognition

[OptimusPrimeCao]

Hi,@nicholas-leonard
I'm trying to implement the [https://arxiv.org/pdf/1412.7755v2.pdf] for multiple object recognition in this repository
I have made some changes with RecurrentAttention and VRClassReward(to multiple one). To do same small change to mnist dataSet and SequencerCriterion(to SequencerCriterionMod, change the clone part)
And here is my model structure:

Agent :
nn.Sequential {
input -> (1) -> (2) -> (3) -> output
(1): nn.Convert
(2): nn.MulRecurrentAttention {
action : nn.Recursor @ nn.Sequential {
input -> (1) -> (2) -> (3) -> (4) -> (5) -> (6) -> output
(1): nn.FastLSTM(512 -> 512)
(2): nn.Linear(512 -> 2)
(3): nn.HardTanh
(4): nn.ReinforceNormal
(5): nn.HardTanh
(6): nn.MulConstant
}
input : nn.Recursor @ nn.Sequential {
input -> (1) -> (2) -> (3) -> (4) -> (5) -> (6) -> output
(1): nn.ConcatTable {
input
(1): nn.Sequential {
input -> (1) -> (2) -> (3) -> output
(1): nn.SelectTable(2)
(2): nn.Linear(2 -> 256)
(3): nn.ReLU
}
(2): nn.Sequential {
input -> (1) -> (2) -> (3) -> (4) -> output
(1): nn.SpatialGlimpse
(2): nn.Collapse
(3): nn.Linear(64 -> 256)
(4): nn.ReLU
}
... output
}
(2): nn.JoinTable
(3): nn.Linear(512 -> 512)
(4): nn.ReLU
(5): nn.Linear(512 -> 512)
(6): nn.FastLSTM(512 -> 512)
}
output : nn.Sequential {
input -> (1) -> (2) -> output
(1): nn.Linear(512 -> 10)
(2): nn.LogSoftMax
}
}
(3): nn.ParallelTable {
input
(1): nn.ConcatTable {
input
(1): nn.Identity
(2): nn.ConcatTable {
input
(1): nn.Identity
(2): nn.Sequential {
input -> (1) -> (2) -> output: nn.Constant
(2): nn.Add
}
... -> output
}
... -> output
}
... -> output
}
}

After running mul-obj-attention.lua, It seems that the loss becomes larger and doesn't converge in trainning. Any mistake I made? Many thanks !