Include Joerg's reinforcement learning w/ goal model

[ndphillips]

This model is a hack of reinforcement learning to try and take goals into account.

Basically, the idea is to change the observed outcome with the difference between the observed outcome, and the amount you'd need to achieve the goal. E.g.; if you have 10 trials and need 20 points, then you need 2 points on average to reach the goal. If the next outcome is 3, then this looks positive, while if it is 1, it looks negative.

[mdsteiner]

I just tried to implement the simulations and stuff but one problem with this new model is that no mean or something is taken, i.e. the new impression is imp_prior[option_i] + alpha_par * points_ref, which can be problematic, because the value of an impression can grow to infinity and if the numbers are relatively large this will kill the softmax because it can't handle too large or low numbers. So I would suggest that we fix alpha between 0 an 1 and use (1 - alpha_par) * imp_prior[option_i] + alpha_par * points_ref to split the learning, this I think would keep the impressions at reasonable levels.

What do you think?

[mdsteiner]

Or else I could also use weight_new <- 1 / i ^ alpha_par, with i being the current trial, to have the same rule as in the other RL model. I think I will implement that. I can still change it back...

[ndphillips]

Oh yes alpha_par definitely needs to be restricted to fall between 0 and 1!

I haven't fully tested the model yet though, I'm not sure if it's implemented in the correct way or not, so don't worry about using it just yet. I'll continue working on it today

[mdsteiner]

oh oops ok, just a second, I'll push the updates to github (I already changed the things in the simulation and fitting files and also in the functions)

[mdsteiner]

quick question about the parameter range. I'll have to use a very rough grid for the model because otherwise there are too many parameter combinations. With the following grid I could reduce it to 2000 combinations for the RLGoal model:

alpha_par_v <- seq(0, 1, .1) # Alpha parameter for reinforcement learning Impression
phi_par_v <- seq(0.0, 3, .3) # Phi parameter for softmax choice
curvature_par_v <- seq(0.0, 1, 0.2) # curvature parameter utility function RLGoal Impression
lambda_par_v <- c(0.5, 1, 1.5, 2.25) # loss aversion parameter utility function RLGoal Impression

but for lambda we'd only have four different values. This will still take some time to fit but will hopefully run through...

[ndphillips]

alpha is good
curvature is good

But we have an issue with phi. The problem is that the scale of the impressions matters a great deal. For small impression values (e.g.; 0 to 1), small values of phi won't make too extreme predictions. For medium to large impression values (e.g.; -10 to +10), even small values of phi can make very extreme predictions.

For example, here is the same parameter applied to impression values on a different scale:

vs.

This is a pain because our impression models have different scales. The SampEx_Int model has impressions that strictly fall between 0 and 1, while the other models have impressions that can fall between, say -10 and +10

If you can, it would be better to have a wider phi range for the SampEx_Int model (e.g.; [0, 1]) and a narrower range for the other models (e.g.; [0, .5]).

Or to make it easier, just set the range to be seq(0, 1, .05) (20 steps) for all models, this is both wide and fine-grained enough for all models (I hope)

What range did you use for phi in the existing model fits?

I think 4 steps for the lambda parameter is fine. But I think the lowest value should be 1. We should always assume some loss aversion (e.g.; lambda >= 1)

[mdsteiner]

Here's what I've used before:

N_par_v <- 1:15 # N paramter for SampEx Impression
alpha_par_v <- seq(0, 3, .1) # Alpha parameter for reinforcement learning Impression
phi_par_v <- seq(0.0, 5, .2) # Phi parameter for softmax choice

Yes I can try to make a finer grid but smaller range for the SampEx model...

Also I'm not sure about lambda, I could get rid of the 0.5 which would save 500 runs, and I could save another 300 by using curvature_par_v <- seq(0.0, 1, 0.25) but then we'd only have four different curvature parameters...

with the above grid it has now been running for 45 minutes on 4 cores for 4 participants and is still not done, so this might be an issue...

[mdsteiner]

Ok smaller grid for SampEx is implemented... I'll just go with speed and use only 4 different curvature parameters and only the three lambda parameters. If at some point we want to explore this in more detail, we can still use the optim function for all but the SampEx model...

[mdsteiner]

One solution for this might be to use an optimization function and for the SampEx model, as we've discussed at some point, use optim for each possible Memory_N and then take the smallest g2. I will put the thins on the server and run it and in the mean time start working on an optim version of the fitting. If I can get this done soon enough I could then put that one on the server and hopefully get it through by thursday...

yep that would be a good solution too!

[mdsteiner]

Ok a version with optim is up (it's in the Optim_Fit.R file). I'm now running this file on the server. It's a little risky since I couldn't fully test it (I only had one testrun because for 4 participants it took around 50 minutes) but if it works out we aren't so vulnerable to the argument that we only covered a very small part of the parameter space.