supporting offset in poisson

[apoorvalal]

this gives the gist; having an offset argument would involve constraining the coefficient to 1 and being able to model rates (in contrast to the regular use to model counts). Statsmodels supports it under an exposure arg.

more bg

[s3alfisc]

Yes, nice one. Have never used an offset before in my life, so hadn't bothered with this yet 😀 looks like it might not be too hard to add! fixest::feglm() has an "offset" argument, so we should maybe stick with that instead of statsmodels kwarg solution?

[bradhackinen]

I'd love to have this feature for a project I am working on. Is there anything I can do to help?

[s3alfisc]

Hi @bradhackinen , thanks for bringing this back to my attention! =) Just thought about this for a few minutes, and this is what I think we'd need to do:

I would add an offset argument to the fepois function, following the fixest example.

From the r-fixest docs:

offset
A formula or a numeric vector. An offset can be added to the estimation. If equal to a formula, it should be of the form (for example) ~0.5*x**2. This offset is linearly added to the elements of the main formula 'fml'.

For the beginning, I would only support a scalar input, no formulas.

We'd first have to add the offset argument to fepois. From there, we pass it via FixestMulti to the Fepois method, and then we'd have to adjust for it in every WLS fit in the iterated weighted least squares fitting step.

Would you be up to take a stab at this @bradhackinen? Of course I'm more than happy to help along the way =)

[bradhackinen]

Yeah, I was looking at the code yesterday and I think I can figure out most of it. Adding an argument for a scalar input is no problem, but I'm still working on understanding how to modify the WLS fit, particularly how that works with the demeaning step. If you have a clear idea of how that should be done, I'd be glad to hear some hints. Otherwise I can probably figure it out eventually by deciphering the fixest R code and reading Laurent Berge's paper.

[s3alfisc]

Alright cool, I'll try to give it some though tomorrow / in the next days and put down some notes!

[bradhackinen]

That would be very helpful, thanks! I'll also take another look at it on Monday and see what I can figure out on my own.

[s3alfisc]

So either I am completely off or this might actually be kind of straightforward, at least for the case with no fixed effects? 🤔

We simply adjust reg_Z = reg_Z - offset on top of line

pyfixest/pyfixest/estimation/fepois_.py

Line 267 in 03693d0

# Step 1: weighted demeaning

- hence subtract the offset from the dependent variable in linear space.

Then, everything stays as is for a while until we compute eta, where we have to add back the offset:

eta = Z - resid + offset

Maybe we can build a working solution without fixed effect demeaning first and then iterate on that? And maybe my brain is clouded right now, though I think the above would work with fixed effect demeaning too?

[bradhackinen]

Useful hints, thanks. That makes sense to me for the no-FE case. Intuitively, I would think you need to demean the offset in the FE case. Or maybe transforming Z works there too? I don't have a clear understanding of how the demeaning works in the non-linear model yet.

[bradhackinen]

I think I have it working. The simple modification @s3alfisc proposed seems to work in both the FE and non-FE case. Here is a pull request that implements the feature.

I tested it manually on a few different cases, and the output looks correct in both the FE and non-FE cases. I don't have r2py set up, so I didn't try to update test_poisson.py, but I can share my (very simple) test scripts if that would be helpful.

[s3alfisc]

Hi @bradhackinen, awesome, thank you! I'll take a look this evening and will add a few tests. Thank you!