Update: Most of this issue was just browsers being browsers (I didn't realize how much tab key behavior varies), but one real problem remains: In Safari, pressing tab can sometimes focus and allow interacting with elements that should be inert.
For example, on https://wicg.github.io/inert/demo/, clicking on the page and then pressing tab focuses an input that should be inert. I can send exactly one keypress to that input, but typing further has no effect.

(Safari 12.1.1 on MacOS 10.13)