Consider not consuming all the buffer in one go in the tokenizer module

[pablogsal]

Seems that some tools were relying on the implementation detail that the readline-like callable that is provided to functions in the tokenize module is called as tokens are emitted and not consumed in one go. Although this was never part of the contract and technically we don't need to change, it would make the implementation more efficient as we don't need to hold the entire input in memory at the same time and we won't break these tools.

Linked PRs

• gh-105069: Add a readline-like callable to the tokenizer to consume input iteratively #105070
• [3.12] gh-105069: Add a readline-like callable to the tokenizer to consume input iteratively (GH-105070) #105119

[pablogsal]

CC: @lysnikolaou @mgmacias95