Documentation enhancement proposal: explainer for asyncio

[anordin95]

Documentation

I've used Python's asyncio a couple times now, but never really felt confident in my mental model of how it fundamentally works and therefore how I can best leverage it. The official docs provide decent documentation for each specific function in the package, but, in my opinion, lack a cohesive overview of the systems design and architecture. Something that could help the user understand the why and how behind the recommended patterns. And a way to help the user make informed decisions about which tool in the asyncio toolkit they ought to grab, or to recognize when asyncio is the entirely wrong toolkit.

I spent a long time digging into the internals and then decided to take a stab at filling that perceived gap by writing a fairly thorough long-form article: A conceptual overview of asyncio.

I also submitted the article to HackerNews where it got some traction: https://news.ycombinator.com/item?id=44638710

I imagine there's a few ways forward here.

• Linking to the Github article from the asyncio docs
• Integrating the article directly into the Python docs (along with stylistic & content modifications to match)
• Y'all decide the article's mediocre at best and you don't want it merged. (Fair enough!).

Either way, let me know what y'all figure makes the most sense. :)

Linked PRs

• gh-137026: Add an explainer guide for asyncio #137215
• [3.14] gh-137026: Add an explainer guide for asyncio (GH-137215) #137581
• [3.13] gh-137026: Add an explainer guide for asyncio (GH-137215) #137582

[StanFromIreland]

I am afraid it will not be possible to merge your article in the current state. From my understanding of your post, a HOWTO for asyncio would solve your problem?

[anordin95]

I don't think a HOWTO would help in this case. In the Diataxis framework that the Python docs prefer, the aim of my article, and the thing I feel is missing from asyncio's docs is an explainer/explanation component. Something that can provide a more thorough, fundamental understanding.

[StanFromIreland]

explainer/explanation component

That would be the purpose of the HOWTO, like the existing ones for other modules, no? It can then be linked in the main doc, which contains the technical documentation?

[AA-Turner]

Thank you for the proposal Alex. We have a mini tutorial in https://docs.python.org/dev/library/asyncio-dev.html, but it could be expanded.

We'd be willing to take more explainer content, but it would need to be merged in small chunks, rather than all 6 pages at once. You would also need to rewrite the content to use reStructuredText and fit into the Python documentation.

I think more documentation on asyncio, and concurrency in general, is certainly welcome -- we now have regular threads, multiprocessing, subinterpreters, async, free threading, and perhaps more (greenlet?).

cc @kumaraditya303 for thoughts.

A

[anordin95]

@StanFromIreland, I feel we may be talking past each other a bit here. I'm working with the below definitions. And based on those, I feel my article falls much more in the explanatory realm. And, such explanatory information is what I feel is missing from asyncio.

Python how-to guides are designed to guide a user through a problem-field. Both tutorials and how-to guides are instructional rather than explanatory and should provide logical steps on how to complete a task. However, how-to guides make more assumptions about the user’s knowledge and focus on the user finding the best way to solve their own particular problem.

Python explanations provide a deeper level of understanding and are naturally more discursive. They aim to deepen the reader’s understanding and answer ‘why’ questions. They should provide context, make connections between topics, and discuss alternative opinions. There is no section dedicated to explanations but these can be found throughout Python’s documentation, for example the Unicode HOWTO.

Taken from: https://devguide.python.org/documentation/style-guide/#diataxis

[anordin95]

@AA-Turner -- gotcha, sounds good! I wonder if you could also provide a rough gauge or check-in on whether the content of this specific article would fit well with what's desired for such explainers or if it's currently very far off of the kind of prose that's commonly accepted.

[AA-Turner]

I've asked our asyncio experts for a second opinion on the technical merits of the content. Based on brief skim of the structure, I think it looks reasonable. I would aim to make some of the code blocks shorter, with explanations interspersed of what is going on. For example, in part 2 you have two blocks that are over 20 lines long -- this is a few full screens on a mobile browser. I would personally make the prose less folksy (e.g. 'regular 'ol function' is cute, but distracts from the content). The metaphor of a conductor/military planner/etc is a nice one, though.

I'm not sure from a pedagodical perspective if it makes sense to introduce the higher level (more frequently used) concepts such as tasks before the low-level building blocks (coroutines). Currently, you go low-level to high-level. Perhaps @treyhunner or @nedbat could opine?

Likewise, the "when to use asyncio" I think could be more prominent than in the conclusion -- it could even deserve a standalone section (or as part of a general 'what concurrency do I want?' FAQ). The 'design choices' section would probably stay in your article rather than moving accross. I like the variety of examples, which is always a good thing to have.

A

[anordin95]

Mmm, sounds good! Though, I do think whether coroutines are low-level is somewhat up for debate. For example, in the current asyncio docs they're listed under the high level API and introduced just before tasks. For reference, I did roughly intend to go from high-level to low-level, but you are right that there are some cases where I don't, like the event-loop.

[AA-Turner]

Also cross-referencing #79012 ("Asyncio Tutorial"), and #9748 as a previous attempt, but was too large to review. We've also had feedback that the detailed internals of how coroutines & await work shouldn't be closely tied to documentation for asyncio.

There's definite interest in this explainer-style prose, but I think past attempts have tried to do too much at once. I would suggest splitting the effort into a series of PRs, perhaps initially adding a small article and then expanding it incrementally. Feel free to reuse any of the content from the closed PRs, too -- as far as I can see, these were closed due to process problems, not because the content was bad.

A