Incompatibility with minted v3

[The-Compiler]

By default, minted v3 requires a second compile to highlight code properly:

highlightmode: Determines when code is highlighted. The default is fastfirst. If a cache for the document exists, then code is highlighted immediately. If a cache for the document does not exist, then typeset a placeholder instead of code and highlight all code at the end of the document. This will require a second compile before code is typeset, but because all code is highlighted at once, there is less overhead and the total time required can be significantly less for documents that include many code snippets. The alternatives are fast (always highlight at end of document, requiring a second compile) and immediate (always highlight immediately, so no second compile is needed). immediate should be used when typesetting code in external temp files that are overwritten during compilation.

However, memoize extracts the externs after the first compilation, which means that all externs still have the <MINTED> placeholder instead of highlighted code:

\documentclass{article}
\usepackage[extract=python]{memoize}
\usepackage{minted}[2024/09/22]
\usepackage{tikz}

\begin{document}
    \begin{tikzpicture}
        \node {\mintinline{text}|test|};
    \end{tikzpicture}
\end{document}

cached result:

The only workarounds I've found are:

• Compile once, delete the cached memoize pictures (but not the minted cache!), compile again
• Disable caching selectively for tikzpictures which use minted (obviously error-prone, especially because the issue depends on two different caches being in the correct state)
• Pass [highlightmode=immediate] to minted, which will however significantly increase the time for the first compile (2min 50s -> 5min 45s for a full `latexmk' run in my case). From what I can gather, there is no way to only set that option for a single minted call either.
• Patching the internal minted command that inserts the placeholder to abort memoizing:

\usepackage{etoolbox}
\makeatletter
\pretocmd{\minted@insertplaceholder}{\mmzAbort}{}{\errmessage{patch failed}}
\makeatother

Is there a good way for memoize to support this properly? Should it perhaps handle minted automatically?

[cfr42]

I would suggest either not memoizing code which includes minted markup or disabling minted's caching mechanism and memoizing minted code instead. Right now, you have two completely independent mechanisms both trying to handle the same (or, worse actually, parts of the same) code. This may be possible, but it strikes me as needlessly complex.

Otherwise, I think you will have to get memoize to replicate the checks minted does and probably its use of cached code, too.

If minted provides a way to disable the caching mechanism locally, you could switch it off for just those bits of code subject to memoization. I couldn't quickly see an option to do this in the docs, but I don't use minted, so I may have missed something. Is there some mechanism you could activate in a hook, say, for all tikzpicture environments (just for example)?

[But Sašo can work magic and I cannot, so ....]

[cfr42]

Maybe something like this, but hopefully less fragile:

\documentclass{article}
\usepackage[extract=python]{memoize}
\usepackage{minted}[2024/09/22]
\usepackage{tikz}
\makeatletter
\mmzset{%
  include context in ccmemo,
  auto = \mintinline{run if memoizing,args={mv},outer handler=\mmz@auto@minted},
}
\NewDocumentCommand \mmz@auto@minted { mv }
{% purloined from memoize.sty & mangled
  \IfFileExists{_minted/\MintedCacheIndexFilename}{%
    \xtoksapp\mmzCCMemo{%
      \expandafter\def\csname memoize:minted@index@filename\endcsname 
      {%
        \noexpand\string\MintedCacheIndexFilename
      }%
    }%
  }{%
    \mmzAbort
  }%
  \ExpandArgs {oo} \AdviceOriginal{#1}{#2}%
}
\makeatother
\begin{document}
\begin{tikzpicture}
  \node {\mintinline{text}|test|};
\end{tikzpicture}
\end{document}

This takes three a minimum of three compilations to 'settle':

1. On the first run, the relevant minted index file doesn't exist, so memoization is simply aborted. In this case, the cmemo is written, but not the ccmemo. No extern is created.
2. On the next run, the index file does exist, so memoization proceeds and an extern is created. A ccmemo is created and the name of the minted index file is added to the context in the ccmemo.
3. On the following run, the extern is extracted and utilised.

Note that this relies on internal implementation details of minted. Although \MintedCacheIndexFilename is a public macro (I assume?), its being available at the right point in the memoization process probably isn't. It would be nice if minted provided some kind public interface for injecting code e.g. hooks. However, as far as I can tell, it does not do so at present, anyway.

Note that prudence probably lies in avoiding this code, if not quite like the plague, at least like a tariff on islands inhabited solely by penguins. 🐧 🐧 🐧