[suggestion] lua script

[atticus-sullivan]

As I read about the issues with python (and perl didn't work for me as well, so I'm also using the python script), I'm wondering whether it might make sense to either supply an additional script written in lua (or even replace the python script with it). The big advantage here would be that texlua (the lua interpreter used by luatex) comes by default with every common texlive distribution (don't know about MiKTeX or other distributions).

As far as I'm aware the only purpose of the pdfrw library is to extract specific pages to individual files (and remove them from the pdf file). Am I right about this?
in this case, I believe (still trying to figure it out, so I'm not 100% on it) this can be done completely with the libraries which come with luatex (namely pdfe, see https://mirrors.ibiblio.org/CTAN/systems/doc/luatex/luatex.pdf on page 292). This would completely avoid the need of installing any additional dependency and would ease the use/setup of memoize.

Is this a way that you think is worth further exploring? (or am I wasting my time and this won't be included anyway (would be also fine, no worries))

So far this is what I've got

local pdf = pdfe.open("./main.pdf")
print("pdf", pdf)

local page = pdfe.getpage(pdf, 1)
print("page", page, #page)
for i=1,#page,1 do
	print(i, pdfe.getfromdictionary(page, i))
end

local box = pdfe.getbox(page, "MediaBox")
print("box", box)
for k,v in pairs(box) do
	print(k,v)
end

creating a new document from one page currently doesn't work out for me, but I'm investigating

local contents, dict = pdfe.getstream(page, "Contents")
print("contents", contents, dict)
print("length", dict.Length)
print("filter", dict.Filter)
print(pdfe.openstream(page.Contents))
local str, n = pdfe.readwholestream(page.Contents)
pdfe.closestream(page.Contents)
-- print(str, n)
local doc = pdfe.new(str, n)
print(doc)

[atticus-sullivan]

So what is possible:

• iterating over the pages
• accessing the data related to a page
• checking the dimensions
• accessing the stream/data/contents related to the page (as is or decompressed)

But I'm afraid there is no way of putting the content of the single page into a new document (the core functionality we'd need).
My approach of using pdfe.new was completely wrong as this expects complete pdf data, so it seems to be more something like pdfe.new(readall("./main.pdf")) == pdfe.open("./main.pdf") as I understand it.

So it seems like this approach doesn't work anyhow and the answer to the question is not really relevant anymore. Sorry for the noise.

[sasozivanovic]

Please do make noise.

I tried this myself when writing Memoize, went through the same ups and downs. You got me thinking, though: perhaps it would be easy enough to write a page extractor in Lua? It could rely on pdfe to read the page, but dress it up "manually" into a complete document. If only I knew more about PDF specs ... but the only thing I really know is that they are complicated.

[atticus-sullivan]

I'm also not quite confident with pdf specs. I explored it quite a while ago, but not in that detail.

I've found https://codeberg.org/catseye/pdf.lua/src/branch/master/pdf.lua which is quite old, but I think we don't need it all and I know my way around lua so I'm confident I could port it (if needed) to a more recent version (having the lib in lua would be nice regarding portabity and shipping the dependency with memoize.

Also we could ask on the luatex mailing list whether we're missing something regarding the included libs and if not if they intend to extend pdfe / add a new one (but I don't have high hopes)

[sasozivanovic]

Synchronicity: I'm looking at this library myself at the moment, and agree with your assessment.

If all else fails, the library is small enough (plus, it could be pruned further) that it could be integrated into the extraction script itself.

I'd be really happy if you are willing to play with this, I'm not proficient in Lua myself.

[atticus-sullivan]

👍 I'll play with the library further, but don't expect the PR tomorrow 😄

Yea probably we'll need to include the library (moreover the parts that we need) in the script (or in the package so it can be found via require)

• This also makes it easy for us to maintain it in case there are troubles
• We need to think about licensing stuff (but at least it sounds very open) (maybe we're even able to reach the author, I mean the latest commit is "only" one year old particularly about licensing)

But now skimming a bit more over the library I'm positive it includes what we need to include the stream obtained via pdfe into a fresh pdf file

[sasozivanovic]

I'm not 😄

I agree with all you said. Happy hacking!

[cfr42]

David Carlisle also mentioned this possibility a while ago. However, I'm still learning Lua, so I haven't explored it at all.

I don't think the Lua in LuaTeX is expected to change much except for bug fixes, so I have my doubts about the PDF functions getting extended. (But it never hurts to ask.)

[atticus-sullivan]

Yea I also have my doubts tbh (but still of course we can ask).

---

⚠️ Off-Topic (sorry but wanted to note this):
Lua is quite a fun language to learn I think. I admit there is also some weird stuff (like with tables you need to be careful if data is stored in the "array-part" or in the "hash-table-part") and sometimes you write stuff over and over again because the stdlib is quite small (there is a reason why nvim and some other tools embedding lua ship some extended stdlib on top 😄).
Also I always miss types a bit, so for larger projects (like the latex-run-script similar to latexmk, not this extraction script), I try to use some typed dialect like teal/tl or so (but that's probably mostly personal preference, I don't quite like python for the same reason mostly).

I think this is classic easy to learn (so if you have time go for it), hard to master (don't be too frustrated if things don't go the way you want/expect in the beginning).

[atticus-sullivan]

For reference I'm tracking my progress during testing in atticus-sullivan/pdf.lua#1 (that repo is only temporary and will be deleted eventually)

[sasozivanovic]

Lua is quite a fun language to learn I think.

I'm saving this for when I have the time to implement Forest's packing algorithm in Lua. 😄

[atticus-sullivan]

Hm, I'm running into more and more issues regarding the fonts (pdf.lua really only has rudimentary features regarding fonts). Thus this will be more work than I expected initially.

I'm still willing to work on extending/modifying pdf.lua but I think it will be quite some time until I have something which is extensive enough so it makes sense to include this into memoize.

Since we need to copy-paste the stream containing the content of a page, this can easily contain some code which pdf.lua did not expect (like a font(number) which we did not copy. So essentially we need to implement the complete pdf spec here (at least that's how I think of it right now). Maybe we've got the difference of writing (just generate "arbitrary" data according to the specs -- what pdf.lua was designed for) and reading (make sense of everything that can be thrown at you according to the specs / emit everything the spec might need you to).
This probably is fun, but also very error prone and also there are some common pdf libraries/tools out there which are maintained way better I could ever do it (also looking into the future).

So in the end I don't currently think writing the extractor completely from scratch is the way to go here (sorry for the back and forth). I also see there is a rungs script included in current texlive distributions (not sure about others) which is used to call ghostscript. Maybe this would also be a possibility (on the other side ghostscript is not a out-of-the-box dependency of texlive as far as I know -- ans we're back at the need to install dependencies (which the need to be found by memoize) :/).

How I'd proceed:

1. Wait if you have more thoughts on this
2. Ask the LaTeX people (luatex mailing list) if they have more thoughts on this / hints for us

---

Don't get me wrong I'd still enjoy digging into the pdf specs and pdf.lua but I don't think it's gonna be stable enough for memoize (stability + feature complete) and I'm not 100% if I can maintain it for a longer time (then it would be quite sad if we had the extractor in lua and it would stop working after some time).

[cfr42]

I also don't think it would make sense to do this without thinking about tagged PDF, which will certainly be an issue for LuaLaTeX and pdfLaTeX.

[On the other hand, I'm not convinced tagging is going to support linguistics or logic at all.]