[GH-395] Port to Python 3

[elsiehupp]

Fixes #395.

EDIT 2: I've removed the entire body of this comment because it's difficult to keep it up to date with progress elsewhere.

EDIT 3: I've personally unsubscribed from this thread because I was getting an email for every single commit.

To interact with this draft pull request, please consult the README on the forked repository. If you run into any problems, opening an issue there will be more effective than commenting about it here.

[nemobis]

Il 09/06/21 23:34, Elsie Hupp ha scritto:

I had to port both `poster` and `wikitools` to Python 3 in order to get this to work, so I included those in their own folders in the repository.

Thank you very much, but can you also post them upstream?

[elsiehupp]

Il 09/06/21 23:34, Elsie Hupp ha scritto:
I had to port both poster and wikitools to Python 3 in order to get this to work, so I included those in their own folders in the repository.
Thank you very much, but can you also post them upstream?

Probably yes? I figured it was easiest to do it here to begin with.

[elsiehupp]

Currently what I’m stuck on is URL encoding, which can probably be simplified by porting to Requests and/or urllib3.

[nemobis]

You probably didn't want to commit .vscode/settings.json

[elsiehupp]

You probably didn't want to commit .vscode/settings.json

I think I just forgot to revert that when I flattened my commits (though the change stopped being relevant once I started using pipenv).

FWIW I think it might be worth my migrating from pipenv to poetry for the purpose of facilitating distribution on, e.g., PyPI.

Do you have any other immediate feedback? IIRC the main issue I was running into was with the test suite, so I haven’t been able to fully validate the new code.

[elsiehupp]

wikitools seems to be abandoned by its maintainer (though I haven’t tried particularly hard to reach him), so I went ahead and published the version from this pull request on PyPI as wikitools3, which allowed me to specify it as an external dependency. I did some digging, and it turned out that someone else had already made a Python 3 version of poster called poster3, so I used that as the dependency for wikitools3.

I migrated wikitools3 to use poetry, which seems to come with a lot of advantages, so I might want to migrate from pipenv to poetry here, as well.

[elsiehupp]

Hi @GreenReaper—can you try the updated version?

In the cloned wikiteam directory, try:

$ git pull
$ poetry install
$ poetry run python dumpgenerator.py --xml --xmlrevisions https://furry.wiki.opencura.com

I ran the above commands myself several times, so the encoding issues should be fixed?

Thanks again for helping me find bugs!

[GreenReaper]

That works, thanks! However I tried it with an older wiki, in an attempt to ensure that encoding was saving correctly, and it seems the --xml case (no --xmlrevisions) is still broken on xmlfile.write in generateXMLDump, both on this wiki and the opencura one.

For this wiki I needed to add |class="mediawiki to the search regex in getWikiEngine, because it was otherwise detected as Unknown (since we removed the generator head lines as superfluous :-) - I tried --force but it didn't seem to do anything:

# poetry run python dumpgenerator.py --xml --curonly https://zh.wikifur.com/ --api https://zh.wikifur.com/w/api.php --index https://zh.wikifur.com/w/index.php
Checking API... https://zh.wikifur.com/w/api.php
API is OK: https://zh.wikifur.com/w/api.php
Checking index.php... https://zh.wikifur.com/w/index.php
index.php is OK
#########################################################################
# Welcome to DumpGenerator 0.4.0-alpha by WikiTeam (GPL v3)                   #
# More info at: https://github.com/WikiTeam/wikiteam                    #
#########################################################################

#########################################################################
# Copyright (C) 2011-2021 WikiTeam developers                           #

# This program is free software: you can redistribute it and/or modify  #
# it under the terms of the GNU General Public License as published by  #
# the Free Software Foundation, either version 3 of the License, or     #
# (at your option) any later version.                                   #
#                                                                       #
# This program is distributed in the hope that it will be useful,       #
# but WITHOUT ANY WARRANTY; without even the implied warranty of        #
# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the         #
# GNU General Public License for more details.                          #
#                                                                       #
# You should have received a copy of the GNU General Public License     #
# along with this program.  If not, see <http://www.gnu.org/licenses/>. #
#########################################################################

Analysing https://zh.wikifur.com/w/api.php
Trying generating a new dump into a new directory...
Loading page titles from namespaces = all
Excluding titles from namespaces = None
20 namespaces found
    Retrieving titles in the namespace 0
    541 titles retrieved in the namespace 0
    Retrieving titles in the namespace 1
    30 titles retrieved in the namespace 1
    Retrieving titles in the namespace 2
    61 titles retrieved in the namespace 2
    Retrieving titles in the namespace 3
    93 titles retrieved in the namespace 3
    Retrieving titles in the namespace 4
    183 titles retrieved in the namespace 4
    Retrieving titles in the namespace 5
    2 titles retrieved in the namespace 5
    Retrieving titles in the namespace 6
    422 titles retrieved in the namespace 6
    Retrieving titles in the namespace 7
    1 titles retrieved in the namespace 7
    Retrieving titles in the namespace 8
    36 titles retrieved in the namespace 8
    Retrieving titles in the namespace 9
    1 titles retrieved in the namespace 9
    Retrieving titles in the namespace 10
    140 titles retrieved in the namespace 10
    Retrieving titles in the namespace 11
    4 titles retrieved in the namespace 11
    Retrieving titles in the namespace 12
    49 titles retrieved in the namespace 12
    Retrieving titles in the namespace 13
    2 titles retrieved in the namespace 13
    Retrieving titles in the namespace 14
    241 titles retrieved in the namespace 14
    Retrieving titles in the namespace 15
    2 titles retrieved in the namespace 15
    Retrieving titles in the namespace 828
    0 titles retrieved in the namespace 828
    Retrieving titles in the namespace 829
    0 titles retrieved in the namespace 829
    Retrieving titles in the namespace 100
    4 titles retrieved in the namespace 100
    Retrieving titles in the namespace 101
    1 titles retrieved in the namespace 101
Titles saved at... zhwikifurcom_w-20210928-titles.txt
1813 page titles loaded
https://zh.wikifur.com/w/api.php
Retrieving the XML for every page from "start"
Traceback (most recent call last):
  File "dumpgenerator.py", line 2850, in <module>
    main()
  File "dumpgenerator.py", line 2841, in main
    createNewDump(config=config, other=other)
  File "dumpgenerator.py", line 2361, in createNewDump
    generateXMLDump(config=config, titles=titles, session=other["session"])
  File "dumpgenerator.py", line 852, in generateXMLDump
    xmlfile.write(bytes(header, 'utf-8'))
TypeError: write() argument must be str, not bytes

Incidentally, it says it saved in "a new directory" but it doesn't say which directory, which can be confusing.

[elsiehupp]

@GreenReaper Okay!

I’m not 100% sure what you’re describing |class="mediawiki, so I didn’t do anything with that.

I fixed two more encoding bugs. I also changed the default path to be a subdirectory of the parent directory rather than the working directory (so that the default path isn’t inside the cloned repository) and added a console message that prints when the --path argument is not used:

No --path argument provided. Defaulting to:
  ../[domain_prefix]-[date]-wikidump
Which expands to:
  ../zhwikifurcom_w-20210928-wikidump

(I could probably make the argument parsing more verbose across the board.)

Anyway, in the cloned wikiteam directory, try:

$ git pull
$ poetry install
$ poetry run python wikiteam3/dumpgenerator.py [args]

Note that dumpgenerator.py is now wikiteam3/dumpgenerator.py (a change that has to do with packaging and isn’t quite relevant here, yet).

I tried the following, and while I didn’t let it run its entire course, I didn’t get any errors for the first minute or two it was running:

$ poetry run python wikiteam3/dumpgenerator.py --xml --curonly https://zh.wikifur.com/ --api https://zh.wikifur.com/w/api.php --index https://zh.wikifur.com/w/index.php

[GreenReaper]

Yeah, I could have been clearer there. I meant getWikiEngine's detection, without which it refused to proceed; I changed to:

    elif re.search(
        '(?im)(alt="Powered by MediaWiki"|<meta name="generator" content="MediaWiki|class="mediawiki)',
        result,
    ):
        wikiengine = "MediaWiki"

There was a similar regex in checkIndex:

     if re.search(
        '(This wiki is powered by|<h2 id="mw-version-license">|meta name="generator" content="MediaWiki|class="mediawiki)',
         raw,
     ):
         return True

I tried the commands above and it worked for a while, then broke (trying to save the constant footer string?):

Downloaded 1810 pages
    新聞:Krystal的三明治在Fur Affinity爆红, 1 edit
    新聞:中文WikiFur的前綴名全面中文化, 1 edit
    新聞:英语 WikiFur 迁入 wikifur.com, 1 edit
    新聞討論:羽鲨, 1 edit
Traceback (most recent call last):
  File "wikiteam3/dumpgenerator.py", line 2854, in <module>
    main()
  File "wikiteam3/dumpgenerator.py", line 2845, in main
    createNewDump(config=config, other=other)
  File "wikiteam3/dumpgenerator.py", line 2365, in createNewDump
    generateXMLDump(config=config, titles=titles, session=other["session"])
  File "wikiteam3/dumpgenerator.py", line 883, in generateXMLDump
    xmlfile.write(bytes(footer, 'utf-8'))
TypeError: write() argument must be str, not bytes

[GreenReaper]

Unfortunately the config file was allegedly not written so it had to start again. In fact, it looks like it was written, but if it's meant to be a text file, it's unreadable, so maybe that bit needs to be changed?

As for the footer, I tried changing the existing line close to the end of generateXMLDump that mentions the footer to

xmlfile.write(str(footer))

and will see how that goes... though on consideration, it really should already be a str, so perhaps that is unnecessary? Anyway, adjusting that line resulted in a completed XML file, so it's definitely the issue.

[elsiehupp]

I’m actually running the test again myself, though I added --delay 1 to avoid a timeout.

You can pull the latest changes again if you want.

Regarding the config file, I ran into the same issue myself, so that’s another thing I need to fix, lol.

Also, by the way, it can be helpful if you refer to line numbers, like, e.g. with the blocks where you added |class="mediawiki. (I was able to find them on my own, but line numbers can make it easier.)

[elsiehupp]

--delay 1 slows things down pretty dramatically, so you might want to try a smaller fraction of a second.

[elsiehupp]

Aaaaand the delay printout doesn’t display fractional seconds, so I fixed that.