Nix flake support

.gitattributes

@@ -1,2 +1,12 @@
 *.com linguist-vendored
 *.org linguist-vendored
+
+*.py text=auto
+*.sh text=auto
+*.json text=auto
+*.txt text=auto
+*.md text=auto
+
+*.html linguist-detectable=false
+
+wikiteam3/dumpgenerator/test/data/* linguist-vendored

.github/workflows/nix.yml

@@ -0,0 +1,34 @@
+name: Nix Build and Check
+
+on:
+  push:
+    branches: [ main, v4-main ]
+  pull_request:
+    branches: [ main, v4-main ]
+
+jobs:
+  nix-build:
+    runs-on: ubuntu-latest
+    steps:
+      - uses: actions/checkout@v4
+
+      - name: Install Nix
+        uses: cachix/install-nix-action@v27
+        with:
+          extra_nix_config: |
+            experimental-features = nix-command flakes
+
+      - name: Check flake
+        run: nix flake check
+
+      - name: Build package
+        run: nix build
+
+      - name: Test wikiteam3dumpgenerator
+        run: |
+          nix run . -- --help
+
+      - name: Verify reproducibility
+        run: |
+          nix build --rebuild
+          nix path-info --json | jq -r '.[].narHash'

.github/workflows/test-dumogenerator.yml

@@ -0,0 +1,43 @@
+# This workflow will install Python dependencies, run tests and lint with a variety of Python versions
+# For more information see: https://docs.github.com/en/actions/automating-builds-and-tests/building-and-testing-python
+
+name: dumogenerator test
+
+on:
+  push:
+  pull_request:
+
+jobs:
+  build:
+
+    runs-on: ubuntu-latest
+    strategy:
+      fail-fast: false
+      matrix:
+        python-version: ["3.9", "3.12", "3.14"]
+
+    steps:
+    - uses: actions/checkout@v4
+    - name: Set up Python ${{ matrix.python-version }}
+      uses: actions/setup-python@v5
+      with:
+        python-version: ${{ matrix.python-version }}
+    - name: Install dependencies
+      run: |
+        sudo apt update
+        sudo apt install -y libxml2-dev libxslt-dev # for lxml
+        python -m pip install --upgrade pip
+        python -m pip install flake8 pytest
+        pip install .
+    - name: Lint with flake8
+      run: |
+        # exit if there are Python syntax errors or undefined names
+        flake8 . --count --select=E9,F63,F7,F82 --show-source --statistics
+        # exit-zero treats all errors as warnings. The GitHub editor is 127 chars wide
+        flake8 . --count --exit-zero --max-complexity=10 --max-line-length=127 --statistics
+    - name: run dumpgenerator
+      run: |
+        python -m wikiteam3.dumpgenerator -h
+    - name: Test with pytest
+      run: |
+        pytest

.gitignore

@@ -1,9 +1,15 @@
 *.pyc
-testing/*.pyc
-testing/dumpgenerator.py
-/.tox
+.pytest_cache
 keys.txt
 batchdownload/keys.txt
 batchdownload/dumpgenerator.py
 batchdownload/uploader.py
+__pycache__
+tests/tmp
+dist/
+.DS_Store
+desktop.ini
 
+.venv
+.vscode
+.idea

.pdm-python

@@ -0,0 +1 @@
+/home/yzqzss/git/wikiteam3/.venv/bin/python

DEV.md

@@ -0,0 +1,89 @@
+# WikiTeam3 internal
+
+## Images.txt structure
+
+```python
+filename + "\t" + url + "\t" + uploader
++ "\t" + (str(size) if size else NULL)
++ "\t" + (str(sha1) if sha1 else NULL)
++ "\t" + (timestamp if timestamp else NULL)
++ "\n"
+```
+
+*optional fields:
+- "null" (magic None value, since wikiteam3 v4.0.0)
+- "False" (magic None value, before wikiteam3 v4.0.0)
+- not present (ancient wikiteam3 versions)
+
+# Snippets
+
+## API Output format
+
+https://www.mediawiki.org/wiki/API:Data_formats#Output
+
+> The standard and default output format in MediaWiki is JSON. All other formats are discouraged.
+> 
+> The output format should always be specified using format=yourformat with yourformat being one of the following:
+> 
+>     json: JSON format. (recommended)
+>     php: serialized PHP format. (deprecated)
+>     xml: XML format. (deprecated)
+>     txt: PHP print_r() format. (removed in 1.27)
+>     dbg: PHP var_export() format. (removed in 1.27)
+>     yaml: YAML format. (removed in 1.27)
+>     wddx: WDDX format. (removed in 1.26)
+>     dump: PHP var_dump() format. (removed in 1.26)
+>     none: Returns a blank response. 1.21+
+
+In our practice, `json` is not available for some old wikis.
+
+## Allpages
+
+https://www.mediawiki.org/wiki/API:Allpages (>= 1.8)
+
+
+## Allimages
+
+https://www.mediawiki.org/wiki/API:Allimages (>= 1.13)
+
+## Redirects
+
+https://www.mediawiki.org/wiki/Manual:Redirect_table
+
+## Logs
+
+https://www.mediawiki.org/wiki/Manual:Logging_table
+
+## Continuation
+
+https://www.mediawiki.org/wiki/API:Continue (≥ 1.26)
+https://www.mediawiki.org/wiki/API:Raw_query_continue (≥ 1.9)
+
+> From MediaWiki 1.21 to 1.25, it was required to specify continue= (i.e. with an empty string as the value) in the initial request to get continuation data in the format described above. Without doing that, API results would indicate there is additional data by returning a query-continue element, explained in Raw query continue.
+> Prior to 1.21, that raw continuation (`query-continue`) was the only option.
+> 
+> If your application needs to use the raw continuation in MediaWiki 1.26 or later, you must specify rawcontinue= to request it. 
+
+# Workarounds
+
+## truncated API response causes infinite loop
+
+https://github.com/mediawiki-client-tools/mediawiki-dump-generator/issues/166
+https://phabricator.wikimedia.org/T86611
+
+wikiteam3 workaround: https://github.com/saveweb/wikiteam3/commit/76465d34898b80e8c0eb6d9652aa8efa403a7ce7
+
+## MWUnknownContentModelException
+
+> "The content model xxxxxx is not registered on this wiki;"
+
+Some extensions use custom content models for their own purposes, but they did not register a handler to export their content.
+
+wikiteam3 workaround: https://github.com/saveweb/wikiteam3/commit/fd5a02a649dcf3bdab7ac1268445b0550130e6ee
+
+## Insecure SSL
+
+https://docs.openssl.org/1.1.1/man1/ciphers/
+https://docs.openssl.org/master/man1/openssl-ciphers/
+
+wikiteam3 workaround: https://github.com/saveweb/wikiteam3/blob/8a054882de19c6b69bc03798d3044b7b5c4c3c88/wikiteam3/utils/monkey_patch.py#L63-L84