Beginners Documentation or Wiki for WhatsApp-Key-DB-Extractor

[connect-mahdi]

@KnugiHK

Hi, I find your projects awesome !
After an extensive search, I couldn't find any other FOSS project to get one's sovereignty back from WhatsApp on our data.

I am very interested in combining the use of WhatsApp-Key-DB-Extractor and WhatsApp-Chat-Exporter

Part 1 - WhatsApp-Key-DB-Extractor
While being a privacy advocate, I have very view technical skills. I've read the README file and can't quite understand some steps.
On WhatsApp-Chat-Exporter there is a detailed Wiki, but not on this project.

Could you and other contributors help me undertake more detailed step-by-step instructions to decrypt the WhatsApp database ?

For the Android and Linux use case:
I could then provide some text basis, even verbose, for a wiki you might add to this project, like you did on WhatsApp-Chat-Exporter.
It would certainly help A LOT of people like me

WhatsApp data is in
Android Huawei P Smart 2019, running EMUI 12 (it might be Android 10 or 11).
No modification and a non-rooted device

Folder WhatsApp: storage/emulated/0/Android/media/com.whatsapp around 14Gib
Question: there are multiple msgstore* files:

• one file: msgstore.db.crypt14
• several files: msgstore-increment*

PC is a Fedora Live Workstation 41 which can be installed within minutes following: https://getfedora.org

I wish if possible not to use my current Google account (which is almost full and only used to install historically WhatsApp from Google Play)
I have a cloud backup of my data on a separated account as per your caution.

My WhatsApp account is mono-device. I haven't set multi-device (yet) for example not yet to have a multi-device account on a Lenovo Android tablet

Those are the steps I can't figure from your README file

By default, this script use the old WhatsApp APK from WhatsApp.com (in Web Archive), use it at your own risk.

It might be more explanatory to indicate here what's on the introduction of the script:

If your WhatsApp version is greater than 2.11.431 (most likely), 
then a legacy version will be installed temporarily in order to get backup 
permissions

So we can better understand how it relates to the later PREREQUISITES and INSTRUCTIONS

On Android:

• USB Debugging is enabled
• Connect Android device to PC via USB
• USB Debugging is still enabled: after connecting via USB, this options sometimes needs to be reactivated
• Accept PC's RSA key for ADB debugging on Android device
• Optional: If a more elaborate command is needed to populate the WhatsApp exporter with the Android's device contacts, those have to be extracted first. Fossify Contacts App is used here to extract all contacts in .vcf format . But any App which can extract in .vcf format would do, I gess (and I don't know any FOSS Contacts App besides this one, feel free to enlighten me !)

On Fedora Workstation:
I've installed the following to comply with your PREREQUISITES
Java: https://www.java.com/en/download/help/linux_install.html#rpm redirects to Java for Fedora
I found just easier to use fedora provided or related repositories

Enabling repositories
(inspired from https://github.com/mrrfv/open-android-backup#linux thanks to mrrfv !)

sudo dnf install -y https://mirrors.rpmfusion.org/free/fedora/rpmfusion-free-release-$(rpm -E %fedora).noarch.rpm https://mirrors.rpmfusion.org/nonfree/fedora/rpmfusion-nonfree-release-$(rpm -E %fedora).noarch.rpm
sudo dnf config-manager setopt fedora-cisco-openh264.enabled=1
sudo dnf install -y https://github.com/rpmsphere/noarch/raw/master/r/rpmsphere-release-40-1.noarch.rpm

Installing packages

Java
sudo dnf install java which is around 204/300 MiB

ADB (Android Debug Bridge) Drivers environment
sudo dnf install -y p7zip p7zip-plugins adb curl newt pv secure-delete

INSTRUCTIONS:

• Downloading script: from https://github.com/KnugiHK/WhatsApp-Key-DB-Extractor/releases/tags
https://github.com/KnugiHK/WhatsApp-Key-DB-Extractor/archive/refs/tags/v5.4.zip would download WhatsApp-Key-DB-Extractor-5.4.zip

• Extract files (here in $HOME/Downloads) to a WhatsApp-Key-DB-Extractor-5.4 folder with File Roller or Nautilus
  /home/liveuser/Downloads/WhatsApp-Key-DB-Extractor-5.4

Note:
WhatsAppKeyDBExtract.sh needs to be executed within the extracted folder

Else the download and installation of legacy Whatsapp will ultimately fail:
Warning: Failed to open the file tmp/LegacyWhatsApp.apk: No such file or directory
[FATAL] Curl failed with 23
the script would continue including uninstalling the device's original WhatsApp App
but will fail: adb: failed to stat tmp/LegacyWhatsApp.apk: No such file or directory

In command line:
$ cd /home/liveuser/Downloads/WhatsApp-Key-DB-Extractor-5.4
$ source /home/liveuser/Downloads/WhatsApp-Key-DB-Extractor-5.4/WhatsAppKeyDBExtract.sh

Follow the scripts instructions...
Q: What about the warnings for an upcoming depreciated adb backup command ?

Anyhow, when the script is finished, command line window closes
Open a new window:

$ cd /home/liveuser/Downloads/WhatsApp-Key-DB-Extractor-5.4
$ ls -lh extracted/
total 200M
-rw-r-----. 1 liveuser liveuser 2.5M Feb 20 01:29 axolotl.db
-rw-r-----. 1 liveuser liveuser 36K Feb 20 01:29 chatsettings.db
-rw-------. 1 liveuser liveuser 196M Feb 20 01:29 msgstore.db
-rw-r-----. 1 liveuser liveuser 1.2M Feb 20 01:29 wa.db
-rw-------. 1 liveuser liveuser 158 Feb 20 01:29 whatsapp.cryptkey

...TA DAA...
Q: Could you elaborate on the uses of all the files extracted above ?
axolotl.db chatsettings.db msgstore.db wa.db whatsapp.cryptkey

What kind of data is in those files ? What should be saved or thrown away or might contain private data ?

Part 2 - WhatsApp-Chat-Exporter.
I've followed Fedora Project recommendations for a python3 virtual environment called WhatsApp-Chat-Exporter_venv:
https://developer.fedoraproject.org/tech/languages/python/pypi-installation.html

The best practice is using pip in the virtual environment. It will keep all modules for one project at one place and it will not break your local system. Another advantage is that you can have more versions of the same module in different virtual environments.

$ python3 -m venv WhatsApp-Chat-Exporter_venv

If you want to work in the virtual environment, you have to activate it.

$ source WhatsApp-Chat-Exporter_venv/bin/activate
(WhatsApp-Chat-Exporter_venv) $ python -m pip install whatsapp-chat-exporter
Collecting whatsapp-chat-exporter
Downloading whatsapp_chat_exporter-0.11.2-py3-none-any.whl.metadata (15 kB)
Collecting jinja2 (from whatsapp-chat-exporter)
Downloading jinja2-3.1.5-py3-none-any.whl.metadata (2.6 kB)
Collecting bleach (from whatsapp-chat-exporter)
Downloading bleach-6.2.0-py3-none-any.whl.metadata (30 kB)
Collecting webencodings (from bleach->whatsapp-chat-exporter)
Downloading webencodings-0.5.1-py2.py3-none-any.whl.metadata (2.1 kB)
Collecting MarkupSafe>=2.0 (from jinja2->whatsapp-chat-exporter)
Downloading MarkupSafe-3.0.2-cp313-cp313-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (4.0 kB)
Downloading whatsapp_chat_exporter-0.11.2-py3-none-any.whl (48 kB)
Downloading bleach-6.2.0-py3-none-any.whl (163 kB)
Downloading jinja2-3.1.5-py3-none-any.whl (134 kB)
Downloading MarkupSafe-3.0.2-cp313-cp313-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (23 kB)
Downloading webencodings-0.5.1-py2.py3-none-any.whl (11 kB)
Installing collected packages: webencodings, MarkupSafe, bleach, jinja2, whatsapp-chat-exporter
Successfully installed MarkupSafe-3.0.2 bleach-6.2.0 jinja2-3.1.5 webencodings-0.5.1 whatsapp-chat-exporter-0.11.2

[notice] A new release of pip is available: 24.2 -> 25.0.1
[notice] To update, run: pip install --upgrade pip
(WhatsApp-Chat-Exporter_venv) $
(WhatsApp-Chat-Exporter_venv) $ mkdir working_wts
(WhatsApp-Chat-Exporter_venv) $ cd working_wts/

With adb is still active so we can pull the files and get the folder structure as required in Unencrypted WhatsApp database

Reminder: you are in a virtual environment (WhatsApp-Chat-Exporter_venv) and inside the working_wts folder

$ pwd
/home/liveuser/working_wts

Pulling the WhatsApp folder and files from the source Android device:
$ adb pull storage/emulated/0/Android/media/com.whatsapp/WhatsApp/

storage/emulated/0/Android/media/com.whatsapp/WhatsApp/: 13000 files pulled, 0 skipped. 22.4 MB/s (13823659717 bytes in 589.459s)

Now ls command should show you your working folder content compliant with the requirement for Whatsapp-Chat-Exporter scripts to work on the Unencrypted WhatsApp database: see here

$ ls
wa.db WhatsApp whatsapp.cryptkey

We can now launch in the working folder the simple command $ wtsexporter -a

If a more elaborate command is needed to populate the exporter with the Androids contacts, those have to be extracted first. Steps that might work:

• On the Android source device, I've installed an active, quality FOSS Contacts App : Fossify Contacts.
• Start the app, grant permissions to the Contacts
• Go to Settings, Export all Contacts, choose whatever account. I guess WhatsApp account and the main Account would suffice. I've extracted all accounts. Export to a location on the device i.e. bluetooth. Then adb pull the file. i.e.
  $ adb pull storage/emulated/0/bluetooth/Fossify_Export_for_WhatsApp-DB-Explorer_contacts_2025_02_20_04_04_45.vcf /run/media/liveuser/0C2E-ECF3/WhatsApp-Chat-Exporter/

storage/emulated/0/bluetooth/Fossify_Export_for_WhatsApp-DB-Explorer_contacts_2025_02_20_04_04_45.vcf: 1 file pulled, 0 skipped. 1.2 MB/s (2643471 bytes in 2.081s)

• You might want to provide you prefered default-country-code but that did not help very much.

Thus the simple command $ wtsexporter -a becomes more elaborate:

$ wtsexporter -a --enrich-from-vcard Fossify_Export_for_WhatsApp-DB-Explorer_contacts_2025_02_20_04_04_45.vcf --default-country-code 33

No contacts profiles found in the default database, consider using --enrich-from-vcards for adopting names from exported contacts from Google
####### You can ignore the latest comment, our command is tailored to enrich the exporter with the names from the exported contacts

Processing messages...(136878/136878)
Processing media...(16545/16545)
Processing vCards...(92/92)
Processing calls...(2765)
Generating chats...(522/522)
Copying media directory...

The last part will take A VERY LONG TIME and A LOT OF DISK SPACE

Then again... Works like a charm... thank you @KnugiHK !!!!

[KnugiHK]

Detailed documentation

Thanks for the information provided! I'm sure a more detailed usage can be provided, and will see how the information can be incorporated into the documentation. However, covering every operating system or distro may not be practical, as it would take considerable effort.

Additionally, the scripts were never intended for novice end-users. They simply automate tasks you'd typically perform manually, and there are risks involved, such as potential data loss or issues with the WhatsApp app on your phone. It requires a certain level of skill, at least basic knowledge of ADB or the ability to look up relevant information on Google to account for those issues should they occur.

As for your questions

Those are the steps I can't figure from your README file

By default, this script use the old WhatsApp APK from WhatsApp.com (in Web Archive), use it at your own risk.

It might be more explanatory to indicate here what's on the introduction of the script:

This is a warning about the source of the APK being installed via this script, as it does not come from WhatsApp's official website. While the risk of using an APK from the Wayback Machine is minimal, users should still be aware of the security implications and make an informed decision on whether to proceed. This warning is here for that reason. If you're willing to accept the risk, you can ignore it. Since this is unrelated to the script itself, additional documentation on this matter isn't necessary.

Q: Could you elaborate on the uses of all the files extracted above ?
axolotl.db chatsettings.db msgstore.db wa.db whatsapp.cryptkey

• axolotl.db: Manages encryption keys for WhatsApp and is unrelated to WhatsApp-Chat-Exporter.
• chatsettings.db: Stores chat settings such as muting, with no connection to WhatsApp-Chat-Exporter.
• msgstore.db: The main message database and the most important file for WhatsApp-Chat-Exporter.
• wa.db: Contains contact information, helpful for WhatsApp-Chat-Exporter, although it appears to no longer be populated.
• whatsapp.cryptkey: The encryption key for crypt12 and crypt14 backups, as well as your account key. This is required when using WhatsApp-Chat-Exporter with crypt12/14 backups.

Q: What about the warnings for an upcoming depreciated adb backup command?

Well, there are no plans to address this at the moment, unless, to my surprise, things change. See also the maintenance notes below.

Maintenance notes

Btw, since this method no longer works on Android 14+ and older versions will only receive security updates, I expect the usage of these scripts to decline over time (as fewer users will be able to extract WhatsApp files using this method) and likely fewer code changes are required to adapt the changes in Android. Therefore, I plan to discontinue maintaining the scripts in the future, though I don't have a specific timeline for when that will occur. Perhaps next year? In 5 years? 10 years? Honestly, I've no idea.

In case you can't extract the files using this scripts and you still want to retrieve chat history using WhatsApp-Chat-Exporter, you can create a crypt15 (end-to-end) backup, provide the decryption key to the exporter, and it will work like a charm.👍

[connect-mahdi]

@KnugiHK

I'm very grateful for your answers and insights.

I've edited my comment, and I'll try to update it in a How-to / Tutorial style.
If possible I'll try using a generic Linux use case (Fedora/rpm and Ubuntu/deb) and a Windows case
I'll do my best to provide screenshots, as for now they are in French, so it won't be very helpful :-)

I'll switch to your other project for further comments and proposals since it regards the WhatsApp-Chat-Exporter

Do you have any knowledge of a similar and reliable way to extract and export to a browser readable client for Signal App or Telegram ?

[KnugiHK]

Do you have any knowledge of a similar and reliable way to extract and export to a browser readable client for Signal App or Telegram ?

There are many projects available for Signal backup, which you can explore here: https://github.com/search?q=Signal%20backup&type=repositories.

As for Telegram, the official apps already offer a robust solution for exporting chats. In fact, this project was inspired by Telegram's features.

[KnugiHK]

If possible, you could create a TUTORIAL.md and make a PR (still subject to a review). This way, you'll be properly credited.

[connect-mahdi]

If possible, you could create a TUTORIAL.md and make a PR (still subject to a review). This way, you'll be properly credited.

Thank you for your kind proposal @KnugiHK . I would gladly do so. Sorting all the step might help others. This may require some time though on my side, least of it I do not know how to make a pull request nor upload a PR proposal with a .md file associated with screenshots

[KnugiHK]

Thank you for your kind proposal @KnugiHK . I would gladly do so. Sorting all the step might help others. This may require some time though on my side, least of it I do not know how to make a pull request nor upload a PR proposal with a .md file associated with screenshots

No rush! If I need to create a beginner's tutorial, it will be a low priority compared to other features and maintenance efforts. So, you'll have plenty of time to work on it!