Persistent storage solution

.env.example

@@ -1,5 +1,13 @@
-DISCORD_BOT_TOKEN=MTE3
-ETH_ADDRESS=0x5B3eADC3
-ETH_PRIVATE_KEY=0x834q5nmsgkseg4wg4
-PRIMARY_RPC_URL=https://sepolia.drpc.org
-SECONDARY_RPC_URL=https://1rpc.io/sepolia
+DISCORD_BOT_TOKEN=your_discord_bot_token_here
+ETH_ADDRESS=your_eth_address
+ETH_PRIVATE_KEY=your_eth_private_key
+PRIMARY_RPC_URL=your_primary_RPC-url
+SECONDARY_RPC_URL=your_secondary_rpc_url
+
+# Database Configuration
+DB_USER=bloom
+DB_PASSWORD=bloom
+DB_NAME=bloombot
+
+# Environment
+ENV=DEV

.github/workflows/test.yml

@@ -0,0 +1,29 @@
+name: Tests
+
+on: [push, pull_request]
+
+jobs:
+  test:
+    runs-on: ubuntu-latest
+
+    steps:
+    - uses: actions/checkout@v2
+
+    - name: Set up Python
+      uses: actions/setup-python@v2
+      with:
+        python-version: '3.11'
+
+    - name: Install pipenv
+      run: |
+        python -m pip install --upgrade pip
+        pip install pipenv
+
+    - name: Install dependencies
+      run: pipenv install --dev
+
+    - name: Run tests
+      env:
+        DB_PASSWORD: test_password
+        ENV: TEST
+      run: pipenv run python -m pytest tests/ -v

.gitignore

@@ -1,3 +1,6 @@
+#env
+.env
+
 # Custom
 testing.py
 

CHANGELOG.md

@@ -7,6 +7,12 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
 
 ## [Unreleased]
 
+### Changed
+- Migration script from JSON file storage to PostgreSQL database
+- Added database models for Contributors, Events, OngoingVotes, and ConcludedVotes
+- Updated configuration to use environment variables for database credentials
+- Added database query script for viewing stored data
+
 ## [0.2.1] - 28-2-2024
 
 ### Added

Dockerfile

@@ -1,25 +1,19 @@
 FROM python:3.11
 
-# Install pipenv and compilation dependencies
 RUN pip install pipenv
 RUN apt-get update && apt-get install -y --no-install-recommends gcc
 
-# Install Node.js and npm
 RUN curl -sL https://deb.nodesource.com/setup_18.x | bash -
 RUN apt-get install -y nodejs
 
 WORKDIR /app
 
-# Copy Pipfile and install dependencies
 COPY Pipfile .
 RUN pipenv install --deploy
 
-# Copy JavaScript dependencies files and install
 COPY snapshot/package*.json /app/snapshot/
 RUN cd /app/snapshot && npm install
 
-# Copy the entire application
 COPY . /app
 
-# Command to run the application
-CMD ["pipenv", "run", "python", "main.py"]
+CMD ["pipenv", "run", "python", "main.py"]

Pipfile

@@ -5,12 +5,15 @@ name = "pypi"
 
 [packages]
 "discord.py" = "*"
-"pytz" = "*"
-"requests" = "*"
-"configparser" = "*"
-"Web3" = "*"
+requests = "*"
+configparser = "*"
+web3 = "*"
+sqlalchemy = "*"
+psycopg2-binary = "*"
 
 [dev-packages]
+pytest = "*"
+pytest-asyncio = "*"
 
 [requires]
 python_version = "3.11"

README.md

@@ -255,49 +255,65 @@ Contributor added successfully!
 
 # Manually publishing proposals via the bot
 
-In the event that the bot goes down, or there is some other issue with a proposal being pushed to Snapshot, the following steps can be taken. These steps can be done locally, utilizing the test server
+In the event that the bot goes down, or there is some other issue with a proposal being pushed to Snapshot, the following steps can be taken. These steps can be done locally, utilizing the test server:
 
 1. Replicate the proposal locally with the bot (Copy / Paste the text & Markdown from the posted proposal in Discord)
 
-2. Create / Publish the proposal. by using /vote_draft & /publish_draft locally.
+2. Create / Publish the proposal by using /vote_draft & /publish_draft locally.
 
 3. Stop the local running of the bot in the test server
 
-4. Access ./data/ongoing_votes.json, copy the proposal object
-
-**Example:**
-
+4. Use the query_db.py script to verify the proposal data:
+```bash
+python scripts/query_db.py --table ongoing_votes
 ```
 
+Example proposal data structure:
+```python
 {
-    "1232974623113744386": {
-        "draft": {
-            "member_id": 316765092689608706,
-            "title": "Annual BVI Services Fee",
-            "type": "budget",
-            "abstract": "**Authors**\nSarahtonein\n\n**Type**\nImbursement\n\n**Definitions**\n**British Virgin Islands LTD (BVI LTD):** Refers to a limited liability company formed and registered in the British Virgin Islands.\n\n**Flag Theory:** A strategic framework used for globalizing personal and business operations. The term refers to the service provider engaged by Bloom Studio to assist in the establishment, maintenance, and legal compliance of the BVI LTD.\n\n**EVM Address:** Ethereum Virtual Machine Address. A unique identifier associated with the blockchain wallet used for transactions.\n\n**Abstract**\nThis proposal seeks approval for the annual service fee payment to Flag Theory, essential for the maintenance and compliance of the British Virgin Islands LTD associated with Bloom Studio. The document details the service fee amount and the necessary payment details, providing a basis for members to vote on the continued support of these essential corporate services.",
-            "background": "**Background**\nBloom Studio has engaged Flag Theory for the establishment and ongoing maintenance of a corporate entity in the British Virgin Islands (BVI). This strategic partnership ensures compliance with BVI legal and regulatory requirements, which is crucial for maintaining our corporate presence and operational efficiency in the region. To uphold these standards and continue benefiting from Flag Theory's expert services, an annual payment is necessary. This proposal outlines the specifics of the payment and the value derived from this continued collaboration.\n\n**Imbursement Specifics**\n- Amount - $1925\n- EVM Address - 0x9970ff60d7924d4cbaa161c6334fcab9fd859050\n- Invoice details - https://bafybeidujjzbmtiwxhmgpzlxf6bj4gt736vb4ojyhgq6s2qsxkfwew3eie.ipfs.w3s.link/Invoice_INV-7037.pdf",
-            "additional": "**Voting Options**\n```\nAdopt\nReassess\nAbstain\n```"
-        },
-        "end_time": 1713962888,
-        "yes_count": 0,
-        "title": "Bloom Budget Proposal (BBP) #2: Annual BVI Services Fee",
-        "channel_id": "1191283982768287795",
-        "thread_id": "1232152965490413630",
-        "message_id": "1232152968225357895"
-    }
+    "proposal_id": "1232974623113744386",
+    "draft": {
+        "member_id": 316765092689608706,
+        "title": "Annual BVI Services Fee",
+        "type": "budget",
+        "abstract": "...",
+        "background": "...",
+        "additional": "..."
+    },
+    "end_time": 1713962888,
+    "title": "Bloom Budget Proposal (BBP) #2: Annual BVI Services Fee",
+    "channel_id": "1191283982768287795",
+    "thread_id": "1232152965490413630",
+    "message_id": "1232152968225357895"
 }
-
 ```
 
-5. Modify the end_time value so that it is *any* time in the past as long as it is 48+ hours from when the proposal that failed to be published concluded. 
+5. If needed, you can modify the proposal data using the database CLI:
+   - Modify `end_time` to any time in the past (48+ hours from when the proposal concluded)
+   - Update channel_id to match the correct channel (improvement-props: 1191283932096905307 or budgetary-props: 1191283982768287795)
+   - Update `thread_id` and `message_id` to match the Discord thread
 
-6. Modify channel_id, thread_id, message_id so that it matches the channel ID (improvement-props OR budgetary-props 1191283932096905307 and 1191283982768287795 respectively), thread_id (the thread_id of the thread the discord bot created within the forum channel), and the message_id (the message the bot sent that contains the voting options (yes / reasses / abstain))
+6. Build & run the bot
 
-7. Modify the BGP / BBP # under the title property, as it will be a default of 0 most likely. Check the Discord thread for the BGP / BBP # if you do not know what number we are up to. 
+The bot will run the scheduled task check_concluded_proposals_task in 5 minutes time. If all has been done correctly, you should see:
+- The concluded message in the bot logs
+- The conclusion message under the Discord proposal thread
+- The proposal created in Snapshot
 
-8. Copy & paste the updated json object into the VM repository ./data/ongoing_votes.json
+## Database Environment Setup
+
+Required environment variables:
+- `DB_PASSWORD`: Database password for PostgreSQL
+- `ENV`: Set to "DEV" for testing/development
+
+Example `.env`:
+```bash
+DB_PASSWORD=your_password_here
+ENV=DEV
+```
 
-9. Build & run the bot
+The bot connects to PostgreSQL using:
+- `DATABASE_URL` if provided (e.g., Heroku's database URL)
+- Default URL `postgresql://bloom:{DB_PASSWORD}@localhost:5432/bloombot` if only `DB_PASSWORD` is set
 
-The bot should run the scheduled task check_concluded_proposals_task in 5 minutes time, if all has been done correctly you should see the concluded message in the bot logs, and under the thread that has the Discord proposal. You should also see the proposal created in Snapshot. 
+Note: PostgreSQL is required for both development and production environments. Only test suites use SQLite in-memory databases.