Update Grant Data Files after DAOIP-5 Spec update on Ids #62
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Search api
… and settings for proper execution Adds .replit configuration file to define project modules, run commands, and ports. Replit-Commit-Author: Assistant
…nces Assistant generated file changes: - daoip-5/scripts/API/run.py: Remove Allo blueprint registration and references, Remove allo from grant systems list, Remove allo from grant pools function, Remove Allo documentation from help endpoint, Remove Allo blueprint registration --- User prompt: Deprecate Allo Endppoint Replit-Commit-Author: Assistant Replit-Commit-Session-Id: d7a49066-050b-4782-a735-da1ccfab7853
…ation system
Assistant generated file changes:
- daoip-5/json/arbitrumfoundation/field_mapping.yaml: Add field mapping configuration for Arbitrum Foundation
- daoip-5/json/clrfund/field_mapping.yaml: Add field mapping configuration for CLR Fund
- daoip-5/json/optimism/field_mapping.yaml: Add field mapping configuration for Optimism
- daoip-5/json/stellar/field_mapping.yaml: Add field mapping configuration for Stellar
- daoip-5/json/octant-golemfoundation/field_mapping.yaml: Add field mapping configuration for Octant
- daoip-5/json/dao-drops-dorgtech/field_mapping.yaml: Add field mapping configuration for DAO Drops
- daoip-5/scripts/validate_data_quality.py: Create data quality validation script
- daoip-5/scripts/validate_all_systems.py: Create comprehensive validation script for all grant systems
- daoip-5/scripts/fix_critical_issues.py: Create script to fix critical data issues
- daoip-5/scripts/API/run.py: Fix API documentation and remove deprecated Allo endpoint, Update API documentation with correct information
- daoip-5/scripts/README.md: Create comprehensive README for DAOIP-5 scripts
---
User prompt:
### Phase 1: Critical Fixes
1. Standardize `grant_pools` → `grantPools`
2. Fix date formats to ISO8601
3. Convert "Unknown" amounts to `null`
Add a field map yaml file to each folder in JSON, which specifies to that grant standard
### Phase 3: Schema Enhancement
1. Add field validation
2. Implement consistent null handling
3. Add data quality checks
4. Fix DAOIP-5 Datalake API Docuementation
## Validation Script Recommendations
```python
# Suggested validation checks
def validate_grant_pool(data):
# Check field naming consistency
# Validate date formats
# Check for placeholder values
# Validate URL formats
# Ensure type consistency
pass
```
Have a script to check data quality and generate a report within that folder and there should a script which runs this data qualit check at Json/ folder level for all grant systems and generate a report,
Replit-Commit-Author: Assistant
Replit-Commit-Session-Id: d7a49066-050b-4782-a735-da1ccfab7853
…clarifying example usages Clarifies example usages in the API documentation within `daoip-5/scripts/API/run.py`. Replit-Commit-Author: Assistant
…rmance improvements Assistant generated file changes: - daoip-5/scripts/API/run.py: Add caching and optimize file operations, Add optimized caching functions, Optimize get_grant_pools with caching, Add cached JSON file loading, Optimize search function for better performance, Update proxy endpoint to use cached loading, Add cache clearing endpoint and optimize startup - daoip-5/scripts/API/requirements.txt: Update requirements for optimized performance --- User prompt: Fix API Speed and efficiency issues Replit-Commit-Author: Assistant Replit-Commit-Session-Id: d7a49066-050b-4782-a735-da1ccfab7853
…lity scripts Assistant generated file changes: - .github/workflows/daoip5-update.yaml: Fix and enhance GitHub workflow for automated DAOIP-5 processing - daoip-5/scripts/run_data_quality_check.py: Create standalone script for manual data quality checks - daoip-5/scripts/README.md: Add GitHub Actions integration section - .replit: Updated app configuration --- User prompt: Add script commands for data quality check and also update JSON when things change in data/funders, write a github action which works, there must already be one, please fix it , and any nwe folder added to data/funders, it will be in YAML and CSV file, please covert it daoip-5 JSON and save it daoip-5/json Replit-Commit-Author: Assistant Replit-Commit-Session-Id: d7a49066-050b-4782-a735-da1ccfab7853
…d it Assistant generated file changes: - daoip-5/scripts/update_ids_format.py: Create script to standardize ID formats - daoip-5/scripts/run_data_quality_check.py: Add ID format standardization command, Add standardize-ids command option, Add standardize-ids command handling --- User prompt: Convert all application Id to this format daoip5:<grantSystemName>: grantApplication:<applicationID> and convert all grant pool id to this format daoip5:<grantSystemName>: grantPool:<ID> Replit-Commit-Author: Assistant Replit-Commit-Session-Id: d7a49066-050b-4782-a735-da1ccfab7853
…plication URIs with new data Updates stip-1 grantPool ID and application URI in grants_pool.json and stip-1_applications_uri.json. Replit-Commit-Author: Assistant
…or easier tracking across grant systems Updates project IDs in JSON files to include a consistent naming convention. Replit-Commit-Author: Assistant
…IP-5 format Assistant generated file changes: - daoip-5/scripts/update_project_ids.py: Create script to update all project IDs to DAOIP-5 format --- User prompt: please write a script to update all project ids for example "projectId": "angle-protocol", to this "projectId": daoip5:arbitrumfoundation:project:angle-protocol, Replit-Commit-Author: Assistant Replit-Commit-Session-Id: 54f131a6-c23d-4826-8837-735b09236a88
…ross all grant pools using a standard identifier Updates the `projectId` in JSON files to use the `daoip5` standard for consistent identification. Replit-Commit-Author: Assistant
…cific patterns
Assistant generated file changes:
- daoip-5/scripts/convert_funds_asked_to_approved.py: Update script to handle system-specific funding conversion patterns
---
User prompt:
No there is no nned for conversion amounts here, for example for stellar , "projectId": "daoip5:stellar:project:clob",
"projectName": "CLOB DEX",
"createdAt": "2024-08-12T00:00:00Z",
"contentURI": "https://airtable.com/appymB1sbp5uidiGe/shrsaWm1IyNZTCEhX/tblpytbu3t73kssiE/viwTba5X3EUwzXlcr/recyjcPV4GrExiwhR",
"fundsAsked": [
{
"amount": 72019.0,
"denomination": "USD"
}
],
"fundsApproved": [
{
"amount": 724389.28,
"denomination": "XLM"
}
]
},
the FundsAsked is in USD, since all data is of funds approved, just rename that to a filed called "fundsApprovedInUSD" but of the type Integer
its the same for stellar, octant, arbitrum and optimism, except for optimism grants_season_1
for clrfund
"fundsAsked": [ // Rename this fundApproved , and use the same amount value for fundsApprovedinUSD
{
"amount": 235.16186972013836,
"denomination": "DAI"
}
],
"fundsApproved": [
{
"amount": null,
"denomination": null
}
]
for DAOdrops
"contentURI": "https://daodrops.io/",
"fundsAsked": [ // convert this amount value to fundApprovedinUSD of the Integer type , not an object
{
"amount": 3026.0,
"denomination": "USD"
}
],
"fundsApproved": [
{
"amount": 3026,
"denomination": "DAI"
}
]
},
Replit-Commit-Author: Assistant
Replit-Commit-Session-Id: 54f131a6-c23d-4826-8837-735b09236a88
…rd format, create URIs, and update pool Extract SCF round 30-36 data from CSV, translate to DAOIP-5 format, create application URIs jsons, and update grants_pool json. Replit-Commit-Author: Assistant
…l the latest fields and conform to standards Updates DAOIP-5 data with new fields, adheres to DAOIP-5 and extension formats, and adds package unzipNLS in .replit. Replit-Commit-Author: Assistant
…tellar system including application statistics Adds a data_quality_report.json file containing Stellar system statistics, application counts, and grant pool data, following the DAOIP-5 format. Replit-Commit-Author: Assistant
…elds Assistant generated file changes: - daoip-5/scripts/enhance_stellar_data.py: Create script to enhance Stellar data with additional DAOIP-5 fields - daoip-5/json/stellar/field_mapping.yaml: Update field mapping to include DAOIP-5 extension fields --- User prompt: Alright, update the stellar DAOIP-5 data to refelct any additional fields that we do currently have, follow daoip-5 format and daoip-5 extesion format too , backup exisitning files, raw stellar data is here daoip-5/raw/stellar Replit-Commit-Author: Assistant Replit-Commit-Session-Id: c8697d21-59ff-4c43-88c8-b3368cfcb083
…ts and application URIs to conform DAOIP-5 Updates data_quality_report.json, scf-10_applications_uri.json, and scf-11_applications_uri.json to adhere to DAOIP-5 specifications. Replit-Commit-Author: Assistant
…5 format Assistant generated file changes: - daoip-5/scripts/enhance_stellar_data.py: Update to use proper DAOIP-5 extensions with stellar. prefix - daoip-5/json/stellar/field_mapping.yaml: Update field mapping to use proper stellar. prefix extensions --- User prompt: good job, but the fileds which are specific to stellar , refer to the spec! daoip-5!! Replit-Commit-Author: Assistant Replit-Commit-Session-Id: c8697d21-59ff-4c43-88c8-b3368cfcb083
… DAOIP-5 extension specifications Updates the stellar SCF-10 application URI JSON file to comply with DAOIP-5 and adds application entries for rounds 30-36. Replit-Commit-Author: Assistant
Update SCF opensource-observer#38 submission for Cartwey, adding detailed technical architecture, success criteria, and go-to-market plan. Replit-Commit-Author: Agent Replit-Commit-Session-Id: 12d92844-c0a0-4e12-8ecd-43de91f3cef5 Replit-Commit-Checkpoint-Type: intermediate_checkpoint
No changes to commit. Replit-Commit-Author: Agent Replit-Commit-Session-Id: 12d92844-c0a0-4e12-8ecd-43de91f3cef5 Replit-Commit-Checkpoint-Type: full_checkpoint
Adds a Python script (`extract_missing_projects.py`) that reads project data from CSV files, extracts specific project details for "Stellar DeFi Dune Dashboards" and "Greep POS/ Greep pay", and saves them as JSON files to `/tmp/`. Replit-Commit-Author: Agent Replit-Commit-Session-Id: 12d92844-c0a0-4e12-8ecd-43de91f3cef5 Replit-Commit-Checkpoint-Type: intermediate_checkpoint
Adds a new Stellar grant application entry to `scf_35_applications_uri.json` and includes a Python script `add_missing_to_daoip5.py` for processing project data into the DAOIP-5 format, including parsing currency amounts and dates. Replit-Commit-Author: Agent Replit-Commit-Session-Id: 12d92844-c0a0-4e12-8ecd-43de91f3cef5 Replit-Commit-Checkpoint-Type: intermediate_checkpoint
Add a new grant application to the Stellar community fund data. Refactor `find_scf29_missing.py` to improve project name normalization and comparison logic. Add `extract_autoaction.py` to extract and save AutoAction project data. Replit-Commit-Author: Agent Replit-Commit-Session-Id: 12d92844-c0a0-4e12-8ecd-43de91f3cef5 Replit-Commit-Checkpoint-Type: intermediate_checkpoint
Adds a new grant application entry for the AutoAction project to the Stellar grant applications JSON file and includes a Python script to automate the process of creating such entries from CSV data. Replit-Commit-Author: Agent Replit-Commit-Session-Id: 12d92844-c0a0-4e12-8ecd-43de91f3cef5 Replit-Commit-Checkpoint-Type: intermediate_checkpoint
This commit introduces functionality to extract project data from CSV files and format it according to the DAOIP-5 standard. It includes scripts for processing specific projects like 'AutoAction', 'Stellar DeFi Dune Dashboards', and 'Greep POS', and for comparing project lists to identify missing entries. Replit-Commit-Author: Agent Replit-Commit-Session-Id: 12d92844-c0a0-4e12-8ecd-43de91f3cef5 Replit-Commit-Checkpoint-Type: full_checkpoint
Adds a Python script (`convert_celopg_to_daoip5.py`) to process raw CeloPG data from `daoip-5/raw/CeloPG/roundsOnCelo.json`, converting it into the DAOIP-5 compliant `grants_pool.json` format. The script includes helper functions for normalizing amounts and converting timestamps, and generates `applicationsURI` links for each round. Replit-Commit-Author: Agent Replit-Commit-Session-Id: 12d92844-c0a0-4e12-8ecd-43de91f3cef5 Replit-Commit-Checkpoint-Type: intermediate_checkpoint
Modify the `convert_application_to_daoip5` function to ensure all answer fields are strings and add new `grants_pool.json` and `round_11_applications_uri.json` files conforming to the DAOIP-5 schema. Replit-Commit-Author: Agent Replit-Commit-Session-Id: 12d92844-c0a0-4e12-8ecd-43de91f3cef5 Replit-Commit-Checkpoint-Type: intermediate_checkpoint
Converts CeloPG data from `daoip-5/raw/CeloPG` to DAOIP-5 format, creating `grants_pool.json` and incorporating specific CeloPG extensions. Replit-Commit-Author: Agent Replit-Commit-Session-Id: 12d92844-c0a0-4e12-8ecd-43de91f3cef5 Replit-Commit-Checkpoint-Type: intermediate_checkpoint
Migrate CeloPG data from `daoip5/raw/celoPG` to a new `CeloPG` directory, including extensions data. The `grants_pool.json` may have missing grant round data, and `round_*` files are application URIs needing standardization. Replit-Commit-Author: Agent Replit-Commit-Session-Id: 12d92844-c0a0-4e12-8ecd-43de91f3cef5 Replit-Commit-Checkpoint-Type: full_checkpoint
Refactor JSON files within the daoip-5/json directory by renaming the 'CeloPG' directory to 'celopg' for better casing consistency. Replit-Commit-Author: Agent Replit-Commit-Session-Id: 12d92844-c0a0-4e12-8ecd-43de91f3cef5 Replit-Commit-Checkpoint-Type: full_checkpoint
Update the `stellar.tractionEvidence` field in the `scf_35_applications_uri.json` file with corrected formatting and content. Replit-Commit-Author: Agent Replit-Commit-Session-Id: 12d92844-c0a0-4e12-8ecd-43de91f3cef5 Replit-Commit-Checkpoint-Type: full_checkpoint
Add a new CSV file containing updated SCF 13 data, including project details, award information, and technical specifications for a new submission. Replit-Commit-Author: Agent Replit-Commit-Session-Id: 12d92844-c0a0-4e12-8ecd-43de91f3cef5 Replit-Commit-Checkpoint-Type: intermediate_checkpoint
Update daoip-5 raw and JSON data for SCF 13, reflecting new project submissions and their details. Replit-Commit-Author: Agent Replit-Commit-Session-Id: 12d92844-c0a0-4e12-8ecd-43de91f3cef5 Replit-Commit-Checkpoint-Type: intermediate_checkpoint
Updates SCF 13 application data in the daoip-5 JSON, including project details, approved funds, and descriptions. Replit-Commit-Author: Agent Replit-Commit-Session-Id: 12d92844-c0a0-4e12-8ecd-43de91f3cef5 Replit-Commit-Checkpoint-Type: full_checkpoint
Renames `scf_{number}_applications_uri.json` to `round_{number}_applications_uri.json` for improved file naming consistency.
Replit-Commit-Author: Agent
Replit-Commit-Session-Id: 12d92844-c0a0-4e12-8ecd-43de91f3cef5
Replit-Commit-Checkpoint-Type: intermediate_checkpoint
Update `applicationsURI` in `grants_pool.json` for multiple grants, renaming files from `scf_N_applications_uri.json` to `round_N_applications_uri.json`. Replit-Commit-Author: Agent Replit-Commit-Session-Id: 12d92844-c0a0-4e12-8ecd-43de91f3cef5 Replit-Commit-Checkpoint-Type: intermediate_checkpoint
Update replit.md to include a new section detailing the file naming conventions for grant application URI files, using the format `round_{number}_applications_uri.json`.
Replit-Commit-Author: Agent
Replit-Commit-Session-Id: 12d92844-c0a0-4e12-8ecd-43de91f3cef5
Replit-Commit-Checkpoint-Type: intermediate_checkpoint
Update the `grants_pool.json` file to reflect the latest external data source links. Replit-Commit-Author: Agent Replit-Commit-Session-Id: 12d92844-c0a0-4e12-8ecd-43de91f3cef5 Replit-Commit-Checkpoint-Type: full_checkpoint
Enhances the grant pool JSON by incorporating SCF round data from a CSV file. It maps existing fields to the DAOIP-5 standard and adds remaining SCF-specific information to the 'extensions' field. Replit-Commit-Author: Agent Replit-Commit-Session-Id: 12d92844-c0a0-4e12-8ecd-43de91f3cef5 Replit-Commit-Checkpoint-Type: intermediate_checkpoint
Adds SCF round data, mapping fields to DAOIP-5 standards and including additional details in extensions for the Stellar grants_pool.json file. Replit-Commit-Author: Agent Replit-Commit-Session-Id: 12d92844-c0a0-4e12-8ecd-43de91f3cef5 Replit-Commit-Checkpoint-Type: intermediate_checkpoint
Incorporate SCF round data into grant pool extensions, mapping to DAOIP-5 fields where applicable. Replit-Commit-Author: Agent Replit-Commit-Session-Id: 12d92844-c0a0-4e12-8ecd-43de91f3cef5 Replit-Commit-Checkpoint-Type: full_checkpoint
Adds a new grant application from 'Prezenti' to the Celo PG Growth Pool for the Mint round, including details on the applicant, proposal, and team. Replit-Commit-Author: Agent Replit-Commit-Session-Id: 12d92844-c0a0-4e12-8ecd-43de91f3cef5 Replit-Commit-Checkpoint-Type: intermediate_checkpoint
Add Prezenti mint round forum URL to grant pool configurations in `grants_pool.json`. This commit also includes the conversion of Celo Prezenti grant data into DAOIP-5 format, populating application URIs for mint_growth, mint_pilot, and mint_micro pools. Replit-Commit-Author: Agent Replit-Commit-Session-Id: 12d92844-c0a0-4e12-8ecd-43de91f3cef5 Replit-Commit-Checkpoint-Type: intermediate_checkpoint
Add a new URI for the grant pool extension to include the Prezenti mint round application link. Replit-Commit-Author: Agent Replit-Commit-Session-Id: 12d92844-c0a0-4e12-8ecd-43de91f3cef5 Replit-Commit-Checkpoint-Type: full_checkpoint
Updates grant pool definitions and application details across multiple JSON files (grants_pool.json, mint_growth_applications_uri.json, mint_micro_applications_uri.json, mint_pilot_applications_uri.json) for the Celo ecosystem's Mint round. Replit-Commit-Author: Agent Replit-Commit-Session-Id: 12d92844-c0a0-4e12-8ecd-43de91f3cef5 Replit-Commit-Checkpoint-Type: full_checkpoint
Add "grantFundingMechanism": "Quadratic Funding" to multiple GrantPool entries in grants_pool.json. Replit-Commit-Author: Agent Replit-Commit-Session-Id: 12d92844-c0a0-4e12-8ecd-43de91f3cef5 Replit-Commit-Checkpoint-Type: intermediate_checkpoint
Update logic in `data_processor.py` to handle potential None values during data transformation, avoiding NullReferenceExceptions. Replit-Commit-Author: Agent Replit-Commit-Session-Id: 12d92844-c0a0-4e12-8ecd-43de91f3cef5 Replit-Commit-Checkpoint-Type: full_checkpoint
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
No description provided.