Skip to content
Permalink

Comparing changes

Choose two branches to see what’s changed or to start a new pull request. If you need to, you can also or learn more about diff comparisons.

Open a pull request

Create a new pull request by comparing changes across two branches. If you need to, you can also . Learn more about diff comparisons here.
base repository: Placekey/placekey-py
Failed to load repositories. Confirm that selected base ref is valid, then try again.
Loading
base: v0.0.18
Choose a base ref
...
head repository: Placekey/placekey-py
Failed to load repositories. Confirm that selected head ref is valid, then try again.
Loading
compare: main
Choose a head ref
  • 20 commits
  • 8 files changed
  • 3 contributors

Commits on Jun 18, 2024

  1. Added better handling for occurrence of 504 (#39)

    * Added better handling for occurance of 504
    
    * Added explicit on_exception for RequestException
    
    * Added 503 and 504 status handling
    
    * Updated json parse
    jarredSafegraph authored Jun 18, 2024

    Verified

    This commit was created on GitHub.com and signed with GitHub’s verified signature.
    Copy the full SHA
    589cbba View commit details
  2. Update __version__.py

    jarredSafegraph authored Jun 18, 2024

    Verified

    This commit was created on GitHub.com and signed with GitHub’s verified signature.
    Copy the full SHA
    abc50bb View commit details
  3. Updated Bulk Request Limit to reflect 1k upper bound

    jarredSafegraph authored Jun 18, 2024

    Verified

    This commit was created on GitHub.com and signed with GitHub’s verified signature.
    Copy the full SHA
    42d4275 View commit details
  4. Update __version__.py

    jarredSafegraph authored Jun 18, 2024

    Verified

    This commit was created on GitHub.com and signed with GitHub’s verified signature.
    Copy the full SHA
    945ab66 View commit details

Commits on Nov 23, 2024

  1. Locking h3 version on supported 3.6.1 to 3.7.7 (#42)

    * Locking h3 version on supported 3.6.1 to 3.7.7
    
    * Fix test
    jarredSafegraph authored Nov 23, 2024

    Verified

    This commit was created on GitHub.com and signed with GitHub’s verified signature.
    Copy the full SHA
    8584f1c View commit details
  2. Update __version__.py

    inc
    jarredSafegraph authored Nov 23, 2024

    Verified

    This commit was created on GitHub.com and signed with GitHub’s verified signature.
    Copy the full SHA
    b3f0363 View commit details
  3. Update __version__.py

    jarredSafegraph authored Nov 23, 2024

    Verified

    This commit was created on GitHub.com and signed with GitHub’s verified signature.
    Copy the full SHA
    1247e8e View commit details

Commits on Jan 28, 2025

  1. Giving users access to free datasets

    sindhu-ranga committed Jan 28, 2025
    Copy the full SHA
    0a9f08f View commit details
  2. Revert "Giving users access to free datasets"

    This reverts commit 0a9f08f.
    sindhu-ranga committed Jan 28, 2025
    Copy the full SHA
    060c422 View commit details

Commits on Jan 29, 2025

  1. Give users ability to get free-datasets locations (#44)

    * Give users ability to get free-datasets locations
    
    * Update python-package.yml
    sindhu-ranga authored Jan 29, 2025

    Verified

    This commit was created on GitHub.com and signed with GitHub’s verified signature.
    Copy the full SHA
    9ac72ce View commit details
  2. Update README.md

    sindhu-ranga committed Jan 29, 2025
    Copy the full SHA
    b95debe View commit details

Commits on Jan 31, 2025

  1. Updated list and return free datasets location to read from S3 instea…

    …d of hard-coded values (#45)
    sindhu-ranga authored Jan 31, 2025

    Verified

    This commit was created on GitHub.com and signed with GitHub’s verified signature.
    Copy the full SHA
    6962482 View commit details
  2. Fixed boto3 dependency installation issue (#46)

    sindhu-ranga authored Jan 31, 2025

    Verified

    This commit was created on GitHub.com and signed with GitHub’s verified signature.
    Copy the full SHA
    59aa35f View commit details

Commits on Feb 6, 2025

  1. Updating build behavior to include dependencies that do not have whee…

    …ls (#47)
    sindhu-ranga authored Feb 6, 2025

    Verified

    This commit was created on GitHub.com and signed with GitHub’s verified signature.
    Copy the full SHA
    a9a9d96 View commit details
  2. Update __version__.py (#48)

    sindhu-ranga authored Feb 6, 2025

    Verified

    This commit was created on GitHub.com and signed with GitHub’s verified signature.
    Copy the full SHA
    9b254c3 View commit details
  3. Fixing build breaking in v.25 (#49)

    sindhu-ranga authored Feb 6, 2025

    Verified

    This commit was created on GitHub.com and signed with GitHub’s verified signature.
    Copy the full SHA
    6b0bd37 View commit details
  4. Adding verbose logging (#50)

    sindhu-ranga authored Feb 6, 2025

    Verified

    This commit was created on GitHub.com and signed with GitHub’s verified signature.
    Copy the full SHA
    06b1ede View commit details

Commits on Feb 7, 2025

  1. Installing h3 and ratelimit dependencies locally (#51)

    * Installing h3 and ratelimit dependencies locally
    
    * Just include it in requirements.txt
    sindhu-ranga authored Feb 7, 2025

    Verified

    This commit was created on GitHub.com and signed with GitHub’s verified signature.
    Copy the full SHA
    de6eb8e View commit details
  2. Update build-and-deploy-release-pypi.yml (#52)

    sindhu-ranga authored Feb 7, 2025

    Verified

    This commit was created on GitHub.com and signed with GitHub’s verified signature.
    Copy the full SHA
    7bfc928 View commit details

Commits on Feb 11, 2025

  1. Reverted to working version (#54)

    This reverts commit 59aa35f.
    sindhu-ranga authored Feb 11, 2025

    Verified

    This commit was created on GitHub.com and signed with GitHub’s verified signature.
    Copy the full SHA
    c20b5fc View commit details
Showing with 106 additions and 25 deletions.
  1. +2 −2 .github/workflows/build-and-deploy-release-pypi.yml
  2. +1 −1 .github/workflows/python-package.yml
  3. +12 −0 README.md
  4. +1 −1 placekey/__version__.py
  5. +48 −17 placekey/api.py
  6. +39 −1 placekey/placekey.py
  7. +1 −1 placekey/tests/test_api.py
  8. +2 −2 setup.py
4 changes: 2 additions & 2 deletions .github/workflows/build-and-deploy-release-pypi.yml
Original file line number Diff line number Diff line change
@@ -36,10 +36,10 @@ jobs:
env:
TWINE_PASSWORD: ${{ secrets.TEST_PYPI_API_TOKEN}}
run: |
twine upload --repository testpypi dist/*
twine upload --repository testpypi dist/* --verbose
- name: Upload to PyPi
env:
TWINE_PASSWORD: ${{ secrets.PYPI_API_TOKEN}}
run: |
twine upload dist/*
twine upload dist/* --verbose
2 changes: 1 addition & 1 deletion .github/workflows/python-package.yml
Original file line number Diff line number Diff line change
@@ -8,7 +8,7 @@ jobs:
runs-on: ubuntu-latest
strategy:
matrix:
python-version: ['3.7', '3.8', '3.9', '3.10', '3.11']
python-version: ['3.8', '3.9', '3.10', '3.11']

steps:
- uses: actions/checkout@v2
12 changes: 12 additions & 0 deletions README.md
Original file line number Diff line number Diff line change
@@ -78,7 +78,19 @@ True
>>> pk.placekey_format_is_valid('@123-456-789')
False
```
You can now access the locations of placekey’s free datasets in S3 using placekey-py! Use these two functions:

```python
print(pk.list_free_datasets())

print(pk.return_free_datasets_location_by_name('chipotle-locations'))
```

1. List Free Datasets: Returns a list of all names of Placekey’s available free datasets

2. Return Free Datasets Location By Name: Using one of the names from List Free Datasets above, returns the publicly accessible S3 URI of said dataset.

You can use these locations to download files programmatically (with boto3) or directly in Spark.
## API Client

This package also includes a client for the Placekey API. The methods in the client are automatically rate limited.
2 changes: 1 addition & 1 deletion placekey/__version__.py
Original file line number Diff line number Diff line change
@@ -1 +1 @@
__version__ = '0.0.18'
__version__ = '0.0.32'
65 changes: 48 additions & 17 deletions placekey/api.py
Original file line number Diff line number Diff line change
@@ -1,12 +1,12 @@
import hashlib
import itertools
import json
import logging
import itertools
from typing import List
from json import JSONDecodeError

import backoff
import requests
from ratelimit import limits, RateLimitException
from backoff import on_exception, fibo

from .__version__ import __version__

console_log = logging.StreamHandler()
@@ -54,7 +54,7 @@ class PlacekeyAPI:
REQUEST_WINDOW = 60

BULK_URL = 'https://api.placekey.io/v1/placekeys'
BULK_REQUEST_LIMIT = 100
BULK_REQUEST_LIMIT = 10
BULK_REQUEST_WINDOW = 60
MAX_BATCH_SIZE = 100

@@ -141,7 +141,7 @@ def lookup_placekey(self,

result = self.make_request(payload)

return json.loads(result.text)
return self._safe_parse_json(result.text)

def lookup_placekeys(self,
places,
@@ -205,6 +205,10 @@ def lookup_placekeys(self,
self.logger.error(
'Fatal error encountered. Returning processed items at size %s of %s', i, len(places))
break
except requests.exceptions.RequestException:
self.logger.error(
'Fatal error encountered. Returning processed items at size %s of %s', i, len(places))
break

# Catch case where all queries in batch having an error,
# and generate rows for individual items.
@@ -260,14 +264,34 @@ def _lookup_batch(self, places,

result = self.make_bulk_request(batch_payload)

return json.loads(result.text)
return self._safe_parse_json(result.text)

def _validate_query(self, query_dict):
query_dict_keys = query_dict.keys()
top_level_check = set(query_dict_keys).issubset(self.QUERY_PARAMETERS)
place_metadata_check = set(query_dict.get(self.PLACE_METADATA_CONSTANT).keys()).issubset(self.PLACE_METADATA_PARAMETERS) if(self.PLACE_METADATA_CONSTANT in query_dict_keys) else True
place_metadata_check = set(query_dict.get(self.PLACE_METADATA_CONSTANT).keys()).issubset(
self.PLACE_METADATA_PARAMETERS) if (self.PLACE_METADATA_CONSTANT in query_dict_keys) else True
return top_level_check and place_metadata_check

def _safe_parse_json(self, result):
"""
Safely parse JSON response.
Parameters:
result (str): JSON string.
Returns:
dict: Parsed JSON dictionary or empty list if parsing fails.
"""
try:
return json.loads(result)
except JSONDecodeError:
self.logger.error("JSONDecodeError parsing, returning empty list")
return []
except Exception as e:
self.logger.error(f"Error parsing: {e}, returning empty list")
return []

def _get_request_function(self, url, calls, period, max_tries):
"""
Construct a rate limited function for making requests.
@@ -278,18 +302,25 @@ def _get_request_function(self, url, calls, period, max_tries):
:param max_tries: the maximum number of retries before giving up
"""

@on_exception(fibo, RateLimitException, max_tries=max_tries)
@backoff.on_exception(backoff.fibo, (RateLimitException, requests.exceptions.RequestException),
max_tries=max_tries)
@limits(calls=calls, period=period)
def make_request(data):
response = requests.post(
url, headers=self.headers,
data=json.dumps(data).encode('utf-8')
)
try:
response = requests.post(
url, headers=self.headers,
data=json.dumps(data).encode('utf-8')
)

if response.status_code == 429:
raise RateLimitException("Rate limit exceeded", 0)
if response.status_code == 429:
raise RateLimitException("Rate limit exceeded", 0)
elif response.status_code == 503:
raise requests.exceptions.RequestException("Service Unavailable")
elif response.status_code == 504:
raise requests.exceptions.RequestException("Gateway Timeout")

# Assumption: A code other than 429 is handled by calling function
return response
return response
except requests.exceptions.RequestException as e:
raise e

return make_request
40 changes: 39 additions & 1 deletion placekey/placekey.py
Original file line number Diff line number Diff line change
@@ -14,7 +14,9 @@
from shapely.ops import transform
from shapely.strtree import STRtree
from shapely.wkt import loads as wkt_loads

import boto3
from botocore import UNSIGNED
from botocore.config import Config

RESOLUTION = 10
BASE_RESOLUTION = 12
@@ -48,6 +50,42 @@
'^' + '-'.join([FIRST_TUPLE_REGEX, TUPLE_REGEX, TUPLE_REGEX]) + '$')
WHAT_REGEX_V1 = re.compile('^[' + ALPHABET + ']{3,}(-[' + ALPHABET + ']{3,})?$')
WHAT_REGEX_V2 = re.compile('^[01][abcdefghijklmnopqrstuvwxyz234567]{9}$')
s3 = boto3.client("s3", config=Config(signature_version=UNSIGNED))

def list_free_datasets():
"""
:return: The names of every free placekey'd dataset Placekey offers
"""
folders = set()
paginator = s3.get_paginator("list_objects_v2")
for page in paginator.paginate(Bucket='placekey-free-datasets', Prefix='', Delimiter="/"):
for common_prefix in page.get("CommonPrefixes", []):
folders.add(common_prefix["Prefix"].replace("/", ""))
return folders

def return_free_datasets_location_by_name(name: str, url: bool = False):
"""
Get the S3 location of a free dataset by its name. Find names using list_free_datasets. Raises ValueError if name is not correct.
:param name: Dataset Name (str)
:param name: Return a URL or S3 URI? Default is False (S3 URI)
:return: The public S3 location of the placekey'd dataset
"""
response = s3.list_objects_v2(Bucket='placekey-free-datasets', Prefix=name+'/csv')

# Extract files from the response
files = [obj["Key"] for obj in response.get("Contents", [])]

if len(files) == 1:
if url:
return "https://placekey-free-datasets.s3.us-west-2.amazonaws.com/"+files[0]
else:
return "s3://placekey-free-datasets/"+files[0]
elif len(files) == 0:
print()
raise FileNotFoundError("No files found in the specified S3 directory. Please notify Placekey.")
else:
raise ValueError(f"Something went wrong. Please notify Placekey.")

def _get_header_int():
"""
2 changes: 1 addition & 1 deletion placekey/tests/test_api.py
Original file line number Diff line number Diff line change
@@ -99,7 +99,7 @@ def test_lookup_placekeys(self):
self.assertListEqual(
self.pk_api.lookup_placekeys(places),
[
{'query_id': 'place_0', 'placekey': '0rsdbudq45@5vg-7gq-5mk'},
{'query_id': 'place_0', 'placekey': '22g@5vg-7gq-5mk'},
{'query_id': 'thisqueryidaloneiscustom', 'placekey': '227-223@5vg-82n-pgk'},
{'query_id': 'place_2', 'placekey': '@5vg-82n-kzz'}
]
4 changes: 2 additions & 2 deletions setup.py
Original file line number Diff line number Diff line change
@@ -28,11 +28,11 @@ def get_version():
long_description_content_type="text/markdown",
url="https://github.com/Placekey/placekey-py",
packages=setuptools.find_packages(),
install_requires=['h3', 'shapely', 'requests', 'ratelimit', 'backoff'],
install_requires=['h3>=3.6.1,<4', 'shapely', 'requests', 'ratelimit', 'backoff', 'boto3'],
classifiers=[
"Programming Language :: Python :: 3",
"License :: OSI Approved :: Apache Software License",
"Operating System :: OS Independent",
],
python_requires=">=3.6",
python_requires=">=3.8",
)