Add importers for various services and refactor base importer class#62
Open
aquarion wants to merge 70 commits into
Open
Add importers for various services and refactor base importer class#62aquarion wants to merge 70 commits into
aquarion wants to merge 70 commits into
Conversation
![github-advanced-security[bot]](https://avatars.githubusercontent.com/in/57789?s=60&v=4){kind=small}
…mproved connection handling
…new helper methods
… functionality and backward compatibility
…ic into foursquare.py; remove obsolete foursquare_oauth.py
…t code_fetcher as CodeFetcher9000' across multiple modules
…; update documentation and configuration files accordingly
…implement job execution functions for scheduler
…or caching in FoursquareAPI
…e logging and error handling across modules
…e features, quick start, configuration, and running imports sections
- Consolidated local imports in `steambadges.py`, `switchbot.py`, `tumblr.py`, `tweets.py`, `wordpress.py`, and `wow.py`. - Enhanced readability by adjusting line breaks and spacing in various files. - Updated exception handling to be more specific in `tumblr.py` and `wow.py`. - Added `flake8` as a dependency in `pyproject.toml` for code style enforcement. - Cleaned up unused imports and improved consistency in `scheduler.py`, `tests/conftest.py`, and other test files. - Ensured all changes adhere to PEP 8 style guidelines for better maintainability.
- Introduced BaseImporter class to standardize importer functionality. - Added FlickrImporter for importing photos from Flickr. - Implemented GithubCommitsImporter to fetch commits from GitHub repositories. - Created LastfmImporter to import recent scrobbles from Last.fm. - Developed MastodonImporter for importing toots from Mastodon instances. - Added SteamImporter to import achievements from Steam. - Implemented SwitchbotImporter for temperature and humidity data from SwitchBot sensors. - Created WordpressImporter to import posts from WordPress sites. - Refactored tests to accommodate changes in module structure and improve clarity.
- Introduced BaseImporter class to standardize importer functionality. - Added FlickrImporter for importing photos from Flickr. - Implemented GithubCommitsImporter to fetch commits from GitHub repositories. - Created LastfmImporter to import recent scrobbles from Last.fm. - Developed MastodonImporter for importing toots from Mastodon instances. - Added SteamImporter to import achievements from Steam. - Implemented SwitchbotImporter for temperature and humidity data from SwitchBot sensors. - Created WordpressImporter to import posts from WordPress sites. - Refactored tests to accommodate changes in module structure and improve clarity.
…e unused graph variable
…t code_fetcher as CodeFetcher9000' across multiple modules
- Consolidated local imports in `steambadges.py`, `switchbot.py`, `tumblr.py`, `tweets.py`, `wordpress.py`, and `wow.py`. - Enhanced readability by adjusting line breaks and spacing in various files. - Updated exception handling to be more specific in `tumblr.py` and `wow.py`. - Added `flake8` as a dependency in `pyproject.toml` for code style enforcement. - Cleaned up unused imports and improved consistency in `scheduler.py`, `tests/conftest.py`, and other test files. - Ensured all changes adhere to PEP 8 style guidelines for better maintainability.
…ror notification formatting
- jobs.py: Narrow ImportError catch to package import only; call importer.run() directly instead of execute() so exceptions propagate to scheduler notifications - scheduler.py: Dispatch shell jobs (! prefix) in run_job_now(); guard grace= option parsing against malformed values - base.py: Add ConfigurationError; require_config() and get_feed_url() raise instead of sys.exit() so scheduler catches config failures correctly; execute() catches ConfigurationError and returns exit code 5 for CLI use - mastodon_toots.py, wordpress.py: Raise ConfigurationError on missing config and on empty sites list instead of silently returning success - flickr.py: Replace raw get_connection()/get_cursor() with self.entry_store; respects no_db mode; no longer leaks connection on exception - notifications.py: Only include traceback when error is an Exception, not a plain string (fixes NoneType: None in shell job failure emails) - __init__.py: Add missing imports for __all__ entries; expose core submodules so `from lifestream import cache` works as a module reference - pytest.ini: Fix [tool:pytest] → [pytest] so pythonpath = src is applied - tests/test_db.py, tests/test_notifications.py: Fix patches to match actual module APIs; fix two orphaned finally: syntax errors Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
…e file reading logic Co-authored-by: Copilot <copilot@github.com>
aquarion
force-pushed
the
feature/restructure
branch
from
May 4, 2026 17:57
3839dd2 to
ab9e630
Compare
Avoids CodeQL flagging the word "secrets" in function names on every review. The config key (secrets_dir in config.ini) is unchanged. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
- Rename imports/lifestream/ → imports/lifestream_legacy/ to prevent sys.path collision with the installed src/lifestream/ package - Update all legacy import scripts to use lifestream_legacy - Wire scheduler.py to use src/lifestream/core/ directly instead of the legacy jobs.py (which lacked new-style IMPORTERS dispatch) - Remove legacy scripts that have been converted to new-style importers: atom, flickr, github_commits, lastfm, mastodon_toots, steam, switchbot, wordpress - Fix F402: rename loop variable config → job_config in list_jobs() Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
- flickr: HTTP → HTTPS for static image URLs - github_commits: add pagination loop (repos + commits), add timeout, fix return type annotation, use r.json() directly - steam_api: add 30s timeout to make_steam_call() - switchbot: add 30s timeout and raise_for_status() to API calls - oauth_utils: create token files with 0o600 permissions via os.open() - cache: handle pickle.UnpicklingError/EOFError on corrupted cache files - foursquare: fix CALLBACK_URL missing https:// scheme Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
historic.py, tumblr.py: suppression tag moved to the expression line lifestream_legacy/oauth_utils.py: explanation comment separated above write calls tests/test_notifications.py: explanation comment separated above assert Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Contributor
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 83 out of 92 changed files in this pull request and generated 11 comments.
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
Comment on lines
+17
to
+19
| # Static URL template: farm, server, id, secret, size | ||
| STATIC_URL_TEMPLATE = "http://farm%s.staticflickr.com/%s/%s_%s_%s.jpg" | ||
| MAX_PAGES = False |
Owner
Author
There was a problem hiding this comment.
Fixed in a previous commit (cf3eccb) — the review was on an older revision.
Comment on lines
+33
to
+41
| if method == "post": | ||
| headers["content-type"] = "application/json; charset=utf8" | ||
| r = requests.post(url, json=data, headers=headers) | ||
| elif method == "get": | ||
| r = requests.get(url, params=data, headers=headers) | ||
| else: | ||
| raise ValueError(f"Unknown method: {method}") | ||
|
|
||
| return r.json() |
Owner
Author
There was a problem hiding this comment.
Fixed in a previous commit (cf3eccb) — the review was on an older revision.
Comment on lines
+32
to
+47
| def github_call(self, path: str, page: int = 1, per_page: int = 100) -> dict: | ||
| """Make an authenticated GitHub API call.""" | ||
| token = self.get_config("auth_token") | ||
| gh_url = f"https://api.github.com/{path}?page={page}&per_page={per_page}" | ||
| headers = {"Authorization": f"token {token}"} | ||
|
|
||
| self.logger.debug("Calling %s", path) | ||
| r = requests.get(gh_url, headers=headers) | ||
|
|
||
| if r.status_code != 200: | ||
| self.logger.error(f"GitHub API error: {r.status_code}") | ||
| self.logger.error(f"URL: {r.url}") | ||
| self.logger.error(f"Response: {r.text}") | ||
| raise Exception(f"GitHub API error: {r.status_code}") | ||
|
|
||
| return json.loads(r.text) |
Owner
Author
There was a problem hiding this comment.
Fixed in a previous commit (cf3eccb) — the review was on an older revision.
Comment on lines
+49
to
+59
| def run(self) -> None: | ||
| """Import commits from all user repositories.""" | ||
| username = self.get_config("username") | ||
|
|
||
| repos = self.github_call("user/repos") | ||
|
|
||
| for repo in repos: | ||
| self.logger.debug("Processing repo: %s", repo["name"]) | ||
|
|
||
| commits = self.github_call(f"repos/{repo['full_name']}/commits") | ||
|
|
Owner
Author
There was a problem hiding this comment.
Fixed in a previous commit (cf3eccb) — the review was on an older revision.
Comment on lines
+74
to
+84
| if os.path.exists(cachefile): | ||
| modified = os.path.getmtime(cachefile) | ||
| logger.debug(f"Found cache file at '{cachefile}'") | ||
| now = time.time() | ||
| if now > (modified + maxage): | ||
| logger.debug(f"Ignoring old cache file '{cachefile}'") | ||
| else: | ||
| with open(cachefile, "rb") as cachehandle: | ||
| logger.info(f"Using cached result from '{cachefile}'") | ||
| return pickle.load(cachehandle) | ||
|
|
Owner
Author
There was a problem hiding this comment.
Fixed in a previous commit (cf3eccb) — the review was on an older revision.
Comment on lines
+21
to
+26
| """Make a call to the Steam API.""" | ||
| url = f"{self.BASE_URL}{interface}/{method}/v{int(version):04d}/" | ||
| params["key"] = self.api_key | ||
| response = requests.get(url, params=params) | ||
| response.raise_for_status() | ||
| return response.json() |
Owner
Author
There was a problem hiding this comment.
Fixed in a previous commit (cf3eccb) — the review was on an older revision.
Comment on lines
+45
to
+53
| if importer_cls is not None: | ||
| importer = importer_cls() | ||
| importer._args = importer.parse_args([]) | ||
| if not importer.validate_config(): | ||
| raise RuntimeError(f"Config validation failed for {job_name}") | ||
| importer.run() | ||
| duration = (datetime.now() - start_time).total_seconds() | ||
| logger.info(f"Completed job: {job_name} in {duration:.1f}s") | ||
| return |
Owner
Author
There was a problem hiding this comment.
We can't use execute() here because it catches all exceptions internally and returns exit codes — the scheduler needs exceptions to propagate so it can send failure notifications and mark the job as failed. Added a comment in commit bcbd9c1 to explain this.
Comment on lines
+93
to
+100
| result = subprocess.run( | ||
| command, | ||
| shell=True, | ||
| cwd=basedir, | ||
| capture_output=True, | ||
| text=True, | ||
| ) | ||
|
|
Owner
Author
There was a problem hiding this comment.
Fixed in commit bcbd9c1 — added timeout=3600 to subprocess.run(). One hour is a reasonable ceiling for any scheduled shell job; it can be tightened per-job if needed.
Comment on lines
+191
to
+196
| mastodon_toots = */24 * * * * | ||
| ; tweets = */10 * * * * | ||
| tumblr = 37 */2 * * * | ||
|
|
||
| ; Code/productivity | ||
| github_commits = 34 * * * * |
Owner
Author
There was a problem hiding this comment.
Fixed in commit bcbd9c1 — updated config.example.ini to use the canonical names (mastodon, github). Also added backward-compatible aliases in IMPORTERS so any existing configs using the old names still work.
Comment on lines
+38
to
+48
| # Registry of all importers by name | ||
| IMPORTERS = { | ||
| "atom": AtomImporter, | ||
| "flickr": FlickrImporter, | ||
| "github": GithubCommitsImporter, | ||
| "lastfm": LastfmImporter, | ||
| "mastodon": MastodonImporter, | ||
| "steam": SteamImporter, | ||
| "switchbot": SwitchbotImporter, | ||
| "wordpress": WordpressImporter, | ||
| } |
Owner
Author
There was a problem hiding this comment.
Fixed in commit bcbd9c1 — added github_commits and mastodon_toots as alias keys in IMPORTERS pointing to the same importers, so legacy schedule names continue to resolve correctly.
- Add legacy aliases github_commits and mastodon_toots to IMPORTERS so existing user configs continue to work after the importer rename - Update config.example.ini to use canonical names (github, mastodon) - Add timeout=3600 to subprocess.run() in run_shell_command() to prevent hung shell jobs from blocking scheduler workers indefinitely - Add comment explaining why _args is set directly instead of using execute() (execute() swallows exceptions the scheduler needs) Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
…dates Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
…down, move argparse
| oauth_token + "\n" | ||
| ) # codeql[py/clear-text-storage-sensitive-data] | ||
| oauth_file.write( | ||
| oauth_token_secret + "\n" |
Comment on lines
+123
to
+126
| "hooks.slack.com" | ||
| in call_args[0][ | ||
| 0 | ||
| ] # codeql[py/incomplete-url-substring-sanitization] |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
config.ini [schedules], persists job state in Redis, coalesces missed runs, supports shell jobs viacmd=optionsrc/lifestream/with propercore/submodules (config,db,cache,notifications,jobs,utils)get_redis_connection()andfile_cache()inlifestream.core.cacheEntryStoreclass centralises database writes withno_dbmode for dry-runsConfigurationErrorexception replacessys.exit()in importers — safe to call from scheduler contextlifestream.core.notificationsget_secrets_dir→get_credentials_dirto stop CodeQL flagging on every reviewprint(token, file=f)withf.write()inwrite_token_file; added CodeQL suppression comments for intentional credential-file writesTHE_FALLEN.mdreplacesthe_fallen/directory — dead importers removed, indexed with git commit for recovery; fitbit_day.py addedworking-directoryset tocontrib/ffxiv-upload-achievement-icons/so Poetry installs the right depsimports/*.pyscripts — they encourage running outside the virtualenv, which fails; the correct workflow ispoetry runor via the schedulerimports/*.pyscripts genuinely fixed by extracting helper functions (no suppressions); new-style importers insrc/lifestream/importers/retain# noqa: C901on a few complex orchestration methods (process_site,run) where the complexity is inherent to the API/pagination logic# noqaplacement fixed; isort ordering;facebook_posts.some_action()refactoredper_pageparam andhtml_url; Mastodon pagination direction (max_idnotmin_id); SwitchBot POST body JSON-encoded;e.message→str(e)in legacy importers; scheduler pre-parses importer args with[]to avoid sys.argv leakage; Steamvalidate_confignow checks all required keysImporters added as structured classes
FlickrImporter— photos from FlickrGithubCommitsImporter— commits from GitHub repositoriesLastfmImporter— scrobbles from Last.fmMastodonImporter— toots from Mastodon instancesSteamImporter— achievements from SteamSwitchbotImporter— temperature/humidity from SwitchBot sensorsWordpressImporter— posts from WordPress sitesStill to do (tracked as follow-up issues)
ffxiv,atproto_posts,instagram,gw2,oyster,oyster_csvfoursquare,tweets,tumblr,historicfacebook_page,facebook_postsTest plan
poetry run pytest— all tests passpython scheduler.py --listagainst a config with a[schedules]sectionpython scheduler.py --run <job>for an import job and a!shelljobpoetry run flickr) with and without--no-db🤖 Generated with Claude Code