Skip to content

Make ipdevpoll able to respond to refresh events #3348

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Draft
wants to merge 8 commits into
base: master
Choose a base branch
from

Conversation

lunkwill42
Copy link
Member

Pulled this one out the "unfinished" archives and rebased to latest master. Will likely still need some work, so keeping it in draft mode.

Fixes #2626

@lunkwill42 lunkwill42 self-assigned this Apr 28, 2025
Copy link

github-actions bot commented Apr 28, 2025

🦙 MegaLinter status: ✅ SUCCESS

Descriptor Linter Files Fixed Errors Warnings Elapsed time
✅ PYTHON black 4 0 0 0.78s
✅ PYTHON ruff 4 0 0 0.15s

See detailed report in MegaLinter reports
Set VALIDATE_ALL_CODEBASE: true in mega-linter.yml to validate all sources, not only the diff

MegaLinter is graciously provided by OX Security

Copy link

github-actions bot commented Apr 28, 2025

Test results

   12 files     12 suites   11m 54s ⏱️
2 178 tests 2 177 ✅ 0 💤 1 ❌
6 009 runs  6 006 ✅ 0 💤 3 ❌

For more details on these failures, see this check.

Results for commit 79341e7.

♻️ This comment has been updated with latest results.

@lunkwill42 lunkwill42 force-pushed the feature/ipdevpoll-refresh-events branch from 61fc148 to aa0ee5f Compare April 28, 2025 13:27
@lunkwill42 lunkwill42 added the nonews No news fragment is necessary for this PR (e.g. refactoring, cleanups, workflow/development changes) label Apr 28, 2025
@lunkwill42
Copy link
Member Author

lunkwill42 commented Apr 30, 2025

Things that seem to still be missing from this PR (I will update this comment as I find more):

  • There should be a command line program to trigger refreshes from the running ipdevpoll (which could also be poc code for implementing in the web front-end)
  • Stale refresh events are not cleaned up or reacted to when ipdevpoll first starts
  • Refresh events for jobs that don't exist are never logged or removed (i.e. events with blank subid are logged and removed, but when subid contains a value that is not a valid job name, it is just ignored)
  • Receipt events should be sent back to the source once a refresh job finishes

@lunkwill42 lunkwill42 force-pushed the feature/ipdevpoll-refresh-events branch from aa0ee5f to ef62b95 Compare April 30, 2025 09:55
lunkwill42 and others added 7 commits April 30, 2025 11:56
EventEngine used to be the only subscriber to eventq update
notifications from PostgreSQL, now ipdevpoll will be there too.

It's nice for the targeted subsystem to be able to discern from the
notification payload itself whether it needs to check the eventq table
for new records or not.
This adds code for ipdevpoll to subscribe to event notifications from
PostgreSQL, which it can use as a trigger for looking for refresh events
targeted at it.
This is just for testing that the notification mechanism works.
We want to be able to easily map job names to existing JobHandler
instances for incoming refresh events (so we can look up which
JobHandler potentially needs to be told to reschedule a job for
an immediate run)
This still doesn't reschedule anything, but it filters out incoming
events that this process doesn't want to handle, and ignores/deletes
events that no ipdevpoll process will handle.

It also groups incoming events by netbox+jobname, in case there are
multiple requests for the same job to be re-run (i.e. someone impatient
keeps sending refresh events, or maybe ipdevpoll was down for a while
and events queued up, or two separate users just happened to send the
same refresh request at the same time).
This looks up the appropriate schedulers for each incoming refresh
request and asks them to reschedule the next job to run immediately.

This still does not provide receipts to the event source, that will have
to come later.
@lunkwill42 lunkwill42 force-pushed the feature/ipdevpoll-refresh-events branch 3 times, most recently from 88d4aac to 52afa56 Compare April 30, 2025 10:08
This program can be used to send refresh events to ipdevpoll "by hand".
@lunkwill42 lunkwill42 force-pushed the feature/ipdevpoll-refresh-events branch from 52afa56 to 79341e7 Compare April 30, 2025 10:57
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
nonews No news fragment is necessary for this PR (e.g. refactoring, cleanups, workflow/development changes)
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Make ipdevpoll able to respond to refresh events
1 participant