merge with milva master - #1
Open
bbesli wants to merge 66 commits into
Open
Conversation
Introduce cursor-based pagination for job occurrences and enhance workflow triggering with per-step job data overrides. * Add `.env` and `.env.local` to `.gitignore` for local environment management. * Add `GetJobOccurenceListCursorQuery` and handler for cursor-based pagination. * Update `JobsController` with `GetJobOccurrencesCursorAsync` endpoint. * Add `StepJobData` to `WorkflowRuns` via `WorkflowJobDataOverride` migration. * Enhance `WorkflowStepDto` with `WorkerId` and `JobNameInWorker`. * Refactor `GetWorkflowDetailQueryHandler` to include job info for steps. * Add `TriggerWorkflowCommand` and handler for per-step job data overrides. * Implement `MergeJobData` method in `WorkflowEngineService`. * Update `OccurrenceTable` and `ExecutionList` for cursor-based pagination. * Add `TriggerWorkflowModal` for workflow triggering with step overrides. * Update `WorkflowDetail` and `WorkflowList` to integrate the modal. * Enhance `occurrenceService` and `workflowService` for new features. * Upgrade NuGet dependencies for improved functionality and compatibility. * Update unit tests to reflect changes in `JobDataMap` handling. * Add new CSS styles for `TriggerWorkflowModal` and pagination updates.
Introduce `BasePath` for hosting under a sub-path. * Add `BasePath` property to `MilvaionConfig.cs` for configuration. * Update `docker-compose.yml` and `Dockerfile` to support `VITE_BASE_PATH`. * Modify `Program.cs` to use `UsePathBase` middleware for routing. * Update `vite.config.js` to dynamically configure `base` and proxy settings. * Update `.env` files to include `VITE_BASE_PATH` for local and production builds. * Refactor React components to use `useNavigate` for base path compatibility. * Update `api.js` and `signalRService.js` to construct URLs dynamically. * Add integration tests (`BasePathIntegrationTests.cs`) to validate sub-path hosting. * Document `BasePath` configuration in `06-configuration.md`. * Update PWA settings to respect the base path. These changes enable deployment flexibility by allowing the application to be hosted under a configurable sub-path.
Refactor startup recovery to improve dispatcher responsiveness: * Add `CleanupZombieOccurrencesAsync` to handle zombie cleanup in the background. * Remove inline zombie cleanup logic from `PerformStartupRecoveryAsync`. * Update log messages to indicate background zombie cleanup. * Enhance XML documentation for `PerformStartupRecoveryAsync` and add documentation for `CleanupZombieOccurrencesAsync`. This change allows the dispatcher to start immediately while zombie cleanup runs asynchronously, improving system performance and maintainability.
Introduce /config.js endpoint to serve runtime base path and settings for the frontend. Update React app to read base path from window.__MILVAION_CONFIG__ for routing, API, and SignalR. Adjust documentation and docker-compose.yml to reflect new runtime configuration approach.
- Added ManagementEnabled and ManagementPort to RabbitMQOptions and appsettings.json. - Introduced RabbitMQManagementClient for querying queue stats via the Management HTTP API. - Updated QueueDepthMonitor to use the Management API for detailed queue statistics and dynamic queue discovery, with fallback to AMQP when unavailable. - Updated DI registration and XML docs accordingly.
Updated Milvasoft SDK and template package versions across projects. ApiWorker and ConsoleWorker now reference the latest SDK.Worker. Improved SyncOrchestratorService logging to only log when there are actual synced or failed items.
…ers. - Fix user activity tags.
- Added `JobTestRunner` and `Milvasoft.Milvaion.Sdk.Worker.Testing` for in-process local job testing without external services - Documented local job testing in `23-local-job-testing.md` - Updated worker templates to generate test projects by default - Added sample test projects and example tests for workers - Upgraded NuGet dependencies and added SQLitePCLRaw for .NET 10 support - Added root `dotnet-tools.json` for tool manifest
- Bump Milvaion template version to 10.1.4 - Update Microsoft.AspNetCore.OpenApi and Microsoft.Extensions.Hosting to 10.0.9 - Switch test imports to Milvasoft.Milvaion.Sdk.Worker.Testing - Remove unused FluentAssertions import in ConsoleWorker.Tests
Refactored AlertNotifier to properly dispose the timeout CancellationTokenSource and handle unregistered channels more cleanly. Updated WorkflowEngineServiceTests to use a polling wait instead of a fixed delay, ensuring all workflow runs have settled before proceeding.
- Implemented infinite scroll for the card (list) view in JobList using IntersectionObserver and a sentinel element. - Refactored job loading logic to support both paginated table and infinite scroll card views. - Updated CSS for infinite scroll UI elements, including loading spinner and end-of-list message. Improved refresh and error handling for both views.
- Introduce MilvaionApiKey entities, migrations, and CQRS features for create/update/revoke/delete/list with permission and audit tracking - Implement ApiKeyAuthenticationHandler (X-ApiKey), Redis cache, and immediate cache invalidation; permissions as role claims for seamless [Auth] support - Add ApiAuthAttribute and update default policy to support both JWT and API key authentication - Add /mcp Model Context Protocol server with job, workflow, and failure tools, enforcing permissions via McpPermissionGuard - Add React UI for API key management (create/edit/delete/revoke, permission selection, one-time key reveal), update navigation and styles - Add docs for API keys and MCP server, including security model and client integration - Support partial workflow updates (UpdateProperty<T>), improve validation, add tests, and optimize job occurrence indexes - Bump version to 1.2.0
Introduce mandatory method annotations (`ReadOnly`, `Destructive`, `Idempotent`) for MCP server tools to clarify intent. Add explicit filtering and sorting to list tools with a max page size of 100. Throw `McpException` for not-found cases. Require explicit permissions for write tools and enforce domain safeguards. Refactor tool methods for improved parameter descriptions and filtering. Define structured prompts in `MilvaionPrompts` and register them with endpoints. Update documentation for conventions, filtering, prompt usage, and error handling.
Added MilvaionInsightTools.cs with 40+ read-only MCP server tools for metric reports, infrastructure health, and configuration. Updated documentation tables and descriptions to reflect new endpoints and permissions, clarifying tool usage and integration details.
Introduce GetWorkflowRunAnalysisQuery and GetMetricReportSummaryListQuery for MCP tools API, providing flattened workflow run analysis and efficient metric report metadata listing. Update MCP tool endpoints to use these queries, improving programmatic consumption and performance. Add new DTOs, EF Core projections, and validators. Update documentation and XML comments to explain new endpoints, DTOs, and design rationale.
…ode, fix startup recovery bug. - Add migration for nullable CompletedAt on ScheduledJobs to track one-time job completion and prevent rescheduling. - Introduce DTOs for config, upcoming executions, and job occurrence logs. - Implement queries, handlers, and validators for job logs, summaries, and upcoming executions. - Add CronProjector and StartupScheduleResolver with unit tests for accurate scheduling. - Add workflow condition expression parser/evaluator with tests. - Add React components: collapsible section, localStorage state hook, Upcoming Executions page, workflow DAG canvas, and visual condition builder. - Refactor detail/page header CSS for consistency. - Update EF migration snapshot for CompletedAt. - Ensure unit test coverage for new features.
- Major refactor to standardize all table/list pages using new shared React components (TableParts, TableActions, Pagination, ViewToggle, JobSelect) and a single, prefixed CSS system. - Replaces per-page table markup and styles with reusable layouts, toolbars, filters, selection bars, bulk actions, and pagination. - Table cells, row actions, and status indicators are now consistent across the app. Improves mobile responsiveness, maintainability, and code reuse. - Updates Vite config for better chunking. - Cleans up global CSS and adds utility functions for durations/statuses.
- Major redesign of Dashboard, Monitoring, and Configuration pages for clarity and navigation. - Adds unified JsonView for JSON handling, refactors trigger result dialogs, and improves delete actions with busy states. - Updates sticky element CSS, CollapsibleSection icons, and table action feedback. - Backend adds /admin/resource-usage endpoint for live metrics. - Documentation and comments updated for new behaviors.
Significantly enhance mobile usability and visual consistency: - Align headers, navigation, and actions beside hamburger menu - Hide job list filters behind toggle on small screens - Improve wrapping/alignment for titles, badges, and chips - Enable horizontal scrolling for tables and logs on mobile - Center/stack profile and card layouts for mobile - Fix sidebar/form header order in job forms - Refactor grid/flex layouts to prevent overflow/overlap - Add/adjust CSS comments for clarity - Minor JS changes for UI behaviors (e.g., filter toggle) - Update notification service sorting logic Addresses overflow, alignment, and visibility issues on mobile.
- Added RabbitMQPublisherTests to verify IRabbitMQPublisher's ability to detect routable and unroutable jobs via mandatory-return. Tests simulate both catch-all queue presence and absence, ensuring accurate detection and preventing silent message loss. - Background service metric fix. - Add worker memory and cpu tracking.
Introduce runtime-configurable branding and notification rules via new settings infrastructure, including AppSetting entity, Redis-synced cache, and admin/public API endpoints. Add background monitors for API key expiry and job misfires, with enhanced alerting using dynamic rules and new alert types. Extend metric reports with period/size fields and metadata. Improve UI with dynamic branding, settings page, period badges, and live table updates. Add migrations, update tests for settings provider, and enhance documentation.
Updated all relevant .csproj files to use package version 10.1.6. Updated the VERSION file from 1.2.2 to 1.2.3 to reflect the new release.
At-least-once delivery has two halves and only the consuming one was described. Adds the publishing side: which channels carry confirms, what a throwing publish means for the caller, and why consume-only channels are left without them.
Publisher confirms on every publishing channel, plus the documentation of what the guarantee covers.
Adds MilvaionConfig:RabbitMQ:QueueType and the worker-side equivalent, defaulting to Classic. Every QueueDeclareAsync now routes its arguments through BuildQueueArguments so the type is applied consistently on both sides. The API and every connected worker must agree: RabbitMQ rejects a redeclare whose arguments differ from the stored queue with PRECONDITION_FAILED, and the API treats that as fatal, so a mismatch stops the whole application rather than degrading. Switching an existing queue also requires deleting it first - RabbitMQ cannot convert a queue's type in place.
Documents QueueType on both the API and the worker table, including that the two must agree and that an existing queue has to be deleted before its type can change.
Records that the whole topology is declared with the configured queue type, that API and workers must agree or the API stops at startup with PRECONDITION_FAILED, and that an existing queue cannot change type in place.
Configurable RabbitMQ queue type across the API and the worker SDK, with the documentation of the constraint that both sides must agree.
Redis ACLs are granted on key patterns, not on database numbers, so the database index cannot isolate one tenant from another. A prefix every key is built from is what makes isolation expressible: each user gets an ACL rule for their own namespace and can reach nothing else. That only holds if no key escapes the prefix. Worker registry, stats and API key entries still wrote to bare keys such as "workers:index", which left them shared across every deployment pointed at the same instance and outside any per-namespace ACL. They now hang off MilvaionConfig:Redis:KeyPrefix, which was already configurable but only partly honoured. Workers gain a matching Redis:KeyPrefix, and CancellationChannel derives from it unless set explicitly, so the cancellation channel lines up with the API without configuring the full name twice.
Replaces the worker's CancellationChannel row with KeyPrefix, from which the channel name is now derived, and keeps CancellationChannel documented as the explicit override. Adds the rationale the setting was missing. Redis grants ACLs on key patterns rather than on database numbers, so selecting a different Database isolates nothing and the prefix is what makes per-deployment isolation expressible at all. It only works because no key is written outside it: one unprefixed key is readable by every tenant sharing the instance and invisible to any per-namespace rule. Corrects three defaults in the API's Redis table that had been copied from the row above and read as 0: SyncTimeout, KeyPrefix and DefaultLockTtlSeconds.
The key tree listed a single "workers" HASH, which has not matched the implementation for some time: the registry is an index set plus per-worker and per-instance keys, with a job_counts hash under each instance. Adds the api key entries, which were missing entirely. Spells out why the prefix covers every one of them. Redis grants ACLs on key patterns rather than on database numbers, so isolation has to be expressed as a pattern and a pattern only isolates if nothing is written outside it.
Every Redis key is now built from a configurable prefix, so a deployment can be isolated with a single ACL pattern, plus the documentation of why that only works when nothing is written outside it.
MilvaionUI was declared as a "Website" project carrying .NET Framework 4.8 precompile properties. It is a Vite frontend, not a C# project, and the entry made dotnet build fail on the solution with MSB4249: the ASP.NET compiler is only available on the .NET Framework MSBuild. The worker testing SDK, which exists on disk, was missing. The compose file lives at the repository root, so the copy under build/ is removed rather than kept in two places. The README told readers to cd into build/ before running it, which would have stopped working.
Adds a dialog behind a Generate button on the cron field, covering the frequencies the scheduler supports: intervals with optional hour windows and weekday filters, weekly, monthly by day or by nth weekday, yearly. Always emits six fields, matching CronFormat.IncludeSeconds on the API side, and previews the next runs in UTC because the dispatcher evaluates against TimeZoneInfo.Utc. Rejects dates that never occur, such as 31 February: Cronos parses them but yields no occurrence, and the dispatcher deletes a job whose expression has no future occurrences. The disabled prop was already being passed by JobForm for externally scheduled jobs but never declared, so the field stayed editable under a section marked as managed elsewhere. It is now honoured by the input, the presets and the new button.
A visual builder behind the cron field, emitting the six-field format the scheduler parses and previewing the next runs in UTC.
favicon, logo and apple-touch-icon were hardcoded to "/" and 404'd under a non-root base path. The manifest was declared twice: once prefix-less here, once with the base path applied by vite-plugin-pwa.
Vite freezes asset URLs into the bundle, so the image was tied to one prefix and VITE_BASE_PATH had to be kept equal to MilvaionConfig:BasePath by hand. The frontend now builds with a placeholder that the API substitutes at startup, making the image prefix-agnostic and leaving the server-side value as the single source of truth. Adds PublicBasePath for reverse proxies that strip the prefix before forwarding: the browser resolves assets, router basename and API calls against it, while BasePath stays the prefix this application receives. It defaults to BasePath, so a pass-through proxy needs only the one value. BREAKING CHANGE: the VITE_BASE_PATH build arg is gone. Set MilvaionConfig__BasePath at run time instead of rebuilding the image.
The PublicBasePath section of 06-configuration.md belongs here: the note that the base path is applied at startup rather than baked into the bundle, that VITE_BASE_PATH is not a run-time setting, and the example for a reverse proxy that strips the prefix. It is already in the tree. Staging the whole file while amending fd1b746 ("Document the worker Redis key prefix") swept those hunks in, so that commit carries documentation its own message never mentions, and it reached master through the redis-key-prefix merge. This commit changes nothing. It exists so the history says where that content came from, instead of leaving it to whoever next reads fd1b746 and wonders why a commit about Redis keys documents a base path.
The SPA base path is applied at startup rather than frozen into the bundle, so one image serves any prefix, with PublicBasePath for reverse proxies that strip it before forwarding.
Brings in the upstream 1.2.3 release - runtime settings, alerting and the running job reconciliation - alongside the base path work.
- Eliminated all UserType-related enums, claims, DTOs, validation, and usages from backend and frontend. - Authorization now relies solely on [Auth] and role/permission claims, replacing [UserTypeAuth]. User creation, update, and token claims no longer handle UserType. - Updated tests, documentation, and UI to reflect this change.
Publisher confirms were hardcoded on every publishing channel. They are the right default - a publish that returns without them says nothing about whether the broker accepted the message - but the extra round trip per publish is not always the trade an operator wants to make. Adds PublisherConfirms to both the API and the worker SDK, defaulting to true so existing deployments are unaffected. The two are independent: unlike QueueType, this is a per-channel client concern and the sides do not have to agree. On the API the connection factory combines the caller's request with the setting, so consume-only channels stay unaffected. On the worker side the six call sites now go through BuildChannelOptions, mirroring how BuildQueueArguments already centralises the queue type.
The API, Application and Infrastructure projects point DocumentationFile at a path inside the source tree rather than under bin/, so the generated XML is tracked in git. Its content comes entirely from the /// comments, which means every commit that touched one left the tracked copy describing code that had already moved on - and left anyone who builds the solution staring at three permanently modified files in git status. This brings all three back in step with the sources. The drift had accumulated across several changes, so it covers the runtime base path options, the queue type helper and the publisher confirms setting, not just the last of them. Nothing reads these files at run time and they are not shipped in any package, so the alternative would be to stop tracking them and let each build produce its own. That is a wider decision than this branch, so they are regenerated here rather than removed.
Publisher confirms become an opt-out on both the API and the worker SDK, defaulting to on, following review feedback that they were unconditional.
Removing the UserType concept from the codebase left the column behind in the database. EF builds its INSERT from the model, so it stopped naming a column the model no longer knows about, but InitialCreate had declared that column NOT NULL with no default: Postgres substitutes NULL and rejects the row with 23502. Any installation that upgrades to this version can no longer create a user, and everything behind authentication fails with it, which is 167 integration tests. This migration exists to bring those installations back into a usable state without touching a single existing row, and to leave earlier versions able to run against the same database. Defaulting the column to AppUser (2), the value the removed property carried, lets the database fill in what the insert omits. The column is kept rather than dropped, so the historical values survive and a downgrade to a version that still reads UserType finds every row populated, including the rows written while this migration was applied. Down only drops the default. The model snapshot no longer carries UserType either, so it matches the model again and an unrelated migration won't pick up a stray DropColumn.
The key-prefix work renamed two private members in ApiKeyStore from _cacheKeyPrefix and _lastUsedKeyPrefix to PascalCase while turning them from constants into properties, and later work added four more PascalCase private members. The repository names private members _camelCase: of the roughly fifty private static readonly declarations and the private constants, the only PascalCase ones were these. Restores the two original names and brings the other four in line. Rename only - no behaviour changes.
The builder renders .modal-overlay and the other shared modal classes but imported only its own stylesheet. Those classes are defined in Modal.css, which routes pull in lazily along with their chunk, so on a page whose route never renders a Modal - the job and workflow forms - the overlay got no positioning and rendered inline under the field instead of over the page. It happened to work when the page was reached from a route that had already loaded Modal.css, which is why it looked intermittent. Imports Modal.css from the builder so the rules travel with the component. The two builder rules that share an element with a shared class are now qualified as .modal-content.cb-modal and .modal-body.cb-body: the two stylesheets land in separate chunks and Modal.css loads second, so equal specificity would otherwise have clamped the builder to the narrower .modal-content width.
Production hardening: clustered RabbitMQ, Redis namespacing, runtime base path — plus a visual cron builder
Refactored private property names in ApiKeyStore for C# convention consistency. Bumped PackageVersion to 10.2.0 across SDK and worker projects. Updated ApiWorker and ConsoleWorker to reference the new SDK.Worker version. Updated VERSION file.
Refactored RedisStatsService and RedisWorkerService to use PascalCase properties for key fields, improving consistency and readability. Updated all references accordingly. Wrapped LogInformation in RuntimeBasePathAssets with a logger level check to reduce overhead. Suppressed ASP0018 warning in MapRuntimeBasePathFallback with [SuppressMessage]. Refactored MakeUserTypeOptional migration to use expression-bodied members and clarified XML documentation placement.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
Please provide a brief description of the changes in this PR.
Related Issue
Fixes #(issue number)
Type of Change
Changes Made
Testing
Describe the tests you ran to verify your changes:
Test Configuration:
Checklist
Screenshots (if applicable)
Add screenshots to help explain your changes.
Additional Notes
Add any additional notes for reviewers.