Feature/webdav sync by jdkruzr · Pull Request #189 · Ethran/notable

jdkruzr · 2025-12-25T07:22:01Z

Outstanding Issues:

Needs image/other attachment hashing solution to avoid duplication (requires messing with database, tabling for now)
Needs folder deletion/cascade solution (also requires messing with database)

Implements Phase 1 of WebDAV synchronization feature: - Add dependencies: WorkManager, OkHttp, security-crypto - Add network permissions (INTERNET, ACCESS_NETWORK_STATE) - Create SyncSettings data class with sync configuration - Implement CredentialManager for encrypted credential storage - Implement WebDAVClient with full WebDAV operations - Basic authentication support - PROPFIND, PUT, GET, DELETE, MKCOL methods - Directory creation and file streaming support

Implements Phase 2 of WebDAV synchronization: - FolderSerializer: Convert folder hierarchy to/from folders.json - NotebookSerializer: Convert notebooks/pages/strokes/images to/from JSON - Handles manifest.json for notebook metadata - Handles per-page JSON with all strokes and images - Converts absolute URIs to relative paths for WebDAV storage - Supports ISO 8601 timestamps for conflict resolution Phase 2 complete. Next: SyncEngine for orchestrating sync operations.

Creates skeleton implementations for remaining sync components: Core Sync Components: - SyncEngine: Core orchestrator with stub methods for sync operations - ConnectivityChecker: Network state monitoring (complete) - SyncWorker: Background periodic sync via WorkManager - SyncScheduler: Helper to enable/disable periodic sync UI Integration: - Add "Sync" tab to Settings UI - Stub SyncSettings composable with basic toggle All components compile and have proper structure. Ready to fill in implementation details incrementally. TODOs mark where logic needs to be added.

- Fix KvProxy import path (com.ethran.notable.data.db.KvProxy) - Replace HTTP_METHOD_NOT_ALLOWED with constant 405 - Correct package imports in SyncEngine

Add full-featured sync settings interface with: - Server URL, username, password input fields - Test Connection button with success/failure feedback - Enable/disable sync toggle - Auto-sync toggle (enables/disables WorkManager) - Sync on note close toggle - Manual "Sync Now" button - Last sync timestamp display - Encrypted credential storage via CredentialManager - Proper styling matching app's design patterns All settings are functional and persist correctly. UI is ready for actual sync implementation.

showHint takes (text, scope) not (context, text)

Log URL and credentials being used, response codes, and errors to help diagnose connection issues

Use Dispatchers.IO for network calls (Test Connection, Sync Now). Switch back to Dispatchers.Main for UI updates using withContext. Fixes: NetworkOnMainThreadException when testing WebDAV connection

Core sync implementation: - syncAllNotebooks(): Orchestrates full sync of folders + notebooks - syncFolders(): Bidirectional folder hierarchy sync with merge - syncNotebook(): Per-notebook sync with last-write-wins conflict resolution - uploadNotebook/uploadPage(): Upload notebook data and files to WebDAV - downloadNotebook/downloadPage(): Download notebook data and files from WebDAV - Image and background file handling (upload/download) Database enhancements: - Add getAll() to FolderDao/FolderRepository - Add getAll() to NotebookDao/BookRepository Sync features: - Timestamp-based conflict resolution (last-write-wins) - Full page overwrite on conflict (no partial merge) - Image file sync with local path resolution - Custom background sync (skips native templates) - Comprehensive error handling and logging - Resilient to partial failures (continues if one notebook fails) Quick Pages sync still TODO (marked in code).

- Remove context parameter from ensureBackgroundsFolder/ensureImagesFolder - Fix image URI updating (create new Image objects instead of reassigning val) - Use updatedImages when saving to database - Handle nullable URI checks properly

Safety Features for Initial Sync Setup: - forceUploadAll(): Delete server data, upload all local notebooks/folders - forceDownloadAll(): Delete local data, download all from server UI: - "Replace Server with Local Data" button (orange warning) - "Replace Local with Server Data" button (red warning) - Confirmation dialogs with clear warnings - Prevents accidental data loss on fresh device sync Use cases: - Setting up sync for first time - Adding new device to existing sync - Recovering from sync conflicts - Resetting sync environment

Log notebook discovery, download attempts, and directory listings to diagnose sync issues

Features: - SyncLogger: Maintains last 50 sync log entries in memory - Live log display in Settings UI (last 20 entries) - Color-coded: green (info), orange (warning), red (error) - Auto-scrolls to bottom as new logs arrive - Clear button to reset logs - Monospace font for readability Makes debugging sync issues much easier for end users without needing to check Logcat.

Fixes: - forceUploadAll: Delete only notebook directories, not entire /Notable folder - Add detailed SyncLogger calls throughout force operations - Add logging to upload/download operations with notebook titles Log viewer now shows: - Exactly which notebooks are being uploaded/downloaded - Success/failure for each notebook - Number of pages per notebook - Any errors encountered This makes debugging sync issues much easier and prevents accidentally wiping the entire sync directory.

Add auto-sync trigger when switching pages in editor: - Hook into EditorControlTower.switchPage() - Pass context to EditorControlTower constructor - Trigger SyncEngine.syncNotebook() when leaving a page - Only syncs if enabled in settings - Runs in background on IO dispatcher - Logs to SyncLogger for visibility Now sync-on-close setting is functional.

Show clearly in sync log: - ↑ Uploading (local newer or doesn't exist on server) - ↓ Downloading (remote newer) - Timestamp comparison for each decision - Which path is taken for each notebook This will help diagnose why sync only goes up and never down.

Create AppRepository instance to properly access PageRepository in triggerSyncForPage method.

Previous logic: equal timestamps → upload New logic: equal timestamps → skip (no changes needed) Now properly handles three cases: - Remote newer → download - Local newer → upload - Equal → skip This prevents unnecessary re-uploads when nothing has changed.

Move sync trigger to EditorView's DisposableEffect.onDispose which fires when navigating away from the editor. Now sync-on-close actually works when you: - Navigate back to library - Switch to a different notebook - Exit the app Will show "Auto-syncing on editor close" in sync log.

Use new CoroutineScope instead of composition scope in onDispose. The composition scope gets cancelled during disposal, causing "rememberCoroutineScope left the composition" error. Now sync-on-close will actually complete.

Log when credentials are loaded or missing to help diagnose AUTH_ERROR issues.

Show full Date.time millisecond values and difference to diagnose why timestamps that appear equal are being treated as different. This should reveal if there are sub-second differences causing unnecessary uploads.

Add null-safe access to remoteUpdatedAt.time

Problem: ISO 8601 serialization loses milliseconds, causing local timestamps to always be slightly newer (100-500ms). Solution: Ignore differences < 1 second (1000ms) - Difference < -1000ms: remote newer → download - Difference > 1000ms: local newer → upload - Within ±1 second: no significant change → skip This prevents unnecessary re-uploads of unchanged notebooks while still detecting real changes.

After syncing local notebooks, now scans server for notebooks that don't exist locally and downloads them. Flow: 1. Sync folders 2. Sync all local notebooks (upload/download/skip) 3. Discover new notebooks on server 4. Download any that don't exist locally This enables proper bidirectional sync - devices can now discover and download notebooks created on other devices.

…c to a new service and introduce an abstraction for WebDAV client creation. - Extract notebook reconciliation and synchronization logic from `SyncOrchestrator` into the new `NotebookReconciliationService`. - Introduce `WebDavClientFactoryPort` and `WebDavClientFactoryAdapter` to abstract the creation of `WebDAVClient` instances, facilitating better testability and dependency injection. - Update `SyncOrchestrator` and `SyncForceService` to use the injected `WebDavClientFactoryPort` instead of direct instantiation. - Add comprehensive unit tests for `FolderSerializer`, `SyncPaths`, `SyncPorts`, and `SyncOrchestrator` components. - Fix minor formatting issues and remove unused imports in sync-related classes.

…s on server --AI-- - **WebDAVClient**: Added `DownloadedFile` data class to track content and ETags; updated `getFileWithMetadata` to capture ETags and `putFile` to support optional `If-Match` headers. - **Error Handling**: Introduced `PreconditionFailedException` to handle HTTP 412 status codes and added a corresponding `CONFLICT` state to `SyncError` and `SyncState`. - **Folder/Notebook Sync**: Updated `FolderSyncService` and `NotebookReconciliationService` to validate ETags before overwriting remote files, ensuring atomic updates. - **Sync Orchestration**: Updated `SyncOrchestrator` to catch and propagate conflict errors to the UI and `SyncWorker`. - **Code Quality**: Fixed formatting and updated method signatures in `NotebookSyncService` and `SyncForceService` to support the new synchronization logic.

… for improved dark mode support and UI consistency.

Ethran · 2026-04-03T21:35:52Z

@jdkruzr

I mostly corrected architecture of the syncing: the syncEngine was doing way too much. I'm still not sure if its the best that can be done, so feel free to adjust the file structure.

There also might be some bugs introduced during the refactor, currently I don't have any available webdav server.

Please, review the changes, make sure that there is no bugs, and other problems.

Ethran · 2026-04-04T12:51:36Z

ok, I got access to nextcloud server, I will be testing the implementation soon. Its very slow, but its might be perfect for caching race condition and other bugs.

If you will be making some changes, please push them as they are ready, or give a heads up on whats will you change in case of bigger changes.

…ed background tasks --AI-- ### Sync & WorkManager - **SyncScheduler**: Added `triggerImmediateSync` to enqueue `OneTimeWorkRequest` with support for various sync types (`syncAll`, `forceUpload`, `uploadDeletion`, etc.) and unique work names. - **SyncWorker**: Expanded to handle multiple sync actions from input data, returning detailed results and error statuses to `WorkManager`. - **SyncOrchestrator**: Refactored to use `@IoDispatcher`; updated state transitions to ensure `Idle` state is set after successful operations and improved error reporting for configuration issues. ### Dependency Injection & Scopes - **CoroutinesModule**: Introduced `CoroutinesModule` providing `@IoDispatcher` and a `@ApplicationScope` (using `SupervisorJob`) for tasks that must outlive UI components. - **Hilt**: Integrated new qualifiers across ViewModels and services to standardize thread and lifecycle management. ### UI & UX Improvements - **SnackState**: Converted to a `@Singleton` injectable component; added `runWithSnack` helper to manage long-running operations with coordinated snackbar feedback. - **SettingsViewModel**: Refactored manual sync and force actions to use `appScope` and `SnackState`, ensuring operations continue if the user navigates away. - **Editor/NotebookConfig**: Migrated "auto-sync" and "notebook deletion" tasks to use `ApplicationScope` or `SyncScheduler` for reliable background execution. - **MainActivity**: Updated app startup to trigger the initial sync via `SyncScheduler` rather than a direct coroutine.

--AI-- ### Sync & Background Tasks - **SyncScheduler**: Updated periodic and immediate sync requests to include a `sync_trigger` metadata field. - **SyncWorker**: Implemented logic to detect periodic syncs and display UI feedback via `SnackState`. - **SyncWorker**: Added a `try-finally` block to ensure sync completion snackbars are shown regardless of the operation's outcome. ### UI & Localization - Added `sync_scheduled_started` and `sync_scheduled_completed` strings to English and Polish resources.

Ethran · 2026-04-04T19:26:29Z

Problems to be addressed:

The sync progress bar is misleading. I suggest dividing it into specific steps (e.g., downloading/uploading changes, checking for deletions, etc.) with a percentage indicator for each stage. It would also be helpful to show the title of the notebook or the filename of the media currently being uploaded. In the current implementation, it hangs at 30% most of the time while uploading. I usually only know the app is still working by monitoring the server's CPU usage, as it’s hard to tell if it has errored out or is just uploading a large file.

1.5) Sync indicators in the Library view. It would be great to have a visual indicator for syncing in the Library view—perhaps a small icon on the notebook covers? The same could be done in the Pages View (the notebook pages overview).

Separate Server URL and WebDAV path. I would consider collecting the Server URL and the WebDAV path in separate fields. This would allow us to propose default options, such as remote.php/webdav for Nextcloud and others.
LastSyncStartTime vs. LastSyncTime. I'm not entirely sure about the current behavior, but when checking for remote changes, it should compare the notebook's modification date with the sync timestamp. We must be careful with changes occurring in the middle of a synchronization; therefore, I suggest using lastSyncStartTime as the reference point rather than the completion time.
Improved sync failure visibility. Failures should be better reported. My server stopped responding in the middle of a sync, yet the logs showed that the sync "completed" in 1,805,295ms. Since I have around 0.5GB of data, it takes a while to sync, and the app needs to distinguish between a timeout and a successful finish.
Consistent Snackbar notifications. It’s a good idea to show Snackbars when starting and finishing a sync. I’ve added these for scheduled syncing, but I think the "sync on notebook exit" trigger still doesn't show them.

Note: I haven't tested this with more than two devices yet; currently, I am testing the upload flow on my phone only.

Ethran · 2026-04-05T10:39:08Z

maybe it might be worth to add "upload only" option for the sync. Before it is fully tested I'm a little bit hesitated to allow sync changing the db.
And, as I have only one tablet, I plan to use sync only for backup, and having the view-only version on the phone.
I splitted the sync engine into smaller files. Please review this change, and let me know if its a good split -- I'm still a little hesitated if it is the best that it can be. I don't want too many files, but SyncEngine in its first form were too big, and hard to navigate around,

Other then the above comments it seems ok. I will for now focus on fixing the bugs in the repo introduced by recent refactor, and will look into this PR after your changes, @jdkruzr .

jdkruzr · 2026-04-05T22:58:01Z

Problems to be addressed:

The sync progress bar is misleading. I suggest dividing it into specific steps (e.g., downloading/uploading changes, checking for deletions, etc.) with a percentage indicator for each stage. It would also be helpful to show the title of the notebook or the filename of the media currently being uploaded. In the current implementation, it hangs at 30% most of the time while uploading. I usually only know the app is still working by monitoring the server's CPU usage, as it’s hard to tell if it has errored out or is just uploading a large file.

1.5) Sync indicators in the Library view. It would be great to have a visual indicator for syncing in the Library view—perhaps a small icon on the notebook covers? The same could be done in the Pages View (the notebook pages overview).

Separate Server URL and WebDAV path. I would consider collecting the Server URL and the WebDAV path in separate fields. This would allow us to propose default options, such as remote.php/webdav for Nextcloud and others.

LastSyncStartTime vs. LastSyncTime. I'm not entirely sure about the current behavior, but when checking for remote changes, it should compare the notebook's modification date with the sync timestamp. We must be careful with changes occurring in the middle of a synchronization; therefore, I suggest using lastSyncStartTime as the reference point rather than the completion time.

Improved sync failure visibility. Failures should be better reported. My server stopped responding in the middle of a sync, yet the logs showed that the sync "completed" in 1,805,295ms. Since I have around 0.5GB of data, it takes a while to sync, and the app needs to distinguish between a timeout and a successful finish.

Consistent Snackbar notifications. It’s a good idea to show Snackbars when starting and finishing a sync. I’ve added these for scheduled syncing, but I think the "sync on notebook exit" trigger still doesn't show them.

Note: I haven't tested this with more than two devices yet; currently, I am testing the upload flow on my phone only.

good, just saw this, will look tonight.

Ethran · 2026-04-12T10:18:56Z

good, just saw this, will look tonight.

Hi @jdkruzr, following up on this. I'd really like to get this merged soon! The main branch is moving forward, and I don't want you to have to deal with a bunch of messy merge conflicts. Let me know when you've had a chance to look.

Splits sync progress into step-level + per-notebook reporting so the progress bar advances within the SYNCING_NOTEBOOKS band instead of hanging at 30%. Adds a new SyncProgressReporter Hilt singleton that owns the SyncState StateFlow; services report per-item progress via beginItem/endItem. SyncSettingsTab renders a step label, flat e-ink progress bar, and "Notebook X of Y · Name" line. Addresses Ethran's 2026-04-04 feedback item Ethran#1. Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>

Ethran · 2026-04-18T18:34:18Z

Great to see progress, but please review code written by agents.

Can you also resolve marge conflicts?
Snacks changed a little bit, and I decided to add AppResult, here are the docs:

https://github.com/Ethran/notable/blob/main/docs/result-and-error-handling.md
https://github.com/Ethran/notable/blob/main/docs/snacks.md

Pure additions from upstream/main. SnackDispatcher (Hilt singleton app-wide snack API) and AppEventBus coexist with the existing SnackState.globalSnackFlow pattern so call sites can migrate incrementally before the upstream merge. Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>

Replace SnackState.globalSnackFlow.emit with SnackDispatcher.showOrUpdateSnack for scheduled-sync start/finish notifications. Expose SnackDispatcher via SyncOrchestratorEntryPoint so the non-Hilt Worker can reach it through EntryPointAccessors. Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>

Replace snackState.runWithSnack() with a direct SnackDispatcher sequence: emit a during-snack with a stable id and duration=null, run the sync action inside try/catch, then update the same snack with the result text and a 3s duration. Gains: SnackDispatcher is the upstream app-wide API; also preserves exception safety when the block throws. Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>

Inject SnackDispatcher and replace the three SnackState.globalSnackFlow call sites (handleExport, fixNotebook, updateOpenedPage) with snackDispatcher.showOrUpdateSnack. logAndShowError companion calls are left in place for now; they still compile against the existing SnackState and will be resolved when upstream's SnackBar.kt replaces ours in the merge step. Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>

Brings in upstream's Hilt/DI-era refactor: SnackDispatcher as the app-wide snack API, AppEventBus + AppEventUiBridge for domain-level event publication, EditorControlTower constructor overhaul (viewModel + clipboardStore), and the EditorViewModel showHint/fixNotebook reshaping. Conflict resolutions: - EditorViewModel: merged constructors (syncOrchestrator + snackDispatcher + historyFactory + appScope); took upstream's fixNotebook with "Remove bad page" action; added syncFromPageId delegate. - EditorControlTower: dropped state/syncOrchestrator params, adopted upstream's viewModel + clipboardStore; triggerSyncForPage now routes via viewModel.syncFromPageId. - EditorView: removed the EntryPointAccessors hop. - MainActivity: kept sync boot (CredentialManager, restore/trigger sync); took upstream SnackDispatcher + AppEventUiBridge; dropped obsolete snackState.register* calls. - SettingsViewModel: merged constructors; removed SyncSettingsEffect + syncEffects flow (redundant with SnackDispatcher). - Settings.kt: took upstream's openInBrowser(onError) signature; removed the now-orphaned syncEffects collector. - SnackBar.kt: take-theirs. Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>

Before, refreshUi calls made before SurfaceHolder.Callback.surfaceCreated fired silently no-op (lockCanvas returns null), but the finally block still logged "Canvas refreshed", hiding the fact that nothing painted. The canvas then stayed blank until an unrelated event (toolbar toggle, pen menu close) triggered refreshUi again on the now-drawable surface. Manifested as a blank canvas area on note re-open after the upstream merge: the app was otherwise responsive but the drawing region showed black until the user tapped a tool. Schedule a repaint from surfaceCreated so the first drawable surface is always populated. Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>

jdkruzr added 30 commits December 11, 2025 15:59

stub commits for synchronization, plus new serialization code

7115027

Fix compilation errors

f56bde7

- Fix KvProxy import path (com.ethran.notable.data.db.KvProxy) - Replace HTTP_METHOD_NOT_ALLOWED with constant 405 - Correct package imports in SyncEngine

Fix showHint argument order

8dfd9d8

showHint takes (text, scope) not (context, text)

Add debug logging for connection testing

876bbe5

Log URL and credentials being used, response codes, and errors to help diagnose connection issues

Fix NetworkOnMainThreadException in sync operations

6a60635

Use Dispatchers.IO for network calls (Test Connection, Sync Now). Switch back to Dispatchers.Main for UI updates using withContext. Fixes: NetworkOnMainThreadException when testing WebDAV connection

Fix SyncEngine compilation errors

a72ae42

- Remove context parameter from ensureBackgroundsFolder/ensureImagesFolder - Fix image URI updating (create new Image objects instead of reassigning val) - Use updatedImages when saving to database - Handle nullable URI checks properly

Add detailed logging for sync debugging

24be07e

Log notebook discovery, download attempts, and directory listings to diagnose sync issues

Fix EditorControlTower repository access

5ab3f86

Create AppRepository instance to properly access PageRepository in triggerSyncForPage method.

Fix coroutine scope cancellation in onDispose

8f3c044

Use new CoroutineScope instead of composition scope in onDispose. The composition scope gets cancelled during disposal, causing "rememberCoroutineScope left the composition" error. Now sync-on-close will actually complete.

Add credential loading debug logging

5562949

Log when credentials are loaded or missing to help diagnose AUTH_ERROR issues.

Add millisecond-precision timestamp logging

6a500c0

Show full Date.time millisecond values and difference to diagnose why timestamps that appear equal are being treated as different. This should reveal if there are sub-second differences causing unnecessary uploads.

Fix null safety in timestamp logging

9f912f6

Add null-safe access to remoteUpdatedAt.time

Fix bug in sync order.

839d93e

Deletion semantics update.

28042b1

Fix bug with timestamp updates not happening in manifest.

66b89b9

Fix deletion synchronization order of operations.

ea9fb36

Ethran added 5 commits April 3, 2026 22:35

update docs

5cf0715

update docs

d2b847f

Update SyncSettingsTab to use MaterialTheme colors and components…

021266a

… for improved dark mode support and UI consistency.

Ethran added 2 commits April 3, 2026 23:37

format code

10964f8

Restore .idea to match origin/main

fa00f28

Ethran added 7 commits April 4, 2026 15:43

fixed auto cleaning of broken notebook.

01eb26d

Remove symbols from sync logs and UI strings

331b307

do not repopulate password field from db, and button to show password

5906890

Improve Sync Settings UI with E-Ink inspired design and improved layout

b7fa920

Added configuration of sync interval and fix sync on startup.

5438689

Ethran and others added 2 commits April 6, 2026 00:34

Merge branch 'main' into feature/webdav-sync

2ae367e

finish marge of recent changes to main branch

99d6a3c

jdkruzr and others added 6 commits April 18, 2026 13:45

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Feature/webdav sync#189

Feature/webdav sync#189
jdkruzr wants to merge 92 commits intoEthran:mainfrom
jdkruzr:feature/webdav-sync

jdkruzr commented Dec 25, 2025

Uh oh!

Ethran commented Apr 3, 2026

Uh oh!

Ethran commented Apr 4, 2026

Uh oh!

Ethran commented Apr 4, 2026

Uh oh!

Ethran commented Apr 5, 2026

Uh oh!

jdkruzr commented Apr 5, 2026

Uh oh!

Ethran commented Apr 12, 2026

Uh oh!

Ethran commented Apr 18, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Uh oh!

Conversation

jdkruzr commented Dec 25, 2025

Uh oh!

Ethran commented Apr 3, 2026

Uh oh!

Ethran commented Apr 4, 2026

Uh oh!

Ethran commented Apr 4, 2026

Uh oh!

Ethran commented Apr 5, 2026

Uh oh!

jdkruzr commented Apr 5, 2026

Uh oh!

Ethran commented Apr 12, 2026

Uh oh!

Ethran commented Apr 18, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants