feat(import): add optional parallel CSV import - #2908
Merged
openai0229 merged 48 commits intoSep 16, 2026
Merged
openai0229 merged 48 commits into
openai0229 merged 48 commits into
Conversation
Execute insert statements in bounded JDBC batches while committing only transactions opened by the executor. Preserve caller-owned transactions, roll back failed executor-owned chunks, discard unusable connections after rollback or auto-commit restoration failures, and keep task cancellation as the primary failure. Add focused coverage for chunking, caller-managed transactions, cancellation, rollback failures, and connection restoration failures.
…ransactions fix(spi): preserve JDBC transaction ownership in batch imports
Execute insert statements in bounded JDBC batches while committing only transactions opened by the executor. Preserve caller-owned transactions, roll back failed executor-owned chunks, discard unusable connections after rollback or auto-commit restoration failures, and keep task cancellation as the primary failure. Add focused coverage for chunking, caller-managed transactions, cancellation, rollback failures, and connection restoration failures.
Store task state, events, resume journals, and artifact metadata in the embedded task database with migration from the legacy file store. Preserve lifecycle semantics across restart and expose preview, resume, artifact-list, and artifact-download contracts through the task API. Add task-center and log interfaces for resuming interrupted work and opening each generated artifact. Keep the legacy SQL workflow available in this batch so the task storage change remains independently deployable. Cover storage migration, lifecycle recovery, controller conversion, artifact handling, and task-center rendering; verify the Community frontend build at this commit.
…ntrols Execute row-oriented imports through adaptive batching, bounded concurrency, durable resume journals, deterministic column mapping, and explicit error policies. Retain staged SQL sources until successful completion so interrupted ordered imports remain recoverable. Run a mandatory read-only admission scan before creating workers. Reject unreadable, compressed, multiline, structurally ambiguous, or unsupported parallel inputs; downgrade small CSV files; and allow operators to acknowledge only strong relationship and ordering risks while preserving hard blockers and audit events. Expose import file selection, preview, mapping, mode switching, inline risk confirmation, and explainable admission verdicts in the shared wizard. Keep advanced export formats hidden and the legacy SQL export dialog mounted until the export batch supplies its backend contracts. Cover batching, concurrency, resume, admission, staging, mapping, request conversion, and wizard parameters; verify lint and the complete Community Web build at this commit.
…zard Stream CSV, Excel, JSON, NDJSON, Markdown, and SQL output through format sinks with rate limiting, optional GZIP compression, resumable checkpoints, keyset shard planning, deterministic SQL value serialization, and large-cell handling. Route table, schema, database, and data-source export actions through the shared wizard. Let desktop users select a destination directory, let all users set a suggested file name, expose compression and checkpoint controls only with the supporting backend, and keep browser artifact downloads available. Remove the superseded SQL export dialogs and dispatch helper only after the unified route is active. Preserve ordered fallback when a database cannot provide a safe keyset capability. Cover sinks, extensions, checkpoint resume, shard planning, rate limiting, SQL serialization, request conversion, and wizard parameters; verify lint and the complete Community Web build at this commit.
Opt the H2 plugin into the generic keyset-sharding export path after the default database capability was made conservative. Add a focused plugin regression test so H2 cannot silently fall back to serial-only exports.
Opt the MySQL plugin into keyset-sharded exports and establish REPEATABLE READ with a consistent snapshot before parallel readers are opened. Cover the dialect-owned SQL and capability flag with focused unit tests. Add credential-gated local MySQL round-trip and ten-million-row stress fixtures, with the connector confined to test scope.
Conflicts were limited to the task artifact APIs. Kept upstream's non-overwriting publication path (CREATE_NEW target creation plus the ARTIFACT_PUBLICATION_STARTED listener) and the batch's resumable multi-artifact APIs, so exports publish without clobbering existing files while interrupted runs still resume and clean up their drafts.
The upstream 50MB staging cap blocked parallel imports on files large enough to benefit and prevented web users from selecting bigger sources. Raise the backend staging cap to 2GB and the wizard's client-side guard to 2048MB to match. Verified against a local MySQL 8.3 target: a 10,000,000-row / 798MB CSV staged through the upload endpoint imports in about 19 minutes and exports back to CSV in about 73 seconds.
The JDBC writer chunked a row batch into 500-statement commits, so a failure in the middle of a batch left an already-committed prefix behind while the batch-granular resume watermark still pointed at the batch start. A resume then replayed those durable rows and rejected them as duplicates. Import batches are now executed as a single transaction (the new chunk-size overload keeps the 500-statement default for other callers), so a failed batch rolls back entirely and the watermark stays authoritative.
The fast mode now starts at a baseline of 4 concurrent batches of 20000 rows, may shrink to 1 batch of 100 rows when the target is slow, and grows without a configured ceiling while the measured throughput keeps improving: the batch sizer hill-climbs on rows per second instead of an absolute latency band, the AIMD gate floor drops to one permit, the import worker pool grows on demand to match the gate (cached pool plus one queue per worker), and the export shard plan and process budget are derived from the machine instead of a fixed 16. Tests were adapted to the new baseline, including a stronger resume assertion that a failed batch leaves no row above the durable watermark.
…m pin Two defects surfaced by the larger baseline batch: - The journal/checkpoint/snapshot cadence only counted batches, so with 20000-row batches a crash could leave up to 1.28M durable rows above the storage watermark and the resume replayed them all as duplicate rejections. Each layer now also fires on a row interval (journal 20k, checkpoint and snapshot 50k rows), keeping the resume window bounded whatever the tuned batch size is. - An explicit chat2db.task.import.parallelism value documented a pinned fan-out but the unbounded gate could grow past it. The pin now caps the gate ceiling; only the default keeps growing without a limit.
…allelism The adaptive gate could grow without any ceiling, which risks asking for more worker threads than the computer can actually run. The fan-out is now bounded by Runtime.availableProcessors(): the AIMD tuning moves inside [4 baseline, processor count], an explicit chat2db.task.import.parallelism pin is clamped by the same ceiling, and the export shard plan and process budget use the processor count instead of a fixed multiplier. The batch size stays unbounded; only the thread count follows the machine.
The adaptive sizer grew the export sink batch into the millions of rows (2.56M observed on a 10M-row export): a single writer buffering that much per flush slowed the run down and the AIMD gate then collapsed to one permit. Sink batches are now bounded by the shard page size, so the tuner still adapts inside [baseline, 50000] while the import keeps its unbounded growth.
A failed batch is no longer replayed row by row: it is bisected, healthy halves are applied as batches again, and each offending row is isolated with its own cause. Isolated rows are screened with their exact file row numbers (reject event and artifact) and the import continues, so k bad rows cost O(k log n) executions instead of replaying the whole batch. On a resumed run a duplicate-key collision means the earlier run had already applied the row, so the new ResumeDuplicatePolicy decides what happens: RECONCILE (default) records it in its own reconciliation artifact and event, never charges it against maxErrors, and finishes; REJECT keeps the historical counting; FAIL aborts on the first such row. Fresh imports keep the historical behaviour, and the summaries report imported, already-applied and rejected rows separately.
The wizard gains a "duplicate rows on resume" choice next to the error handling controls: record as already applied (default), count as a rejected row, or stop the task. The value travels with the import options, and the labels are translated in every shipped locale.
…atch1-fast-mode The fork's main only changed DefaultSQLExecutor.java (the JDBC transaction-ownership fix already carried byte-identically by this branch), so the branch version - newer upstream plus the chunk-size overload - was kept and the merge is content-neutral.
Keeps the fork's own history (the PR #1 JDBC transaction-ownership fix) and brings main up to upstream so the batch-1 PR shows only the feature diff instead of the whole upstream delta.
…iour The fast-mode tuning contract leaked into standard mode: the plain import used the 20000-row batch baseline with one transaction per batch, and the serial export used the 20000-row sink baseline. Standard mode now keeps exactly the historical behaviour - 500-row batches with the 500-statement chunked commits (so its resume watermark stays exact because batch and chunk sizes match) and the 500-row sink batch - while fast mode keeps the 20000-row baselines, the atomic batch and the adaptive sizing. Only the fast-mode switch selects the new machinery.
The per-dialect snapshot was wired behind a property that defaulted to off, so no dialect ever pinned its shard reads to a consistent transaction. It is now on by default: every shard read runs inside the dialect's snapshot, and a dialect without a snapshot implementation or one whose statement the server rejects (SQL Server without ALLOW_SNAPSHOT_ISOLATION, for example) degrades to the plain auto-commit read with a warning. Opt out with -Dchat2db.task.shard.consistent-snapshot=false.
…-batch1-fast-mode
liushikuan63
requested review from
a team,
Aias00 and
openai0229
as code owners
September 13, 2026 08:48
# Conflicts: # chat2db-community-server/chat2db-community-domain/chat2db-community-domain-core/src/main/java/ai/chat2db/community/domain/core/impl/task/RunningTask.java
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Related issue
N/A — no linked issue was supplied.
Summary
CSV imports can opt into
FASTafter file preview and confirmation that their rows have no ordering dependencies. The importer executes batches on dedicated JDBC connections and adjusts concurrency and batch size using measured throughput. Workers inherit the existing task statement guards and request/logging context. Completed batches report progress; failure or cancellation cancels registered statements and waits for every worker to exit. Dedicated connections are closed once and never returned to the pool. Worker startup and expansion failures, including thread-creation errors, stop pending work and release started workers before the original failure is propagated; errors while converting CSV rows do not flush buffered SQL during cleanup. The default mode isSTANDARD. Mode values are the uppercase names ofTaskExecutionMode(STANDARD,FAST), while request fields remain strings. Converters pass mode through without normalization; only exactFASTenables parallel execution. The ordinary CSV implementation remains behind a single explicit mode dispatch. CSV and Excel retain the existing ordinary import behavior; their row-to-SQL conversion is extracted unchanged intoImportRowSqlBuilderand shared with fast CSV.ImportRowBatcheraccepts generated SQL.ImportSqlExecutormatches the base branch, and all existingDefaultSQLExecutormethods retain their original bodies. Only the fast CSV path calls the newexecuteJdbcBatchInsertmethod.The existing file picker, CSV settings, preview, and column mapping remain the workflow. SQL-file import UI and generic submission helpers match the base branch. Execution uses the existing
CsvOptionsand mapping contract. Errors fail the import. Fast JDBC batches use the connection's existing transaction settings; failed or cancelled execution can leave rows already committed by the driver. Cancellation tracks parallel statements and unregisters closed statements.This PR is scoped to imports. Export implementations, export request models, database plugins, and export UI behavior match the base branch. Task storage uses the existing file implementation and single-output contract.
There is no pre-import diagnosis, full-file admission scan, generated admission report, or server-side relationship acknowledgement. The existing UI warning and confirmation remain. CSV uses the existing parser validation and streaming execution passes; quoted embedded newlines are supported in fast mode. SQL, Excel, and JSON retain serial execution.
Batch size grows only up to 50,000 rows. Batches also flush at 2,097,152 SQL characters (about 4 MiB of UTF-16 text); a single larger row is sent alone. Column names preserve whitespace consistently with the existing preview and ordinary importer. File staging keeps the base-branch 50 MiB limit. There is no additional import-directory allowlist. Worker batches require a concurrency permit and keep responding to cancellation while waiting; timeouts never allow execution without a permit.
Affected surfaces
Verification
mvn -B -f chat2db-community-server/pom.xml -pl :chat2db-community-domain-core,:chat2db-community-web -am -Dmaven.test.skip=false -DskipTests=false '-Dsurefire.includes=**/*Test.java' -Dsurefire.failIfNoSpecifiedTests=false -Dmaven.test.failure.ignore=false package: 701 tests, zero failures/errors/skips; BUILD SUCCESS. Fault-injection coverage includes partial worker startup with interruption, rejected startup, expansion failures during accept/flush, and conversion errors with buffered rows; four error paths fail before the fix and pass afterward. All eight extracted conversion method bodies were compared against the previous implementation and match after ignoring whitespace.mvn -B -f chat2db-community-server/pom.xml -pl :chat2db-community-storage,:chat2db-community-domain-core,:chat2db-community-web -am -Dmaven.test.skip=false -DskipTests=false '-Dsurefire.includes=**/*Test.java' -Dsurefire.failIfNoSpecifiedTests=false -Dmaven.test.failure.ignore=false package: 752 tests, zero failures/errors/skips; BUILD SUCCESS. Coverage includes worker cancellation and termination after failure, user cancellation during blocked JDBC execution, dedicated connection closure and absence of pool entries, propagated task guards and request context, monotonically increasing progress, row/text batch bounds, whitespace-preserving column mappings, ordinary-mode compatibility, parallel import of 200,000 rows, and no added transaction control.chat2db-community-client:yarn run test:import-preview,yarn run lint, andyarn run build:web:community --app_version=0.0.0: passed. The build also runs its existing frontend regression suite.mvn -B package -Dmaven.test.skip=true -Dchat2db.finalName=chat2db-community -f chat2db-community-server/pom.xml -pl chat2db-community-start -am: executable packaging passed; this command skips tests.git diff --check: passed.Risk and compatibility
mode; the confirmation is a UI interaction. CSV settings and mappings use the existing contract. Task storage and export APIs retain their base-branch models.Reviewer map
ImportMappingContent,ImportModeControl,DbMappedImportServiceImpl,ParallelCSVImporter,ImportRowBatcher,RunningTask,TaskExtensionManager, andDefaultSQLExecutor.executeJdbcBatchInsert. Lifecycle regression coverage is inParallelImportLifecycleTest.STANDARD) to use the original import execution path.Contributor declaration
AI assistance: Codex assisted the maintainer revisions, scope reduction, conflict resolution, and verification.