Skip to content

TODO.md

Latest commit

 

History

History
318 lines (259 loc) · 15.1 KB

TODO.md

File metadata and controls

318 lines (259 loc) · 15.1 KB

OpenADS — TODO

Items are ordered by priority within each section. Check off completed work and commit the file update so it stays current.

Data Dictionary (DD) support

Done

  • Text-format DD parse + save (# OpenADS Data Dictionary v1): TABLE, INDEX, USER, MEMBER, LINK, RI, DBPROP, USERPROP rows; atomic write-then-rename save.
  • AdsConnect60 opens a .add path — parent dir becomes data dir.
  • Full DD ABI CRUD: AdsDDAddTable, AdsDDRemoveTable, AdsDDCreateUser, AdsDDDeleteUser, AdsDDAddUserToGroup, AdsDDRemoveUserFromGroup, AdsDDCreate{,Drop,Modify}Link, AdsDDCreate/RemoveRefIntegrity, AdsDDSet/GetDatabaseProperty, AdsDDGet/SetUserProperty, AdsDDCreate, AdsDDAddTable90, AdsDDCreateRefIntegrity62.
  • Binary .add reader — DataDict::load_add_binary_(). Parses all Table records from real ADS proprietary files. Fixture: tests/fixtures/adi/pmsys.add. (2026-05-24)

Open

  • Binary .add write support — full round-trip mutations on ADS proprietary Data Dictionary files. save_add_binary_() serializes all records back in exact binary layout (524-byte records, 2200-byte header updated in-place). add_table, remove_table, add_index_file, remove_index_file, create_user, delete_user all dispatch through the binary path when binary_format_ is set. 4 round-trip tests added to tests/unit/data_dict_test.cpp. (2026-05-24)

  • AdsDDGetTableProperty / AdsDDSetTableProperty. Exported. Handles: RELATIVE_PATH (211), TABLE_PATH (205), TABLE_TYPE (204, inferred from extension), CHAR_TYPE (212), OBJ_ID (208), FIELD_COUNT (206, returns 0), and the numeric-zero boolean properties. Set returns AE_FUNCTION_NOT_AVAILABLE. 5 unit tests in tests/unit/abi_dd_table_prop_test.cpp. (2026-05-24)

  • AdsDDSetUserProperty — implemented and exported. Property code dispatch: 1102 (GROUP_MEMBERSHIP) → add_user_to_group; 1103 (BAD_LOGINS) → read-only no-op; all other codes (including 1101 PASSWORD, 1 COMMENT) stored as "prop_N" string properties, symmetrical with AdsDDGetUserProperty. AdsDDCreateUser now stores password (prop_1101), description (prop_1), and optional group membership. 7 tests in tests/unit/abi_dd_user_prop_test.cpp. Cross-checked against f:\harbour3.2-bcc7.3\contrib\rddads\. (2026-05-24)

  • ADI level-2 page navigation — fixed. Character-key ADI indexes (CICHAR/CHAR fields) use level-2 dense-leaf pages instead of level-3. Branch entries use a different format: padded_key[(len+3)&~3] + cum[4 LE] + page[1] vs numeric's key[8 BE] + cum[4 BE] + page[4 BE]. Compound-key tags (e.g. "F2;F14") are now parsed and their total key length computed correctly. is_dense_leaf() now accepts both level-2 and level-3 pages. encode_adt_key handles CICHAR/CHAR (returns raw field bytes). 2 new unit tests in abi_adi_smoke_test.cpp. (2026-05-24)

  • RI enforcement at write time. INSERT (AdsWriteRecord after AdsAppendRecord): validates FK exists in parent table via linear scan; blank FK skips the check (NULL semantics). DELETE (AdsDeleteRecord): enforces delete_opt — RESTRICT (2) rejects if children exist; CASCADE (1) marks child rows deleted; SETNULL (3) / SETDEFAULT (4) blank the child FK field. pending_appends() set tracks in-flight appends; in_ri_check() flag prevents recursive enforcement on cascade actions. UPDATE enforcement deferred (see below). 8 tests in tests/unit/abi_dd_ri_test.cpp. (2026-05-25)

  • RI enforcement on UPDATE (parent key change). PK snapshot captured at navigation time (AdsGotoRecord/Top/Bottom/Skip/Seek) via pk_snapshots() map keyed by Table*. At AdsWriteRecord, the new PK (from dirty buffer) is compared to the old PK (from snapshot). If they differ, update_opt is enforced: RESTRICT rejects and restores the old value on disk; CASCADE updates child FK fields; SETNULL/SETDEFAULT blanks child FK fields. On-disk rollback on RESTRICT: set_field writes immediately so ri_enforce_update re-writes the old value when rejecting. 4 tests in tests/unit/abi_dd_ri_test.cpp. (2026-05-26)

  • DD authentication for local connections. AdsConnect60 checks the DD's ADS_DD_LOG_IN_REQUIRED property (prop_5). When set, the supplied pucUser/pucPwd are validated against the DD: unknown user or password mismatch both return AE_LOGIN_FAILED (7077). On success the authenticated username is stored on Connection::username_ for future permission checks. 6 tests in tests/unit/abi_dd_auth_test.cpp. (2026-05-25)

  • Per-table access control / permission checking. DD stores TABLEPERM <table>;<user_or_group>=<level> entries (0=none, 1=read, 2=write, 3=delete, 4=full). Effective level is max of direct user perm and any group memberships; tables with no ACL default to full access. AdsOpenTable with usCheckRights≠0 enforces the level (1 for ADS_READONLY, 2 otherwise). SQL AdsExecuteSQLDirect with check_rights≠0 enforces per-operation (SELECT→1, INSERT/UPDATE→2, DELETE→3). AdsDDGetTableProperty for property 216 returns effective level. New AdsDDSetUserTableRights / AdsDDGetUserTableRights manage per-user/group permissions programmatically. 7 tests in tests/unit/abi_dd_perms_test.cpp. (2026-05-26)

  • AdsDDGetFieldProperty / AdsDDSetFieldProperty — per-field metadata read/write. Structural props (name 301, type 302, length 303, decimals 304) read live from the table file via a brief open; stored props (required 305, default 306, rule 307, msg 308, comment 309) stored in DataDict::field_props_ and persisted as FIELDPROP rows in the text format. 6 tests in tests/unit/abi_dd_field_prop_test.cpp. (2026-05-26)

  • DD triggers — AdsDDCreateTrigger / AdsDDDropTrigger / AdsDDGetTriggerProperty / AdsDDSetTriggerProperty. Model: TriggerEntry { name, table_alias, event_mask, container, procedure, priority, enabled, comment } in DataDict::triggers_. Persisted as TRIGGER rows in the text format. Event mask uses ADS_BEFORE/AFTER_INSERT/UPDATE/DELETE bits. Execution is a no-op stub (trigger definition is stored and queryable; user code not called). 5 tests in tests/unit/abi_dd_trigger_test.cpp. (2026-05-26)

  • DD stored procedures — AdsDDCreateProcedure / AdsDDDropProcedure / AdsDDGetProcProperty / AdsDDSetProcProperty. Model: ProcEntry { name, container, procedure, input_params, output_params, comment } in DataDict::procs_. Persisted as PROC rows. Execution is a no-op stub. 4 tests in tests/unit/abi_dd_proc_view_test.cpp. (2026-05-26)

  • DD views — AdsDDCreateView / AdsDDDropView / AdsDDGetViewProperty / AdsDDSetViewProperty. Model: ViewEntry { name, sql, comment } in DataDict::views_. Persisted as VIEW rows. AdsOpenTable expansion of view alias → AdsExecuteSQLDirect deferred (see system.* SQL item below). 5 tests in tests/unit/abi_dd_proc_view_test.cpp. (2026-05-26)

  • system.* SQL virtual tables — SELECT * FROM system.tables and 10 other aliases: system.indexes, system.users, system.usergroups, system.permissions, system.relations, system.links, system.triggers, system.storedprocedures, system.views, system.dictionary. Each builds an in-memory temp DBF from DataDict state and opens it as a read-only cursor. AdsOpenTable view-alias expansion: opening a DD view name executes the view's SQL via AdsExecuteSQLDirect and returns the cursor. Both DBF and ADT table types are reflected in system.tables.TABLE_TYPE. 9 tests in tests/unit/abi_sql_system_tables_test.cpp. (2026-05-26)

  • AdsDDGetIndexProperty / AdsDDSetIndexProperty — per-index metadata read from open index bindings. Properties: file name (401), expression (402), unique (403), descending (404), condition (405, returns ""), key length (406), type (407, returns 0). AdsDDSetIndexProperty returns AE_FUNCTION_NOT_AVAILABLE. (2026-05-26)

  • DD-related SQL statements — complete: CREATE DATABASE "path" [PASSWORD ... DESCRIPTION ... ENCRYPT ...]; GRANT right [("col")] ON object TO principal; REVOKE right [("col")] ON object FROM principal; 17 built-in sp_* stored procedures via EXECUTE PROCEDURE: sp_CreateUser/DropUser, sp_CreateGroup/DropGroup, sp_AddUserToGroup/RemoveUserFromGroup, sp_ModifyUserProperty/ModifyGroupProperty, sp_AddTableToDatabase/AddIndexFileToDatabase, sp_ModifyTableProperty/ModifyFieldProperty, sp_CreateReferentialIntegrity/DropReferentialIntegrity, sp_CreateLink/DropLink, sp_ModifyDatabase. system.iota (1-row scalar table) and system.columns (per-field metadata) added to virtual-table set. DataDict gains explicit GROUP storage + create_group/delete_group. 10 tests in tests/unit/abi_sql_dd_sql_test.cpp. (2026-05-26)

  • ADS proprietary ADT encryption — out of scope for now. We use our own AES encryption (M11.2). ADS-original per-table encryption format not reversed; tables with that flag will open but return garbled values. Deferred indefinitely.

ADT / ADM driver

Done

  • AdsPackTable / AdsZapTable for ADT. platform::File::truncate() added (Win32 + POSIX). AdtDriver::zap() truncates the file to hdr_len after zeroing the record count, so Table::pack() (zap + re-append survivors) leaves no stale bytes. Tests in tests/unit/abi_zap_pack_test.cpp verify record count, field values, and exact file size post-operation. (2026-05-24)

Open

VFP driver

Open

  • VFP table support (DBF 0x30 / 0x31). table.cpp:49 returns an error for VFP-typed DBF files. Was in the original M4 plan but deferred. Needs a VfpDriver that handles the _NullFlags system field for NULL bitmap and the VFP autoinc field type.

SQL engine

Open

  • CONTAINS / LIKE in join-cursor and aggregate WHERE. Both the join-cursor compile path and the aggregate FILTER compile path only handle Cmp / AND / OR / NOT — anything else returns AE_FUNCTION_NOT_AVAILABLE. CONTAINS and LIKE are the most common missing operators. Fixed in ace_exports.cpp: join-cursor compile lambda, aggregate FILTER cf lambda, and CASE WHEN compile_cond lambda all now support LIKE (strip trailing spaces + sql_like_match) and CONTAINS (load .fts via Fts::load/search before building the lambda; capture the hit set by shared_ptr). 3 new tests in tests/unit/abi_sql_contains_test.cpp. (2026-05-24)

  • CASE WHEN conditions beyond Cmp/AND/OR/NOT. Fixed: compile_cond lambda in ace_exports.cpp now handles Kind::In (literal list), IsNull/IsNotNull, and Between in addition to the existing Cmp/AND/OR/NOT. (2026-05-25)

  • FTS query-time token lookup. Already wired: CONTAINS(col, expr) is handled in all four SQL compile lambdas (main SELECT WHERE, join-cursor, aggregate FILTER, CASE WHEN). Token lookup hits the .fts inverted index at compile time and captures the record set. (2026-05-24)

Encryption

Open

  • AdsDecryptTable / AdsEncryptRecord / AdsDecryptRecord — out of scope. We use our own encryption model (M11.2 AES). ADS-original per-record encryption format not reversed. Stubs return AE_FUNCTION_NOT_AVAILABLE.

  • Key derivation hardening. Connection::set_encryption_password zero-pads the password to 32 bytes (noted in connection.cpp:510). Should use PBKDF2 or Argon2 once a SHA-256 implementation is in the tree. Low urgency for local-use scenarios; critical before any multi-user deployment that stores encrypted tables long-term.

Transactions

Open

  • WAL crash recovery — end-to-end crash recovery implemented and tested. Connection::open → recover_orphan_tx_() reads the WAL, identifies transactions with no COMMIT/ABORT (orphans), writes before-images back for UPDATE records, and marks appended rows deleted for APPEND records. LsnMap sidecar makes recovery crash-safe across interrupted passes. WAL APPEND record type added so orphan appends are tracked persistently. 3 tests in tests/unit/abi_m5x_recovery_test.cpp. (2026-05-26)

ABI completeness

Open

  • AdsEval*Expr family — server-side expression evaluation used by Harbour/X# ADSRDD.prg server-side query path. Implemented: AdsEvalLogicalExpr (AOF boolean expression at current record via aof::evaluate_record), AdsEvalNumericExpr (field read or numeric literal parse), AdsEvalStringExpr (field read or string literal passthrough). (2026-05-25)

  • AdsStmt* helpers — per-statement table-open settings (table_type, lock_type, char_type, read_only, check_rights, disable_enc, collation, passwords) stored in SqlStatement and threaded into AdsExecuteSQLDirect for SELECT/UPDATE/DELETE/INSERT. All 9 setter functions implemented. (2026-05-25)

  • AdsRestructureTable type conversion — CHANGE path now supports C↔N, C/N→L, L→C/N, and D↔C conversions. Raw-copy fallback for other pairs. Test updated. (2026-05-25)

  • AdsContinue (LOCATE/CONTINUE loop). Implemented: filter-aware skip(1) on the underlying Table — since Table::skip() already walks past non-matching records when a filter or AOF bitmap is active, AdsContinue is a single-step forward move with *pbFound = eof() ? 0 : 1. Test in tests/unit/abi_aof_test.cpp. (2026-05-24)

  • Table-management stubs. AdsCopyTableContent (field-name-matched copy between two open tables), AdsCloneTable (full structural clone including deleted records into a temp DBF; returns new handle), and AdsCopyTableStructure (schema-only copy, 0 records). All implemented and tested. (2026-05-25)

  • Enumeration stubs. AdsGetAllTables enumerates all table handles owned by a connection (iterates HandleRegistry for HandleKind::Table, filters via owns_table_ptr). AdsGetAllIndexes enumerates all index handles bound to a table (iterates index_bindings()). AdsGetFTSIndexes returns 0 — FTS indexes have no persistent handles in OpenADS. 3 tests in tests/unit/abi_enum_test.cpp. (2026-05-25)

Wire protocol

Open

  • Forward-only prefetch (M12.21) — disabled in M12.21b after cursor-drift regressions on indexed scans. Re-enable once the indexed-scan drift is understood and fixed.