diff --git a/src/.vuepress/config.js b/src/.vuepress/config.js index 860dfb1d4..86fd59817 100644 --- a/src/.vuepress/config.js +++ b/src/.vuepress/config.js @@ -209,6 +209,15 @@ module.exports = config({ children: [ ['applications/modules/bootstrap/tags', 'Tags'], ['applications/modules/bootstrap/dynamic-attributes', 'Dynamic Attributes'], + { + title: 'Data', + collapsable: true, + children: [ + ['applications/modules/bootstrap/data/general', 'General'], + ['applications/modules/bootstrap/data/data-checks', 'Checks'], + ['applications/modules/bootstrap/data/data-transformations', 'Transformations'] + ] + } ] }, ['preparing-content', 'Commerce Start'], diff --git a/src/.vuepress/public/.attachments/applications/modules/bootstrap/data/data-check-rule-form.png b/src/.vuepress/public/.attachments/applications/modules/bootstrap/data/data-check-rule-form.png new file mode 100644 index 000000000..1d450f9a2 Binary files /dev/null and b/src/.vuepress/public/.attachments/applications/modules/bootstrap/data/data-check-rule-form.png differ diff --git a/src/.vuepress/public/.attachments/applications/modules/bootstrap/data/data-transformation-step-form.png b/src/.vuepress/public/.attachments/applications/modules/bootstrap/data/data-transformation-step-form.png new file mode 100644 index 000000000..6d2494c14 Binary files /dev/null and b/src/.vuepress/public/.attachments/applications/modules/bootstrap/data/data-transformation-step-form.png differ diff --git a/src/en/developer-guide/applications/modules/bootstrap/data/data-checks.md b/src/en/developer-guide/applications/modules/bootstrap/data/data-checks.md new file mode 100644 index 000000000..cb95b74ee --- /dev/null +++ b/src/en/developer-guide/applications/modules/bootstrap/data/data-checks.md @@ -0,0 +1,108 @@ +--- +title: Data Check Rules +tagline: Define and execute validation rules to identify incorrect or incomplete records. +author: David Wasserbauer +--- + +## Introduction + +Data Check Rules allow administrators to identify records that do not comply with expected data quality criteria. Each rule specifies a target entity, a query to identify records of interest, and an optional condition for advanced validation logic. + +When a rule is executed, it generates a set of log records representing failed validations. These logs can be reviewed, manually resolved, or serve as the basis for further automated actions. + +![Data Check Rule Form](/.attachments/applications/modules/bootstrap/data/data-check-rule-form.png) + +## Entity Definitions + +### Data Check Rule (`talxis_datacheckrule`) + +| Display Name | Logical Name | Description | +| -------------------- | -------------------- | ---------------------------------------------------------------------------- | +| Name | `talxis_name` | Name of the rule, typically includes its purpose or target field. | +| Entity Name | `talxis_entityname` | Logical name of the entity to which the rule applies. | +| Filter Query | `talxis_query` | FetchXML query identifying relevant records. | +| Condition (Function) | `talxis_conditionid` | Optional reference to a PowerFx-based function used for more advanced logic. | +| Description | `talxis_description` | Optional explanation of what the rule validates or checks. | + +### Data Check Log (`talxis_datachecklog`) + +| Display Name | Logical Name | Description | +| ---------------------- | ---------------------------- | -------------------------------------------------------------- | +| Data Record | `talxis_datarecordid` | Polymorphic lookup to the affected record. | +| Data Check Rule | `talxis_ruleid` | Reference to the rule that created this log. | +| Status (Reason) | `statuscode` | Current processing status (see table below). | + +## Status Values + +| Code | Status Name | Meaning | +| --------- | --------------- | ------------------------------------------------------ | +| 742070000 | Evaluation | Awaiting result of condition evaluation. | +| 742070001 | Pending | Condition passed or not defined, needs user attention. | +| 742070002 | Checked | Confirmed by user, optionally after manual correction. | +| 742070003 | Cancelled | Condition evaluated to false. | +| 742070004 | Error | Error occurred during evaluation. | + +## Rule Execution + +Rules can be triggered in two ways: + +### 1. Automatically + +* As part of a broader processing pipeline (e.g., after data import). +* System filters rules by entity name and runs them on newly imported data. + +> ⚠️ This applies only to Albert VAT App. See [User Guide](https://thenetworg.sharepoint.com/:w:/r/teams/pct20004/Shared%20Documents/General/_DOCS/250509-DC-01%20VAT%20App%20-%20User%20Guide.docx?d=w621b4d97c59f40be860062708c44bbd1&csf=1&web=1&e=NfOERk) to get more info. + +### 2. Manually + +* Rules can be run on demand from the admin UI. +* The "Run Rule" button is available in the grid view of active rules. + +Each rule execution creates one or more log entries in `talxis_datachecklog`, depending on the number of matching records. + +## PowerFx Conditions + +Rules may include an optional condition defined in a PowerFx-based Custom API (a Function). This enables scenarios that FetchXML cannot support, such as: + +* Calculations +* Multi-entity logic +* Value comparisons + +### Requirements + +* Input: `RecordId` (string) +* Output: `Result` (boolean) + +Example function body: + +```plaintext +{ + Result: + LookUp('Invoice Lines','Line ID' = GUID(RecordId)).Amount > 1000 +} +``` + +If the condition returns `true`, a log entry is marked for review. + +If `false`, the `statuscode` is set to `cancelled`. + +## Manual Resolution + +Users can resolve check logs via the following options: + +* **Mark as Checked**: Indicates that the data is acceptable despite the check failure. +* **Perform Correction**: Launches a dialog to apply a correction directly to the source record. This creates a related Data Transformation Log. + +## Related Plugins + +* **ValidateCustomApiCondition** – When a check rule includes a condition, this plugin executes the corresponding Function and updates the log status. + +## Automation Integration + +Data Check Rules can be used as part of larger workflows. Typical integration points include: + +* Post-import validation +* Pre-submission data quality assurance +* Audit trail creation + +Each check log entry provides a clear indication of where data failed to meet defined criteria, enabling both reactive and preventive data governance. \ No newline at end of file diff --git a/src/en/developer-guide/applications/modules/bootstrap/data/data-transformations.md b/src/en/developer-guide/applications/modules/bootstrap/data/data-transformations.md new file mode 100644 index 000000000..ae525e7ac --- /dev/null +++ b/src/en/developer-guide/applications/modules/bootstrap/data/data-transformations.md @@ -0,0 +1,148 @@ +--- +title: Data Transformation Steps +tagline: Define and execute rules that correct values in target records. +author: David Wasserbauer +--- + +## Introduction + +Data Transformation Steps provide a structured mechanism for correcting data in target records. + +Each step defines which records to modify, which field to update, and what new value to apply. Execution results are logged for full auditability. + +![Data Transformation Step Form](/.attachments/applications/modules/bootstrap/data/data-transformation-step-form.png) + +## Entity Definitions + +### Data Transformation Step (`talxis_datatransformationstep`) + +| Display Name | Logical Name | Description | +| ----------------------------| --------------------------| ----------------------------------------------------------| +| Name | `talxis_name` | Descriptive name of the transformation step. | +| Entity Name | `talxis_entityname` | Logical name of the entity being transformed. | +| Target Field | `talxis_attributename` | Field name to update (logical name). | +| Step Number | `talxis_stepnumber` | Execution order; lower numbers are processed first. | +| Filter Query | `talxis_query` | FetchXML to select records for update. | +| Condition (Function) | `talxis_conditionid` | Optional PowerFx condition. | +| Transformation (Function) | `talxis_transformationid` | PowerFx function that computes the applies new value. | +| Description | `talxis_description` | Optional explanation of what the step does. | + +### Data Transformation Log (`talxis_datatransformationlog`) + +| Display Name | Logical Name | Description | +| ----------------------- | -------------------------------------- | -------------------------------------------------------------| +| Data | `talxis_datarecordid` | Polymorphic reference to the target record. | +| Transformation Step | `talxis_stepid` | Reference to the originating step. | +| Original Value | `talxis_originalvalue` | Value before the transformation. | +| New Value | `talxis_newvalue` | Value after the transformation. | +| Type (Manual/Automatic) | `talxis_datatransformationlogtypecode` | Indicates how the change was triggered. | +| Reason Description | `talxis_reasondescription` | Used to capture the reason for manual transformations. | +| Error Message | `talxis_errormessage` | Stores error messages generated during custom API execution. | +| Status | `statuscode` | Processing result (see below). | + +## Status Values + +| Code | Status Name | Meaning | +| --------- | -----------------------| ---------------------------- | +| 742070000 | Evaluation | Condition pending. | +| 742070001 | Waiting for processing | Ready for execution. | +| 742070002 | Processed | Change applied successfully. | +| 742070003 | Cancelled | Condition returned false. | +| 742070004 | Error | Error during execution. | + +## Execution Flow + +### Step Ordering + +Each transformation step includes a `Step Number` that defines its place in the execution sequence. The system: + +1. Sorts steps by this number (ascending) +2. Waits for all steps with lower numbers to finish before running the next group + +The flow `talxis_waitforpreviousinstancesandlogsfinished` handles this sequencing. + +> ⚠️ This applies when multiple steps are executed at the same time. + +### Automatic Execution + +Triggered after data import or other orchestration points, the steps run: + +* Based on their target entity (matched against the import type) +* Only if their FetchXML query returns results +* Conditionally, if a Function is defined + +> ⚠️ This applies only to Albert VAT App. See [User Guide](https://thenetworg.sharepoint.com/:w:/r/teams/pct20004/Shared%20Documents/General/_DOCS/250509-DC-01%20VAT%20App%20-%20User%20Guide.docx?d=w621b4d97c59f40be860062708c44bbd1&csf=1&web=1&e=NfOERk) to get more info. + +### Manual Execution + +From the admin interface, a user can trigger a specific step using the **Execute Step** button. The system will: + +* Evaluate FetchXML +* Create logs for eligible records +* Evaluate condition (if present) +* Run the plugin to apply the change + +## PowerFx Conditions + +Used to determine whether a record qualifies for update. Function requirements: + +* Input: `RecordId` (string) +* Output: `Result` (boolean) + +Example: + +```plaintext +{ + Result: + LookUp('Invoice Lines','Line ID' = GUID(RecordId)).Country = "CZ" +} +``` + +## PowerFx Value Computation + +In contrast to checks, transformations can also compute the actual value to be applied. A dedicated PowerFx function returns: + +* Input: `RecordId` +* Output: `NewValue` (string) + +Example: + +```plaintext +{ + NewValue: + Text( + Patch( + 'Invoice Lines', + LookUp('Invoice Lines', 'Line ID' = GUID(RecordId)), + { Country: "DE" } + ).Country + ) +} +``` + +## Manual Corrections via Dialog + +Manual corrections allow users to: + +* Select a field to update (validated for writability) +* Enter a new value based on data type +* Confirm the change + +This creates a `talxis_datatransformationlog` with: + +* Type: Manual +* Step: *(not linked to any step)* + +Plugin `ResolveDataTransformationLog` applies the change based on the new value. + +## Related Plugins + +* **ResolveDataTransformationLog** – Applies the new value to the record based on the transformation log. + +## Automation Integration + +Transformation steps are typically triggered as part of a post-import process, but may also run independently. Each step provides: + +* Controlled correction mechanism +* Traceable log per change +* Flexible logic with conditions and value computation \ No newline at end of file diff --git a/src/en/developer-guide/applications/modules/bootstrap/data/general.md b/src/en/developer-guide/applications/modules/bootstrap/data/general.md new file mode 100644 index 000000000..a4f5a64b9 --- /dev/null +++ b/src/en/developer-guide/applications/modules/bootstrap/data/general.md @@ -0,0 +1,145 @@ +--- +title: Data Quality Feature +tagline: Overview of the validation and transformation feature and how to use it. +author: David Wasserbauer +--- + +## Introduction + +The Data Quality Feature enables administrators to evaluate and correct data records. This feature is based on two parallel concepts: + +* **Data Check Rules** – identify data inconsistencies using queries and optional conditions +* **Transformation Steps** – automatically or manually correct values in records + +Both mechanisms share a common structure and can be triggered manually. + +## Core Entities and Relationships + +### Data Check Rules + +| Display Name | Logical Name | Description | +| --------------- | ---------------------- | --------------------------------------------------------------------------------- | +| Data Check Rule | `talxis_datacheckrule` | Defines a validation rule using a FetchXML filter and optional PowerFx condition. | +| Data Check Log | `talxis_datachecklog` | Result log of a check rule execution on a specific record. | + +### Transformation Steps + +| Display Name | Logical Name | Description | +| ------------------------ | ------------------------------- | ------------------------------------------------------------------------------------ | +| Data Transformation Step | `talxis_datatransformationstep` | Defines a correction rule, including FetchXML query, attribute to update, and logic. | +| Data Transformation Log | `talxis_datatransformationlog` | Result log containing the original and new value. | + +### Shared Relationships + +Both `talxis_datachecklog` and `talxis_datatransformationlog` include a polymorphic lookup: + +| Field | Description | +| --------------------- | ------------------------------------------------------------------------------------ | +| `talxis_datarecordid` | Reference to the target record affected by the rule (e.g., document header or line). | + +## Execution Flow + +### Automatic Execution (Albert VAT App only) + +After records are imported (e.g. via Dataflows or Power Automate Flows), the main orchestration logic runs in the following sequence: + +1. Pre-transformation data adjustments (e.g. categorization, enrichment) +2. All matching Transformation Steps (filtered by entity) +3. All matching Data Check Rules (filtered by entity) + +This ensures records are corrected before checks are run, and issues are flagged only if still unresolved. + +> ⚠️ This applies only to Albert VAT App. See [User Guide](https://thenetworg.sharepoint.com/:w:/r/teams/pct20004/Shared%20Documents/General/_DOCS/250509-DC-01%20VAT%20App%20-%20User%20Guide.docx?d=w621b4d97c59f40be860062708c44bbd1&csf=1&web=1&e=NfOERk) to get more info. + +### Manual Execution + +Both check rules and transformation steps can be manually triggered from the admin section. The buttons "Execute Rule" and "Execute Step" are available in the grid of the respective entity. + +### Manual Corrections + +Users can also manually edit records using a dedicated dialog. This action: + +* Lets the user select the field and input a new value +* Creates a transformation log with type **Manual** +* Applies the change using plugin logic (`ResolveDataTransformationLog`) + +## Filtering and Expressions + +### FetchXML Filtering + +Each rule/step includes a FetchXML query to identify relevant records. Complex logic can be applied using related entities and groups. + +### Conditions via PowerFx (Functions) + +To express more advanced conditions (e.g. math, combined fields), custom PowerFx-based APIs are used: + +* Written as low-code plug-ins +* Require input `RecordId` and output `Result` (boolean) + +Example syntax: + +```plaintext +{ + Result: + LookUp('Purchase Invoice Lines','Line ID' = GUID(RecordId)).Code = "0" +} +``` + +### Transformation Logic via PowerFx + +For automated corrections, PowerFx functions can return a `NewValue` string to be applied to the target field. + +Example: + +```plaintext +{ + NewValue: + Text( + Patch( + 'Purchase Invoice Lines', + LookUp('Purchase Invoice Lines', 'Line ID' = GUID(RecordId)), + { Code: "XX"} + ).Code + ) +} +``` + +## Status and Logs + +### Check Log Statuses (`talxis_datachecklog.statuscode`) + +| Value | Meaning | +| --------- | ---------------------------------- | +| 742070000 | Evaluating condition | +| 742070001 | Awaiting review | +| 742070002 | Checked (user confirmed/corrected) | +| 742070003 | Cancelled (condition false) | +| 742070004 | Failed (custom API error) | + +### Transformation Log Statuses (`talxis_datatransformationlog.statuscode`) + +| Value | Meaning | +| --------- | --------------------------- | +| 742070000 | Evaluating condition | +| 742070001 | Awaiting processing | +| 742070002 | Processed | +| 742070003 | Cancelled (condition false) | +| 742070004 | Failed (custom API or plugin error) | + +## Related Plugins and Flows + +### Plugins + +* **ValidateCustomApiCondition** – evaluates PowerFx condition and updates status +* **ResolveDataTransformationLog** – applies new value based on log record and updates status + +### Power Automate Flows + +* `talxis_datalog_creatorhandler` – creates transformation/check logs in batches +* `talxis_waitforpreviousinstancesandlogsfinished` – ensures steps run in defined order +* `talxis_executedatacheckrule_trigger` – triggers evaluation of a Data Check Rule +* `talxis_executedatatransformationstep_trigger` – triggers execution of a Data Transformation Step + +## Summary + +The Data Quality Feature enables structured validation and correction of records. It supports both automated (see [VAT APP User Guide](https://thenetworg.sharepoint.com/:w:/r/teams/pct20004/Shared%20Documents/General/_DOCS/250509-DC-01%20VAT%20App%20-%20User%20Guide.docx?d=w621b4d97c59f40be860062708c44bbd1&csf=1&web=1&e=NfOERk) for example) and manual workflows and provides a transparent, traceable mechanism to ensure data accuracy prior to downstream processing. \ No newline at end of file