DOCS-3198: Add offline data pipelines, SDK docs, hot data store fixes (#4440)

Co-authored-by: Jack Damon <[email protected]>
Co-authored-by: Naomi Pentrel <[email protected]>

Author: 3 people

.github/workflows/sdk_protos_map.csv

@@ -425,6 +425,11 @@ data,GetDatabaseConnection,,get_database_connection,,getDatabaseConnection,getDa
 data,ConfigureDatabaseUser,,configure_database_user,,configureDatabaseUser,configureDatabaseUser
 data,AddBinaryDataToDatasetByIDs,,add_binary_data_to_dataset_by_ids,,addBinaryDataToDatasetByIds,addBinaryDataToDatasetByIds
 data,RemoveBinaryDataFromDatasetByIDs,,remove_binary_data_from_dataset_by_ids,,removeBinaryDataFromDatasetByIds,removeBinaryDataFromDatasetByIds
+data,GetDataPipeline,,get_data_pipeline,,,getDataPipeline
+data,ListDataPipelines,,list_data_pipelines,,,listDataPipelines
+data,CreateDataPipeline,,create_data_pipeline,,,createDataPipeline
+data,DeleteDataPipeline,,delete_data_pipeline,,,deleteDataPipeline
+data,ListDataPipelineRuns,,list_data_pipeline_runs,,,listDataPipelineRuns
 
 ## Dataset
 dataset,CreateDataset,,create_dataset,,createDataset,createDataset

docs/data-ai/capture-data/advanced/advanced-data-capture-sync.md

@@ -15,7 +15,24 @@ Some data use cases require advanced configuration beyond the attributes accessi
 You can use raw JSON to configure additional attributes for both data management and data capture.
 You can also configure data capture for remote parts.
 
-### Advanced data management service configurations
+## Cloud data retention
+
+Configure how long your synced data remains stored in the cloud:
+
+- **Retain data up to a certain size (for example, 100GB) or for a specific length of time (for example, 14 days):** Set `retention_policies` at the resource level.
+  See the `retention_policy` field in [data capture configuration attributes](/data-ai/capture-data/advanced/advanced-data-capture-sync/#click-to-view-data-capture-attributes).
+- **Delete data captured by a machine when you delete the machine:** Control whether your cloud data is deleted when a machine or machine part is removed.
+  See the `delete_data_on_part_deletion` field in the [data management service configuration attributes](/data-ai/capture-data/advanced/advanced-data-capture-sync/#click-to-view-data-management-attributes).
+
+## Sync optimization
+
+**Configurable sync threads:** You can control how many concurrent sync operations occur by adjusting the `maximum_num_sync_threads` setting.
+Higher values may improve throughput on more powerful hardware, but raising it too high may introduce instability on resource-constrained devices.
+
+**Wait time before syncing arbitrary files:** If you choose to sync arbitrary files (beyond those captured by the data management service), the `file_last_modified_millis` configuration attribute specifies how long a file must remain unmodified before the data manager considers it for syncing.
+The default is 10 seconds.
+
+## Advanced data management service configuration
 
 To configure the data manager in JSON, see the following example configurations:
 
@@ -96,7 +113,7 @@ The following attributes are available for the data management service:
 
 You can edit the JSON directly by switching to **JSON** mode in the UI.
 
-### Advanced data capture configurations
+## Advanced data capture configuration
 
 {{< alert title="Caution" color="caution" >}}
 
@@ -524,62 +541,14 @@ The following attributes are available for data capture configuration:
 | `capture_frequency_hz` | float   | **Required** | Frequency in hertz at which to capture data. For example, to capture a reading every 2 seconds, enter `0.5`. |
 | `method` | string | **Required** | Depends on the type of component or service. See [Supported components and services](/data-ai/capture-data/capture-sync/#click-to-see-resources-that-support-data-capture-and-cloud-sync). **Note:** For tabular data, Viam enforces a maximum size of 4MB for any single reading. |
 | `retention_policy` | object | Optional | Option to configure how long data collected by this component or service should remain stored in the Viam Cloud. You must set this in JSON mode. See the JSON example for a camera component. <br> **Options:** `"days": <int>`, `"binary_limit_gb": <int>`, `"tabular_limit_gb": <int>`. <br> Days are in UTC time. Setting a retention policy of 1 day means that data stored now will be deleted the following day **in UTC time**. You can set either or both of the size limit options and size is in gigabytes. The `retention_policy` does not affect logs. For information about logs, see [Logging](/operate/reference/viam-server/#logging). |
-| `recent_data_store` | object | Optional | Configure a rolling time frame of recent data to store in a [hot data store](#capture-to-the-hot-data-store) for faster access. Example: `{ "stored_hours": 24 }` |
+| `recent_data_store` | object | Optional | Configure a rolling time frame of recent data to store in a [hot data store](/data-ai/data/hot-data-store/) for faster access. Example: `{ "stored_hours": 24 }` |
 | `additional_params` | depends | depends | Varies based on the method. For example, `ReadImage` requires a MIME type. |
 
 {{< /expand >}}
 
 You can edit the JSON directly by switching to **JSON** mode in the UI.
 
-### Capture to the hot data store
-
-If you want faster access to your most recent sensor readings, you can configure hot data storage.
-The hot data store keeps a rolling window of hot data for faster queries.
-All historical data remains in your default storage.
-
-To configure the hot data store:
-
-1. Use the `recent_data_store` attribute on each capture method in your data manager service.
-2. Configure your queries' data source to the hot data store by passing the `use_recent_data` boolean argument to [tabularDataByMQL](/dev/reference/apis/data-client/#tabulardatabymql).
-
-{{% expand "Click to view a sample configuration" %}}
-
-The following sample configuration captures data from a sensor at 0.5 Hz.
-`viam-server` stores the last 24 hours of data in a shared recent-data database, while continuing to write all data to blob storage:
-
-```json {class="line-numbers linkable-line-numbers" data-line="17-19"}
-{
-  "components": [
-    {
-      "name": "sensor-1",
-      "api": "rdk:component:sensor",
-      "model": "rdk:builtin:fake",
-      "attributes": {},
-      "service_configs": [
-        {
-          "type": "data_manager",
-          "attributes": {
-            "capture_methods": [
-              {
-                "method": "Readings",
-                "capture_frequency_hz": 0.5,
-                "additional_params": {},
-                "recent_data_store": {
-                  "stored_hours": 24
-                }
-              }
-            ]
-          }
-        }
-      ]
-    }
-  ]
-}
-```
-
-{{% /expand%}}
-
-### Capture directly to your own MongoDB cluster
+## Capture directly to your own MongoDB cluster
 
 You can configure direct capture of tabular data to a MongoDB instance alongside disk storage on your edge device.
 This can be useful for powering real-time dashboards before data is synced from the edge to the cloud.
@@ -699,20 +668,3 @@ Failing to write to MongoDB doesn't affect capturing and syncing data to cloud s
   If your use case needs to support very high capture rates, this feature may not be appropriate.
 
 {{< /alert >}}
-
-### Cloud data retention
-
-Configure how long your synced data remains stored in the cloud:
-
-- **Retain data up to a certain size (for example, 100GB) or for a specific length of time (for example, 14 days):** Set `retention_policies` at the resource level.
-  See the `retention_policy` field in [data capture configuration attributes](/data-ai/capture-data/advanced/advanced-data-capture-sync/#click-to-view-data-capture-attributes).
-- **Delete data captured by a machine when you delete the machine:** Control whether your cloud data is deleted when a machine or machine part is removed.
-  See the `delete_data_on_part_deletion` field in the [data management service configuration attributes](/data-ai/capture-data/advanced/advanced-data-capture-sync/#click-to-view-data-management-attributes).
-
-### Sync optimization
-
-**Configurable sync threads:** You can control how many concurrent sync operations occur by adjusting the `maximum_num_sync_threads` setting.
-Higher values may improve throughput on more powerful hardware, but raising it too high may introduce instability on resource-constrained devices.
-
-**Wait time before syncing arbitrary files:** If you choose to sync arbitrary files (beyond those captured by the data management service), the `file_last_modified_millis` configuration attribute specifies how long a file must remain unmodified before the data manager considers it for syncing.
-The default is 10 seconds.