Refactor C2D docs. #1513
Open
Refactor C2D docs. #1513
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
giurgiur99
reviewed
May 23, 2025
.gitbook/assets/vscode/setup.png
Outdated
There was a problem hiding this comment.
Can we update with latest screenshot without paid resources? Thanks.
| In the C2D workflow, the following steps are performed: | ||
|
|
||
| 1. The consumer initiates a compute-to-data job by selecting the desired data asset and algorithm, and then, the orders are validated via the dApp used. | ||
| 2. A dedicated and isolated execution pod is created for the C2D job. |
| 3. The execution pod loads the specified algorithm into its environment. | ||
| 4. The execution pod securely loads the selected dataset for processing. | ||
| 5. The algorithm is executed on the loaded dataset within the isolated execution pod. | ||
| 6. The results and logs generated by the algorithm are securely returned to the user. |
There was a problem hiding this comment.
We can mention that we use a web3 auto-generated/custom PK and this is how we ensure private access to results
|
|
||
| Now, let's delve into the inner workings of the Provider. Initially, it verifies whether the Consumer has sent the appropriate datatokens to gain access to the desired data. Once validated, the Provider interacts with the Operator-Service, a microservice responsible for coordinating the job execution. The Provider submits a request to the Operator-Service, which subsequently forwards the request to the Operator-Engine, the actual compute system in operation. | ||
|
|
||
| The Operator-Engine, equipped with functionalities like running Kubernetes compute jobs, carries out the necessary computations as per the requirements. Throughout the computation process, the Operator-Engine informs the Operator-Service of the job's progress. Finally, when the job reaches completion, the Operator-Engine signals the Operator-Service, ensuring that the Provider receives notification of the job's successful conclusion. |
| - `GetComputeEnvironments` - returns list of environments that can be selected to run the algorithm on | ||
| - `InitializeCompute` - generates provider fees necessary for asset's ordering | ||
| - `FreeStartCompute` - runs algorithms without necessary publish the assets on-chain (dataset and algorithm), using free resources from the selected environment | ||
| - `PaidStartCompute` - runs algorithms with on-chain assets (dataset and algorithm), using paid resources from the selected environment. The payment is requested at every start compute call, being handled by `Escrow` contract. |
There was a problem hiding this comment.
startCompute
COMPUTE_START: 'startCompute',
| - `FreeStartCompute` - runs algorithms without necessary publish the assets on-chain (dataset and algorithm), using free resources from the selected environment | ||
| - `PaidStartCompute` - runs algorithms with on-chain assets (dataset and algorithm), using paid resources from the selected environment. The payment is requested at every start compute call, being handled by `Escrow` contract. | ||
| - `ComputeGetStatus` - retrieves compute job status. | ||
| - `ComputeStop` - stops compute job execution when the job is `Running`. |
| - `PaidStartCompute` - runs algorithms with on-chain assets (dataset and algorithm), using paid resources from the selected environment. The payment is requested at every start compute call, being handled by `Escrow` contract. | ||
| - `ComputeGetStatus` - retrieves compute job status. | ||
| - `ComputeStop` - stops compute job execution when the job is `Running`. | ||
| - `ComputeGetResult` - returns compute job results when job is `Finished`. |
There was a problem hiding this comment.
COMPUTE_GET_RESULT: 'getComputeResult',
|
|
||
| One of its responsibility revolves around fetching and preparing the required assets and files, ensuring a smooth and seamless execution of the job. By meticulously handling the environment configuration, the **C2D Engine** guarantees that all necessary components are in place, setting the stage for a successful job execution. | ||
|
|
||
| 1. **Fetching Dataset Assets**: It downloads the files corresponding to datasets and saves them in the location `/data/inputs/DID/`. The files are named based on their array index ranging from 0 to X, depending on the total number of files associated with the dataset. |
There was a problem hiding this comment.
Dataset can be did/url/arweave/ipfs
|
|
||
| ## Prerequisites | ||
|
|
||
| The prerequisite for this flow is the algorithm code which can be input for consumers components: Ocean CLI and it is open for integration with other systems (e.g. Ocean Enterprise Marketplace). |
There was a problem hiding this comment.
CLI and vscode extension, might be useful to add links
developers/vscode/README.md
Outdated
There was a problem hiding this comment.
New dataset here will be url/did/ipfs/arweave.
We will now use the c2d_examples not algo_dockers repo
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Fixes #1515 .
Changes proposed in this PR: