Merge pull request #416 from OpenFn/sprint-2-docs

docs updates to implementation checklist, tutorials, and running workflows

Author: aleksa-krolls

docs/build/workflows.md

@@ -22,9 +22,10 @@ Check out the video overview below to learn how to create a Workflow.
 <iframe width="784" height="441" src="https://www.youtube.com/embed/HmE_wp_g1RY?si=Pud7DPS0BevAjStp" title="YouTube video player" frameborder="0" allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture; web-share" allowfullscreen></iframe>
 
 ## Run Workflows
-To run a Workflow, you can either activate the Trigger (e.g., send a request to the Webhook Event Trigger's URL, or wait for the cron timer to be activated), or run your workflow manually. 
+To run a Workflow, you can either activate the Trigger (e.g., send a request to the Webhook Event Trigger's URL, or wait for the cron timer to be activated), or run your workflow manually. Check out the video below for how to run your workflow manually. 
+
+<iframe width="784" height="441" src="https://www.youtube.com/embed/dssixE3Sukc?si=n3Jpdiu_aiBLXuHb" title="YouTube video player" frameborder="0" allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture; web-share" allowfullscreen></iframe>
 
-<!-- TODO: ADD DOCS ON HOW TO MANUALLY RUN A WORKFLOW -->
 
 ## Turn off Workflows
 To "turn off" or disable a Workflow: 

docs/get-help/support.md

@@ -2,10 +2,16 @@
 title: Support for OpenFn Implementations
 sidebar_label: Support
 ---
-:::warning Under construction
 
-This docs page is under construction. Check back later for the complete docs, or check out the Docs Version "Platform (v1)". 
+## Ask the Community! 
+If you need help getting started, have questions, or product feedback, first check out our **[Community](https://community.openfn.org)**. Our core team and other OpenFn implementers monitor all posts to help each other out, share examples, and circulate product updates. 
 
-:::
+## Have a question about your project on OpenFn.org? 
+If you're using the hosted OpenFn platform SaaS, and have a private question about your project, account, or billing, contact our core team at [[email protected]](mailto://[email protected]). 
+
+## Need helping hands?
+The OpenFn core team and our certified partners offer enterprise support, implementation & developer services, and training to jump-start your team. Check out our website: 
+- About [OpenFn services & pricing](https://www.openfn.org/pricing)
+- About our [certified partners](https://www.openfn.org/partners)
 
 <!--Add link to OpenFn community, or need a hand - contact us -->

docs/get-started/implementation-checklist.md

@@ -3,16 +3,18 @@ sidebar_label: Implementation Checklist
 title: Implementation Checklist for planning your next integration project
 ---
 
+
 # Implementation Checklist
 
-This checklist draws from experience of implementing interoperability projects
+This [Implementation Checklist](https://docs.google.com/spreadsheets/d/1_XY0nx0OLNUsogrIHnRaSTyZ-KdcSXks-tqwm3ZfMc4/edit#gid=72612093) checklist draws from experience of implementing interoperability projects
 with in-country government agencies (incl. UNICEF country offices, Ministry of
 Social Services Cambodia, MoH Thailand) to offer an implementation & planning
 guide covering key milestones in most interoperability and integration projects.
 
 While this checklist should be tailored for each implementation, the tasks
-outlined here provide a template workplan that can help any organization prepare
-for their upcoming implementation. 
+outlined provide a template workplan that can help any organization prepare
+for their upcoming implementation. **The implementation process is broken up into the seven phases summarized below. Check out the checklist for detailed steps.**
+
 
 :::tip
 
@@ -24,94 +26,83 @@ partner NGOs:
 
 :::
 
-:::info
 
-The XLS version of this template can be found [here](https://docs.google.com/spreadsheets/d/1_XY0nx0OLNUsogrIHnRaSTyZ-KdcSXks-tqwm3ZfMc4/edit#gid=72612093).
-
-:::
 
 
 ## (1) Preparing for the Implementation
+Set the project up for success by creating a project plan, defining roles responsibilities, documenting the business value of the implementation, and confirming the technical feasibility of the implementation. 
 
-- [ ] Point of contacts identified for each target system (incl. system administrators, folks who can speak to the functional and technical requirements)
-- [ ] Data sharing agreement finalized (if required; common for cross-organization workflows)
-- [ ] Business value assessed & documented
-- [ ] High-level workflow requirements gathered & documented (in diagram)
-- [ ] Technical feasibility assessment completed to verify integration approach, available connection points, and OpenFn deployment option and adaptors
-- [ ] Capacity assessment completed
+Key Outputs:
+- Business Value Assessment
+- High-level workflow requirements
+- Technical Feasibility Assessment 
+- Capacity Assessment
 
 
 ## (2) Discovery & Design - Functional Workflow Requirements
+Gather and document user stories and functional workflow requirements.
+
+Key Outputs:
+- Solution Architecture Diagram
+- Workflow Diagrams (functional) 
+- Data Element Mapping Specifications (functional)
 
-- [ ] User stories documented to capture business value & desired outcomes
-      Learn more about user stories [here.](https://docs.openfn.org/documentation/design/design-overview#1-capture-requirements-as-user-stories)
-- [ ] Workflow BPMN diagram capturing functional steps of the business process finalized
-      Learn more about diagrams & BPMN notation [here.](https://docs.openfn.org/documentation/design/design-overview#2-diagram-the-business-process)
-- [ ] Request list of data elements from administrators of target systems
-      Read about mapping specs [here.](https://docs.openfn.org/documentation/design/design-overview#3-map-data-elements-to-be-exchanged).
-- [ ] Data element mapping specifications finalized (functional/business-friendly version)
-- [ ] Client sign-offs on workflow diagram & mapping specs
-- [ ] Workflow assumptions documented (e.g., what human, manual steps does the workflow rely on; what are the unique identifiers) 
-- [ ] Testing scenarios drafted 
 
 ## (3) Discovery & Design - Technical Specifications
+Iterate on workflow requirements to define technical specifications for how the workflow will be implemented. For instance, consider specific API endpoints to access and HTTP methods/operations to use for each.
 
-- [ ] Documentation on APIs or target connection points secured
-- [ ] Connection points & authentication methods confirmed
-- [ ] Access secured to developer/sandbox environments for testing
-- [ ] Authentication and authorization methods & credentials tested
-- [ ] Target API endpoints determined based on functional specifications & review of API docs
-- [ ] Target API endpoints tested to validate the functional data element specifications can be delivered
-- [ ] Workflow BPMN diagram capturing the technical steps of the workflow finalized 
-- [ ] Technical version of data element mapping specifications created 
-- [ ] Workflow assumptions updated to include any technical considerations 
-- [ ] Test scenarios updated to include any technical considerations
-- [ ] Project Security Configuration Checklist drafted to capture data security & compliance considerations
-- [ ] Github repository created
-- [ ] Job specifications written for developers 
+Key Outputs:
+- Solution Architecture Diagram
+- Workflow Diagrams (technical) 
+- Data Element Mapping Specifications (technical)
 
 ## (4) Build
+Configure the workflow on OpenFn.org and develop and test the jobs and adaptors to be used in the workflow. 
+
+
+Key Outputs:
+- OpenFn Project configuration
+- Jobs
+- Adaptors - new/updated (if needed)
+- Drafted “Project Security Configuration Checklist” to document config settings implemented
+
 
-- [ ] OpenFn platform: project space created & relevant users invited
-- [ ] OpenFn platform: Jobs, triggers, and credentials configured 
-- [ ] OpenFn platform: Version control configured to connect Github repo
-- [ ] Jobs written & pushed to branch on Github
-- [ ] Job code review complete and merged to main branch on Github
-- [ ] OpenFn platform: Github paths updated on each job to link to source file
-- [ ] Test data created (if needed)
-- [ ] Engineer updates mapping specifications (if needed)
 
 ## (5) Testing
+Create a test suite and conduct UAT. After UAT, Incorporate any feedback and iterate on the testing process. 
+
+Key Outputs:
+
+- Completed test suite
+- Backlog of new requests (if feedback identified for future phases)
+- Completed “Project Security Configuration Checklist”
+
 
-- [ ] Testing Round 1: Developers run jobs locally with sample data provided
-- [ ] Testing Round 2: Analysts complete Test Scenarios & run jobs on platform
-- [ ] Iteration: Analysts submit feedback to developers & re-test
-- [ ] UAT Round 1: Client completes Test Scenarios
-- [ ] Iteration: Analysts submit feedback to developers & re-test
-- [ ] UAT Round 2 (if needed): Client completes Test Scenarios
-- [ ] Iteration: Analysts submit feedback to developers & re-test
 
 ## (6) Training & Prep for Go-Live
+Train OpenFn administrators and target system end users and document what was implemented. This is also the phase where the configuration and code is migrated to production environments. 
+
+Key Outputs:
+- Published documentation 
+- Training video recording
+- Signed-off “Project Security Configuration Checklist”
+- Ready-to-go OpenFn project
 
-- [ ] Training materials drafted for client administrators
-- [ ] Documentation drafted, and all project artefacts/docs linked
-- [ ] Project Security Configuration Checklist reviewed to determine if any configuration changes or settings need to be implemented post-testing
-- [ ] Confirm that production system(s) have been configured
-- [ ] Production credentials secured & tested (authentication & authorization)
-- [ ] OpenFn platform: "production" project created (cloned from "staging" project), job configuration migrated, & jobs connected to production credentials
-- [ ] OpenFn administrator users & access levels confirmed and granted?
-- [ ] Support POCs identified for each target system?
-- [ ] Establish support structures & governance model for change management 
-- [ ] Training session delivered to designated OpenFn and target system administrators and any other ToTs
 
-### Rollout & Support
 
-- [ ] Go Live: Turn "on" OpenFn jobs in production platform project so that the workflow is now live in production systems
-- [ ] Confirm administrators have OpenFn notifications turned on to "Each Time" so that they will receive failure notifications (see OpenFn Account Settings)
-- [ ] Communicate to end users as needed about the go-live and its implications
+## (7) Rollout & Support
+Turn “on” OpenFn workflows to go-live and establish support structures & a governance model for change management.
+
+Key Outputs:
+- “Live” OpenFn project
+- Documented support model
+
+
 
 ## Questions or feedback?
 
-If you have any inputs, comments, or questions–please contribute! Submit a pull
-request to this documentation page or leave a comment in
+If you have any inputs, comments, or questions—please contribute! Submit a pull
+request to this documentation page on Github or leave a comment in
 [OpenFn Community](https://community.openfn.org/).
+
+Interested in receiving **training on the OpenFn implementation process**? Contact [[email protected]](mailto://[email protected]). 

docs/tutorials/commcare-to-db.md

@@ -0,0 +1,192 @@
+---
+sidebar_label: CommCare to PostgreSQL
+title:
+ Syncing your CommCare form submissions to a PostgreSQL database
+---
+
+**Before starting this tutorial please make sure:**
+
+- You have signed up for [OpenFn.org](http://openfn.org) (it takes less than a
+  minute!)
+- You have checked out our glossary and have an understanding of basic OpenFn
+  and API terminology. Check out the pages below to get started
+  - [OpenFn Concepts](../get-started/terminology.md)
+  - [A glossary for data integration](../get-started/glossary.md)
+- You have a CommCare application with at least one form configured. This is
+  your source system.
+- You have a PostgreSQL database configured. This is your destination system.
+
+**If you don’t have a CommCare application or PostgreSQL database setup, you can
+also follow along with the prebuilt solution. Follow along at the links below:**
+
+1. [Mapping specifications document](https://docs.google.com/spreadsheets/d/1pi_oxImakhtaCCCIENkjTPZeuyWhpFEcNmH7hfvTBgo/edit?usp=sharing)
+2. Commcare application to download:
+   - Username: testuser
+   - Password: 123
+
+![install_cc_app](/img/install_cc_app.png)
+
+3. [Public report that shows records in the PostgreSQL database](https://analytics.openfn.org/public/question/095449a9-5696-463c-a4fb-24614c9f08a5)
+
+## Getting started
+
+In this walkthrough, we will be setting up an **automatic data sync between
+CommCare and a PostgreSQL database**. We will be syncing submissions coming from
+a CommCare `Maternal and Newborn Health` application that has a
+`Register a New Patient` form.
+
+:::tip
+
+Whenever a CommCare user registers a new patient, the patient details will
+automatically be synced to an already configured PostgreSQL database to enable
+real-time monitoring and analytics on data collected in the field. For example,
+this database can quickly be connected to a dashboard that collects aggregate
+data on patients registered!
+
+:::
+
+![cc-postgres](/img/cc-postgres.png)
+
+**This integration can be broken up into two parts:**
+
+1. Getting data from your source system into OpenFn to trigger your workflow 
+2. Transforming and loading this data to your destination system
+
+… let’s get started!
+
+## Getting data from CommCare
+
+**There are two ways to get your CommCare form submissions in OpenFn.**
+
+### Option 1: Webhook to forward cases and/or forms in real-time from CommCare to OpenFn using REST service
+
+CommCareHQ has a native data forwarding feature that provides a webhook/REST
+service that can be pointed to the destination of your choice (i.e., your OpenFn
+workflow). When a webhook is configured, any Commcare forms submitted are
+**_automatically forwarded_** to the designated endpoint, such as your OpenFn
+workflow. After data forwarding is set up, it happens automatically, **_in
+real-time for all forms and cases_**. Learn more about configuring a webhook
+[here](/adaptors/commcare#webhook-forward-cases-andor-forms-from-commcare-to-openfn-using-rest-service).
+
+![option1](/img/option1.png)
+
+### Option 2: Extracting Commcare data via the REST API
+
+CommCare provides a robust
+[REST API](https://confluence.dimagi.com/display/commcarepublic/List+Forms) for
+extracting and loading data. This second option involves configuring a step in
+OpenFn to fetch CommCare submissions via a `GET` HTTP request with parameters to
+filter your data query. CommCare API access requires a paid CommCare plan.
+
+
+   The main advantage of using the webhook is that your data is forwarded to the
+   destination system in real-time. However, the List Forms API is also
+   advantageous because it enables users to extract data in bulk on a scheduled
+   basis, for syncing historical data every month on the 30th, for example.
+   Deciding on which option to go with depends on your business requirements.
+
+### Set up a workflow using Option 1
+
+1. **Open up an existing project and create a new workflow**
+
+![create_new_workflow](/img/create-new-workflow.gif)
+
+2. **Create a new “Webhook” trigger to schedule this extract job.**
+
+![create_trigger](/img/create_trigger.gif)
+
+Make sure you have copied the webhook URL from your OpenFn workflow into CommCare. Each form submitted in CommCare will be automatically sent to OpenFn and will trigger your new workflow.
+
+## Transforming and loading CommCare data to a PostgreSQL database
+
+1. **You should have a database configured and a username provided for OpenFn to
+   read and write data in your target DB tables.** For this demo, we have
+   configured the database
+   [like this](https://docs.google.com/spreadsheets/d/1pi_oxImakhtaCCCIENkjTPZeuyWhpFEcNmH7hfvTBgo/edit?usp=sharing)
+   to capture the CommCare form data. Check out the
+   [this page](../design/mapping-specs)
+   for how to create your own `mapping specification document` to map data
+   elements to be exchanged.
+
+![db_config](/img/db_config.png)
+
+
+
+2. **Create a new step with the `postgresql` adaptor for loading the CommCare
+   data into your destination database.**
+
+![configure_job_postgres](/img/create-job.gif)
+
+3. **Create a PostgreSQL credential which will be used by the step to
+   authenticate with the database.**
+
+![add_credential_postgres](/img/postgresql-cred.gif)
+
+4. **Writing the step:** For this step we will use the upsert operation to
+insert/update records in the destination `patient` table and use `patient_id` as
+the primary key. An `upsert` will update an existing row if a specified value
+already exists in a table, and insert a new row if the specified value doesn't
+already exist.
+
+```js
+upsert('patient', 'ON CONSTRAINT patient_pk', {
+  patient_id: dataValue('data.patient_name'),
+  patient_name: dataValue('data.patient_name'),
+  village_name: dataValue('data.village_name'),
+  last_menstrual_period: dataValue('data.last_menstrual_period'),
+  expected_delivery_date: dataValue('data.expected_delivery_date'),
+  children_alive: dataValue('data.children_alive'),
+  living_children: dataValue('data.living_children'),
+  feeling_sick: dataValue('data.feeling_sick'),
+  total_children: dataValue('data.Total_children'),
+  risk_level: dataValue('data.Risk_level'),
+});
+```
+
+Feel free to modify the code above to reflect your CommCare and database
+configuration according to your mapping specifications. 
+
+![create-job](/img/create_job_db.gif)
+
+
+## Time to test!
+
+1. Submit a form in CommCare
+2. If you have enabled data forwarding, your workflow should should be triggered automatically.
+3. If you have not enabled data forwarding and set up a FETCH step instead, run
+   the step (ensure the `received_on_start` and `received_on_start` dates in the
+   FETCH are appropriate).
+4. Run the FETCH step. If the fetch step passes, the “Load to DB” step should
+   automatically run.
+5. Check out the `History` and ensure that the work order was successful.
+
+![activity_history_final](/img/activity_history_success.png)
+
+:::info
+
+**What do do if your run fails:**
+
+1. Open the run to inspect the error log
+2. Adjust the step to resolve the issue and re-run the step as needed by clicking the
+   "rerun" button in `History` or the "Re-run from here" button on the `Inspector`
+3. Check out the [PostgreSQL common errors](/adaptors/postgresql/#common-errors)
+   page for more details!
+
+:::
+
+
+4. **Finally, refresh your database and check out the new submission data!**
+
+![metabase](/img/metabase.png)
+
+While this guide is specifically for PostgreSQL databases, you can generally
+follow these same steps for other database types (e.g., MS SQL or MySQL)—simply
+leverage a different adaptor in your step configuration.
+ 
+**Other resources to check out:**
+
+1. OpenFn Job Library
+2. OpenFn Docs ‘App’ pages for CommCare and Postgres
+
+**Any questions? Comments? New configuration ideas? Please reach out to us with
+a post on the [OpenFn Community](https://community.openfn.org/) forum.**