Expose modules to support creation of CDP DW Database Catalog and Virtual Warehouses by raju-saravanan · Pull Request #19 · cloudera-labs/cloudera.cloud

raju-saravanan · 2021-07-15T09:32:28Z

Changes in this PR:

Support private load balancers while creating a DW cluster.
Support the creation of DW database catalog.
Support the creation of DW virtual warehouse.

Changes in this PR are done in conjunction with the following PRs:

Deploy : Add CDW playbook for DW datacatalog and virutal warehouse creation cloudera-deploy#46
Exe : Support creation of CDP DW Database Catalog and Virtual Warehouses cloudera.exe#27
CDPY: Support creation of CDP DW Database Catalog and Virtual Warehouses cdpy#23

Chaffelson · 2021-07-22T08:18:21Z

+            cluster_id=dict(required=True, type='str', aliases=['cluster_id']),
+            dbc_id=dict(required=False, type='str', aliases=['dbc_id']),
+            vw_type = dict(required=False, type='str', aliases=['vw_type']),
+            name = dict(required=True, type='str', aliases=['name']),


shouldn't be 'name' with an alias of 'name'

Chaffelson · 2021-07-22T08:20:41Z

+                        self.target = self.cdpy.sdk.wait_for_state(
+                            describe_func=self.cdpy.dw.describe_dbc,
+                            params=dict(cluster_id=self.cluster_id,dbc_id=self.name),
+                            state='Running', delay=self.delay, timeout=self.timeout


state should be self.cdpy.sdk.STARTED_STATES

Chaffelson · 2021-07-22T08:24:58Z

+                            self.target = self.cdpy.sdk.wait_for_state(
+                                describe_func=self.cdpy.dw.delete_vw,
+                                params=dict(cluster_id=self.cluster_id, vw_id=self.name),
+                                state='Running', delay=self.delay, timeout=self.timeout


self.cdpy.sdk.STARTED_STATES

Chaffelson · 2021-07-22T08:25:37Z

+                                self.target = self.cdpy.sdk.wait_for_state(
+                                    describe_func=self.cdpy.dw.describe_vw,
+                                    params=dict(cluster_id=self.cluster_id, vw_id=self.name),
+                                    state='Running', delay=self.delay, timeout=self.timeout


self.cdpy.sdk.STARTED_STATES

wmudge

Looks good! I would like to clarify the module names and return values. In addition, the documentation sections need review, esp. the dw_vw.py module. The contents do not reflect the code, and I suspect the format of applicationConfig, for example, needs updating, too.

wmudge · 2021-07-22T17:40:11Z

+    )
+
+    result = DwDbc(module)
+    output = dict(changed=False, dbcs=result.dbcs)


dbcs -> data_catalogs

wmudge · 2021-07-22T18:00:23Z

+    description: Maximum number of available nodes for Virtual Warehouse autoscaling.
+    type: int
+    required: False
+  config:


There is no config parameter group in the code -- commonConfigs, applicationConfigs, ldapGroups, and enableSSO are all "top level" module parameters in the code. Moreover, the parameters are lowercase with underscores.

My bad, will fix this.

raju-saravanan · 2021-07-23T10:02:27Z

@wmudge : Address your comments, can you take another look ?

wmudge

The documentation and module specification sections need alignment and further detail.

wmudge · 2021-07-23T14:22:30Z

+    description: Configurations that are applied to every application in the service.
+    type: dict
+    required: False
+    contains:


Apparently, contains has now been replaced by suboptions -- for example: https://github.com/ansible-collections/azure/blob/dev/plugins/modules/azure_rm_securitygroup.py#L49-L52

Full docs are here: https://docs.ansible.com/ansible/latest/dev_guide/developing_modules_documenting.html#documentation-block

wmudge · 2021-07-23T14:23:06Z

+    required: False
+    contains:
+      configBlocks: List of ConfigBlocks for the application.
+        type: list


Lists require elements, which would be dict in this case, IIRC

wmudge · 2021-07-23T14:27:28Z

+            required: False
+          content:
+            description: Contents of a ConfigBlock.
+            type: obj


wmudge · 2021-07-23T14:27:42Z

+            contains:
+              keyValues: 
+                description: Key-value type configurations. 
+                type: obj


wmudge · 2021-07-23T14:29:26Z

+  - cloudera.cloud.cdp_auth_options
+'''
+
+EXAMPLES = r'''


Could you add an example or two of using the various nested configurations?

wmudge · 2021-07-23T14:35:26Z

+def main():
+    module = AnsibleModule(
+        argument_spec=CdpModule.argument_spec(
+            id=dict(required=False, type='str', default=None),


Only use default when necessary. The _get_param() should handle a non-existent parameter.

Same with required -- only needed if True

wmudge · 2021-07-23T14:56:39Z

+            autoscaling_min_nodes=dict(required=False, type='int', default=None),
+            autoscaling_max_nodes=dict(required=False, type='int', default=None),
+            common_configs=dict(required=False, type='dict', default=None),
+            application_configs=dict(required=False, type='dict', default=None),


Both common_configs and application_configs need to be expanded to match documentation rules above.

wmudge · 2021-07-23T14:57:15Z

+    )
+
+    result = DwVw(module)
+    output = dict(changed=False, virtual_warehouses=result.virtual_warehouses)


changed is invalid -- this module can make changes.

raju-saravanan · 2021-07-27T17:17:39Z

@wmudge : I have expanded the commonConfigs in argument_spec, but applicationConfig have some dynamic keys in the uppermost level so, couldn't expand beyond 'dict'. Eg :

 "applicationConfigs": {
        "das-webapp": {
        "configBlocks": [
           {
               "id": "hive-kerberos-config",
               "format": "TEXT",
               "content": {
                "text": "\n[libdefaults]\n\trenew_lifetime = 7d"
              }
          }
        ] 
      }   
    }

Here "das-webapp" is dynamic key, we can have configs for other services as well.

Signed-off-by: Saravanan Raju <saravanan.footloose@gmail.com>

wmudge · 2021-10-06T15:38:27Z

Superseded by #29

raju-saravanan force-pushed the cdw-level-2 branch from f8299df to 54d4676 Compare July 15, 2021 09:38

Chaffelson suggested changes Jul 15, 2021

View reviewed changes

Chaffelson suggested changes Jul 22, 2021

View reviewed changes

raju-saravanan force-pushed the cdw-level-2 branch 2 times, most recently from 7e43f33 to 898d275 Compare July 22, 2021 11:48

Chaffelson suggested changes Jul 22, 2021

View reviewed changes

Comment thread plugins/modules/dw_vw.py Outdated

raju-saravanan force-pushed the cdw-level-2 branch from 1c81d3d to 8122b01 Compare July 22, 2021 13:15

Chaffelson approved these changes Jul 22, 2021

View reviewed changes

wmudge requested changes Jul 22, 2021

View reviewed changes

raju-saravanan force-pushed the cdw-level-2 branch 2 times, most recently from 0ac1f4e to a647c61 Compare July 23, 2021 09:55

wmudge requested changes Jul 23, 2021

View reviewed changes

raju-saravanan force-pushed the cdw-level-2 branch from 4213ebb to 5d07cde Compare July 27, 2021 16:59

raju-saravanan added 8 commits August 5, 2021 16:14

Support CDW DBC and VW creation

dd79a8f

Signed-off-by: Saravanan Raju <saravanan.footloose@gmail.com>

Fix class docs

df6bff9

Signed-off-by: Saravanan Raju <saravanan.footloose@gmail.com>

Make vw and dbc modules idempotent

aca177d

Signed-off-by: Saravanan Raju <saravanan.footloose@gmail.com>

Make the dbc and vw module idempotent

0c0fbdc

Signed-off-by: Saravanan Raju <saravanan.footloose@gmail.com>

Use started state in wait for state function

71fdbb7

Signed-off-by: Saravanan Raju <saravanan.footloose@gmail.com>

Fix typo

6e504d3

Signed-off-by: Saravanan Raju <saravanan.footloose@gmail.com>

Rename vh and dbc module

a9eb2be

Signed-off-by: Saravanan Raju <saravanan.footloose@gmail.com>

Expand common_config and application_config

7032e26

Signed-off-by: Saravanan Raju <saravanan.footloose@gmail.com>

raju-saravanan force-pushed the cdw-level-2 branch from 5d07cde to 7032e26 Compare August 5, 2021 10:45

wmudge mentioned this pull request Aug 19, 2021

Add and update CDW modules #29

Merged

wmudge closed this Oct 6, 2021

Uh oh!

Conversation

raju-saravanan commented Jul 15, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

wmudge left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

raju-saravanan commented Jul 23, 2021

Uh oh!

wmudge left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

raju-saravanan commented Jul 27, 2021

Uh oh!

wmudge commented Oct 6, 2021

Uh oh!

Reviewers

Assignees

Labels

Milestone

Development

Uh oh!

3 participants

raju-saravanan commented Jul 15, 2021 •

edited

Loading