-
-
Notifications
You must be signed in to change notification settings - Fork 34
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
- Loading branch information
Showing
6 changed files
with
246 additions
and
126 deletions.
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -1,88 +1,5 @@ | ||
| # Contributing to `dbterd` | ||
|
|
||
| `dbterd` is open source software. It is what it is today because community members have opened issues, provided feedback, and contributed to the knowledge loop. Whether you are a seasoned open source contributor or a first-time committer, we welcome and encourage you to contribute code, documentation, ideas, or problem statements to this project. | ||
| [](https://dbterd.datnguyen.de/latest/nav/development/contributing-guide.html) | ||
|
|
||
| 1. [About this document](#about-this-document) | ||
| 2. [Getting the code](#getting-the-code) | ||
| 3. [Setting up an environment](#setting-up-an-environment) | ||
| 4. [Testing dbterd](#testing) | ||
| 5. [Submitting a Pull Request](#submitting-a-pull-request) | ||
|
|
||
| ## About this document | ||
|
|
||
| There are many ways to contribute to the ongoing development of `dbterd`, such as by participating in discussions and issues. We encourage you to first read our higher-level document: ["Expectations for Open Source Contributors"](add-document). | ||
|
|
||
| The rest of this document serves as a more granular guide for contributing code changes to `dbterd` (this repository). It is not intended as a guide for using `dbterd`, and some pieces assume a level of familiarity with Python development with `poetry`. Specific code snippets in this guide assume you are using macOS or Linux and are comfortable with the command line. | ||
|
|
||
| - **Branches:** All pull requests from community contributors should target the `main` branch (default). If the change is needed as a patch for a minor version of dbt that has already been released (or is already a release candidate), a maintainer will backport the changes in your PR to the relevant "latest" release branch (`1.0.<latest>`, `1.1.<latest>`, ...). If an issue fix applies to a release branch, that fix should be first committed to the development branch and then to the release branch (rarely release-branch fixes may not apply to `main`). | ||
| - **Releases**: Before releasing a new minor version, we prepare a series of beta release candidates to allow users to test the new version in live environments. This is an important quality assurance step, as it exposes the new code to a wide variety of complicated deployments and can surface bugs before official release. Releases are accessible via pip. | ||
|
|
||
| ## Getting the code | ||
|
|
||
| ### Installing git | ||
|
|
||
| You will need `git` in order to download and modify the `dbterd` source code. On macOS, the best way to download git is to just install [Xcode](https://developer.apple.com/support/xcode/). | ||
|
|
||
| ### External contributors | ||
|
|
||
| You can contribute to `dbterd` by forking the `dbterd` repository. For a detailed overview on forking, check out the [GitHub docs on forking](https://help.github.com/en/articles/fork-a-repo). In short, you will need to: | ||
|
|
||
| 1. Fork the `dbterd` repository | ||
| 2. Clone your fork locally | ||
| 3. Check out a new branch for your proposed changes | ||
| 4. Push changes to your fork | ||
| 5. Open a pull request against `datnguye/dbterd` from your forked repository | ||
|
|
||
|
|
||
| ## Setting up an environment | ||
|
|
||
| There are some tools that will be helpful to you in developing locally. While this is the list relevant for `dbterd` development, many of these tools are used commonly across open-source python projects. | ||
|
|
||
| ### Tools | ||
|
|
||
| We will buy `poetry` in `dbterd` development and testing. | ||
|
|
||
| So first install poetry via pip: | ||
| ```bash | ||
| python3 -m pip install poetry --upgrade | ||
| ``` | ||
|
|
||
| then, start installing the local environment: | ||
| ```bash | ||
| python3 -m poetry install | ||
| python3 -m poetry shell | ||
| poe git-hooks | ||
| pip install -e . | ||
| dbterd -h | ||
| ``` | ||
|
|
||
|
|
||
| ## Testing | ||
|
|
||
| Once you're able to manually test that your code change is working as expected, it's important to run existing automated tests, as well as adding some new ones. These tests will ensure that: | ||
| - Your code changes do not unexpectedly break other established functionality | ||
| - Your code changes can handle all known edge cases | ||
| - The functionality you're adding will _keep_ working in the future | ||
|
|
||
| #### `pytest` | ||
|
|
||
| Finally, you can also run a specific test or group of tests using [`pytest`](https://docs.pytest.org/en/latest/) directly. With a virtualenv active and dev dependencies installed you can do things like: | ||
|
|
||
| ```bash | ||
| poe test | ||
| ``` | ||
|
|
||
| Run test with coverage report: | ||
| ```bash | ||
| poe test-cov | ||
| ``` | ||
|
|
||
| > See [pytest usage docs](https://docs.pytest.org/en/6.2.x/usage.html) for an overview of useful command-line options. | ||
| ## Submitting a Pull Request | ||
|
|
||
| Code can be merged into the current development branch `main` by opening a pull request. A `dbterd` maintainer will review your PR. They may suggest code revision for style or clarity, or request that you add unit or integration test(s). These are good things! We believe that, with a little bit of help, anyone can contribute high-quality code. | ||
|
|
||
| Automated tests run via GitHub Actions. If you're a first-time contributor, all tests (including code checks and unit tests) will require a maintainer to approve. Changes in the `dbterd` repository trigger integration tests against Postgres. dbt Labs also provides CI environments in which to test changes to other adapters, triggered by PRs in those adapters' repositories, as well as periodic maintenance checks of each adapter in concert with the latest `dbterd` code changes. | ||
|
|
||
| Once all tests are passing and your PR has been approved, a `dbterd` maintainer will merge your changes into the active development branch. And that's it! Happy developing :tada: | ||
| _(please help to check [here](./docs/nav/development/contributing-guide.md) if the above link doesn't work)_ |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -1,21 +1,3 @@ | ||
| MIT License | ||
| [](https://dbterd.datnguyen.de/latest/license.html) | ||
|
|
||
| Copyright (c) 2022 Dat Nguyen | ||
|
|
||
| Permission is hereby granted, free of charge, to any person obtaining a copy | ||
| of this software and associated documentation files (the "Software"), to deal | ||
| in the Software without restriction, including without limitation the rights | ||
| to use, copy, modify, merge, publish, distribute, sublicense, and/or sell | ||
| copies of the Software, and to permit persons to whom the Software is | ||
| furnished to do so, subject to the following conditions: | ||
|
|
||
| The above copyright notice and this permission notice shall be included in all | ||
| copies or substantial portions of the Software. | ||
|
|
||
| THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR | ||
| IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, | ||
| FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE | ||
| AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER | ||
| LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, | ||
| OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE | ||
| SOFTWARE. | ||
| _(please help to check [here](./docs/license.md) if the above link doesn't work)_ |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -1 +1,124 @@ | ||
| ../README.md | ||
| # dbterd | ||
| CLI to generate Diagram-as-a-code file ([DBML](https://dbdiagram.io/d), [Mermaid](https://mermaid-js.github.io/mermaid-live-editor/)) from dbt artifact files (required: `manifest.json`, `catalog.json`) | ||
|
|
||
| [](https://pypi.org/project/dbterd/) | ||
|  | ||
| [](https://opensource.org/licenses/MIT) | ||
| [](https://www.python.org) | ||
| [](https://codecov.io/gh/datnguye/dbterd) | ||
|
|
||
| ``` | ||
| pip install dbterd --upgrade | ||
| ``` | ||
|
|
||
| Verify installed version: | ||
| ``` | ||
| dbterd --version | ||
| ``` | ||
|
|
||
|
|
||
| ```bash | ||
| dbterd -h | ||
| Usage: dbterd [OPTIONS] COMMAND [ARGS]... | ||
|
|
||
| Tools for producing diagram-as-code | ||
|
|
||
| Options: | ||
| --version Show the version and exit. | ||
| -h, --help Show this message and exit. | ||
|
|
||
| Commands: | ||
| debug Inspect the hidden magics | ||
| run Run the convert | ||
|
|
||
| Specify one of these sub-commands and you can find more help from there. | ||
| ``` | ||
|
|
||
| ## Quick examine with existing samples | ||
| ```bash | ||
| # select all models in dbt_resto | ||
| dbterd run -ad "samples/dbtresto" -o "target" | ||
| # select all models in dbt_resto, Select multiple dbt resources | ||
| dbterd run -ad "samples/dbtresto" -o "target" -rt "model" -rt "source" | ||
| # select only models in dbt_resto excluding staging | ||
| dbterd run -ad "samples/dbtresto" -o "target" -s model.dbt_resto -ns model.dbt_resto.staging | ||
| # select only models in schema name "mart" excluding staging | ||
| dbterd run -ad "samples/dbtresto" -o "target" -s schema:mart -ns model.dbt_resto.staging | ||
| # select only models in schema full name "dbt.mart" excluding staging | ||
| dbterd run -ad "samples/dbtresto" -o "target" -s schema:dbt.mart -ns model.dbt_resto.staging | ||
|
|
||
| # other samples | ||
| dbterd run -ad "samples/fivetranlog" -o "target" | ||
| dbterd run -ad "samples/fivetranlog" -o "target" -rt "model" -rt "source" | ||
|
|
||
| dbterd run -ad "samples/facebookad" -o "target" | ||
| dbterd run -ad "samples/facebookad" -o "target" -rt "model" -rt "source" | ||
|
|
||
| dbterd run -ad "samples/shopify" -o "target" | ||
| dbterd run -ad "samples/shopify" -o "target" -rt "model" -rt "source" | ||
|
|
||
| # your own sample without commiting to repo | ||
| dbterd run -mp "samples/local" -o "target" -rt "model" -rt "source" | ||
| ``` | ||
|
|
||
| ## Decide to exclude Relationship Tests from ERD generated | ||
| Add `ignore_in_erd` attribute into your test's meta: | ||
| ```yml | ||
| version: 2 | ||
|
|
||
| models: | ||
| - name: your_model | ||
| columns: | ||
| - name: your_column | ||
| tests: | ||
| - relationships_test: | ||
| to: ref('your_other_model') | ||
| field: your_other_column | ||
| meta: | ||
| ignore_in_erd: 1 | ||
| ``` | ||
| ## Quick DEMO | ||
| #### 1. Produce your manifest json | ||
| In your dbt project (I am using dbt-resto/[integration_tests](https://github.com/datnguye/dbt-resto) for demo purpose), try to build the docs: | ||
| ```bash | ||
| dbt docs generate | ||
| ``` | ||
|
|
||
| #### 2. Generate DBML | ||
| Copy `manifest.json` into a specific folder, and run | ||
| ``` | ||
| dbterd run -mp "/path/to/dbt/target" -o "/path/to/output" | ||
| # dbterd run -mp "./target/v4-dbtresto" -o "./target" -s model.dbt_resto -ns model.dbt_resto.staging | ||
| ``` | ||
|
|
||
| File `./target/output.dbml` will be generated as the result | ||
|
|
||
| #### 3. Build database docs site (Optional) | ||
| Assuming you're already familiar with [dbdocs](https://dbdocs.io/docs#installation) | ||
| ``` | ||
| dbdocs build "/path/to/output/output.dbml" | ||
| # dbdocs build "./target/output.dbml" | ||
| ``` | ||
|
|
||
| Your terminal should provide the info as below: | ||
| ```bash | ||
| √ Parsing file content | ||
| ? Project name: poc | ||
| ‼ Password is not set for 'poc' | ||
| √ Done. Visit: https://dbdocs.io/datnguye/poc | ||
| ``` | ||
|
|
||
| The site will be looks like: | ||
|
|
||
|  | ||
|
|
||
| Result after applied Model Selection: | ||
|  | ||
|
|
||
|
|
||
| ## Contributing ✨ | ||
| If you've ever wanted to contribute to this tool, and a great cause, now is your chance! | ||
|
|
||
| See the contributing docs [CONTRIBUTING.md](https://github.com/datnguye/dbterd/blob/main/CONTRIBUTING.md) for more information |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -1 +1,21 @@ | ||
| ../LICENSE | ||
| MIT License | ||
|
|
||
| Copyright (c) 2022 Dat Nguyen | ||
|
|
||
| Permission is hereby granted, free of charge, to any person obtaining a copy | ||
| of this software and associated documentation files (the "Software"), to deal | ||
| in the Software without restriction, including without limitation the rights | ||
| to use, copy, modify, merge, publish, distribute, sublicense, and/or sell | ||
| copies of the Software, and to permit persons to whom the Software is | ||
| furnished to do so, subject to the following conditions: | ||
|
|
||
| The above copyright notice and this permission notice shall be included in all | ||
| copies or substantial portions of the Software. | ||
|
|
||
| THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR | ||
| IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, | ||
| FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE | ||
| AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER | ||
| LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, | ||
| OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE | ||
| SOFTWARE. |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -1 +1,88 @@ | ||
| ../../../CONTRIBUTING.md | ||
| # Contributing to `dbterd` | ||
|
|
||
| `dbterd` is open source software. It is what it is today because community members have opened issues, provided feedback, and contributed to the knowledge loop. Whether you are a seasoned open source contributor or a first-time committer, we welcome and encourage you to contribute code, documentation, ideas, or problem statements to this project. | ||
|
|
||
| 1. [About this document](#about-this-document) | ||
| 2. [Getting the code](#getting-the-code) | ||
| 3. [Setting up an environment](#setting-up-an-environment) | ||
| 4. [Testing dbterd](#testing) | ||
| 5. [Submitting a Pull Request](#submitting-a-pull-request) | ||
|
|
||
| ## About this document | ||
|
|
||
| There are many ways to contribute to the ongoing development of `dbterd`, such as by participating in discussions and issues. We encourage you to first read our higher-level document: ["Expectations for Open Source Contributors"](add-document). | ||
|
|
||
| The rest of this document serves as a more granular guide for contributing code changes to `dbterd` (this repository). It is not intended as a guide for using `dbterd`, and some pieces assume a level of familiarity with Python development with `poetry`. Specific code snippets in this guide assume you are using macOS or Linux and are comfortable with the command line. | ||
|
|
||
| - **Branches:** All pull requests from community contributors should target the `main` branch (default). If the change is needed as a patch for a minor version of dbt that has already been released (or is already a release candidate), a maintainer will backport the changes in your PR to the relevant "latest" release branch (`1.0.<latest>`, `1.1.<latest>`, ...). If an issue fix applies to a release branch, that fix should be first committed to the development branch and then to the release branch (rarely release-branch fixes may not apply to `main`). | ||
| - **Releases**: Before releasing a new minor version, we prepare a series of beta release candidates to allow users to test the new version in live environments. This is an important quality assurance step, as it exposes the new code to a wide variety of complicated deployments and can surface bugs before official release. Releases are accessible via pip. | ||
|
|
||
| ## Getting the code | ||
|
|
||
| ### Installing git | ||
|
|
||
| You will need `git` in order to download and modify the `dbterd` source code. On macOS, the best way to download git is to just install [Xcode](https://developer.apple.com/support/xcode/). | ||
|
|
||
| ### External contributors | ||
|
|
||
| You can contribute to `dbterd` by forking the `dbterd` repository. For a detailed overview on forking, check out the [GitHub docs on forking](https://help.github.com/en/articles/fork-a-repo). In short, you will need to: | ||
|
|
||
| 1. Fork the `dbterd` repository | ||
| 2. Clone your fork locally | ||
| 3. Check out a new branch for your proposed changes | ||
| 4. Push changes to your fork | ||
| 5. Open a pull request against `datnguye/dbterd` from your forked repository | ||
|
|
||
|
|
||
| ## Setting up an environment | ||
|
|
||
| There are some tools that will be helpful to you in developing locally. While this is the list relevant for `dbterd` development, many of these tools are used commonly across open-source python projects. | ||
|
|
||
| ### Tools | ||
|
|
||
| We will buy `poetry` in `dbterd` development and testing. | ||
|
|
||
| So first install poetry via pip: | ||
| ```bash | ||
| python3 -m pip install poetry --upgrade | ||
| ``` | ||
|
|
||
| then, start installing the local environment: | ||
| ```bash | ||
| python3 -m poetry install | ||
| python3 -m poetry shell | ||
| poe git-hooks | ||
| pip install -e . | ||
| dbterd -h | ||
| ``` | ||
|
|
||
|
|
||
| ## Testing | ||
|
|
||
| Once you're able to manually test that your code change is working as expected, it's important to run existing automated tests, as well as adding some new ones. These tests will ensure that: | ||
| - Your code changes do not unexpectedly break other established functionality | ||
| - Your code changes can handle all known edge cases | ||
| - The functionality you're adding will _keep_ working in the future | ||
|
|
||
| #### `pytest` | ||
|
|
||
| Finally, you can also run a specific test or group of tests using [`pytest`](https://docs.pytest.org/en/latest/) directly. With a virtualenv active and dev dependencies installed you can do things like: | ||
|
|
||
| ```bash | ||
| poe test | ||
| ``` | ||
|
|
||
| Run test with coverage report: | ||
| ```bash | ||
| poe test-cov | ||
| ``` | ||
|
|
||
| > See [pytest usage docs](https://docs.pytest.org/en/6.2.x/usage.html) for an overview of useful command-line options. | ||
| ## Submitting a Pull Request | ||
|
|
||
| Code can be merged into the current development branch `main` by opening a pull request. A `dbterd` maintainer will review your PR. They may suggest code revision for style or clarity, or request that you add unit or integration test(s). These are good things! We believe that, with a little bit of help, anyone can contribute high-quality code. | ||
|
|
||
| Automated tests run via GitHub Actions. If you're a first-time contributor, all tests (including code checks and unit tests) will require a maintainer to approve. Changes in the `dbterd` repository trigger integration tests against Postgres. dbt Labs also provides CI environments in which to test changes to other adapters, triggered by PRs in those adapters' repositories, as well as periodic maintenance checks of each adapter in concert with the latest `dbterd` code changes. | ||
|
|
||
| Once all tests are passing and your PR has been approved, a `dbterd` maintainer will merge your changes into the active development branch. And that's it! Happy developing :tada: |