Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Switches for views, external schemas, remote DBs, etc. #214

Open
mateuszboryn opened this issue Apr 19, 2023 · 2 comments
Open

Switches for views, external schemas, remote DBs, etc. #214

mateuszboryn opened this issue Apr 19, 2023 · 2 comments
Labels
enhancement New feature or request

Comments

@mateuszboryn
Copy link
Contributor

It would be good to have switches in command line options (and in API of course):

  • to enable or disable analyzing database views - usually views provide redundant data to tables, thus we waste computation time
  • to enable or disable analyzing of external schemas (in case of Redshift), or schemas that are mapped from remote DBs - to avoid deep scanning of external data sources which may incur costs.
@jhecking jhecking added the enhancement New feature or request label May 12, 2023
@nicolepng
Copy link
Contributor

Hi @mateuszboryn piicatcher currently utilises the amundson package to retrieve data from the databases and the sql queries are on a tabular level [https://github.com/amundsen-io/amundsen/blob/main/databuilder/databuilder/extractor/postgres_metadata_extractor.py]. Hence, we are unable to filter out the database views and create a switch for that.

@vrajat
Copy link
Member

vrajat commented May 31, 2023

Another option is to use include/exclude lists: https://docs.tokern.io/piicatcher/include_exclude_lists

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request
Projects
None yet
Development

No branches or pull requests

4 participants