Harshal Sheth
954397a078
fix(ingest): remove dbt disable_dbt_node_creation and load_schema options ( #5877 )
2022-09-09 14:07:55 -07:00
Marcin Szymański
57996049fb
fix(ingest): sql - add missing trino types ( #5870 )
...
Co-authored-by: Shirshanka Das <shirshanka@apache.org>
2022-09-09 14:07:00 -07:00
Shirshanka Das
5074035c4d
feat(ingest): looker - improve defaults for usage extraction ( #5893 )
...
Co-authored-by: Ravindra Lanka <rslanka@gmail.com>
2022-09-09 13:35:49 -07:00
Shirshanka Das
056add128d
fix(ingest): datahub-api - move instantiation to the right config class ( #5878 )
2022-09-09 13:34:21 -07:00
Pedro Silva
e7b33c595b
feat(cli): make docker compose quiet ( #5869 )
2022-09-08 14:55:03 -07:00
Harshal Sheth
da46dcdc22
refactor(ingest): add ALL_ENV_TYPES constant ( #5866 )
2022-09-08 14:07:22 -07:00
Harshal Sheth
6063484714
fix(ingest): avrogen handling for missing fields with default values ( #5844 )
2022-09-08 14:05:28 -07:00
Harshal Sheth
2115d5bf1d
fix(ingest): remove dbt delete_tests_as_datasets option ( #5865 )
...
This option was added for backwards compat and can be dropped now.
2022-09-08 11:22:25 -07:00
Harshal Sheth
39fa9321c1
feat(ingest): support reading config file from stdin ( #5847 )
2022-09-08 11:04:20 -07:00
Harshal Sheth
eb57bebf48
fix(ingest): include urns in rest sink failure logs ( #5848 )
2022-09-08 10:29:40 -07:00
Shirshanka Das
b9068ffd2e
fix(m1): preflight works with brew postgres@14 ( #5868 )
2022-09-08 15:02:30 +05:30
Shirshanka Das
d911b15da5
docs(ingest): improve doc gen, docs for snowflake, looker ( #5867 )
2022-09-08 00:38:10 -07:00
Harshal Sheth
6f09f1025e
ci(ingest): test with python 3.10 ( #5863 )
2022-09-07 17:08:49 -07:00
Harshal Sheth
1e8a1038d8
chore(ingest): remove outdated GE compatibility hack ( #5862 )
2022-09-07 17:06:02 -07:00
Tamas Nemeth
f9917ea5fb
Fixing sharded regexp pattern config ( #5861 )
2022-09-07 15:36:37 -07:00
Shirshanka Das
2fea3d26df
fix(ingest): reporting should work with timestamps ( #5860 )
2022-09-07 21:09:08 +02:00
Harshal Sheth
abddc01877
fix(ingest): fix doc generation import ordering issue with postgres ( #5846 )
...
Relying on the correct import directly, rather than going through
SQLAlchemy's import wrapper (in their dialect.py) allows us to bypass
this strange error in doc generation.
2022-09-07 21:44:33 +05:30
Harshal Sheth
08622f25ef
feat(ingest): add utility for converting MCEs to MCPs ( #5812 )
2022-09-06 15:25:48 -07:00
mohdsiddique
2f65e2f226
feat(transformers): Add semantics & transform_aspect support in transformers ( #5514 )
...
Co-authored-by: MohdSiddique Bagwan <mohdsiddique.bagwan@gslab.com>
Co-authored-by: Harshal Sheth <hsheth2@gmail.com>
2022-09-06 14:44:14 -07:00
Pedro Silva
20138a32e5
fix(standalone-consumers): Exclude Solr from spring boot application config & make them run on M1 ( #5827 )
...
* fix(standalone-consumers) Removes Solr from spring boot application config
* Adds standalone consumer option to datahub quickstart cli
* Rename files
* Make dockerize platform agnostic & change docker compose utility to work with M1
* Fix MAE/MCE dockerfiles for arm & make smoke tests use standalone consumers
* Fix identation
* Use master code
* Adds ARM image publishing for consumers
* Fix linter
* Fix lint
2022-09-06 13:55:31 -05:00
Shirshanka Das
f4eea5dfd5
fix(ingest): profiling - memory usage reduction ( #5830 )
2022-09-05 23:37:19 -07:00
Mayuri Nehate
a8c1397e5e
feat(ingest): tableau - improve lineage, workbooks query, fix pagination ( #5756 )
2022-09-05 23:19:22 -07:00
Harshal Sheth
7e16ce0126
feat(ingest): looker - add test connection ( #5768 )
...
Co-authored-by: Shirshanka Das <shirshanka@apache.org>
2022-09-05 21:11:34 -07:00
Harshal Sheth
bee27e8468
fix(ingest): use AwsConnectionConfig instead of AwsSourceConfig ( #5813 )
2022-09-05 21:10:33 -07:00
Harshal Sheth
6d83eb9449
refactor(ingest): prefer as imports instead of pegasus2avro files ( #5820 )
2022-09-05 21:09:07 -07:00
David Sanchez
a092a770a3
docs(ingest): bigquery - add needed delete role to list ( #5826 )
2022-09-05 21:04:33 -07:00
Tamas Nemeth
d283e2b588
fix(ingest): presto-on-hive - customize filter query for postgres ( #5825 )
2022-09-05 21:01:55 -07:00
Shirshanka Das
23b929ea10
feat(cli): report memory usage during ingest ( #5828 )
2022-09-05 11:53:39 -07:00
Shirshanka Das
172441beae
fix(ingest): looker - report dashboard scanning correctly ( #5829 )
2022-09-05 11:10:47 -07:00
Mayuri Nehate
4bf4236e29
feat(ingest): snowflake-beta - populate size, add table level profiles for large tables ( #5774 )
2022-09-04 22:37:57 -07:00
Shirshanka Das
84b279a933
feat(ingest):looker - reduce mem usage, misc reporting improvements ( #5823 )
2022-09-04 15:43:57 -07:00
Aseem Bansal
b643b34d30
feat(restore-indices): add endpoint for restore indices, add basic check for graph ( #5805 )
2022-09-02 15:24:40 +05:30
Shirshanka Das
ae4fb7c2e1
feat(ingest): looker - reduce memory requirements ( #5815 )
2022-09-02 10:45:51 +02:00
Tamas Nemeth
c91e29b052
fix(ingest): bigquery-beta - Fixing dependencies ( #5814 )
...
* Fixing bigquery beta dependencies
* Removing unneded comment
2022-09-02 13:47:14 +05:30
Shirshanka Das
9afda47085
feat(cli): add support for sampled reporting to keep logs manageable ( #5800 )
2022-09-01 14:47:28 -07:00
Harshal Sheth
c05f3970fd
feat(ingest): cli - add rewrite option for metadata file check ( #5763 )
2022-09-01 14:30:00 -07:00
Harshal Sheth
e884b69b0e
chore(ingest): remove archived docs ( #5793 )
2022-09-01 12:10:03 -07:00
Ravindra Lanka
af1fc8d91d
fix(ingest): snowflake - fix oauth config validation ( #5796 )
2022-08-31 17:51:13 -07:00
mohdsiddique
dbcd3099d4
fix(ingestion): looker - fix dashboard and chart-usage calculations ( #5791 )
...
Co-authored-by: Shirshanka Das <shirshanka@apache.org>
Co-authored-by: MohdSiddique Bagwan <mohdsiddique.bagwan@gslab.com>
2022-08-31 16:22:11 -07:00
Gabe Lyons
570367ce4a
docs(lookml): updating github_info in lookml docs ( #5779 )
2022-08-31 10:23:10 -07:00
Shirshanka Das
134ca67369
feat(cli): add spinner to indicate progress ( #5769 )
2022-08-30 13:25:03 -07:00
Harshal Sheth
138846acca
fix(ingest): add pydantic version upper bound ( #5775 )
2022-08-30 11:07:23 -07:00
mohdsiddique
1f1056d7cf
feat(ingestion): looker - chart usage statistics ( #5652 )
...
Co-authored-by: Shirshanka Das <shirshanka@apache.org>
2022-08-30 00:21:31 -07:00
Tamas Nemeth
4572c96d60
feat(ingestion): bigquery - Bigquery beta connector - first cut ( #5663 )
2022-08-30 07:33:24 +02:00
Harshal Sheth
24e4ee1746
feat(ingest): add config to extractor interface ( #5761 )
2022-08-29 19:16:17 -07:00
Harshal Sheth
80edf7c8ed
fix(ingest): add databricks dep for hive ( #5764 )
2022-08-29 19:14:04 -07:00
Mayuri Nehate
cc2a86853b
fix(ingest): snowflake-beta - fix missing initialization of variable ( #5757 )
2022-08-29 16:35:36 -07:00
Harshal Sheth
eb87db9813
fix(ingest): proper null skip logic in serialization ( #5749 )
2022-08-29 16:34:58 -07:00
Tamas Nemeth
7e15947a37
fix(doc): presto-on-hive - Removing new lines from docs to fix doc generation ( #5755 )
2022-08-29 13:42:07 +02:00
Harshal Sheth
ee43262f7c
chore(ingest): add vulture config and remove dead code ( #5745 )
2022-08-28 23:41:15 -07:00
