Aseem Bansal
5a59d5a1dd
fix(ingestion): Adding missing init.py ( #4659 )
2022-04-13 11:02:57 +02:00
Aseem Bansal
155209f0e1
fix(ingestion): add missing workunit ids ( #4657 )
2022-04-13 10:19:37 +02:00
Tamas Nemeth
f99d27fd8c
feat(ingest): airflow - add support to capture airflow executions, add high level dataflow jobs api to python sdk ( #4615 )
...
Co-authored-by: Gabe Lyons <itsgabelyons@gmail.com>
2022-04-12 23:19:39 -07:00
Gabe Lyons
c92990d32b
feat(ml): bringing ml screens up to date w/ the modern ui layout & improving ml lineage ( #4651 )
...
* backend ml changes
* updating ml model UI
* more work on the UI
* ml primary key joining the party
* more progress on UI
* making progress on lineage
* finalizing UI experience
* remove irrelevant test
* fixing lint
* fixups
* add tests and fix what the issues they discovered
* internal > core
2022-04-12 22:42:12 -07:00
Kevin Hu
08c34bfe15
feat(ingest): capture MSSQL table+column descriptions ( #4579 )
...
* feat(ingest): capture MSSQL table+column descriptions
Co-authored-by: Ravindra Lanka <rlanka@acryl.io>
2022-04-12 17:49:56 -07:00
David Sanchez
9a950ef231
fix(tableau): avoid duplicate schema in URNs for upstream tables ( #4645 )
...
* fix(tableau): avoid duplicate schema in URNs for upstream tables
* Fix(lint)
2022-04-12 16:26:52 -07:00
Dexter Lee
c9f2c453d2
Fix issue where browse cards are missing ( #4652 )
2022-04-12 16:19:05 -07:00
Meenakshi Kamalaseshan Radha
e75e2f8bbf
fix(ingest): Fix snowflake KEY_PAIR auth ( #4638 )
...
* fix(ingest): Fix snowflake KEY_PAIR auth to work with stateful ingestion.
Co-authored-by: Ravindra Lanka <rlanka@acryl.io>
2022-04-12 15:58:53 -07:00
Zach Bluhm
ff685b7feb
feat: Enable the ingestion of bigquery audit logs to parse usage info… ( #4441 )
...
* feat: Enable the ingestion of bigquery audit logs to parse usage information
Co-authored-by: Ravindra Lanka <rlanka@acryl.io>
2022-04-12 14:58:34 -07:00
Ravindra Lanka
9226e3e27f
Enable lower-casing of the name part of dataset urn via an environment vairable. ( #4649 )
2022-04-12 12:54:22 -07:00
Dyana Rose
5b22d96e04
fix(ingestion): looker - extract explore views from join name ( #4627 )
...
Co-authored-by: Dyana Rose <dyanarose@gmail.com>
Co-authored-by: Shirshanka Das <shirshanka@apache.org>
2022-04-12 08:20:10 -07:00
Aseem Bansal
23ece3b1a4
fix(ingestion): ensure source/sink reports are always logged ( #4592 )
2022-04-12 05:00:59 -07:00
Xu Wang
7b1487135a
feat(ingest): add Urn python library for DataJob, DataFlow, Domain and Tag ( #4618 )
...
* feat(ingest): add python library for DataJobUrn
* add DataFlowUrn lib and fix DataJobUrn
* fix create_from_str method
* fix lint error and unit test
* add DomainUrn and TagUrn
Co-authored-by: Xu Wang <xu.wang@grandrounds.com>
2022-04-12 09:02:28 +02:00
Tamas Nemeth
902f525307
[model]: Dataprocess instance entity to model datajob/jobflow runs ( #4459 )
...
* Initial model for dataprocess instance to model datajob/jobflow runs
* Fixing typos
* Removing dataprocessinstance from entity registry
* Add dataprocessinstance to entity registry
Adding DataprocessInstancekey aspect
* Fixing relationships
* Updating the model
* Removing EditableDataProcessInstanceProperties
* Makint try number optional
* Updating model
* Fixing typo
* visualizing job runs for a task
* add pagination
* show utc time on hover
* add dataset side
* dataprocessinstance > dataprocessinst
* add cypress test
* Removing jobs from DataFlowInfo.pdl as it should be part of another change if we really want it
* Adding missing files
* Fix moved import
* Remove duplicate import
* fixing yarn test
* Update DatasetRunsResolver.java
* Update TaskRunsResolver.java
* Update DatasetRunsResolver.java
* responding to comments
Co-authored-by: Gabe Lyons <itsgabelyons@gmail.com>
Co-authored-by: John Joyce <john@acryl.io>
2022-04-12 08:52:07 +02:00
Javier Sotelo
fa4cc3ad31
fix(metadata-service): stop overriding kafka registry props with empty values ( #4604 )
...
Co-authored-by: jsotelo <javier.sotelo@viasat.com>
2022-04-11 22:02:47 -07:00
Marcin Szymański
e7c5eb357c
feat(ingest): add trino platform for great expectations ( #4594 )
2022-04-11 19:48:15 -07:00
jchen0824
524d183d93
feat: add presto-on-hive metadata ingestion source ( #4625 )
...
* feat(metadata ingestion source): add presto-on-hive metadata ingestion source
Co-authored-by: Houren Chen <houren.chen@grabtaxi.com>
Co-authored-by: Ravindra Lanka <rlanka@acryl.io>
2022-04-11 17:46:44 -07:00
Dexter Lee
90bc00c697
feat(search): Only reindex if the mappings for an existing field changed ( #4629 )
...
* Just update mappings as much as possible
* Fix checkstyle
2022-04-11 16:06:41 -07:00
Dexter Lee
2a8ae27261
Revert "feat: adding gradle, pip cache via gh cache, docker cache via dockerhub ( #4387 )" ( #4637 )
...
This reverts commit b6de051d9a5240cefa1a7b217690cfe3a85a70f5.
2022-04-11 16:00:16 -07:00
Aditya Radhakrishnan
80001d2206
fix(cypress) - add waits for cypress search test to remove flakiness ( #4640 )
2022-04-11 15:56:43 -07:00
Tamas Nemeth
e2a617f183
Restricting pytest docker version ( #4639 )
2022-04-11 23:35:34 +02:00
Aseem Bansal
04fd8fcbe8
doc(cli): add env variables and options for ingest command ( #4598 )
2022-04-11 09:18:51 +02:00
BZ
5637e73ca5
feat(glue): add CatalogId parameter for cross-account access ( #4608 )
...
* Update glue.py
* Update glue.md
* Update glue.py
2022-04-11 09:08:25 +02:00
Aseem Bansal
aa0fe3636a
doc(scheduling): make it easier to find ui ingestion ( #4610 )
2022-04-08 10:26:41 -07:00
Aseem Bansal
b6de051d9a
feat: adding gradle, pip cache via gh cache, docker cache via dockerhub ( #4387 )
...
* attempt cache
* change cache keys
* use docker repository for caching
* change buildcache tag
* change tag
* fix cache key
* take docker cache from head buildcache
* try not pushing when not publishing
* change to use templating
* change docker caching
* test change org name
* check using cache
* revert changes
* remove gradle cache from build and test workflow
2022-04-08 09:19:58 -07:00
Aseem Bansal
61a95f41ae
chore: fix lint and remove incorrect integration mark from unit tests ( #4621 )
...
* chore: fix lint and remove incorrect integration mark from unit tests
* add to test requirements
* revert athena source tests
2022-04-08 17:18:48 +02:00
Abhiram98
cd43a4a543
doc(redshift): Add grant statements ( #4559 )
2022-04-08 16:30:43 +02:00
Marcin Szymański
7c3ad3d293
feat(ingest): enable connection string for all sqlalchemy datasources ( #4508 )
...
* feat(ingest): enable connection string for all sqlalchemy datasources
* Update sql_common.py
* fix types
* update docs
* rename variable to sqlalchemy_uri
Co-authored-by: Harshal Sheth <hsheth2@gmail.com>
2022-04-07 23:11:52 -04:00
John Joyce
45e09ca824
feat(ui): Adding GraphQL queries to fetch entity deprecation status ( #4614 )
2022-04-07 19:17:24 -07:00
John Joyce
060f965577
fix(ui): Fix dashboard tags display ( #4611 )
2022-04-07 19:17:12 -07:00
leifker
aa05faad32
fix(protobuf) Fix test instability by deterministrically selecting field ( #4612 )
2022-04-07 13:35:44 -07:00
Aseem Bansal
336a628c5b
fix(bigquery): fix lineage bug, improve docs, add dataset filter config ( #4607 )
...
* fix(bigquery): fix metadata from exported logs, doc missing permission, improve logging, add tests
Co-authored-by: Ravindra Lanka <rslanka@gmail.com>
2022-04-07 13:10:21 -07:00
David Haglund
0785ed6143
fix: urlencode slash in urns too ( #4527 )
...
* fix: urlencode slash in urns too + tests
Co-authored-by: Ravindra Lanka <rlanka@acryl.io>
2022-04-07 13:04:57 -07:00
Gabe Lyons
112589db32
feat(tableau): add some logic to normalize table names in tableau ( #4609 )
...
* add some logic to normalize table names in tableau
2022-04-07 12:15:41 -07:00
Ravindra Lanka
5e25cd1e22
feat(ingestion): Redshift Usage Source - simplify OperationalStats workunit generation. ( #4585 )
...
* feat(ingestion): Redshift Usage Source - simplify OperationalStats workunit generation.
2022-04-07 11:24:26 -07:00
Dexter Lee
55f0412a33
feat(search): Add SearchScore annotation to use fields for search ranking ( #4596 )
...
* Add SearchScore annotation
* Add back test-model
* Remove search features
* Fix to John's comments
* simplify ranker
* Fix checkstyle
2022-04-07 11:07:27 -07:00
Fernando Falci
5d7d4c5d92
docs(GraphQL): fix typo ( #4605 )
2022-04-07 09:52:15 -07:00
Aseem Bansal
25de224a3f
doc: add page for handling deprecations, breaking changes etc. ( #4590 )
2022-04-07 09:48:47 -07:00
Dexter Lee
12f9cc2575
Reduce browse aggregation size ( #4601 )
2022-04-06 22:07:08 -07:00
Dexter Lee
32324a6476
fix(search): Make sure home page and search pages are consistent ( #4588 )
...
* Centralize entity count cache
* Fix to John's comment
* Fix test
* Fix tests
2022-04-06 22:06:55 -07:00
John Joyce
7080798825
feat(metadata-service): Supporting a configurable Authorizer Chain ( #4584 )
2022-04-06 19:13:39 -07:00
Aseem Bansal
5ebb37ab4c
fix(bigquery): incorrect lineage when views are present ( #4568 )
...
* fix(bigquery): incorrect lineage when views are present
Co-authored-by: Ravindra Lanka <rlanka@acryl.io>
2022-04-06 17:29:02 -07:00
David Haglund
3a5cf8eded
fix(doc): try to fix broken link ( #4593 )
2022-04-06 10:38:39 -07:00
Aseem Bansal
df1d8ad07e
doc(snowflake): add example of table pattern ( #4580 )
2022-04-05 16:23:21 -07:00
Aditya Radhakrishnan
aeafa7e63f
feat(okta) - add support for filtering/searching when ingesting Okta groups and users ( #4586 )
2022-04-05 16:15:34 -07:00
Aseem Bansal
32349bf405
chore: update default cli for managed ingestion ( #4581 )
2022-04-05 15:33:13 -07:00
mayurinehate
0a97fa22f9
fix(tableau): fix for incorrect schema returned by tableau api for snowflake connectionType ( #4577 )
2022-04-05 14:56:35 -07:00
Aditya Radhakrishnan
8610c29e49
feat(search) - add DATETIME FieldType ( #4407 )
...
* feat(search) - add DATETIME FieldType
2022-04-05 14:41:54 -07:00
John Joyce
bc1c06069a
Adding PRE to graphql ( #4582 )
2022-04-05 11:29:06 -07:00
Ravindra Lanka
fe5f24c2b3
fix(ingestion): Refactor redshift_usage source: simplify, annotate & fix bugs. ( #4572 )
2022-04-05 09:21:27 -07:00
