yujunjun/datahub
2
0
Fork 0
mirror of https://github.com/datahub-project/datahub.git synced 2025-07-29 20:39:56 +00:00
Code Issues Packages Projects Releases Wiki Activity
11,753 Commits 202 Branches 123 Tags
Commit Graph

3904 Commits

Author SHA1 Message Date
Aseem Bansal
1478489c9c
fix(bigquery): missing dependency (#4567) 
* fix(bigquery): missing dependency

* code review feedback
 2022-04-04 08:34:34 -07:00
mayurinehate
58e4364354
fix(tableau): gracefully stop ingestion if tableau sign in not successful (#4548) 
* fix(tableau): gracefully stop ingestion if tableau sign in not successful

* Update tableau.md

* Update tableau.md

* docs(tableau): update doc, add caveats, use env variables in credentials

Co-authored-by: John Joyce <john@acryl.io>
 2022-04-04 13:15:08 +02:00
Dexter Lee
57ea768e7f
fix(docs): fix logo links on ingestion docs (#4564) 2022-04-01 22:02:24 -07:00
Abhiram98
26742728a6
feat(ingestion): schema, table filtering for redshift-usage (#4396) 
* Filter based on table/schema pattern + documentation

Co-authored-by: Ravindra Lanka <rlanka@acryl.io>
 2022-04-01 20:48:23 -07:00
darapuk
a05d798939
(fix): Update path generated when creating LookML URL (#4554) 
* (fix): Update path generated when creating LookML URL
 2022-04-01 11:54:36 -07:00
Aseem Bansal
40188d27c9
fix: dependencies for usage sources (#4561) 2022-04-01 09:56:52 -07:00
Corentin
2fc3a48bc5
feat(ingest): indent sql queries for usage sources (#3782) 
* feat(ingest): indent sql queries for usage connectors.

Co-authored-by: EC2 Default User <ec2-user@ip-172-31-22-140.eu-west-1.compute.internal>
Co-authored-by: Ravindra Lanka <rlanka@acryl.io>
 2022-03-31 15:15:09 -07:00
Aseem Bansal
94890c1e71
fix(ingest): snowflake-usage - log warning instead of error out (#4544) 2022-03-31 08:16:15 -07:00
mayurinehate
c09834d52b
fix(kafka-connect): add platform for default case in jdbc connector, update tests for platform instance map (#4545) 2022-03-31 08:13:09 -07:00
mayurinehate
467ea7917c
fix(kafka-connect): fix lineage for postgres-like 3-level hierarchy d… (#4375) 
* fix(kafka-connect): fix lineage for postgres-like 3-level hierarchy dialects in jdbc source connnector

Co-authored-by: Ravindra Lanka <rlanka@acryl.io>
 2022-03-30 20:59:02 -07:00
RyanHolstien
d311067384
fix(cli): delete - handle case insensitive entity types (#4492) 2022-03-30 20:37:14 -07:00
Sergio Gómez Villamor
bdf17f551e
feat(ingest): glue - adds platform instance capability (#4130) 2022-03-30 18:50:26 -07:00
mohdsiddique
57002c766d
feat(stateful dbt): add stateful ingestion capability in dbt source (#4456) 
* feat(stateful dbt): add stateful ingestion capability in dbt source

Co-authored-by: MohdSiddiqueBagwan <mohdsiddique.bagwan@gslab.com>
Co-authored-by: Ravindra Lanka <rlanka@acryl.io>
 2022-03-30 18:09:02 -07:00
Pedro Silva
306ddff13e
feat(platform): adds side-effect report for rollbacks (#4482) 
Co-authored-by: Shirshanka Das <shirshanka@apache.org>
 2022-03-30 17:33:35 -07:00
Maggie Hays
0be0689093
docs: Ingestion Source Docs Template (#4275) 
* testing img.shield for status

* update to hyperlink

* changing link format

* adding status options

* updating prerequisities and quickstart

* update to ingestion docs

* updating template with collapse details

* adding linebreak between pip install commands

* Removed incomplete sentence

* typo fix

* pushing current changes

* testing logos in markdown table

* markdown table fix

* markdown table fix

* adding in additional logos

* transposing markdown table

* settling on final table format

* adding commented-out source template to sidebar.js

* moving reference sidebar and adding trailing comma

* fixing docs build
 2022-03-30 15:36:39 -07:00
mayurinehate
9ba36100ab
feat(tableau): emit lineage edge from embedded datasource to upstream… (#4470) 
* feat(tableau): emit lineage edge from embedded datasource to upstream published datasource

Co-authored-by: Ravindra Lanka <rlanka@acryl.io>
 2022-03-30 15:32:15 -07:00
Sunil Patil
36e9552d61
feat(ingestion): Support pluggable Schema Registry for Kafka Source (#4535) 
* Support for pluggable schema registry for the Kafka source.
Co-authored-by: Sunil Patil <spatil@twilio.com>
Co-authored-by: Ravindra Lanka <rlanka@acryl.io>
 2022-03-30 13:20:23 -07:00
Aseem Bansal
5d9c146bf7
doc: add example of profiling in default example (#4532) 2022-03-30 10:07:32 -07:00
Kevin Hu
1bad3c7bc9
fix(ingest): mssql - support database_alias (#4523) 2022-03-29 20:47:43 -07:00
Arun Vasudevan
c79c778270
feat(ingest): kafka-connect - support mapping for multiple DB instances (#4501) 2022-03-29 20:46:07 -07:00
John Joyce
c69310522b
feat(metadata service): Introducing Platform Events (#4477) 2022-03-29 18:32:04 -07:00
cuong-pham
4833452192
fix(ingest): make tableau ingestion more resilient to error (#4494) 2022-03-29 17:44:50 -07:00
Aseem Bansal
a137631cb5
doc(ingestion): add examples of running in docker and Kubernetes (#4519) 2022-03-29 17:43:49 -07:00
Andres Lowrie
8564d45404
docs(metadata-ingestion): callout props in para (#4485) 2022-03-29 17:43:12 -07:00
mayurinehate
e2723bc981
feat(assertion): update python example, assertion entity doc (#4507) 2022-03-29 13:25:51 -07:00
Kevin Hu
838982abb6
feat(ingestion): detect and disable telemetry in CI (#4513) 2022-03-29 13:21:53 -07:00
Ravindra Lanka
4f7d0f3281
Fix: Snowflake Table to View lineage (#4520) 2022-03-29 09:42:43 -07:00
Aseem Bansal
d30b6e1465
feat(ingest): Add config to improve user exp for initial ingestion and fix docs (#4510) 
* feat(snowflake): change defaults to improve user experience for initial ingestion + documentation
 2022-03-29 07:15:36 -07:00
Aseem Bansal
f36cf69360
feat(ingest): option for number of workunits in preview (#4517) 
* feat(ingest): option for number of workunits in preview + documentation update
 2022-03-29 07:10:22 -07:00
Tamas Nemeth
4358d8fb01
feat(ingest): athena - set Athena location as upstream (#4503) 2022-03-29 07:06:48 -07:00
MugdhaHardikar-GSLab
37aedfc87c
feat(s3): add s3 source (#4490) 
* feat(data-lake): add containers and folder level dataset support

* docs(data-lake): Update readme for data lake

* doc(data-lake): fix examples, update doc

* lint fix

* feat(s3): add s3 source, restore old data-lake source

Co-authored-by: Mayuri N <mayuri.nehate@gslab.com>
 2022-03-29 11:52:57 +02:00
Aseem Bansal
6b04dff913
docs: add example of database and schema allow/deny patterns (#4505) 2022-03-28 13:06:40 +02:00
Aseem Bansal
a702265824
fix(snowflake): allow/deny patterns (#4504) 
* fix(snowflake): allow/deny patterns

* fix lint failures

* fix linting
 2022-03-28 12:48:00 +02:00
Shirshanka Das
a69eac8247
feat(ingest): dbt,looker,sql_common,kafka - moving sources to produce display names and subtypes more consistently (#4496) 2022-03-27 18:49:26 -05:00
Aseem Bansal
ab36ac0f12
feat(snowflake): stop querying for usage data when no mix/max dates (#4480) 2022-03-25 11:02:38 +01:00
Aseem Bansal
7f1a1a3dcf
fix(snowflake-usage): do not ingest for stage as a dataaset (#4483) 2022-03-25 10:59:37 +01:00
Aseem Bansal
1a20f76225
fix(ingestion): pin Jinja2 to version < 3.1.0 (#4489) 2022-03-24 12:26:38 -07:00
Aseem Bansal
f770ed5fea
fix(ingestion): stop CLI build failures (#4484) 2022-03-24 17:03:56 +01:00
Aseem Bansal
9596e73706
doc: add caveats to snowflake doc (#4467) 2022-03-24 16:24:38 +01:00
Aseem Bansal
611cc2ddb5
fix(snowflake): don't recommend accountadmin role for snowflake (#4481) 2022-03-24 07:29:52 -07:00
Aseem Bansal
dcd4af51bb
fix: change log level to debug (#4479) 2022-03-24 07:27:25 -07:00
Xu Wang
d04092e634
feat(ingest): add python utility classes for NotebookUrn, CorpuserUrn and CorpGroupUrn (#4469) 
* feat: add python utility classes for NotebookUrn, CorpuserUrn and CorpGroupUrn

Co-authored-by: Xu Wang <xu.wang@grandrounds.com>
Co-authored-by: Ravindra Lanka <rslanka@gmail.com>
 2022-03-23 16:07:57 -07:00
Kevin Hu
a7b0275b86
feat(ingest): simplify event IDs for function invocations (#4398) 
* Simplify function call events
Co-authored-by: Ravindra Lanka <rslanka@gmail.com>

Co-authored-by: Ravindra Lanka <rslanka@gmail.com>
 2022-03-23 12:52:29 -07:00
Sergio Gómez Villamor
9fbb521bfe
chore: acryl-data 0.6.12 (#4474) 2022-03-23 10:24:48 -07:00
Tamas Nemeth
5c8017789a
fix(redshift) Properly handling database alias in redshift usage and redshift lineage generation (#4473) 
* Fix database-alias in redshift usage and redshift lineage generation
 2022-03-23 16:01:14 +01:00
Tamas Nemeth
2013d5ddbd
feat(ingest) data-lake: Add s3 properties metadata when ingesting s3 files (#4453) 
* Add s3 porperties for data lake ingestion
 2022-03-22 10:31:18 -07:00
Aseem Bansal
15d5e418fb
fix(snowflake-usage): add more error handling (#4466) 2022-03-22 08:10:03 -07:00
Fernanda de Camargo
e290e6e07e
fix(ingest): add fix to tableau connector when table has None fields (#4445) 
Co-authored-by: Ludmila Ferreira <ludmila.ferreira@elo7.com>
 2022-03-22 14:59:18 +01:00
mayurinehate
885cf26828
docs(hive): update recipe with example to specify kerberos auth (#4457) 2022-03-22 13:38:21 +01:00
pedro-iatzky
6a6d744667
fix(ingest): bigquery - fix ingestion of external tables (#4313) 2022-03-22 13:35:41 +01:00