yujunjun/datahub
2
0
Fork 0
mirror of https://github.com/datahub-project/datahub.git synced 2025-07-06 16:49:03 +00:00
Code Issues Packages Projects Releases Wiki Activity
11,753 Commits 189 Branches 118 Tags
Commit Graph

1604 Commits

Author SHA1 Message Date
Mayuri Nehate
b3d80e57e8
feat(ingest/bigquery): usage for views (#8046) 
Co-authored-by: Tamas Nemeth <treff7es@gmail.com>
Co-authored-by: Andrew Sikowitz <andrew.sikowitz@acryl.io>
 2023-05-24 09:48:58 -07:00
Andrew Sikowitz
8357fc8d64
feat(ingest): Browse Path v2 helper (#8012) 2023-05-23 23:46:46 -07:00
Harshal Sheth
b0f8c3de1e
refactor(ingest): simplify stateful ingestion provider interface (#8104) 
Co-authored-by: Tamas Nemeth <treff7es@gmail.com>
 2023-05-23 12:57:57 -07:00
Harshal Sheth
afd65e16fb
feat(cli): delete cli v2 (#8068) 2023-05-23 14:43:44 -05:00
Harshal Sheth
4873a32e4a
fix(ingest): emitter bug fixes (#8093) 
Co-authored-by: Tamas Nemeth <treff7es@gmail.com>
 2023-05-23 12:04:16 -07:00
Tamas Nemeth
f8be9f6aee
feat(ingest/s3): type aware directory sorting (#8089) 2023-05-23 08:59:46 +02:00
Andrew Sikowitz
a43903bf6d
refactor(ingest): Auto report workunits (#8061) 2023-05-22 17:06:31 -07:00
Harshal Sheth
4e9c652707
feat(ingest): add env to container properties (#8027) 2023-05-22 12:07:16 -07:00
Shubham Jagtap
e6371c8e94
fix(ingestion/powerbi): skip erroneous pages of a report (#8021) 
Co-authored-by: mohdsiddique <mohdsiddiquebagwan@gmail.com>
Co-authored-by: MohdSiddiqueBagwan <mohdsiddique.bagwan@gslab.com>
 2023-05-19 18:02:55 -07:00
Harshal Sheth
98bba52c20
test(sdk): move cli tests into the unit dir (#8028) 2023-05-19 16:13:39 +02:00
Tamas Nemeth
bdd4bc7b92
feat(ingest/s3) - Stateful ingestion and last-updated support (#8022) 2023-05-19 13:10:15 +02:00
Harshal Sheth
00470acc02
test(sdk): better error messages in registry codegen test (#8081) 2023-05-19 11:18:50 +02:00
Harshal Sheth
1902e7d4db
ci(ingest/clickhouse): don't use kernel ephemeral ports (#8060) 2023-05-19 11:17:41 +02:00
Andrew Sikowitz
2e1c3981aa
refactor(ingest): Move source_helpers.py from datahub/utilities -> datahub/api (#8052) 2023-05-17 20:51:06 -07:00
mohdsiddique
ae30be9c25
fix(ingestion/tableau): ingest parent project name in container properties (#8030) 
Co-authored-by: MohdSiddiqueBagwan <mohdsiddique.bagwan@gslab.com>
 2023-05-17 14:19:41 -07:00
Shubham Jagtap
8cc6606e68
feat(ingestion/kafka): add description in dataset properties (#7974) 
Co-authored-by: Harshal Sheth <hsheth2@gmail.com>
Co-authored-by: MohdSiddiqueBagwan <mohdsiddique.bagwan@gslab.com>
Co-authored-by: mohdsiddique <mohdsiddiquebagwan@gmail.com>
 2023-05-17 11:03:08 -07:00
Shirshanka Das
151eab3628
fix(build): fix lint issue (#8066) 2023-05-17 08:17:52 -07:00
Shirshanka Das
b3c790aab6
feat: Add support for Data Products (#8039) 
Co-authored-by: Chris Collins <chriscollins3456@gmail.com>
 2023-05-17 07:17:25 +00:00
Andrew Sikowitz
7ba2d13087
refactor(ingest): Make get_workunits() return MetadataWorkUnits (#8051) 
- Deprecates UsageAggregationClass, /usageStats?action=batchIngest, UsageStatsWorkUnit
- Removes parsing of UsageAggregationClass in file source, all sinks, and WorkUnitRecordExtractor
 2023-05-17 00:01:57 -04:00
Mayuri Nehate
a06c5aee2c
fix(ingest/bigquery): update usage audit log query to include create/drop operations (#7995) 2023-05-16 11:58:20 -07:00
Harshal Sheth
e70d0b3859
fix(ingest/dbt): fix dbt subtypes for sources (#8048) 2023-05-16 11:11:26 -07:00
xiphl
d619cc6b3f
feat(ingest): Allow csv-enricher to update more types (#7932) 
Co-authored-by: xiphl <xiphlerl9@gmail.com>
 2023-05-15 10:38:19 -07:00
Shubham Jagtap
7483d9a4de
fix(ingestion/metabase): metabase connector bigquery lineage fix (#8042) 
Co-authored-by: mohdsiddique <mohdsiddiquebagwan@gmail.com>
 2023-05-15 14:30:20 +02:00
cccs-Dustin
87d32d7377
feat(ingest/superset): add stateful ingestion (#8013) 2023-05-11 21:56:05 -07:00
Andrew Sikowitz
afcf462cb1
feat(ingest/unity): Add profiling support (#7976) 
- Also adds a new databricks sdk
 2023-05-11 10:00:50 -07:00
Tamas Nemeth
dec54bf098
feat(ingest/s3): Inferring schema from the alphabetically last folder (#8005) 2023-05-10 21:55:05 +02:00
Andrew Sikowitz
44406f7adf
fix(ingest/postgres): Allow specification of initial engine database; set default database to postgres (#7915) 
Co-authored-by: Mayuri Nehate <33225191+mayurinehate@users.noreply.github.com>
 2023-05-09 11:11:43 -07:00
Mayuri Nehate
c845c75a2d
feat(ingest/snowflake): add config option to specify deny patterns for upstreams (#7962) 
Co-authored-by: Andrew Sikowitz <andrew.sikowitz@acryl.io>
 2023-05-08 14:13:57 -07:00
Mayuri Nehate
13b1d66170
fix(ingest/bigquery): remove incorrectly used table_pattern filter (#7810) 
Co-authored-by: Andrew Sikowitz <andrew.sikowitz@acryl.io>
 2023-05-08 10:33:42 -07:00
Mayuri Nehate
0131aeefb1
fix(ingest/unity): improve error message if no scheme in workspace_url (#7951) 
Co-authored-by: John Joyce <john@acryl.io>
 2023-05-08 10:13:53 -07:00
Tamas Nemeth
0e69e5a810
fix(ingest/redshift): Enabling autocommit for Redshift connection (#7983) 2023-05-08 10:24:40 +02:00
Andrew Sikowitz
8019d17aa6
fix(ingest/bigquery): Filter projects for lineage and usage (#7954) 2023-05-04 18:14:48 +02:00
Harshal Sheth
ca5dffa54d
refactor(ingest/biz-glossary): simplify business glossary source (#7912) 2023-05-03 17:01:58 -07:00
Reilman79
b6e2cc549a
fix(ldap): properly handle escaped characters in LDAP DNs (#7928) 2023-05-03 13:57:52 -07:00
Felipe Ribeiro
d504cbd1b6
docs(ingest): update max_threads default value (#7947) 
Co-authored-by: Felipe Ribeiro <fribeiro@fanatics.com>
 2023-05-02 22:54:15 -07:00
Mayuri Nehate
a711baa131
fix(ingest/hive): fix containers generation for hive (#7926) 2023-05-02 15:07:51 +02:00
Andrew Sikowitz
5b290c9bc5
feat(ingest/unity): Add usage extraction; add TableReference (#7910) 
- Adds usage extraction to the unity catalog source and a TableReference object to handle references to tables
Also makes the following refactors:
- Creates UsageAggregator class to usage_common, as I've seen this same logic multiple times.
- Allows customizable user_urn_builder in usage_common as not all unity users are emails. We create emails with a default email_domain config in other connectors like redshift and snowflake, which seems unnecessary now?
- Creates TableReference for unity catalog and adds it to the Table dataclass, for managing string references to tables. Replaces logic, especially in lineage extraction, with these references
- Creates gen_dataset_urn and gen_user_urn on unity source to reduce duplicate code
Breaks up proxy.py into implementation and types
 2023-05-01 11:30:09 -07:00
Mayuri Nehate
a0c4e0dd46
feat(ingest): add GCS ingestion source (#7903) 
Co-authored-by: Tamas Nemeth <treff7es@gmail.com>
 2023-04-27 19:03:41 +02:00
Harshal Sheth
916cb21454
test(ingest/biz-glossary): add test for enable_auto_id (#7911) 2023-04-26 19:48:52 -07:00
Mayuri Nehate
031aee4298
fix(ingest/bigquery): fix handling of time decorator offset queries (#7843) 
Co-authored-by: Andrew Sikowitz <andrew.sikowitz@acryl.io>
 2023-04-25 13:51:20 -07:00
Harshal Sheth
19d7c392d6
feat(sdk): support entity types filter in get_urns_by_filter (#7902) 2023-04-25 13:31:55 -07:00
Harshal Sheth
71ecbd6060
fix(ingest/dbt): ensure dbt shows view properties (#7872) 2023-04-25 12:25:07 -07:00
Mayuri Nehate
28986d8081
fix(ingestion/tableau): backward compatibility with version 2021.1 and above (#7864) 
Co-authored-by: Harshal Sheth <hsheth2@gmail.com>
 2023-04-24 11:08:56 -07:00
Mayuri Nehate
3212e74969
feat(ingest/snowflake): optionally emit all upstreams irrespective of recipe pattern (#7842) 2023-04-24 11:01:15 -07:00
Andrew Sikowitz
e9c2f9afcc
feat(ingest/unity): Ingest ownership for containers; lookup service principal display names (#7869) 2023-04-21 11:02:39 -07:00
mohdsiddique
f21eeed6e7
feat(ingestion): lookml refinement support (#7781) 
Co-authored-by: MohdSiddiqueBagwan <mohdsiddique.bagwan@gslab.com>
Co-authored-by: Harshal Sheth <hsheth2@gmail.com>
 2023-04-21 10:55:31 -07:00
Yusuf Mahtab
fa10256c47
feat(glue): allow resource links to be ignored (#7639) 
Co-authored-by: Justas Cernas <justas.cernas@fundingcircle.com>
 2023-04-21 10:42:32 -07:00
Aezo
1a5c716b87
feat(ingest/powerbi): support modified_since, extract_dataset_schema and many more (#7519) 
Co-authored-by: Harshal Sheth <hsheth2@gmail.com>
 2023-04-20 22:58:45 -07:00
Harshal Sheth
6802142f6e
fix(ingest/salesforce): use report timestamp for operations (#7838) 
Co-authored-by: John Joyce <john@acryl.io>
 2023-04-19 20:39:07 -07:00
Harshal Sheth
e461d03d94
feat(ingest/unity): capture create/lastModified timestamps (#7819) 
Co-authored-by: Tamas Nemeth <treff7es@gmail.com>
 2023-04-17 12:18:21 -07:00