yujunjun/datahub
2
0
Fork 0
mirror of https://github.com/datahub-project/datahub.git synced 2025-08-11 10:46:52 +00:00
Code Issues Packages Projects Releases Wiki Activity
11,753 Commits 228 Branches 123 Tags
Commit Graph

3904 Commits

Author SHA1 Message Date
Andrew Sikowitz
668a09ffd4
fix(ingest/bigquery): Deduplicate LogEntries based on insertId, timestamp, logName (#8132) 2023-05-25 17:53:47 -07:00
Harshal Sheth
2d442161c4
ci(ingest/kafka): improve kafka integration test reliability (#8085) 2023-05-25 15:40:56 -07:00
Tamas Nemeth
b216caa83b
fix(ingest/redshift): Fixing Redshift subtypes (#8125) 2023-05-25 19:45:07 +02:00
Vinícius Mello
7059874dec
feat(ingest/bigquery): Add BigQuery Views lineage extraction from Google Data Catalog API (#8100) 2023-05-25 08:37:46 -07:00
Andrew Sikowitz
d3cd4dbb0c
feat(ingest/unity): Allow ingestion without metastore admin role (#8091) 
- Adds more detailed docs and connection test
- Fixes empty username queries
 2023-05-24 15:36:22 -07:00
Tamas Nemeth
fb087c5e35
fix(ingest/redshift):fixing schema filter (#8119) 2023-05-24 22:35:28 +00:00
Mayuri Nehate
84270bcac8
feat(ingest/nifi): kerberos authentication (#8097) 
Co-authored-by: david-leifker <114954101+david-leifker@users.noreply.github.com>
Co-authored-by: Andrew Sikowitz <andrew.sikowitz@acryl.io>
Co-authored-by: Indy Prentice <iprentic@users.noreply.github.com>
 2023-05-24 15:09:01 -07:00
Andrew Sikowitz
fdbc4de695
refactor(ingest): Call source_helpers via new WorkUnitProcessors in base Source (#8101) 2023-05-24 13:36:19 -07:00
Amanda Hernando
0e0d8934ea
feat(ingest): Add GenericAspectTransformer (#7994) 
Co-authored-by: Adrián Pertíñez <khurzak92@gmail.com>
 2023-05-24 13:31:33 -07:00
Mayuri Nehate
b3d80e57e8
feat(ingest/bigquery): usage for views (#8046) 
Co-authored-by: Tamas Nemeth <treff7es@gmail.com>
Co-authored-by: Andrew Sikowitz <andrew.sikowitz@acryl.io>
 2023-05-24 09:48:58 -07:00
Mayuri Nehate
798ce3d6c8
feat(classification): configurable sample size (#8096) 
Co-authored-by: david-leifker <114954101+david-leifker@users.noreply.github.com>
 2023-05-24 00:07:01 -07:00
Andrew Sikowitz
8357fc8d64
feat(ingest): Browse Path v2 helper (#8012) 2023-05-23 23:46:46 -07:00
Tamas Nemeth
287a292fe7
fix(ingest/redshift): Making Redshift source more verbose (#8109) 2023-05-24 08:19:37 +02:00
Harshal Sheth
b0f8c3de1e
refactor(ingest): simplify stateful ingestion provider interface (#8104) 
Co-authored-by: Tamas Nemeth <treff7es@gmail.com>
 2023-05-23 12:57:57 -07:00
Harshal Sheth
afd65e16fb
feat(cli): delete cli v2 (#8068) 2023-05-23 14:43:44 -05:00
Indy Prentice
0612c70a63
fix(sample data): Update timestamps in bootstrap_mce.json to more recent (#8103) 
Co-authored-by: Indy Prentice <indy@Indys-MBP.hsd1.ca.comcast.net>
 2023-05-23 14:09:27 -05:00
Harshal Sheth
4873a32e4a
fix(ingest): emitter bug fixes (#8093) 
Co-authored-by: Tamas Nemeth <treff7es@gmail.com>
 2023-05-23 12:04:16 -07:00
Tamas Nemeth
4ca7a9b50e
fix(ingest/build): setting typing extension <4.6.0 because it breaks tests (#8108) 2023-05-23 18:55:28 +05:30
Tamas Nemeth
f8be9f6aee
feat(ingest/s3): type aware directory sorting (#8089) 2023-05-23 08:59:46 +02:00
Harshal Sheth
fb9a35b58a
feat(cli): support datahub ingest mcps (#7871) 2023-05-22 17:48:51 -07:00
Andrew Sikowitz
a43903bf6d
refactor(ingest): Auto report workunits (#8061) 2023-05-22 17:06:31 -07:00
Harshal Sheth
4e9c652707
feat(ingest): add env to container properties (#8027) 2023-05-22 12:07:16 -07:00
Shubham Jagtap
e6371c8e94
fix(ingestion/powerbi): skip erroneous pages of a report (#8021) 
Co-authored-by: mohdsiddique <mohdsiddiquebagwan@gmail.com>
Co-authored-by: MohdSiddiqueBagwan <mohdsiddique.bagwan@gslab.com>
 2023-05-19 18:02:55 -07:00
Andrew Sikowitz
2ca362e96c
fix(ingest/delta-lake): Walk through directory structure with full path; reduce resource creation (#8072) 2023-05-19 12:42:48 -07:00
Harshal Sheth
d79b28c147
feat(sdk): better auth error messages in the rest emitter (#8025) 2023-05-19 16:16:03 +02:00
Harshal Sheth
98bba52c20
test(sdk): move cli tests into the unit dir (#8028) 2023-05-19 16:13:39 +02:00
alplatonov
999b997223
fix(ingest/s3_data_lake)_ingestor_skips_directories_with_similar_prefix (#8078) 
Co-authored-by: Tamas Nemeth <treff7es@gmail.com>
 2023-05-19 15:32:19 +02:00
Harshal Sheth
bd5f99b6ab
docs(ingest/snowflake): fix authentication type docs (#8059) 2023-05-19 13:19:10 +02:00
Tamas Nemeth
bdd4bc7b92
feat(ingest/s3) - Stateful ingestion and last-updated support (#8022) 2023-05-19 13:10:15 +02:00
Harshal Sheth
00470acc02
test(sdk): better error messages in registry codegen test (#8081) 2023-05-19 11:18:50 +02:00
Harshal Sheth
1902e7d4db
ci(ingest/clickhouse): don't use kernel ephemeral ports (#8060) 2023-05-19 11:17:41 +02:00
Hyejin Yoon
a8b622a016
feat: add docs on column-level linage (#8062) 2023-05-19 07:59:30 +09:00
Harshal Sheth
48b856ccf4
fix(ingest/looker): don't prematurely show connectivity success (#8070) 2023-05-17 22:27:56 -07:00
Mayuri Nehate
43bb84738b
fix(ingest/tableau): don't use unsupported sql condition field (#8065) 
Co-authored-by: Harshal Sheth <hsheth2@gmail.com>
 2023-05-17 22:24:53 -07:00
matwalk
f2683203ab
fix(ingest/snowflake): lowercase user urn when using email (#7767) 
Co-authored-by: mathilde.oustlant <mathilde.oustlant@cdiscount.com>
 2023-05-17 22:09:36 -07:00
Andrew Sikowitz
2e1c3981aa
refactor(ingest): Move source_helpers.py from datahub/utilities -> datahub/api (#8052) 2023-05-17 20:51:06 -07:00
mohdsiddique
ae30be9c25
fix(ingestion/tableau): ingest parent project name in container properties (#8030) 
Co-authored-by: MohdSiddiqueBagwan <mohdsiddique.bagwan@gslab.com>
 2023-05-17 14:19:41 -07:00
Shubham Jagtap
8cc6606e68
feat(ingestion/kafka): add description in dataset properties (#7974) 
Co-authored-by: Harshal Sheth <hsheth2@gmail.com>
Co-authored-by: MohdSiddiqueBagwan <mohdsiddique.bagwan@gslab.com>
Co-authored-by: mohdsiddique <mohdsiddiquebagwan@gmail.com>
 2023-05-17 11:03:08 -07:00
Shirshanka Das
151eab3628
fix(build): fix lint issue (#8066) 2023-05-17 08:17:52 -07:00
Shirshanka Das
b3c790aab6
feat: Add support for Data Products (#8039) 
Co-authored-by: Chris Collins <chriscollins3456@gmail.com>
 2023-05-17 07:17:25 +00:00
Mayuri Nehate
20927e84c9
refractor(classification): simplify classification handler (#8056) 2023-05-16 23:27:02 -07:00
Andrew Sikowitz
7ba2d13087
refactor(ingest): Make get_workunits() return MetadataWorkUnits (#8051) 
- Deprecates UsageAggregationClass, /usageStats?action=batchIngest, UsageStatsWorkUnit
- Removes parsing of UsageAggregationClass in file source, all sinks, and WorkUnitRecordExtractor
 2023-05-17 00:01:57 -04:00
Hyejin Yoon
25450ac82c
feat(docs): add guide on integration ML system via SDKs (#8029) 
Co-authored-by: socar-dini <dini@socar.kr>
 2023-05-17 10:21:39 +09:00
Mayuri Nehate
a06c5aee2c
fix(ingest/bigquery): update usage audit log query to include create/drop operations (#7995) 2023-05-16 11:58:20 -07:00
Harshal Sheth
e70d0b3859
fix(ingest/dbt): fix dbt subtypes for sources (#8048) 2023-05-16 11:11:26 -07:00
xiphl
d619cc6b3f
feat(ingest): Allow csv-enricher to update more types (#7932) 
Co-authored-by: xiphl <xiphlerl9@gmail.com>
 2023-05-15 10:38:19 -07:00
Shubham Jagtap
7483d9a4de
fix(ingestion/metabase): metabase connector bigquery lineage fix (#8042) 
Co-authored-by: mohdsiddique <mohdsiddiquebagwan@gmail.com>
 2023-05-15 14:30:20 +02:00
Harshal Sheth
92af7152c0
feat(ingest): support importing local modules (#8026) 2023-05-13 13:55:12 +05:30
cccs-Dustin
87d32d7377
feat(ingest/superset): add stateful ingestion (#8013) 2023-05-11 21:56:05 -07:00
Harshal Sheth
9a2e990bed
fix(sdk): throw errors on empty gms server urls (#8017) 2023-05-11 21:42:22 -07:00