Jonny Dixon
62557841bc
test fixes
2025-09-13 11:39:03 +01:00
Jonny Dixon
0ba70347b8
adding test coverage
2025-09-12 23:28:55 +01:00
Jonny Dixon
fb8e3d9f0a
Merge origin/master into fivetran-std-edition-support
...
- Resolved conflict in config.py by removing KNOWN_DATA_PLATFORM_MAPPING
- Our dynamic service mapping approach is more robust and maintainable
- All tests pass, linting clean
2025-09-12 19:45:36 +01:00
Jonny Dixon
203e3126a6
Update fivetran.py
2025-09-12 19:42:40 +01:00
Jonny Dixon
e5591e8865
better formatting
2025-09-12 19:28:58 +01:00
Michael Maltese
10649f3f38
feat(ingest/fivetran): map google_cloud_postgresql => postgres ( #14742 )
2025-09-12 17:12:10 +00:00
dependabot[bot]
becfe19fee
chore(deps): bump actions/setup-python from 5 to 6 ( #14701 )
...
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: david-leifker <114954101+david-leifker@users.noreply.github.com>
2025-09-12 12:11:28 -05:00
Jonny Dixon
526f23be28
Merge origin/master into fivetran-std-edition-support
...
Resolved merge conflicts by:
- Keeping improved dynamic service mapping over hardcoded platform mappings
- Maintaining enhanced parallel processing and error handling
- Preserving comprehensive API validation and accessibility checks
- Using updated DataFlow/DataJob constructors with better parameter naming
- Keeping improved golden test expectations that reflect our enhancements
- Cleaned up unused imports (StructuredLogCategory, CorpUserUrn)
All 49 unit tests passing with comprehensive Fivetran connector improvements.
2025-09-12 17:42:27 +01:00
Jonny Dixon
b4b1d10590
feat: Improve Fivetran connector with dynamic service mapping and enhanced parallelization
...
- Remove unsupported platform mappings to only include DataHub-supported platforms
- Add dynamic service mapping using Fivetran API service field instead of hardcoded aliases
- Enhance parallel processing for connector and table-level operations
- Add connector accessibility validation with graceful error handling
- Improve column retrieval with better error handling and fallback mechanisms
- Add comprehensive API endpoint validation
- Update documentation with performance optimization details
- All 49 unit tests passing with comprehensive coverage
2025-09-12 17:30:38 +01:00
dependabot[bot]
d3e8139d53
chore(deps-dev): bump vite from 6.3.5 to 6.3.6 in /datahub-web-react ( #14720 )
...
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2025-09-12 16:24:33 +00:00
dependabot[bot]
46ac6c428a
chore(deps): bump aquasecurity/trivy-action from 0.33.0 to 0.33.1 ( #14699 )
...
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: david-leifker <114954101+david-leifker@users.noreply.github.com>
2025-09-12 16:04:49 +00:00
david-leifker
ef5a5b4fa2
fix(ci): cloudflare workflow cannot run without token ( #14749 )
2025-09-12 10:58:57 -05:00
dependabot[bot]
8fc449b45f
chore(deps): bump actions/stale from 9 to 10 ( #14697 )
...
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2025-09-12 10:52:50 -05:00
dependabot[bot]
17ec667af1
chore(deps): bump aws-actions/configure-aws-credentials from 4 to 5 ( #14698 )
...
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2025-09-12 10:52:40 -05:00
Chris Collins
87c2d14ea2
fix(ui) Improve flakiness of modules and templates cypress tests ( #14748 )
2025-09-12 11:51:44 -04:00
dependabot[bot]
e62719b207
chore(deps): bump actions/download-artifact from 4 to 5 ( #14700 )
...
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2025-09-12 10:51:34 -05:00
Jonny Dixon
6471c32457
feat(fivetran): Add parallel processing and optimize Standard API performance
...
- Add ThreadPoolExecutor for parallel column retrieval and connector processing
- Add max_workers configuration parameter for controlling parallelization
- Improve error handling for 400 Bad Request responses in column retrieval
- Fix per-table mode DPI generation to avoid excessive duplication
- Improve Fivetran edition auto-detection to prefer enterprise mode
- Fix GraphQL operation parameter naming issue
Performance improvements:
- Column retrieval now parallelized within batches
- Connector lineage extraction now parallelized across connectors
- Reduced unnecessary retries for 400 errors (permanent failures)
- Better error logging and handling
This significantly speeds up Standard API ingestion while maintaining reliability.
2025-09-12 15:02:37 +01:00
Abe
dede42d933
docs(dev-setup): Update IntelliJ setup instructions ( #14718 )
2025-09-11 23:52:13 +00:00
david-leifker
c3283ef314
docs(platform-instance): documentation refresh ( #14739 )
2025-09-11 20:17:06 +00:00
david-leifker
030b4ace93
chore(bump): bump msk-iam-auth ( #14744 )
2025-09-11 13:57:50 -05:00
david-leifker
f34abede15
fix(trivy): fix pattern ( #14743 )
2025-09-11 13:32:14 -05:00
Jonny Dixon
57250477bf
feat(access-request): enable groups to be granted to role and grey button when granted ( #14622 )
2025-09-11 19:15:01 +01:00
Jay
4ce1ae8dd1
feat(docs): revise tuning instructions for smart assertions ( #14740 )
2025-09-11 13:42:42 -04:00
Chakru
ac80e8171b
fix(kafka-setup): remove default to enable topicDefaults to be used ( #14738 )
2025-09-11 16:38:11 +00:00
skrydal
3af386b626
chore(ingestion/iceberg): Safe-guard pyiceberg version before pydantic 1->2 transition ( #14736 )
2025-09-11 16:32:33 +00:00
Harshal Sheth
14130701b5
fix(ingest): use sequence for sdk input types ( #14695 )
...
Co-authored-by: Anush Kumar <anush.kumar@datahub.com>
2025-09-11 08:49:09 -07:00
Brock Griffey
4244620e7a
feat(cassandra): Add optional SSL configuration ( #14726 )
2025-09-11 15:37:30 +00:00
Tamas Nemeth
01932d3f87
fix(ingest/pipeline): Fix for slow ingestion and incomplete ingestion report metrics ( #14735 )
2025-09-11 16:07:47 +02:00
Jonny Dixon
d0519ddce3
fix(docs): fixing spelling mistakes ( #14730 )
2025-09-11 07:59:11 +00:00
Aseem Bansal
137ffb7d48
fix(ingest): only add to samples where platform match ( #14722 )
2025-09-11 13:26:39 +05:30
skrydal
5f23652fd3
fix(ingestion/iceberg): Improve iceberg source resiliency to server errors ( #14731 )
2025-09-11 00:57:03 +02:00
skrydal
3fbef4a632
chore(setup): Limit mixpanel dependency <=4.10.1 ( #14725 )
2025-09-10 18:56:10 +02:00
Jay
d67681b08f
feat(web): extended incidents graphql apis to return more context ( #14671 )
2025-09-10 12:21:12 -04:00
Tamas Nemeth
a82d4e0647
fix(ingest/athena): Fix Athena partition extraction and CONCAT function type issues ( #14712 )
2025-09-10 12:33:54 +02:00
Saketh Varma
c7ad3f45ea
feat(ui): Add option to remove asset from an Application ( #14679 )
2025-09-09 16:17:21 -07:00
Abe
c731e725ff
docs(metrics): Add a best practices guide for micrometer metrics library ( #14711 )
2025-09-09 18:22:12 +00:00
Abe
6ec6f0150d
refactor(metrics): Make MetricUtils.registry non-nullable
...
- Make MetricUtils.registry non-nullable with default no-op implementation. This allows us to remove boilerplate for handling the null case
- Rename request context metric names to follow convention
2025-09-09 11:08:16 -07:00
Tamas Nemeth
4ea758da19
chore(ingest/sqlparser): Bump sqlglot to 27.12.0 ( #14673 )
2025-09-09 19:57:52 +02:00
Michael Maltese
3d464941d0
fix(ingest/tableau): retry when getting 'unexpected error occurred' ( #14672 )
2025-09-09 10:25:49 -04:00
Aseem Bansal
5ce93f54b4
feat(ingest): add time taken by compute stats overall ( #14713 )
2025-09-09 14:53:13 +05:30
david-leifker
aee04b5680
feat(build): improve failure logging ( #14696 )
2025-09-08 16:57:20 -05:00
Kevin Karch
6a3f31b093
fix(docs): correct redshift usage stats capability ( #14706 )
2025-09-08 14:40:45 -04:00
Aseem Bansal
3e1e9fcf34
fix(ui): add warning for view all modal ( #14702 )
2025-09-08 19:27:41 +05:30
Benjamin Maquet
9105241bfd
feat(superset/preset): add dataset and column description ( #14426 )
2025-09-08 16:35:43 +09:00
Hyejin Yoon
da1326d047
fix(ui): Improve Model/Model Group description handling (truncate + editableProperties) ( #14595 )
2025-09-08 07:25:00 +00:00
Jonny Dixon
f8a95b0ade
chore(ingestion/file-based lineage): update from com.linkedin.pegasus2avro.dataset to datahub.metadata.schema_classes ( #14690 )
2025-09-08 09:20:07 +02:00
Jesse Jia
55817d14b8
feat(openapi-v3) enhance scroll API ( #14677 )
...
Co-authored-by: jjia <jjia@netflix.com>
Co-authored-by: david-leifker <114954101+david-leifker@users.noreply.github.com>
2025-09-06 14:57:01 +00:00
Abe
80f206290e
feat(metrics): add modern Micrometer metrics PFP-1613 ( #14661 )
...
- Add micrometer metrics in RequestContext
- Add a smoke test that verifies metrics via the prometheus endpoint
2025-09-05 14:53:54 -07:00
david-leifker
8b194cdd28
fix(ci): adjust trivy configuration for pyspark cache ( #14692 )
2025-09-05 14:21:01 -05:00
david-leifker
fa4c866d0b
chore(): bump netty version ( #14691 )
2025-09-05 13:03:06 -05:00
