264 Commits

Author SHA1 Message Date
Harshal Sheth
e70d0b3859
fix(ingest/dbt): fix dbt subtypes for sources (#8048) 2023-05-16 11:11:26 -07:00
david-leifker
cd05f5b174
feat(schema-registry): replace confluent schema registry (#7930)
Co-authored-by: Pedro Silva <pedro@acryl.io>
Co-authored-by: Shirshanka Das <shirshanka@apache.org>
Co-authored-by: Ryan Holstien <ryan@acryl.io>
2023-05-01 13:18:41 -05:00
david-leifker
87aa792f22
fix(consumer): fix datahub usage event topic consumer (#7866) 2023-05-01 09:28:11 -05:00
david-leifker
2ffa39f009
Logging and test models fixes (#7884) 2023-04-21 15:45:20 -05:00
RyanHolstien
cbf0809c4c
chore(snakeyaml): upgrade to snakeyaml 2 (#7786)
Co-authored-by: david-leifker <114954101+david-leifker@users.noreply.github.com>
2023-04-13 12:01:51 -05:00
david-leifker
82e55636a7
feat(search): allow longer customProperties (#7804) 2023-04-12 16:59:11 -05:00
RyanHolstien
8ffb6a3585
feat(config): allow hooks to be enabled/disabled (#7761) 2023-04-07 11:10:05 -05:00
david-leifker
1b15dd484f
lint(test): remove unused imports, other test fixes (#7659) 2023-03-21 16:53:04 -05:00
RyanHolstien
8963e53bbd
fix(revert): remove unnecessary class check (#7658) 2023-03-21 12:53:39 -05:00
Shirshanka Das
fc9ad49821
fix(platform): Ensure time based lineage handles noop changes (#7657)
Co-authored-by: aditya-radhakrishnan <aditya@acryl.io>
Co-authored-by: RyanHolstien <RyanHolstien@users.noreply.github.com>
2023-03-21 10:43:56 -05:00
Bowen Hu
a84ec0ccf8
feat(metadata-io): add support in Neo4jGraphService for lineage time filter (#7375) 2023-03-16 15:54:49 -07:00
david-leifker
72198f98c0
feat(search): searchAcrossEntities/Lineage improvements (#7550)
Co-authored-by: Shirshanka Das <shirshanka@apache.org>
2023-03-12 18:18:51 -07:00
John Joyce
d54e3b81cf
fix(lineage): Fixing Timeline Lineage Filters (#7435) 2023-02-27 16:00:57 -08:00
Pedro Silva
4f8a5c3800
feat(timeBasedLineage): add feature flag for always producing MCL (#7407) 2023-02-22 15:52:51 +00:00
david-leifker
751289c9e3
fix(gradle): fix gradle command referenced in docs (#7318) 2023-02-16 18:27:53 -06:00
david-leifker
267472ccbb
gradle(development): add additional commands for development (#7321) 2023-02-16 10:41:42 -06:00
david-leifker
b4b3a39829
fix(system-update): fixes system-update with more than 1 partition (#7302) 2023-02-09 18:54:20 -06:00
Aseem Bansal
0f8f186bd0
chore(logs): clean logs scanning location (#7261) 2023-02-07 16:41:39 +05:30
Aditya Radhakrishnan
c13e7e09f8
feat(lineage): add time filtering for lineage edges (#7159) 2023-02-02 15:30:49 -08:00
david-leifker
39920bb00f
feat(elasticsearch): Elasticsearch improvements (#6894) 2023-01-31 18:44:37 -06:00
RyanHolstien
af3fa6f22d
fix(diffMode): prevent misconfiguration of diff mode (#7127) 2023-01-24 15:15:41 -08:00
Aseem Bansal
192aac4271
chore: misc fixes (#6966) 2023-01-09 14:12:36 +05:30
david-leifker
122c024bab
refactor(gradle): move the local docker registry to common location (#6881) 2022-12-29 15:08:36 +05:30
david-leifker
b5275acab8
test(mce-consumer): mockbeans (#6878) 2022-12-28 16:18:10 -06:00
david-leifker
cb6314c51d
fix(elasticsearch): fixes out of order runId writes (#6845)
Co-authored-by: leifker <dleifker@gmail.com>
Co-authored-by: Pedro Silva <pedro@acryl.io>
2022-12-27 17:50:11 -06:00
david-leifker
ecc01b9a46
refactor(restli-mce-consumer) (#6744)
* fix(security): commons-text in frontend

* refactor(restli): set threads based on cpu cores
feat(mce-consumers): hit local restli endpoint

* testing docker build

* Add retry configuration options for entity client

* Kafka debugging

* fix(kafka-setup): parallelize topic creation

* Adjust docker build

* Docker build updates

* WIP

* fix(lint): metadata-ingestion lint

* fix(gradle-docker): fix docker frontend dep

* fix(elastic): fix race condition between gms and mae for index creation

* Revert "fix(elastic): fix race condition between gms and mae for index creation"

This reverts commit 9629d12c3bdb3c0dab87604d409ca4c642c9c6d3.

* fix(test): fix datahub frontend test for clean/test cycle

* fix(test): datahub-frontend missing assets in test

* fix(security): set protobuf lib datahub-upgrade & mce/mae-consumer

* gitingore update

* fix(docker): remove platform on docker base image, set by buildx

* refactor(kafka-producer): update kafka producer tracking/logging

* updates per PR feedback

* Add documentation around mce standalone consumer
Kafka consumer concurrency to follow thread count for restli & sql connection pool

Co-authored-by: leifker <dleifker@gmail.com>
Co-authored-by: Pedro Silva <pedro@acryl.io>
2022-12-26 16:09:08 +00:00
Chris Collins
f559dcc282
feat(lineage) Adds ability to edit lineage manually from the UI (#6816) 2022-12-20 10:54:08 -05:00
david-leifker
be9348b518
fix(build): rename conflicting module auth-api (#6728)
* fix(build): rename conflicting module `auth-api`
2022-12-09 18:06:51 -06:00
Chris Collins
b6887d23bb
feat(graph) Add createdOn, createdActor, updatedOn, updatedActor to graph edges (#6615) 2022-12-05 16:56:28 -05:00
david-leifker
2de9d3d5bf
fix(logging): Remove lombok as source of slf4j-api, convert to compileOnly where possible (#6616) 2022-12-04 19:57:47 -08:00
Maciej Bryński
10deee7333
docs(): remove Kafka Streams from documentation (#6596) 2022-12-01 15:24:25 -08:00
John Joyce
7136dd5ae6
feat(platform): Support @Searchable + @Relationship Annotations for Timeseries Aspects (#6455) 2022-11-28 18:17:40 -08:00
Peter Szalai
dc739878a8
fix(loggin): print logging timestamp in ISO8601 format instead of just time (#6474) 2022-11-17 12:33:56 +01:00
david-leifker
33fd876421
feat(elasticsearch): Updates to elasticsearch configuration, dao, and tests (#6269) 2022-11-15 18:03:11 -08:00
Chris Collins
1d944beb0a
feat(lineage) Implement CLL impact analysis for inputFields (#6426) 2022-11-15 16:45:19 -05:00
Aditya Radhakrishnan
873b100a47
refactor(change-events): refactor AspectDiffers to EntityChangeEventGenerators (#6320)
* refactor(change-events): refactor AspectDiffers to EntityChangeEventGenerators

* feat(change-event): add change events for AssertionRunEvent

* feat(change-event): add change events for DataProcessInstanceRunEvent
2022-11-01 09:53:43 -07:00
Chris Collins
cd1331fffe
feat(lineage) Add column-level impact analysis feature (#6272) 2022-10-26 16:43:39 -04:00
RyanHolstien
bfb903cfb8
feat(ingest): add async option to ingest proposal endpoint (#6097)
* feat(ingest): add async option to ingest proposal endpoint

* small tweak to validate before write to K, also keep existing path for timeseries aspects

* avoid double convert

Co-authored-by: Shirshanka Das <shirshanka@apache.org>
2022-10-03 19:56:19 -05:00
RyanHolstien
0728656fbf
feat(patch): initial support of json patch style semantics in MCPs (#5901)
Co-authored-by: Shirshanka Das <shirshanka@apache.org>
2022-09-22 23:52:18 -07:00
Amanda Ng
63ef9c04f9
fix(ownership-differ): standardise ownership change event payload (#5967)
* fix(ownership-differ): standardise ownership change event payload

* fix(ownership-differ): update test case

Co-authored-by: Shirshanka Das <shirshanka@apache.org>
Co-authored-by: John Joyce <john@acryl.io>
2022-09-22 15:06:17 -07:00
Parham Ghazanfari
2712dd1903
fix(patch-entity-registry): Use AspectSpec to retrieve aspect class in order to support custom entity ingestion via patch mechanism. (#5936)
* fix(patch-entity-registry): Remove exception for entities with key aspects.

* test(patch-entity-registry): Fix failing unit test for entities with key aspect

* test(patch-entity-registry): Fix checkstyle for PatchEntityRegistryTest.java

* fix(patch-entity-registry): Remove duplicate semicolon.

* fix(patch-entity-registry): Use AspectSpec to retrieve aspect class in order to support custom entity ingestion via patch mechanism.

* fix(patch-entity-registry): Fixed incorrect merge conflict resolution in BrowsePathUtils.java

* fix(patch-entity-registry): Remove extraneous import from BrowsePathUtils.java

Co-authored-by: Parham Ghazanfari <pghazanfari53@yahoo.com>
2022-09-19 10:47:20 -05:00
Pedro Silva
9434afc78a
feat(hooks): Add toggle for enabling/disabling platform event hook (#5840) 2022-09-06 14:53:58 -05:00
Pedro Silva
20138a32e5
fix(standalone-consumers): Exclude Solr from spring boot application config & make them run on M1 (#5827)
* fix(standalone-consumers) Removes Solr from spring boot application config

* Adds standalone consumer option to datahub quickstart cli

* Rename files

* Make dockerize platform agnostic & change docker compose utility to work with M1

* Fix MAE/MCE dockerfiles for arm & make smoke tests use standalone consumers

* Fix identation

* Use master code

* Adds ARM image publishing for consumers

* Fix linter

* Fix lint
2022-09-06 13:55:31 -05:00
John Joyce
ea9b5f8e21
fix(ui): Improve Error Messaging on the UI (#5752) 2022-08-28 20:08:25 -07:00
John Joyce
57b7ade1f0
feat(ui): Add rich UI ingestion run summary (#5577) 2022-08-08 14:33:57 -07:00
Gabe Lyons
9e58cd6ff1
fix(Siblings): Have sibling hook use entity client (#5279)
* fixing dbt platform issues

* have sibling hook use entity client over entity service

* switching search service as well

* lint

* more lint

* more specific exceptions
2022-06-29 10:02:45 -07:00
Gabe Lyons
baf3f3f33c
feat(dbt): add sibling association logic to associate dbt elements with their target systems (#5190) 2022-06-22 08:31:18 -07:00
John Joyce
e2f1da00ff
refactor(configs): Simplify Kafka Topic name configurations + docs (#5198) 2022-06-17 12:15:51 -07:00
John Joyce
6bc29e6323
refactor(docs): Update Metadata Events Docs (#5173) 2022-06-17 06:29:50 -07:00
RyanHolstien
259a63ab6f
fix(datahub-upgrade): exclude unnecessary configuration from standalone applications (#5047) 2022-05-31 22:59:31 -07:00