308 Commits

Author SHA1 Message Date
david-leifker
ecc01b9a46
refactor(restli-mce-consumer) (#6744)
* fix(security): commons-text in frontend

* refactor(restli): set threads based on cpu cores
feat(mce-consumers): hit local restli endpoint

* testing docker build

* Add retry configuration options for entity client

* Kafka debugging

* fix(kafka-setup): parallelize topic creation

* Adjust docker build

* Docker build updates

* WIP

* fix(lint): metadata-ingestion lint

* fix(gradle-docker): fix docker frontend dep

* fix(elastic): fix race condition between gms and mae for index creation

* Revert "fix(elastic): fix race condition between gms and mae for index creation"

This reverts commit 9629d12c3bdb3c0dab87604d409ca4c642c9c6d3.

* fix(test): fix datahub frontend test for clean/test cycle

* fix(test): datahub-frontend missing assets in test

* fix(security): set protobuf lib datahub-upgrade & mce/mae-consumer

* gitingore update

* fix(docker): remove platform on docker base image, set by buildx

* refactor(kafka-producer): update kafka producer tracking/logging

* updates per PR feedback

* Add documentation around mce standalone consumer
Kafka consumer concurrency to follow thread count for restli & sql connection pool

Co-authored-by: leifker <dleifker@gmail.com>
Co-authored-by: Pedro Silva <pedro@acryl.io>
2022-12-26 16:09:08 +00:00
djordje-mijatovic
e6c48e5f19
feat(kafka): expose default kafka producer mechanism (#6381)
* Expose Kafka Sender Retry Parameters

* Implement KafkaHealthChecker

* feat(kafka): expose default kafka producer mechanism
2022-12-20 14:41:24 -06:00
Chris Collins
f559dcc282
feat(lineage) Adds ability to edit lineage manually from the UI (#6816) 2022-12-20 10:54:08 -05:00
Aditya Radhakrishnan
2475d2bbec
fix(policies): resolve the associated domain for a domain as the domain itself (#6812) 2022-12-20 07:13:30 -08:00
Aditya Radhakrishnan
80041d7586
fix(posts): add deletePost GraphQL endpoint (#6813) 2022-12-20 07:08:34 -08:00
RyanHolstien
f85fd157e9
fix(elasticsearch): build in resilience against IO exceptions on httpclient (#6680)
* fix(elasticsearch): build in resilience against IO exceptions on http client
2022-12-13 18:11:02 -06:00
John Joyce
54230a8d81
feat(): Views Feature Milestone 1 (#6666) 2022-12-12 12:30:09 -08:00
david-leifker
be9348b518
fix(build): rename conflicting module auth-api (#6728)
* fix(build): rename conflicting module `auth-api`
2022-12-09 18:06:51 -06:00
John Joyce
bbcec3b226
refactor(ui): Adding caching for users, groups, and roles (#6673) 2022-12-08 09:23:15 -08:00
Aditya Radhakrishnan
f0f0355a83
feat(onboarding): adds framework and some steps for onboarding steps UI (#6462)
* feat(onboarding): adds models and API for onboarding steps feature

* feat(onboarding): adds backend for onboarding steps feature

* feat(onboarding): adds framework and some steps for onboarding steps UI
2022-12-07 16:21:55 -08:00
mohdsiddique
f2a83f3074
feat(gms): Pluggable Authentication & Authorization Framework (#6634) 2022-12-06 10:22:41 -08:00
syedzoherer
ffea9302d9
Add AWS Postgres Iam Auth jar to GMS (#6371)
* Add AWS Postgres Iam Auth jar to GMS

* Added IAM support for Postgres

* Setting IAM support for Postgres as false by default

* format fix

Co-authored-by: syed.javed <syed.javed@precisely.com>
2022-12-05 13:07:26 -06:00
david-leifker
2de9d3d5bf
fix(logging): Remove lombok as source of slf4j-api, convert to compileOnly where possible (#6616) 2022-12-04 19:57:47 -08:00
david-leifker
83b21b021c
fix(security): security version updates (#6602) 2022-12-01 16:43:15 -06:00
david-leifker
4ca3327d89
fix(security): update ranger commons & dependencies for security vulns (#6577)
* fix(security): update ranger commons & dependencies for security vulns
2022-11-30 17:05:01 -06:00
John Joyce
d1b12243bb
feat(models): Adding 'created', 'lastModified' timestamp to Dataset, Container, Dashboard, Chart (#6527) 2022-11-28 18:18:34 -08:00
mohdsiddique
4d76b415c0
fix(apache-ranger): rome dependency (#6560)
Co-authored-by: MohdSiddique Bagwan <mohdsiddique.bagwan@gslab.com>
2022-11-28 09:21:06 -08:00
Pedro Silva
055196c9d4
fix(security): Bump ranger plugin commons dependency (#6535) 2022-11-23 17:18:27 +00:00
RyanHolstien
848574dbff
fix(platform): patch for entity creation, honor async flag on request (#6504) 2022-11-22 11:05:09 -08:00
syedzoherer
5010e0cf69
Added I AM auth for Opensearch (#6370)
* Added I AM auth for Opensearch

* Code refactor and removed default Region for AWS cred.

* Added missing opensearch region exception

* Update RestHighLevelClientFactory.java

* Fixed stylecheck

* Update RestHighLevelClientFactory.java

* Update RestHighLevelClientFactory.java

Co-authored-by: syed.javed <syed.javed@precisely.com>
2022-11-22 10:49:28 -06:00
Mayuri Nehate
5f0908f81c
docs(gms): update search example, do not use deprecated clause (#6340) 2022-11-21 13:42:21 -05:00
Aditya Radhakrishnan
63036993d1
fix(roles): allows role service to unassign roles (#6434) 2022-11-17 10:59:07 -08:00
david-leifker
33fd876421
feat(elasticsearch): Updates to elasticsearch configuration, dao, and tests (#6269) 2022-11-15 18:03:11 -08:00
Chris Collins
e8aaf2a337
feat(ui) Add upgrade step to enable CLL impact analysis for existing data (#6427) 2022-11-15 16:45:33 -05:00
Corentin
fbcfcfc147
feat(recommendations): add last edited entities (#6329) 2022-11-14 22:15:48 -08:00
Aditya Radhakrishnan
2a4ec99d89
fix(change-event): remove unnecessary dependencies on EntityChangeEventGeneratorRegistryFactory (#6431) 2022-11-14 12:40:48 -08:00
Oleksandr
0a0cf70b6f
feat(ingest): power-bi - add support for on-prem PowerBI Report Server (#5369) 2022-11-01 16:05:42 -07:00
mohdsiddique
fbb1b7271c
feat(ingestion): business-glossary - Add values and relatedTerms support (#6148)
* WIP

* updated snapshot

* merge

* while creating BusinessGlossaryConfig object, the version field always get set to None

* WIP

* lint fix

* remove field

* doc update

* rename the fields

* review comments

* lintFix

* Fix the business glossary example link

* small fixes to examples and docs

* some small tweaks to the model annotations

Co-authored-by: MohdSiddique Bagwan <mohdsiddique.bagwan@gslab.com>
Co-authored-by: Shirshanka Das <shirshanka@apache.org>
Co-authored-by: Gabe Lyons <itsgabelyons@gmail.com>
2022-11-01 15:17:30 -07:00
Aditya Radhakrishnan
873b100a47
refactor(change-events): refactor AspectDiffers to EntityChangeEventGenerators (#6320)
* refactor(change-events): refactor AspectDiffers to EntityChangeEventGenerators

* feat(change-event): add change events for AssertionRunEvent

* feat(change-event): add change events for DataProcessInstanceRunEvent
2022-11-01 09:53:43 -07:00
Tamas Nemeth
0dc2d6af25
feat(ingest): databricks - Unity catalog source (#6292)
Co-authored-by: MohdSiddique Bagwan <mohdsiddique.bagwan@gslab.com>
Co-authored-by: Shirshanka Das <shirshanka@apache.org>
2022-10-29 21:44:41 -07:00
Chris Collins
cd1331fffe
feat(lineage) Add column-level impact analysis feature (#6272) 2022-10-26 16:43:39 -04:00
Gabe Lyons
228c10de43
fix(adv search): two advanced search fixes (#6252)
* two advanced search fixes

* another approach

* adding unit tests

* fixing checkstyle issue
2022-10-26 11:31:38 -07:00
Aditya Radhakrishnan
2a501dead1
feat(telemetry): add telemetry events to the glossary, domains, and managed ingestion pages (#6216)
Co-authored-by: John Joyce <john@acryl.io>
2022-10-18 15:48:10 -07:00
Aditya Radhakrishnan
a58502c5dc
feat(telemetry): add telemetry events to the settings page (#6198) 2022-10-13 11:41:47 -07:00
Aditya Radhakrishnan
e8259788a3
feat(tracking): add telemetry for frontend events (#6129) 2022-10-06 18:56:32 -07:00
david-leifker
928f29438a
build: Update JNA for M1 Mac (#6116) 2022-10-05 14:53:38 -07:00
Aditya Radhakrishnan
ee1a1ef45b
feat(posts): add posts feature to DataHub (#6110) 2022-10-04 15:37:28 -07:00
Gabe Lyons
ce90310dd0
feat(advanced-search): Complete Advanced Search: backend changes & tying UI together (#6068)
* stashing progress

* adding remove option

* more progress

* editing

* further in

* additional rendering improvements

* stashing adv search progress

* stashing more progress

* propagating not filters back to UI

* more frontend progress

* more filters working

* getting ready for data platform selector

* add platform select functionality

* locking out switching btwn advanced and standard filters

* final polish

* remove unneeded code

* added unit and cypress tests

* resolutions after merge

* adding documentation

* cleaning up & refactoring

* removing console.log

* minor ui fix & removing unneeded code

* fixing lineage search

* fixing lints

* fix display of degree

* fixing test

* fixing lint

* responding to comments

* fixing tests

* fix smoke tests

* fixing cypress

* fixing cypress test

* responding to comments
2022-10-04 10:20:04 -07:00
RyanHolstien
bfb903cfb8
feat(ingest): add async option to ingest proposal endpoint (#6097)
* feat(ingest): add async option to ingest proposal endpoint

* small tweak to validate before write to K, also keep existing path for timeseries aspects

* avoid double convert

Co-authored-by: Shirshanka Das <shirshanka@apache.org>
2022-10-03 19:56:19 -05:00
Shirshanka Das
48b46971b8
feat(ingest): remove hardcoded env variable default for cli version (#6075) 2022-10-02 16:46:34 -07:00
Aseem Bansal
79575b2815
feat(retention): change default policy, add API to apply retention (#6088) 2022-09-30 16:11:39 +05:30
Peter Szalai
4792ac56b7
feat(monitoring): track graphql errors in metrics (#6087)
* feat(monitoring): track graphql errors in metrics

* fix(statuscode): use status code

* address linting

* address linting
2022-09-29 21:36:17 +02:00
Tomáš Kubín
596d48448e
fix(elasticsearch) Analytics indices creation on AWS ES (#5502)
* refactor(elasticsearch-setup-job): create-indices.sh readability

The script contains many copy-pasting and is not easy to follow.
Add comments, extract commonly used operations into functions, unify approaches.

* fix(elasticsearch-setup-job): AWS indices creation

Fix the issue where Amazon OpenSearch (AWS ES) indices are incorrectly initialised
and the Analytics screen shows errors only.

* feat(elasticsearch-setup-job): configuration hint

mention USE_AWS_ELASTICSEARCH env value if it seems it's set the wrong way

* fix(elasticsearch-setup-job): silent curl

* fix(elasticsearch-setup-job): better USE_AWS_ELASTICSEARCH hint

* docs(elasticsearch-setup-job): index dropping explained

- more comments
- more defensive approach
- index file renamed

* fix(elasticsearch-setup-job): script fixes

* merge(elasticsearch-setup-job): merging in PR #5937

* merge(elasticsearch-setup-job): merging in PR #5963

* merge(elasticsearch-setup-job): merging in PR #5975

Co-authored-by: Pedro Silva <pedro@acryl.io>
2022-09-29 14:10:51 +01:00
Pedro Silva
164bc1aa53
Revert "chore(setup): change defaults for partitions (#6074)" (#6086) 2022-09-29 15:44:32 +05:30
Aseem Bansal
4cf635e16e
chore(setup): change defaults for partitions (#6074) 2022-09-28 15:04:35 +05:30
Aseem Bansal
1c4e127ff9
feat(retention): disable applying retention on bootstrap (#6066) 2022-09-27 12:24:37 -07:00
leifker
203a6ff57b
refactor(java11) - convert most modules to java 11 (#5836)
Co-authored-by: Shirshanka Das <shirshanka@apache.org>
2022-09-25 08:39:22 -07:00
Aditya Radhakrishnan
325b959ea6
feat(roles): add ability to invite users into a role (#6015) 2022-09-23 16:48:23 -07:00
Shirshanka Das
af6a423f9d
feat(patch): report whether server is patch capable (#6037) 2022-09-23 14:42:45 -07:00
RyanHolstien
0728656fbf
feat(patch): initial support of json patch style semantics in MCPs (#5901)
Co-authored-by: Shirshanka Das <shirshanka@apache.org>
2022-09-22 23:52:18 -07:00