147 Commits

Author SHA1 Message Date
Harshal Sheth
b565a657d2
feat(quickstart): fix broker InconsistentClusterIdException issues (#9148) 2023-10-30 23:35:12 -07:00
david-leifker
3f4ab44a91
feat(kafka): increase kafka message size and enable compression (#9038)
Co-authored-by: Pedro Silva <pedro@acryl.io>
2023-10-29 16:26:05 -05:00
david-leifker
21eb4dfc12
feat(search): update to support OpenSearch 2.x (#8852) 2023-09-21 13:01:55 -05:00
david-leifker
40caf35812
feat(datahub-ingestion): refactor datahub ingestion slim images (#8515) 2023-08-17 10:40:17 +05:30
Harshal Sheth
b8681a7129
feat(quickstart): bump quickstart start periods more (#8573) 2023-08-04 15:17:16 +05:30
RyanHolstien
9f791a3b6e
feat(browseV2): add browseV2 logic to system update (#8506)
Co-authored-by: david-leifker <114954101+david-leifker@users.noreply.github.com>
2023-08-01 12:48:02 -05:00
david-leifker
3640291493
config(ebean): add ebean retry configuration (#8500) 2023-07-28 14:09:44 -05:00
Harshal Sheth
2a3f6c42d4
ci: workarounds for pyyaml installation (#8435) 2023-07-17 15:26:43 -07:00
Harshal Sheth
e5f9ee178c
fix(cli/quickstart): handle docker hangs gracefully (#8211)
Co-authored-by: Pedro Silva <pedro@acryl.io>
2023-06-13 14:34:06 -05:00
Jinlin Yang
166dc90896
Upgrade kafka and its dependencies to 3.4 in docker compose (#8161)
Co-authored-by: Pedro Silva <pedro@acryl.io>
2023-06-13 13:57:54 +01:00
david-leifker
43d37ff442
chore(cp-schema-registry): bump minor version (#8192) 2023-06-09 11:53:18 +01:00
Guillaume Cernier
ef1ada118d
fix(docker-compose): make startup more robust with deterministic services' dependencies (#7880) 2023-05-08 16:42:15 -05:00
david-leifker
cd05f5b174
feat(schema-registry): replace confluent schema registry (#7930)
Co-authored-by: Pedro Silva <pedro@acryl.io>
Co-authored-by: Shirshanka Das <shirshanka@apache.org>
Co-authored-by: Ryan Holstien <ryan@acryl.io>
2023-05-01 13:18:41 -05:00
Harshal Sheth
8f8c11081d
fix(cli): remove duplicate labels from quickstart files (#7886) 2023-04-25 14:35:34 -07:00
Harshal Sheth
7cf3d378a8
feat(quickstart): move quickstart back to master (#7697) 2023-03-28 15:10:57 +02:00
Harshal Sheth
81ec296b06
feat(cli): switch default quickstart to v0.10.0 (#7567) 2023-03-13 21:42:05 -07:00
Peter Szalai
fd1bd51541
Docs update (#7517) 2023-03-08 13:43:46 +01:00
Peter Szalai
1d33392761
feat(cli): introduce remote config for quickstart (#7424)
Co-authored-by: Harshal Sheth <hsheth2@gmail.com>
2023-03-07 13:14:24 +01:00
Peter Szalai
e9f060dae3
Add setup job labels to compose files (#7473) 2023-03-02 15:05:16 +01:00
Shirshanka Das
516aee0aaa
fix(docker): add env variable to control mysql setup image, sort dock… (#7266) 2023-02-07 00:56:08 -08:00
david-leifker
39920bb00f
feat(elasticsearch): Elasticsearch improvements (#6894) 2023-01-31 18:44:37 -06:00
Peter Szalai
8a2230aea4
fix(quickstart): adding back kafka-setup (#7181)
Co-authored-by: Shirshanka Das <shirshanka@apache.org>
2023-01-31 00:34:36 +01:00
Pedro Silva
bef59b0928
feat(quickstart): Remove kafka-setup as a hard deployment requirement (#7073) 2023-01-24 16:12:57 +00:00
david-leifker
9277101a5f
config(quickstart): enable auto-reindex for quickstart (#6983) 2023-01-09 17:58:29 -06:00
david-leifker
ecc01b9a46
refactor(restli-mce-consumer) (#6744)
* fix(security): commons-text in frontend

* refactor(restli): set threads based on cpu cores
feat(mce-consumers): hit local restli endpoint

* testing docker build

* Add retry configuration options for entity client

* Kafka debugging

* fix(kafka-setup): parallelize topic creation

* Adjust docker build

* Docker build updates

* WIP

* fix(lint): metadata-ingestion lint

* fix(gradle-docker): fix docker frontend dep

* fix(elastic): fix race condition between gms and mae for index creation

* Revert "fix(elastic): fix race condition between gms and mae for index creation"

This reverts commit 9629d12c3bdb3c0dab87604d409ca4c642c9c6d3.

* fix(test): fix datahub frontend test for clean/test cycle

* fix(test): datahub-frontend missing assets in test

* fix(security): set protobuf lib datahub-upgrade & mce/mae-consumer

* gitingore update

* fix(docker): remove platform on docker base image, set by buildx

* refactor(kafka-producer): update kafka producer tracking/logging

* updates per PR feedback

* Add documentation around mce standalone consumer
Kafka consumer concurrency to follow thread count for restli & sql connection pool

Co-authored-by: leifker <dleifker@gmail.com>
Co-authored-by: Pedro Silva <pedro@acryl.io>
2022-12-26 16:09:08 +00:00
Navin Sharma
7c3872a2ed
Update Grafana Dashboard (#6076)
* Add Datasource as variable in dashboard

(cherry picked from commit e75b3f7333dd9436f7434eefa76ab9fbfb1babab)

* Update datahub_dashboard.json

(cherry picked from commit 701592697702a99ddf33f75d57826282df999c82)

* Bump docker compose version to 3.8

(cherry picked from commit ff6a97b1692ac21368eacc83a0daed30b0364d5d)

* Update grafana image tag from latest to 9.1.4

(cherry picked from commit 2c88e2a3041b9ddfa70bb92d13e80ca15be0ead2)

* Update old metric name in datahub_dashboard.json

(cherry picked from commit 21b502e25392f37c5053dcf00161fc5c7d35c94d)

* Add panel for new metrics

(cherry picked from commit 194452778bc825820dda87ab2e159b7064a360ef)

Co-authored-by: Peter Szalai <szalaipeti.vagyok@gmail.com>
2022-12-19 22:43:47 +01:00
david-leifker
637b540fa6
fix(kafka-setup): fix return code check (#6782)
* fix(kafka-setup): parallelize topic creation
* Remove -setup from docker compose (not services)
2022-12-16 13:39:18 -06:00
mohdsiddique
f2a83f3074
feat(gms): Pluggable Authentication & Authorization Framework (#6634) 2022-12-06 10:22:41 -08:00
Shirshanka Das
155a68386a
docs(actions): slack, teams (#6632) 2022-12-04 13:27:52 -08:00
Shirshanka Das
4876fdd084
feat(quickstart): add support for env vars for Slack, Teams actions (#6591) 2022-12-03 23:00:50 -08:00
Shirshanka Das
5937b35447
fix(actions): add missing datahub-gms-protocol env var (#6593) 2022-12-01 00:25:43 -08:00
david-leifker
a2dc229c62
fix(tests): Misc updates for tests, auth log level, and quickstart (#6491) 2022-11-29 08:44:55 -08:00
Peter Szalai
f77117a9e0
fix(envs): remove DATASET_ENABLE_SCSI stale env var (#6546) 2022-11-24 21:34:53 +05:30
david-leifker
33fd876421
feat(elasticsearch): Updates to elasticsearch configuration, dao, and tests (#6269) 2022-11-15 18:03:11 -08:00
Aseem Bansal
98eba90edc
fix(build): allow image tag via env, fix requirements (#6237) 2022-10-19 12:26:22 +05:30
Shirshanka Das
e688dd554a
chore(deps): upgrade neo4j to 4.4.x (#6101) 2022-10-02 23:16:45 -07:00
Shirshanka Das
48b46971b8
feat(ingest): remove hardcoded env variable default for cli version (#6075) 2022-10-02 16:46:34 -07:00
leifker
203a6ff57b
refactor(java11) - convert most modules to java 11 (#5836)
Co-authored-by: Shirshanka Das <shirshanka@apache.org>
2022-09-25 08:39:22 -07:00
Pedro Silva
20138a32e5
fix(standalone-consumers): Exclude Solr from spring boot application config & make them run on M1 (#5827)
* fix(standalone-consumers) Removes Solr from spring boot application config

* Adds standalone consumer option to datahub quickstart cli

* Rename files

* Make dockerize platform agnostic & change docker compose utility to work with M1

* Fix MAE/MCE dockerfiles for arm & make smoke tests use standalone consumers

* Fix identation

* Use master code

* Adds ARM image publishing for consumers

* Fix linter

* Fix lint
2022-09-06 13:55:31 -05:00
Peter Szalai
aa146db611
feat(monitoring): enable JMX and OTEL for frontend pods (#5834) 2022-09-06 17:14:53 +05:30
Harshal Sheth
13f20ad2c3
fix(docker): add missing port mappings for non-neo4j quickstart (#5799)
* fix(docker): add missing port mappings for non-neo4j quickstart

* undo cassandra
2022-09-05 15:32:19 +05:30
mohdsiddique
2b4f9400ca
fix(GMS): fix no such classes directory file:///etc/datahub/plugins/auth/resources (#5743) 2022-08-29 12:08:18 -07:00
mohdsiddique
cfaf7a0c20
fix(quickstart): fix for No such classes directory file:///etc/datahub/plugins/auth/resources (#5720)
Co-authored-by: MohdSiddique Bagwan <mohdsiddique.bagwan@gslab.com>
2022-08-24 09:26:13 -07:00
Chris Collins
d64cca5c6b
feat(retention) Enable retention and set max versions for executionRequests (#5704) 2022-08-23 16:25:18 -07:00
Gabe Lyons
f06b8eb4e8
(chore): upgrading ingestion to 0.8.42 (#5562) 2022-08-04 17:06:16 +05:30
John Joyce
4e0da5de29
fix(docker): Fixing dev docker and quickstart (#5550) 2022-08-02 15:02:01 -07:00
mohdsiddique
c31e380489
feat(apache-ranger): Apache Ranger Authorizer support in datahub-gms (#4999) 2022-08-01 09:47:08 -07:00
Aseem Bansal
252992f15e
fix(ci): intermittent failure in github actions (#5452) 2022-07-25 23:02:02 -07:00
Shirshanka Das
941770f3bc
feat(cli): quickstart - experimental support for backup restore (#5418) 2022-07-24 23:52:25 -07:00
Aseem Bansal
acb9879eb4
feat(cli,build): remove deprecated variables GMS_HOST/_PORT (#5451) 2022-07-20 20:54:43 +05:30