658 Commits

Author SHA1 Message Date
david-leifker
d557fd65ff
fix(elasticsearch): fix orphan index clean up pattern, consistent top… (#7242) 2023-02-03 07:09:19 -06:00
Peter Szalai
0a43d250d0
fix(dev.sh): Add context to kafka-setup build (#7234) 2023-02-03 17:45:42 +05:30
david-leifker
39920bb00f
feat(elasticsearch): Elasticsearch improvements (#6894) 2023-01-31 18:44:37 -06:00
Peter Szalai
8a2230aea4
fix(quickstart): adding back kafka-setup (#7181)
Co-authored-by: Shirshanka Das <shirshanka@apache.org>
2023-01-31 00:34:36 +01:00
Peter Szalai
732c837e67
fix(htrace): remove htrace jar (#7126) 2023-01-24 19:43:34 -08:00
Pedro Silva
5db3a1f1f8
fix(kafka-setup): Make topic name consistent with other images (#7103) 2023-01-24 10:48:23 -08:00
Pedro Silva
bef59b0928
feat(quickstart): Remove kafka-setup as a hard deployment requirement (#7073) 2023-01-24 16:12:57 +00:00
tony csoka
1149cea1c8
fix(docker): support non amd64 dockerize in setup containers (#7091)
Co-authored-by: Pedro Silva <pedro@acryl.io>
2023-01-23 10:25:33 +00:00
Pedro Silva
ae2d781359
fix(kafka-setup): Remove reference to non-existing topic (#7019) 2023-01-12 07:25:17 -08:00
david-leifker
9277101a5f
config(quickstart): enable auto-reindex for quickstart (#6983) 2023-01-09 17:58:29 -06:00
Pedro Silva
2fe9e8d962
fix(kafka-setup): Make kafka-setup run with multiple threads (#6970) 2023-01-09 12:27:00 +00:00
Peter Szalai
a629df5de2
fix(kafka-setup): Fix compose context for kafka-setup (#6923) 2023-01-03 08:12:31 -08:00
Peter Szalai
a164bdabe2
fix(es-setup): create data stream on non-aws (#6926) 2023-01-03 09:52:45 +01:00
Harshal Sheth
62a2aa94f6
feat: remove jq requirement + tweak modeldocgen args (#6904)
Co-authored-by: Tamas Nemeth <treff7es@gmail.com>
2022-12-30 14:02:57 -05:00
Aseem Bansal
0bbe6d693e
fix(ci): reduce flakiness in add_users, siblings smoke test (#6883) 2022-12-29 21:47:25 +05:30
david-leifker
122c024bab
refactor(gradle): move the local docker registry to common location (#6881) 2022-12-29 15:08:36 +05:30
Harshal Sheth
8e0f4078b8
chore(ingest): loosen additional requirements (#6868) 2022-12-28 12:08:53 +01:00
Harshal Sheth
d851140048
chore(ingest): loosen requirements file (#6867) 2022-12-27 22:39:12 +01:00
david-leifker
a5b5abf270
fix(kafka): datahub-upgrade job (#6864) 2022-12-27 12:35:48 -06:00
david-leifker
ecc01b9a46
refactor(restli-mce-consumer) (#6744)
* fix(security): commons-text in frontend

* refactor(restli): set threads based on cpu cores
feat(mce-consumers): hit local restli endpoint

* testing docker build

* Add retry configuration options for entity client

* Kafka debugging

* fix(kafka-setup): parallelize topic creation

* Adjust docker build

* Docker build updates

* WIP

* fix(lint): metadata-ingestion lint

* fix(gradle-docker): fix docker frontend dep

* fix(elastic): fix race condition between gms and mae for index creation

* Revert "fix(elastic): fix race condition between gms and mae for index creation"

This reverts commit 9629d12c3bdb3c0dab87604d409ca4c642c9c6d3.

* fix(test): fix datahub frontend test for clean/test cycle

* fix(test): datahub-frontend missing assets in test

* fix(security): set protobuf lib datahub-upgrade & mce/mae-consumer

* gitingore update

* fix(docker): remove platform on docker base image, set by buildx

* refactor(kafka-producer): update kafka producer tracking/logging

* updates per PR feedback

* Add documentation around mce standalone consumer
Kafka consumer concurrency to follow thread count for restli & sql connection pool

Co-authored-by: leifker <dleifker@gmail.com>
Co-authored-by: Pedro Silva <pedro@acryl.io>
2022-12-26 16:09:08 +00:00
raysaka
fcb3242983
chore(ingest): bump python package dependencies to resolve vulns (#6384)
Co-authored-by: John Joyce <john@acryl.io>
2022-12-19 18:12:56 -05:00
Navin Sharma
7c3872a2ed
Update Grafana Dashboard (#6076)
* Add Datasource as variable in dashboard

(cherry picked from commit e75b3f7333dd9436f7434eefa76ab9fbfb1babab)

* Update datahub_dashboard.json

(cherry picked from commit 701592697702a99ddf33f75d57826282df999c82)

* Bump docker compose version to 3.8

(cherry picked from commit ff6a97b1692ac21368eacc83a0daed30b0364d5d)

* Update grafana image tag from latest to 9.1.4

(cherry picked from commit 2c88e2a3041b9ddfa70bb92d13e80ca15be0ead2)

* Update old metric name in datahub_dashboard.json

(cherry picked from commit 21b502e25392f37c5053dcf00161fc5c7d35c94d)

* Add panel for new metrics

(cherry picked from commit 194452778bc825820dda87ab2e159b7064a360ef)

Co-authored-by: Peter Szalai <szalaipeti.vagyok@gmail.com>
2022-12-19 22:43:47 +01:00
david-leifker
637b540fa6
fix(kafka-setup): fix return code check (#6782)
* fix(kafka-setup): parallelize topic creation
* Remove -setup from docker compose (not services)
2022-12-16 13:39:18 -06:00
david-leifker
cca3639848
Optimize kafka setup (#6778)
* fix(kafka-setup): parallelize topic creation
2022-12-15 20:09:29 -06:00
Aseem Bansal
1350bb010c
docs(docker): add section for adding community images (#6770) 2022-12-15 15:26:31 -08:00
Harshal Sheth
8c14dfc617
ci: fix ingestion gradle retry (#6752) 2022-12-13 20:16:59 -05:00
Pedro Silva
c93bdc5e39
fix(setup): Bump setup images to alpine 3.14 with arch based on machine OS. (#6612)
* fix(setup): Bump setup images to alpine 3.14 with arch based on machine OS.
2022-12-09 09:49:24 -06:00
Harshal Sheth
ab24379def
build: add retries to gradle wrapper download in ingestion docker (#6704) 2022-12-08 22:37:29 -05:00
david-leifker
691f802a13
fix(security): version bump to latest minor python image (#6694) 2022-12-08 13:54:21 -05:00
Aseem Bansal
96150cfb10
fix(ci): revert confluent kafka in base image (#6690) 2022-12-08 13:54:13 -05:00
Aseem Bansal
a087a9fbb9
chore(ci): update base ingestion image requirements file (#6687) 2022-12-08 09:59:19 +05:30
John Joyce
15e9475efb
fix(docker): Fix datahub-frontend dockerfile (#6670) 2022-12-06 23:11:54 -08:00
mohdsiddique
b46d52dc0f
feat(gms): Enable auth-api publishing to maven (#6671) 2022-12-06 21:56:36 -08:00
mohdsiddique
f2a83f3074
feat(gms): Pluggable Authentication & Authorization Framework (#6634) 2022-12-06 10:22:41 -08:00
syedzoherer
ffea9302d9
Add AWS Postgres Iam Auth jar to GMS (#6371)
* Add AWS Postgres Iam Auth jar to GMS

* Added IAM support for Postgres

* Setting IAM support for Postgres as false by default

* format fix

Co-authored-by: syed.javed <syed.javed@precisely.com>
2022-12-05 13:07:26 -06:00
Shirshanka Das
155a68386a
docs(actions): slack, teams (#6632) 2022-12-04 13:27:52 -08:00
Shirshanka Das
4876fdd084
feat(quickstart): add support for env vars for Slack, Teams actions (#6591) 2022-12-03 23:00:50 -08:00
Pedro Silva
4f7b5ac5e6
feat(sql): Add SQL index on createdon field (#6522) 2022-12-02 09:13:47 +00:00
Tim Costa
6e31754594
fix: remove set -x from start script (#6589)
Co-authored-by: Tim Costa <timcosta@amazon.com>
2022-12-01 20:33:54 -08:00
Shirshanka Das
5937b35447
fix(actions): add missing datahub-gms-protocol env var (#6593) 2022-12-01 00:25:43 -08:00
david-leifker
ef9a280ba4
fix(verions): Update opentelemetry and updates from pr-5239 (#6563) 2022-11-29 10:59:38 -06:00
david-leifker
a2dc229c62
fix(tests): Misc updates for tests, auth log level, and quickstart (#6491) 2022-11-29 08:44:55 -08:00
Peter Szalai
f77117a9e0
fix(envs): remove DATASET_ENABLE_SCSI stale env var (#6546) 2022-11-24 21:34:53 +05:30
Parham Ghazanfari
269342bec0
feat(kafka-setup): Added support for MSK IAM authentication. (#6435) 2022-11-22 15:37:24 -06:00
Harshal Sheth
2cb1e6576d
fix(ingest): add alias for bigquery-beta (#6521) 2022-11-22 14:15:20 -05:00
syedzoherer
5010e0cf69
Added I AM auth for Opensearch (#6370)
* Added I AM auth for Opensearch

* Code refactor and removed default Region for AWS cred.

* Added missing opensearch region exception

* Update RestHighLevelClientFactory.java

* Fixed stylecheck

* Update RestHighLevelClientFactory.java

* Update RestHighLevelClientFactory.java

Co-authored-by: syed.javed <syed.javed@precisely.com>
2022-11-22 10:49:28 -06:00
Aseem Bansal
317f0b135b
fix(build): do not use neo4j on local (#6501) 2022-11-21 22:19:13 +05:30
david-leifker
33fd876421
feat(elasticsearch): Updates to elasticsearch configuration, dao, and tests (#6269) 2022-11-15 18:03:11 -08:00
skrydal
a274406126
fix(docker): Set proper variable expansion for JMX_OPTS and JAVA_OPTS in MAE docker (#6378)
image
2022-11-15 10:03:33 -08:00
Harshal Sheth
9c04c63813
docs(ingest): airflow - clarify docs around 1.x compat (#6436) 2022-11-14 21:30:42 -08:00