31 Commits

Author SHA1 Message Date
david-leifker
ecc01b9a46
refactor(restli-mce-consumer) (#6744)
* fix(security): commons-text in frontend

* refactor(restli): set threads based on cpu cores
feat(mce-consumers): hit local restli endpoint

* testing docker build

* Add retry configuration options for entity client

* Kafka debugging

* fix(kafka-setup): parallelize topic creation

* Adjust docker build

* Docker build updates

* WIP

* fix(lint): metadata-ingestion lint

* fix(gradle-docker): fix docker frontend dep

* fix(elastic): fix race condition between gms and mae for index creation

* Revert "fix(elastic): fix race condition between gms and mae for index creation"

This reverts commit 9629d12c3bdb3c0dab87604d409ca4c642c9c6d3.

* fix(test): fix datahub frontend test for clean/test cycle

* fix(test): datahub-frontend missing assets in test

* fix(security): set protobuf lib datahub-upgrade & mce/mae-consumer

* gitingore update

* fix(docker): remove platform on docker base image, set by buildx

* refactor(kafka-producer): update kafka producer tracking/logging

* updates per PR feedback

* Add documentation around mce standalone consumer
Kafka consumer concurrency to follow thread count for restli & sql connection pool

Co-authored-by: leifker <dleifker@gmail.com>
Co-authored-by: Pedro Silva <pedro@acryl.io>
2022-12-26 16:09:08 +00:00
david-leifker
637b540fa6
fix(kafka-setup): fix return code check (#6782)
* fix(kafka-setup): parallelize topic creation
* Remove -setup from docker compose (not services)
2022-12-16 13:39:18 -06:00
david-leifker
cca3639848
Optimize kafka setup (#6778)
* fix(kafka-setup): parallelize topic creation
2022-12-15 20:09:29 -06:00
Pedro Silva
c93bdc5e39
fix(setup): Bump setup images to alpine 3.14 with arch based on machine OS. (#6612)
* fix(setup): Bump setup images to alpine 3.14 with arch based on machine OS.
2022-12-09 09:49:24 -06:00
Parham Ghazanfari
269342bec0
feat(kafka-setup): Added support for MSK IAM authentication. (#6435) 2022-11-22 15:37:24 -06:00
Pedro Silva
164bc1aa53
Revert "chore(setup): change defaults for partitions (#6074)" (#6086) 2022-09-29 15:44:32 +05:30
Aseem Bansal
4cf635e16e
chore(setup): change defaults for partitions (#6074) 2022-09-28 15:04:35 +05:30
Pedro Silva
6232447ecf
fix(kafka-setup) Bump kafka version to existing version (#5995) 2022-09-20 14:14:17 +01:00
Pedro Silva
12f462e82b
fix(docker): Bumps JRE 11 to latest (#5875) 2022-09-08 12:44:28 -05:00
Pedro Silva
7d0f5435c8
feat(docker): Update Dockerfiles to use java 11 runtime (#5853) 2022-09-08 17:39:23 +05:30
Pedro Silva
12eb2c6240
feat(kafka) Adds SASL SSL support in kafka setup docker image (#5697) 2022-08-22 16:01:10 -07:00
Shirshanka Das
9d79fd3767
fix(docker): fix kafka-setup command to support same capabilities as previous (#5428) 2022-07-18 13:25:17 -07:00
Shirshanka Das
024797df0d
feat(quickstart): moving to official confluent images for m1 (#5416) 2022-07-17 18:59:09 -07:00
abiwill
7bbac5ef4d
feat(kafka-setup): add options for sasl_plaintext (#5385)
allow sasl_plaintext options using environment variables
2022-07-13 13:47:23 -07:00
Dexter Lee
93145d1591
fix(Kafka-setup): Make sure it doesn't fail when the new envs are not set (#5168) 2022-06-14 10:29:24 -07:00
chen4119
3c32ac7916
fix(kafka-setup): Check if keystore/truststore location env variables are set (#4924)
* OIDC discovery URL will not have none as auth method

* Fix code style

* fix compilation error

* add keystore/truststore type env variables

* check keystore/truststore location is set

Co-authored-by: Dexter Lee <dexter@acryl.io>
2022-05-16 13:51:19 -07:00
chen4119
a0e7262b00
fix(kafka-setup): Add ssl.keystore.type and ssl.truststore.type (#4923) 2022-05-16 10:22:55 -07:00
John Joyce
87c5ad385c
Fix iss #4683 (#4697) 2022-04-19 15:22:49 -07:00
John Joyce
c69310522b
feat(metadata service): Introducing Platform Events (#4477) 2022-03-29 18:32:04 -07:00
Tamas Nemeth
111d6d3520
fix(docker): alpine based multi-platform docker build for kafka-setup (#3991) 2022-01-28 16:25:57 -08:00
Tamas Nemeth
d41c5eba35
feat(docker): add multiplatform docker support for arm64 (m1) (#3537) 2021-11-09 11:50:06 -08:00
Dexter Lee
65d6c474f6
fix(docker): upgrade base image version (#3072) 2021-08-10 22:22:36 -07:00
John Joyce
352a0abf8d
Introducing TimeSeries Aspects + Dataset Profile (Stats) Aspect (#2983)
Co-authored-by: Dexter Lee <dexter@acryl.io>
Co-authored-by: Harshal Sheth <hsheth2@gmail.com>
Co-authored-by: Ravindra Lanka <rlanka@acryl.io>
Co-authored-by: Shirshanka Das <shirshanka@apache.org>
2021-07-30 17:41:03 -07:00
G-nther
634dde5cfb
fix(analytics): use seperate env variable for tracking topic in MAE-Consumer (#2545) 2021-05-12 17:53:40 -07:00
John Joyce
29832e5385
feat(Product Analytics): Introducing In-App Analytics Beta (#2499)
Co-authored-by: Harshal Sheth <harshal@acryl.io>
Co-authored-by: Dexter Lee <dexter@acryl.io>
Co-authored-by: Gabe Lyons <itsgabelyons@gmail.com>
2021-05-11 15:41:42 -07:00
Dexter Lee
c1c587d164
fix(kafka-setup): Fix start script for kafka setup (#2421) 2021-04-19 15:39:38 -07:00
Dexter Lee
1b3a1459e3
fix(kafka-setup): Add the correct context to the git workflow for pushing kafka-setup image (#2414) 2021-04-16 17:51:43 -07:00
G-nther
177161a850
feat(kafka-setup): add option for SSL and topic partition config via environment (#2398) 2021-04-16 09:40:39 -07:00
Dexter Lee
f018dd6f3e
fix: Fix env variable setup for kafka, mysql-setup docker containers (#2402) 2021-04-14 19:07:50 -07:00
Dexter Lee
fa015c5aaa
fix(kafka-topic-convention): Fix DAOs that do not refer to TopicConvention (#2387) 2021-04-13 07:58:31 -07:00
John Plaisted
b8e18b0b5d
refactor(docker): make docker files easier to use during development. (#1777)
* Make docker files easier to use during development.

During development it quite nice to have docker work with locally built code. This allows you to launch all services very quickly, with your changes, and optionally with debugging support.

Changes made to docker files:
- Removed all redundant docker-compose files. We now have 1 giant file, and smaller files to use as overrides.
- Remove redundant README files that provided little information.
- Rename docker/<dir> to match the service name in the docker-compose file for clarity.
- Move environment variables to .env files. We only provide dev / the default environment for quickstart.
- Add debug options to docker files using multistage build to build minimal images with the idea that built files will be mounted instead.
- Add a docker/dev.sh script + compose file to easily use the dev override images (separate tag; images never published; uses debug docker files; mounts binaries to image).
- Added docs/docker documentation for this.
2020-08-06 16:38:53 -07:00