Mars Lan
f4bccaf052
fix(ingestion): set schema registry URL correctly for FMCE producer ( #1839 )
...
Fixes https://github.com/linkedin/datahub/issues/1829
2020-08-31 04:50:54 -07:00
Javier Sotelo
ad4c80a753
feat(kafka-config): Add ability to configure other Kafka props ( #1745 )
...
* Integarte spring-kafka & spring-boot for security props
- Upgrade spring-kafka to 2.1.14
- Use KafkaListener and KafkaTemplates to enable KafkaAutoConfiguration
- Integrates spring-boot's KafkaProperties into spring-kafka's config
* Cleanup imports
* Add DataHub kafka env vars
* Remove kafka-streams dependency
* Add KafkaProperties to gms; Add docs
* Add to Adoption
* Remove KAFKA_BOOTSTRAP_SERVER default
Co-authored-by: jsotelo <javier.sotelo@viasat.com>
Co-authored-by: Kerem Sahin <ksahin@linkedin.com>
2020-07-27 05:22:51 -07:00
Jyoti Wadhwani
ad6f1653e1
metadata-models 62.0.3 -> 72.0.8 ( #1693 )
2020-06-11 10:21:51 -07:00
Kerem Sahin
66271923a7
Update readme for mae-consumer-job
2020-06-09 18:10:29 -07:00
Jyoti Wadhwani
90e449d25e
upgrade the version of neo4j ( #1691 )
2020-06-04 22:14:57 -07:00
Jyoti Wadhwani
8a8dd3de77
move the gradle plugin version to top level build.gradle ( #1685 )
2020-06-01 19:00:19 -07:00
Jyoti Wadhwani
2d06e1de5e
Revert "upgrade version of certain dependencies ( #1675 )" ( #1676 )
...
This reverts commit 81508bcb835df4c46b399fd3029b13259930bcd6.
2020-05-19 18:31:41 -07:00
Jyoti Wadhwani
81508bcb83
upgrade version of certain dependencies ( #1675 )
2020-05-19 09:12:21 -07:00
Mars Lan
f932437742
build: start enforcing checkstyle and fix all violations ( #1670 )
2020-05-11 08:41:02 -07:00
Mars Lan
c43e119b3e
refactor: change the default log location ( #1667 )
...
The original location (/local/log) is not a commonly writable location on Mac & Linux.
The new default location also matches the default data location for docker containers.
2020-05-08 14:39:20 -07:00
Arun Vasudevan
8c9787c19c
feat: convert MAE application to springboot ( #1635 )
...
* Convert MAE to Spring boot
* Fix after testing
* Changes after testing
* Add file appender for gms and update doc type for ESv5.6
* Review comments
* Fix Review Comments
2020-04-10 10:23:24 -07:00
Arun Vasudevan
e56e4eafd2
feat: Converting MCE to a Spring boot Application ( #1629 )
2020-04-06 10:16:41 -07:00
Mars Lan
229272fdc9
Update README.md
2020-02-28 22:08:28 -08:00
Arun Vasudevan
9f119d3898
feat: Enable Failed Metadata Change Event for MCE Processor ( #1575 )
...
* feat: Enable Failed Metadata Change Event for MCE Processor
No Failed MCE was sent when a MCE was rejected by the system
- Added FailedMetadataChangeEvent kafka topic
- Created FailedMCE event from the MCE record
- Added Kafka Producer to send FailedMCE to the FailedMetadataChangeEvent Topic
* fix: Review Comment
- Added Nonnull Annotations
- Made Local variables as final
* bug: added pegasus2avro test json
- Added pegasus2avro test fmce json
* feat: Add Tests for Event Utils Avro2pegasusfmce
- Added Tests for avro2Pegasusfmce
2020-02-26 14:18:08 -08:00
Kerem Sahin
b17b91f24a
Bump gradle to 5.6.4 and pegasus to 27.7.18
2020-02-12 17:10:49 -08:00
Kerem Sahin
6c99764fb1
Rename elasticsearch-index to mae-consumer in MaeStreamTask
2019-12-19 17:46:19 -08:00
Kerem Sahin
165d4aef95
Documentation update part-1
2019-12-18 18:57:18 -08:00
Kerem Sahin
8c0e5a2eb0
Use graph to get downstream dataset relationships instead of search index
2019-11-27 00:51:28 -08:00
Kerem Sahin
7e2e6b5c93
Enable datahub-mae-consumer job to build graph as well
2019-11-26 22:19:46 -08:00
Kerem Sahin
22eefc966b
Rename ElasticSearchStreamTask to MaeStreamTask
2019-11-20 18:22:54 -08:00
Kerem Sahin
c64c5d384d
Rename elasticsearch-index-job to mae-consumer-job
2019-11-20 18:19:31 -08:00
Kerem Sahin
8e809e383d
Fix Kafka topic names
2019-10-02 16:28:03 -07:00
Kerem Sahin
6716e38279
Update documentation
2019-09-09 01:45:49 -07:00
Kerem Sahin
0dc5bd9fe0
Add documentation
2019-09-08 20:25:58 -07:00
Kerem Sahin
e9c557f0e3
Start reading configuration from environment variables for mce-consumer
2019-09-08 00:07:02 -07:00
Kerem Sahin
2584793cb5
Start reading configuration from environment variables for mae-consumer
2019-09-07 23:38:56 -07:00
Kerem Sahin
23339df23a
Initial commit for Data Hub
2019-08-31 20:51:14 -07:00
