* fix(security): commons-text in frontend
* refactor(restli): set threads based on cpu cores
feat(mce-consumers): hit local restli endpoint
* testing docker build
* Add retry configuration options for entity client
* Kafka debugging
* fix(kafka-setup): parallelize topic creation
* Adjust docker build
* Docker build updates
* WIP
* fix(lint): metadata-ingestion lint
* fix(gradle-docker): fix docker frontend dep
* fix(elastic): fix race condition between gms and mae for index creation
* Revert "fix(elastic): fix race condition between gms and mae for index creation"
This reverts commit 9629d12c3bdb3c0dab87604d409ca4c642c9c6d3.
* fix(test): fix datahub frontend test for clean/test cycle
* fix(test): datahub-frontend missing assets in test
* fix(security): set protobuf lib datahub-upgrade & mce/mae-consumer
* gitingore update
* fix(docker): remove platform on docker base image, set by buildx
* refactor(kafka-producer): update kafka producer tracking/logging
* updates per PR feedback
* Add documentation around mce standalone consumer
Kafka consumer concurrency to follow thread count for restli & sql connection pool
Co-authored-by: leifker <dleifker@gmail.com>
Co-authored-by: Pedro Silva <pedro@acryl.io>
* feat(ingest): add async option to ingest proposal endpoint
* small tweak to validate before write to K, also keep existing path for timeseries aspects
* avoid double convert
Co-authored-by: Shirshanka Das <shirshanka@apache.org>
Co-authored-by: Dexter Lee <dexter@acryl.io>
Co-authored-by: Gabe Lyons <itsgabelyons@gmail.com>
Co-authored-by: Shirshanka Das <shirshanka@apache.org>
Move code to linkedin/datahub-gma.
"GMA" (Generalized Metadata Architecture) is the backend of DataHub, and has been moved to its own repository.
This deletes the code that was moved and uses jars that GMA publishes to bintray to load it.
Note that not all of GMA was moved, but most of it. We may still move more things to the other repository in the future.
Requested by a few people in OS. See https://github.com/linkedin/datahub/issues/1840.
Companies need full customization over the topic name. This new class should be easily customizable by using a spring factory.
TODO to finish the implmentation for v5. For right now v5 is LI only and unfinished. Getting this in for v4 so it is useful to other companies now.
TODO AFTER OPEN SOURCE PUSH - make configurable via spring
TODO AFTER SUBMIT - see where else we can use this (jobs, where else?)
- Upgrade to checkstyle 8
- Copy javadoc checks from Google
- Disable missing class and method checks for now, too many warnings. I'll have to figure out how to suppress them instead.
- Fix other issues, which are mostly missing periods at the end of sentences and lack of paragraph tags.
Revert "Reverting the commit range: 8dfdb73ac6c73581ef56c0d81c21a2a92e8a1a02..194bd6f57f4a4d075d2ea1f442397d1139080f7a."
This reverts commit ab178ec1469fa72c0c339f0b842e7ff0850e7c74.
- Upgrade to checkstyle 8
- Copy javadoc checks from Google
- Disable missing class and method checks for now, too many warnings. I'll have to figure out how to suppress them instead.
- Fix other issues, which are mostly missing periods at the end of sentences and lack of paragraph tags.
metadata-models 80.0.0 -> 90.0.13:
90.0.13: Roll forward: Fix the open source build by avoiding URN method that isn't part of the open source URN.
90.0.2: Refactor listUrnsFromIndex method
90.0.0: Start distinguishing between [] aspects vs null aspects input param
89.0.4: Fix the open source build by avoiding URN method that isn't part of the open source URN.
89.0.2: fix some test case name
89.0.0: META-12686: Made the MXE_v5 topics become strictly ACL'ed to avoid the wildcard write ACL as "MetadataXEvent.+"
88.0.6: change DAO to take Storage Config as input
88.0.3: Add a comment on lack of avro generation for MXEv5 + add MXEv5 to the pegasus validation task.
87.0.15: META-12651: Integrate the metadata-models-ext with metadata-models
87.0.13: add StorageConfig to Local DAO
87.0.3: Treat empty aspect vs optional aspect same until all clients are migrated
87.0.2: Treat empty aspect vs optional aspect differently
87.0.1: META-12533: Skip processing unregistered aspect specific MAE.
83.0.6: action method to return list of urns from strong consistent index
83.0.4: Change input param type for batch backfill
83.0.3: Implement batch backfill
83.0.1: Implement support for OR filter in browse query
82.0.10: Throw UnsupportedOperationException for unsupported condition types in search filter
82.0.6: Implement local secondary backfilling index as part of backfill method
82.0.5: [strongly consistent index] implement getUrns method
82.0.4: Add indexing urn fields to the local secondary index
82.0.0: Render Delta fiels in the MCE_v5.
81.0.1: Add pegasus to avro conversion for FMCE
80.0.4: add get all support for BaseSingleAspectEntitySimpleKeyResource
80.0.2: Add a BaseSearchWriterDAO with an ESBulkWriterDAO implementation.
80.0.1: META-12254: Produce aspect specific MAE with always emit option
80.0.0: Convert getNodesInTraversedPath to getSubgraph to return complete view of the subgraph (nodes+edges)
