datahub/metadata-ingestion/docs/sources/s3/s3_recipe.yml

source:
  type: s3
  config:
    path_specs:
      -
        include: "s3://covid19-lake/covid_knowledge_graph/csv/nodes/*.*"

    aws_config:
      aws_access_key_id: *****
      aws_secret_access_key: *****
      aws_region: us-east-2
    env: "PROD"
    profiling:
      enabled: false

# sink configs
feat(ingest): docs - overhaul source connector docs to make it code driven (#4798) Co-authored-by: MugdhaHardikar-GSLab <mugdha.hardikar@gslab.com> 2022-05-02 00:18:15 -07:00			`source:`
			`type: s3`
			`config:`
feat(ingest): s3 - add support for multiple pathspecs in one recipe (#4777) 2022-05-05 19:09:47 +02:00			`path_specs:`
			`-`
			`include: "s3://covid19-lake/covid_knowledge_graph/csv/nodes/."`

feat(ingest): docs - overhaul source connector docs to make it code driven (#4798) Co-authored-by: MugdhaHardikar-GSLab <mugdha.hardikar@gslab.com> 2022-05-02 00:18:15 -07:00			`aws_config:`
			`aws_access_key_id: *****`
			`aws_secret_access_key: *****`
			`aws_region: us-east-2`
			`env: "PROD"`
			`profiling:`
			`enabled: false`

feat(ingest): s3 - add support for multiple pathspecs in one recipe (#4777) 2022-05-05 19:09:47 +02:00			`# sink configs`