diff --git a/openmetadata-docs/content/v1.0.0/connectors/storage/s3/airflow.md b/openmetadata-docs/content/v1.0.0/connectors/storage/s3/airflow.md index 099d28bda1e..9e84465b68b 100644 --- a/openmetadata-docs/content/v1.0.0/connectors/storage/s3/airflow.md +++ b/openmetadata-docs/content/v1.0.0/connectors/storage/s3/airflow.md @@ -26,11 +26,54 @@ For all the buckets that we want to ingest, we need to provide the following: - `s3:ListBucket` - `s3:GetObject` - `s3:GetBucketLocation` +- `s3:ListAllMyBuckets` + +Note that the `Resources` should be all the buckets that you'd like to scan. A possible policy could be: + +```json +{ + "Version": "2012-10-17", + "Statement": [ + { + "Effect": "Allow", + "Action": [ + "s3:GetObject", + "s3:ListBucket", + "s3:GetBucketLocation", + "s3:ListAllMyBuckets" + ], + "Resource": [ + "arn:aws:s3:::*" + ] + } + ] +} +``` ### CloudWatch Permissions Which is used to fetch the total size in bytes for a bucket and the total number of files. It requires: - `cloudwatch:GetMetricData` +- `cloudwatch:ListMetrics` + +The policy would look like: + +```json +{ + "Version": "2012-10-17", + "Statement": [ + { + "Sid": "VisualEditor0", + "Effect": "Allow", + "Action": [ + "cloudwatch:GetMetricData", + "cloudwatch:ListMetrics" + ], + "Resource": "*" + } + ] +} +``` ### Python Requirements diff --git a/openmetadata-docs/content/v1.0.0/connectors/storage/s3/cli.md b/openmetadata-docs/content/v1.0.0/connectors/storage/s3/cli.md index 4e0b0f6a664..9cfe1f84333 100644 --- a/openmetadata-docs/content/v1.0.0/connectors/storage/s3/cli.md +++ b/openmetadata-docs/content/v1.0.0/connectors/storage/s3/cli.md @@ -33,11 +33,54 @@ For all the buckets that we want to ingest, we need to provide the following: - `s3:ListBucket` - `s3:GetObject` - `s3:GetBucketLocation` +- `s3:ListAllMyBuckets` + +Note that the `Resources` should be all the buckets that you'd like to scan. A possible policy could be: + +```json +{ + "Version": "2012-10-17", + "Statement": [ + { + "Effect": "Allow", + "Action": [ + "s3:GetObject", + "s3:ListBucket", + "s3:GetBucketLocation", + "s3:ListAllMyBuckets" + ], + "Resource": [ + "arn:aws:s3:::*" + ] + } + ] +} +``` ### CloudWatch Permissions Which is used to fetch the total size in bytes for a bucket and the total number of files. It requires: - `cloudwatch:GetMetricData` +- `cloudwatch:ListMetrics` + +The policy would look like: + +```json +{ + "Version": "2012-10-17", + "Statement": [ + { + "Sid": "VisualEditor0", + "Effect": "Allow", + "Action": [ + "cloudwatch:GetMetricData", + "cloudwatch:ListMetrics" + ], + "Resource": "*" + } + ] +} +``` ### Python Requirements diff --git a/openmetadata-docs/content/v1.0.0/connectors/storage/s3/index.md b/openmetadata-docs/content/v1.0.0/connectors/storage/s3/index.md index b24016512c1..66c1a5de0c7 100644 --- a/openmetadata-docs/content/v1.0.0/connectors/storage/s3/index.md +++ b/openmetadata-docs/content/v1.0.0/connectors/storage/s3/index.md @@ -52,11 +52,54 @@ For all the buckets that we want to ingest, we need to provide the following: - `s3:ListBucket` - `s3:GetObject` - `s3:GetBucketLocation` +- `s3:ListAllMyBuckets` + +Note that the `Resources` should be all the buckets that you'd like to scan. A possible policy could be: + +```json +{ + "Version": "2012-10-17", + "Statement": [ + { + "Effect": "Allow", + "Action": [ + "s3:GetObject", + "s3:ListBucket", + "s3:GetBucketLocation", + "s3:ListAllMyBuckets" + ], + "Resource": [ + "arn:aws:s3:::*" + ] + } + ] +} +``` ### CloudWatch Permissions Which is used to fetch the total size in bytes for a bucket and the total number of files. It requires: - `cloudwatch:GetMetricData` +- `cloudwatch:ListMetrics` + +The policy would look like: + +```json +{ + "Version": "2012-10-17", + "Statement": [ + { + "Sid": "VisualEditor0", + "Effect": "Allow", + "Action": [ + "cloudwatch:GetMetricData", + "cloudwatch:ListMetrics" + ], + "Resource": "*" + } + ] +} +``` ## Metadata Ingestion diff --git a/openmetadata-docs/content/v1.1.0-snapshot/connectors/storage/s3/airflow.md b/openmetadata-docs/content/v1.1.0-snapshot/connectors/storage/s3/airflow.md index 099d28bda1e..9e84465b68b 100644 --- a/openmetadata-docs/content/v1.1.0-snapshot/connectors/storage/s3/airflow.md +++ b/openmetadata-docs/content/v1.1.0-snapshot/connectors/storage/s3/airflow.md @@ -26,11 +26,54 @@ For all the buckets that we want to ingest, we need to provide the following: - `s3:ListBucket` - `s3:GetObject` - `s3:GetBucketLocation` +- `s3:ListAllMyBuckets` + +Note that the `Resources` should be all the buckets that you'd like to scan. A possible policy could be: + +```json +{ + "Version": "2012-10-17", + "Statement": [ + { + "Effect": "Allow", + "Action": [ + "s3:GetObject", + "s3:ListBucket", + "s3:GetBucketLocation", + "s3:ListAllMyBuckets" + ], + "Resource": [ + "arn:aws:s3:::*" + ] + } + ] +} +``` ### CloudWatch Permissions Which is used to fetch the total size in bytes for a bucket and the total number of files. It requires: - `cloudwatch:GetMetricData` +- `cloudwatch:ListMetrics` + +The policy would look like: + +```json +{ + "Version": "2012-10-17", + "Statement": [ + { + "Sid": "VisualEditor0", + "Effect": "Allow", + "Action": [ + "cloudwatch:GetMetricData", + "cloudwatch:ListMetrics" + ], + "Resource": "*" + } + ] +} +``` ### Python Requirements diff --git a/openmetadata-docs/content/v1.1.0-snapshot/connectors/storage/s3/cli.md b/openmetadata-docs/content/v1.1.0-snapshot/connectors/storage/s3/cli.md index 4e0b0f6a664..9cfe1f84333 100644 --- a/openmetadata-docs/content/v1.1.0-snapshot/connectors/storage/s3/cli.md +++ b/openmetadata-docs/content/v1.1.0-snapshot/connectors/storage/s3/cli.md @@ -33,11 +33,54 @@ For all the buckets that we want to ingest, we need to provide the following: - `s3:ListBucket` - `s3:GetObject` - `s3:GetBucketLocation` +- `s3:ListAllMyBuckets` + +Note that the `Resources` should be all the buckets that you'd like to scan. A possible policy could be: + +```json +{ + "Version": "2012-10-17", + "Statement": [ + { + "Effect": "Allow", + "Action": [ + "s3:GetObject", + "s3:ListBucket", + "s3:GetBucketLocation", + "s3:ListAllMyBuckets" + ], + "Resource": [ + "arn:aws:s3:::*" + ] + } + ] +} +``` ### CloudWatch Permissions Which is used to fetch the total size in bytes for a bucket and the total number of files. It requires: - `cloudwatch:GetMetricData` +- `cloudwatch:ListMetrics` + +The policy would look like: + +```json +{ + "Version": "2012-10-17", + "Statement": [ + { + "Sid": "VisualEditor0", + "Effect": "Allow", + "Action": [ + "cloudwatch:GetMetricData", + "cloudwatch:ListMetrics" + ], + "Resource": "*" + } + ] +} +``` ### Python Requirements diff --git a/openmetadata-docs/content/v1.1.0-snapshot/connectors/storage/s3/index.md b/openmetadata-docs/content/v1.1.0-snapshot/connectors/storage/s3/index.md index b24016512c1..66c1a5de0c7 100644 --- a/openmetadata-docs/content/v1.1.0-snapshot/connectors/storage/s3/index.md +++ b/openmetadata-docs/content/v1.1.0-snapshot/connectors/storage/s3/index.md @@ -52,11 +52,54 @@ For all the buckets that we want to ingest, we need to provide the following: - `s3:ListBucket` - `s3:GetObject` - `s3:GetBucketLocation` +- `s3:ListAllMyBuckets` + +Note that the `Resources` should be all the buckets that you'd like to scan. A possible policy could be: + +```json +{ + "Version": "2012-10-17", + "Statement": [ + { + "Effect": "Allow", + "Action": [ + "s3:GetObject", + "s3:ListBucket", + "s3:GetBucketLocation", + "s3:ListAllMyBuckets" + ], + "Resource": [ + "arn:aws:s3:::*" + ] + } + ] +} +``` ### CloudWatch Permissions Which is used to fetch the total size in bytes for a bucket and the total number of files. It requires: - `cloudwatch:GetMetricData` +- `cloudwatch:ListMetrics` + +The policy would look like: + +```json +{ + "Version": "2012-10-17", + "Statement": [ + { + "Sid": "VisualEditor0", + "Effect": "Allow", + "Action": [ + "cloudwatch:GetMetricData", + "cloudwatch:ListMetrics" + ], + "Resource": "*" + } + ] +} +``` ## Metadata Ingestion diff --git a/openmetadata-ui/src/main/resources/ui/public/locales/en-US/Storage/S3.md b/openmetadata-ui/src/main/resources/ui/public/locales/en-US/Storage/S3.md index f2de6dd9234..647e09f7bc5 100644 --- a/openmetadata-ui/src/main/resources/ui/public/locales/en-US/Storage/S3.md +++ b/openmetadata-ui/src/main/resources/ui/public/locales/en-US/Storage/S3.md @@ -14,11 +14,54 @@ For all the buckets that we want to ingest, we need to provide the following: - `s3:ListBucket` - `s3:GetObject` - `s3:GetBucketLocation` +- `s3:ListAllMyBuckets` + +Note that the `Resources` should be all the buckets that you'd like to scan. A possible policy could be: + +```json +{ + "Version": "2012-10-17", + "Statement": [ + { + "Effect": "Allow", + "Action": [ + "s3:GetObject", + "s3:ListBucket", + "s3:GetBucketLocation", + "s3:ListAllMyBuckets" + ], + "Resource": [ + "arn:aws:s3:::*" + ] + } + ] +} +``` ### CloudWatch Permissions Which is used to fetch the total size in bytes for a bucket and the total number of files. It requires: - `cloudwatch:GetMetricData` +- `cloudwatch:ListMetrics` + +The policy would look like: + +```json +{ + "Version": "2012-10-17", + "Statement": [ + { + "Sid": "VisualEditor0", + "Effect": "Allow", + "Action": [ + "cloudwatch:GetMetricData", + "cloudwatch:ListMetrics" + ], + "Resource": "*" + } + ] +} +``` You can find further information on the Kafka connector in the [docs](https://docs.open-metadata.org/connectors/storage/s3).