yujunjun/OpenMetadata
1
0
Fork 0
mirror of https://github.com/open-metadata/OpenMetadata.git synced 2025-12-05 20:15:15 +00:00
Code Issues Packages Projects Releases Wiki Activity

Update S3 storage permissions (#12064)

Browse Source
This commit is contained in:
Pere Miquel Brull 2023-06-28 00:10:02 +02:00 committed by GitHub
parent d9cd484657
commit 49114cdad4
No known key found for this signature in database
GPG Key ID: 4AEE18F83AFDEB23
7 changed files with 301 additions and 0 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

43
openmetadata-docs/content/v1.0.0/connectors/storage/s3/airflow.md
View File

@ -26,11 +26,54 @@ For all the buckets that we want to ingest, we need to provide the following:
- `s3:ListBucket`
- `s3:GetObject`
- `s3:GetBucketLocation`
- `s3:ListAllMyBuckets`
Note that the `Resources` should be all the buckets that you'd like to scan. A possible policy could be:
```json
{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Action": [
"s3:GetObject",
"s3:ListBucket",
"s3:GetBucketLocation",
"s3:ListAllMyBuckets"
],
"Resource": [
"arn:aws:s3:::*"
]
}
]
}
```
### CloudWatch Permissions
Which is used to fetch the total size in bytes for a bucket and the total number of files. It requires:
- `cloudwatch:GetMetricData`
- `cloudwatch:ListMetrics`
The policy would look like:
```json
{
"Version": "2012-10-17",
"Statement": [
{
"Sid": "VisualEditor0",
"Effect": "Allow",
"Action": [
"cloudwatch:GetMetricData",
"cloudwatch:ListMetrics"
],
"Resource": "*"
}
]
}
```
### Python Requirements

43
openmetadata-docs/content/v1.0.0/connectors/storage/s3/cli.md
View File

@ -33,11 +33,54 @@ For all the buckets that we want to ingest, we need to provide the following:
- `s3:ListBucket`
- `s3:GetObject`
- `s3:GetBucketLocation`
- `s3:ListAllMyBuckets`
Note that the `Resources` should be all the buckets that you'd like to scan. A possible policy could be:
```json
{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Action": [
"s3:GetObject",
"s3:ListBucket",
"s3:GetBucketLocation",
"s3:ListAllMyBuckets"
],
"Resource": [
"arn:aws:s3:::*"
]
}
]
}
```
### CloudWatch Permissions
Which is used to fetch the total size in bytes for a bucket and the total number of files. It requires:
- `cloudwatch:GetMetricData`
- `cloudwatch:ListMetrics`
The policy would look like:
```json
{
"Version": "2012-10-17",
"Statement": [
{
"Sid": "VisualEditor0",
"Effect": "Allow",
"Action": [
"cloudwatch:GetMetricData",
"cloudwatch:ListMetrics"
],
"Resource": "*"
}
]
}
```
### Python Requirements

43
openmetadata-docs/content/v1.0.0/connectors/storage/s3/index.md
View File

@ -52,11 +52,54 @@ For all the buckets that we want to ingest, we need to provide the following:
- `s3:ListBucket`
- `s3:GetObject`
- `s3:GetBucketLocation`
- `s3:ListAllMyBuckets`
Note that the `Resources` should be all the buckets that you'd like to scan. A possible policy could be:
```json
{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Action": [
"s3:GetObject",
"s3:ListBucket",
"s3:GetBucketLocation",
"s3:ListAllMyBuckets"
],
"Resource": [
"arn:aws:s3:::*"
]
}
]
}
```
### CloudWatch Permissions
Which is used to fetch the total size in bytes for a bucket and the total number of files. It requires:
- `cloudwatch:GetMetricData`
- `cloudwatch:ListMetrics`
The policy would look like:
```json
{
"Version": "2012-10-17",
"Statement": [
{
"Sid": "VisualEditor0",
"Effect": "Allow",
"Action": [
"cloudwatch:GetMetricData",
"cloudwatch:ListMetrics"
],
"Resource": "*"
}
]
}
```
## Metadata Ingestion

43
openmetadata-docs/content/v1.1.0-snapshot/connectors/storage/s3/airflow.md
View File

@ -26,11 +26,54 @@ For all the buckets that we want to ingest, we need to provide the following:
- `s3:ListBucket`
- `s3:GetObject`
- `s3:GetBucketLocation`
- `s3:ListAllMyBuckets`
Note that the `Resources` should be all the buckets that you'd like to scan. A possible policy could be:
```json
{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Action": [
"s3:GetObject",
"s3:ListBucket",
"s3:GetBucketLocation",
"s3:ListAllMyBuckets"
],
"Resource": [
"arn:aws:s3:::*"
]
}
]
}
```
### CloudWatch Permissions
Which is used to fetch the total size in bytes for a bucket and the total number of files. It requires:
- `cloudwatch:GetMetricData`
- `cloudwatch:ListMetrics`
The policy would look like:
```json
{
"Version": "2012-10-17",
"Statement": [
{
"Sid": "VisualEditor0",
"Effect": "Allow",
"Action": [
"cloudwatch:GetMetricData",
"cloudwatch:ListMetrics"
],
"Resource": "*"
}
]
}
```
### Python Requirements

43
openmetadata-docs/content/v1.1.0-snapshot/connectors/storage/s3/cli.md
View File

@ -33,11 +33,54 @@ For all the buckets that we want to ingest, we need to provide the following:
- `s3:ListBucket`
- `s3:GetObject`
- `s3:GetBucketLocation`
- `s3:ListAllMyBuckets`
Note that the `Resources` should be all the buckets that you'd like to scan. A possible policy could be:
```json
{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Action": [
"s3:GetObject",
"s3:ListBucket",
"s3:GetBucketLocation",
"s3:ListAllMyBuckets"
],
"Resource": [
"arn:aws:s3:::*"
]
}
]
}
```
### CloudWatch Permissions
Which is used to fetch the total size in bytes for a bucket and the total number of files. It requires:
- `cloudwatch:GetMetricData`
- `cloudwatch:ListMetrics`
The policy would look like:
```json
{
"Version": "2012-10-17",
"Statement": [
{
"Sid": "VisualEditor0",
"Effect": "Allow",
"Action": [
"cloudwatch:GetMetricData",
"cloudwatch:ListMetrics"
],
"Resource": "*"
}
]
}
```
### Python Requirements

43
openmetadata-docs/content/v1.1.0-snapshot/connectors/storage/s3/index.md
View File

@ -52,11 +52,54 @@ For all the buckets that we want to ingest, we need to provide the following:
- `s3:ListBucket`
- `s3:GetObject`
- `s3:GetBucketLocation`
- `s3:ListAllMyBuckets`
Note that the `Resources` should be all the buckets that you'd like to scan. A possible policy could be:
```json
{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Action": [
"s3:GetObject",
"s3:ListBucket",
"s3:GetBucketLocation",
"s3:ListAllMyBuckets"
],
"Resource": [
"arn:aws:s3:::*"
]
}
]
}
```
### CloudWatch Permissions
Which is used to fetch the total size in bytes for a bucket and the total number of files. It requires:
- `cloudwatch:GetMetricData`
- `cloudwatch:ListMetrics`
The policy would look like:
```json
{
"Version": "2012-10-17",
"Statement": [
{
"Sid": "VisualEditor0",
"Effect": "Allow",
"Action": [
"cloudwatch:GetMetricData",
"cloudwatch:ListMetrics"
],
"Resource": "*"
}
]
}
```
## Metadata Ingestion

43
openmetadata-ui/src/main/resources/ui/public/locales/en-US/Storage/S3.md
View File

@ -14,11 +14,54 @@ For all the buckets that we want to ingest, we need to provide the following:
- `s3:ListBucket`
- `s3:GetObject`
- `s3:GetBucketLocation`
- `s3:ListAllMyBuckets`
Note that the `Resources` should be all the buckets that you'd like to scan. A possible policy could be:
```json
{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Action": [
"s3:GetObject",
"s3:ListBucket",
"s3:GetBucketLocation",
"s3:ListAllMyBuckets"
],
"Resource": [
"arn:aws:s3:::*"
]
}
]
}
```
### CloudWatch Permissions
Which is used to fetch the total size in bytes for a bucket and the total number of files. It requires:
- `cloudwatch:GetMetricData`
- `cloudwatch:ListMetrics`
The policy would look like:
```json
{
"Version": "2012-10-17",
"Statement": [
{
"Sid": "VisualEditor0",
"Effect": "Allow",
"Action": [
"cloudwatch:GetMetricData",
"cloudwatch:ListMetrics"
],
"Resource": "*"
}
]
}
```
You can find further information on the Kafka connector in the [docs](https://docs.open-metadata.org/connectors/storage/s3).