-
-
Notifications
You must be signed in to change notification settings - Fork 2.3k
AWS CLI with SeaweedFS
Er2 edited this page Oct 3, 2024
·
7 revisions
See https://aws.amazon.com/cli/
On mac or linux: pip install awscli
Configure for AWS. The key values do not matter.
$ aws configure
AWS Access Key ID [None]: any
AWS Secret Access Key [None]: any
Default region name [None]: us-east-1
Default output format [None]:
Also, make sure using AWS signature version 4:
$ aws configure set default.s3.signature_version s3v4
This is undefined behavior as AWS S3 servers always have sub-domains instead of sub-paths.
Use this only if you can't create (sub-)domain and use other port!
AWS CLI appends sub-path before actual path so need to add X-Forwarded-Prefix
header (set to /s3
for example)
Example for Caddy web server
redir /s3 /s3/
handle_path /s3/* {
reverse_proxy localhost:8333 {
header_up X-Forwarded-Prefix /s3
}
}
# list buckets
$ aws --endpoint-url http://localhost:8333 s3 ls
2019-01-02 01:59:25 newbucket
# list files inside the bucket
$ aws --endpoint-url http://localhost:8333 s3 ls s3://newbucket
2019-01-02 12:52:44 6804 password
# make a bucket
$ aws --endpoint-url http://localhost:8333 s3 mb s3://newbucket3
make_bucket: newbucket3
# add an object
$ aws --endpoint-url http://localhost:8333 s3 cp /etc/passwd s3://newbucket3
upload: ../../../../../etc/passwd to s3://newbucket3/passwd
# copy an object
$ aws --endpoint-url http://localhost:8333 s3 cp s3://newbucket3/passwd s3://newbucket3/passwd.txt
copy: s3://newbucket3/passwd to s3://newbucket3/passwd.txt
# remove an object
$ aws --endpoint-url http://localhost:8333 s3 rm s3://newbucket3/passwd
delete: s3://newbucket3/passwd
# remove a bucket
$ aws --endpoint-url http://localhost:8333 s3 rb s3://newbucket3
remove_bucket: newbucket3
If authentication is enabled, the url is not accessible without proper credentials. But you can presign a url and access it.
# presign url, default to 1 hour
$ aws --endpoint-url http://localhost:8333 s3 presign s3://newbucket/t.txt
http://localhost:8333/newbucket/t.txt?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=some_access_key1%2F20200726%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20200726T161749Z&X-Amz-Expires=3600&X-Amz-SignedHeaders=host&X-Amz-Signature=e0cc153209e414ca8168661f57827aa03ab84e7041ef9270ff639bcc519d24f5
# access the url
$ curl "http://localhost:8333/newbucket/t.txt?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=some_access_key1%2F20200726%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20200726T161749Z&X-Amz-Expires=3600&X-Amz-SignedHeaders=host&X-Amz-Signature=e0cc153209e414ca8168661f57827aa03ab84e7041ef9270ff639bcc519d24f5"
- Replication
- Store file with a Time To Live
- Failover Master Server
- Erasure coding for warm storage
- Server Startup Setup
- Environment Variables
- Filer Setup
- Directories and Files
- Data Structure for Large Files
- Filer Data Encryption
- Filer Commands and Operations
- Filer JWT Use
- Filer Cassandra Setup
- Filer Redis Setup
- Super Large Directories
- Path-Specific Filer Store
- Choosing a Filer Store
- Customize Filer Store
- Migrate to Filer Store
- Add New Filer Store
- Filer Store Replication
- Filer Active Active cross cluster continuous synchronization
- Filer as a Key-Large-Value Store
- Path Specific Configuration
- Filer Change Data Capture
- Cloud Drive Benefits
- Cloud Drive Architecture
- Configure Remote Storage
- Mount Remote Storage
- Cache Remote Storage
- Cloud Drive Quick Setup
- Gateway to Remote Object Storage
- Amazon S3 API
- AWS CLI with SeaweedFS
- s3cmd with SeaweedFS
- rclone with SeaweedFS
- restic with SeaweedFS
- nodejs with Seaweed S3
- S3 API Benchmark
- S3 API FAQ
- S3 Bucket Quota
- S3 API Audit log
- S3 Nginx Proxy
- Docker Compose for S3
- Hadoop Compatible File System
- run Spark on SeaweedFS
- run HBase on SeaweedFS
- run Presto on SeaweedFS
- Hadoop Benchmark
- HDFS via S3 connector
- Async Replication to another Filer [Deprecated]
- Async Backup
- Async Filer Metadata Backup
- Async Replication to Cloud [Deprecated]
- Kubernetes Backups and Recovery with K8up