1. Introduction
In this tutorial, we’ll learn how to interact with the Amazon S3 (Simple Storage Service) storage system programmatically from Java.
Remember that S3 has a very simple structure; each bucket can store any number of objects, which can be accessed using either a SOAP interface or a REST-style API.
Going forward, we’ll use the AWS SDK for Java to create, list, and delete S3 buckets. We’ll also upload, list, download, copy, move, rename and delete objects within these buckets.
2. Maven Dependencies
Before we get started, we need to declare the AWS SDK dependency in our project:
<dependency>
<groupId>software.amazon.awssdk</groupId>
<artifactId>s3</artifactId>
<version>2.20.52</version>
</dependency>
To view the latest version, we can check Maven Central.
3. Prerequisites
To use AWS SDK, we’ll need a few things:
- AWS Account: we need an Amazon Web Services account. If we don’t have one, we can go ahead and create an account.
- AWS Security Credentials: These are our access keys that allow us to make programmatic calls to AWS API actions. We can get these credentials in two ways, either by using AWS root account credentials from the access keys section of the Security Credentials page, or by using IAM user credentials from the IAM console.
- Choosing AWS Region: We also have to select the AWS region(s) where we want to store our Amazon S3 data. Keep in mind that S3 storage prices vary by region. For more details, head over to the official documentation. In this tutorial, we’ll use US East (Ohio, region us-east-2).
4. Creating Client Connection
First, we need to create a client connection to access the Amazon S3 web service. We’ll use the AmazonS3 interface for this purpose:
AWSCredentials credentials = new BasicAWSCredentials(
"<AWS accesskey>",
"<AWS secretkey>"
);
Then we’ll configure the client:
AmazonS3 s3client = AmazonS3ClientBuilder
.standard()
.withCredentials(new AWSStaticCredentialsProvider(credentials))
.withRegion(Regions.US_EAST_2)
.build();
5. Amazon S3 Bucket Operations
5.1. Creating a Bucket
It’s important to note that the bucket namespace is shared by all users of the system. So our bucket name must be unique across all existing bucket names in Amazon S3 (we’ll find out how to check that in just a moment).
In addition, as specified in the official documentation, the Bucket names must comply with the following requirements:
- names shouldn’t contain underscores
- names should be between 3 and 63 characters long
- names shouldn’t end with a dash
- names can’t contain adjacent periods
- names can’t contain dashes next to periods (e.g., “my-.bucket.com” and “my.-bucket” are invalid)
- names can’t contain uppercase characters
Now let’s create a bucket:
String bucketName = "baeldung-bucket";
if(s3client.doesBucketExist(bucketName)) {
LOG.info("Bucket name is not available."
+ " Try again with a different Bucket name.");
return;
}
CreateBucketRequest bucketRequest = CreateBucketRequest.builder()
.bucket(bucketName)
.build();
s3Client.createBucket(bucketRequest);
Before we create a bucket, we have to check whether our bucket name is available or not by using the doesBucketExist() method. If the name is available, then we’ll build a CreateBucketRequest and provide the bucket name. The last step is to pass the bucketRequest to the S3Client’s createBucket of CreateBucketRequest createBucketRequest.
5.2. Listing Buckets
Now that we’ve created a few buckets, let’s print a list of all the buckets available in our S3 environment using the listBuckets() method. This method will return a ListBucketsResponse, containing information about the buckets.
ListBucketsResponse listBucketsResponse = s3Client.listBuckets();
// Display the bucket names
List<Bucket> buckets = listBucketsResponse.buckets();
System.out.println("Buckets:");
for (Bucket bucket : buckets) {
System.out.println(bucket.name());
}
This will list all the buckets that are present in our S3 environment:
baeldung-bucket
baeldung-bucket-test2
elasticbeanstalk-us-east-2
5.3. Deleting a Bucket
It’s important to ensure that our bucket is empty before we delete it. Otherwise, an exception will be thrown.
Firstly, we need to build a DeleBucketRequest instance and pass to it the bucket name. Then, we call the deleteBucket method on the s3Client object, passing the request as an argument.
Also, note that only the owner of a bucket can delete it, regardless of its permissions (Access Control Policies):
try {
DeleteBucketRequest deleteBucketRequest = DeleteBucketRequest.builder()
.bucket(bucketName)
.build();
s3Client.deleteBucket(deleteBucketRequest);
System.out.println("Successfully deleted bucket : " + bucketName);
} catch (S3Exception e) {
System.err.println(e.getMessage());
System.exit(1);
}
6. Amazon S3 Object Operations
A file or collection of data inside an Amazon S3 bucket is known as an object. We can perform several operations on objects like uploading, listing, downloading, copying, moving, renaming and deleting.
6.1. Uploading Objects
Uploading an object is a pretty straightforward process. First, we will build a PutObjectRequest instance, specifying the bucket name and the key. Then, we will pass that request and the path to the file containing the data to s3Client’s putObject method:
PutObjectRequest request = PutObjectRequest.builder()
.bucket(bucketName)
.key(key)
.build();
return s3Client.putObject(request, Path.of(file.toURI()) );
6.2. Listing Objects
We’ll use the listObjects() method to list all the available objects in our S3 bucket:
ListObjectsV2Request listObjectsV2Request = ListObjectsV2Request.builder()
.bucket(bucketName)
.build();
ListObjectsV2Response listObjectsV2Response = s3Client.listObjectsV2(listObjectsV2Request);
List<S3Object> contents = listObjectsV2Response.contents();
System.out.println("Number of objects in the bucket: " + contents.stream().count());
contents.stream().forEach(System.out::println);
To list objects from an AWS S3 bucket, we need to create a ListObjectsV2Request instance and specify the bucket name. Then, we call the listObjectsV2 method on the s3Client object, passing the request as an argument. This method returns a ListObjectsV2Response, containing information about the objects in the bucket.
6.3. Downloading an Object
To download an object, we’ll first use the create a GetObjectRequest instance and pass it the bucket name and key as input parameters. Then, we will provide that to getObjectAsBytes() method and get back the response. Once we get the response, we can extract the array of bytes. The last step is to process the array of bytes:
GetObjectRequest objectRequest = GetObjectRequest.builder()
.bucket(bucketName)
.key(objectKey)
.build();
ResponseBytes<GetObjectResponse> responseResponseBytes = s3Client.getObjectAsBytes(objectRequest);
byte[] data = responseResponseBytes.asByteArray();
// Write the data to a local file.
java.io.File myFile = new java.io.File("/Users/user/Desktop/hello.txt" );
OutputStream os = new FileOutputStream(myFile);
os.write(data);
System.out.println("Successfully obtained bytes from an S3 object");
os.close();
6.4. Copying, Renaming, and Moving an Object
We can copy an object by calling the copyObject() method on our s3client, which accepts a CopyObjectRequest instance. So, CopyObjectRequest accepts four parameters:
- source bucket name
- object key in source bucket
- destination bucket name (it can be same as source)
- object key in destination bucket
CopyObjectRequest copyObjectRequest = CopyObjectRequest.builder()
.sourceBucket(sourceBucketName)
.sourceKey(sourceKey)
.destinationBucket(destinationBucketName)
.destinationKey(destinationKey)
.build();
return s3Client.copyObject(copyObjectRequest);
Note: We can use a combination of the copyObject() method and deleteObject() for performing moving and renaming tasks. This will involve copying the object first and then deleting it from its old location.
6.5. Deleting an Object
To delete an Object, we’ll call the deleteObject() method on the s3client and pass the DeleteObjectRequest instance. For the creation of DeleteObjectRequest instance, we need to pass the key of the object that we want to delete and the bucket name:
DeleteObjectRequest deleteObjectRequest = DeleteObjectRequest.builder()
.bucket(bucketName)
.key(objectKey)
.build();
s3Client.deleteObject(deleteObjectRequest);
6.6. Deleting Multiple Objects
To delete multiple objects at once, we’ll first create the DeleteObjectsRequest object and pass the bucket. Then we’ll pass an ArrayList of all the object keys that we want to delete.
Once we have this DeleteObjectsRequest object, we can pass it to the deleteObjects() method of our s3client as an argument. If successful, it’ll delete all the objects that we supplied:
ArrayList<ObjectIdentifier> toDelete = new ArrayList<>();
for(String objKey : keys) {
toDelete.add(ObjectIdentifier.builder()
.key(objKey)
.build());
}
DeleteObjectsRequest deleteObjectRequest = DeleteObjectsRequest.builder()
.bucket(bucketName)
.delete(Delete.builder()
.objects(toDelete).build())
.build();
s3Client.deleteObjects(deleteObjectRequest);
7. Conclusion
In this article, we focused on the basics of interacting with the Amazon S3 web service, both at the bucket and object level.
As always, the full implementation of this article can be found over on Github.