Java 中进行Amazon S3 Multipart 上传

1. Overview

In this tutorial, we’ll see how to handle multipart uploads in Amazon S3 with AWS Java SDK.

Simply put, in a multipart upload, we split the content into smaller parts and upload each part individually. All parts are re-assembled when received.

Multipart uploads offer the following advantages:

Higher throughput – we can upload parts in parallel
Easier error recovery – we need to re-upload only the failed parts
Pause and resume uploads – we can upload parts at any point in time. The whole process can be paused and remaining parts can be uploaded later

Note that when using multipart upload with Amazon S3, each part except the last part must be at least 5 MB in size.

2. Maven Dependencies

Before we begin, we need to add the AWS SDK dependency in our project:

<dependency>
    <groupId>software.amazon.awssdk</groupId>
    <artifactId>s3</artifactId>
    <version>2.24.9</version>
</dependency>

To view the latest version, check out Maven Central.

3. Performing Multipart Upload

3.1. Creating Amazon S3 Client

First, we need to create a client for accessing Amazon S3. We’ll use the AmazonS3ClientBuilder for this purpose:

AmazonS3 amazonS3 = AmazonS3ClientBuilder
  .standard()
  .withCredentials(new DefaultAWSCredentialsProviderChain())
  .withRegion(Regions.DEFAULT_REGION)
  .build();

This creates a client using the default credential provider chain for accessing AWS credentials.

For more details on how the default credential provider chain works, please see the documentation. If you’re using a region other than the default (US West-2), make sure you replace Regions.DEFAULT_REGION with that custom region.

3.2. Uploading Object

Firstly we need to create a CreateMultipartUploadRequest instance and pass to it the bucket name and the key. Then, we can call the S3Client’s createMultipartUpload() method.

The s3.createMultipartUpload() method is called to initiate the multipart upload. It returns a CreateMultipartUploadResponse object. When we get the response, we can extract the upload Id.

// Initiate a multipart upload
CreateMultipartUploadRequest createRequest = CreateMultipartUploadRequest.builder()
    .bucket(existingBucketName)
    .key(keyName)
    .build();

CreateMultipartUploadResponse createResponse = s3.createMultipartUpload(createRequest);

String uploadId = createResponse.uploadId();

3.3. Prepare and Upload Each Part

Now, we will use the ByteBuffer with a size of 5MB to hold each part of the file. For each part, we will create an UploadPartRequest object with the bucket name, key name, upload ID, part number, and content length. Then, we will call the uploadPart() method and pass it the request. As a result of this method invocation, we will get a UploadPartResponse, extract the information, and add it to the completed parts list.

// Prepare the parts to be uploaded
List<CompletedPart> completedParts = new ArrayList<>();
int partNumber = 1;
ByteBuffer buffer = ByteBuffer.allocate(5 * 1024 * 1024); // Set your preferred part size (5 MB in this example)

// Read the file and upload each part
try (RandomAccessFile file = new RandomAccessFile(filePath, "r")) {
    long fileSize = file.length();
    long position = 0;

    while (position < fileSize) {
        file.seek(position);
        int bytesRead = file.getChannel().read(buffer);

        buffer.flip();
        UploadPartRequest uploadPartRequest = UploadPartRequest.builder()
            .bucket(existingBucketName)
            .key(keyName)
            .uploadId(uploadId)
            .partNumber(partNumber)
            .contentLength((long) bytesRead)
            .build();

        UploadPartResponse response = s3.uploadPart(uploadPartRequest, RequestBody.fromByteBuffer(buffer));

        completedParts.add(CompletedPart.builder()
            .partNumber(partNumber)
            .eTag(response.eTag())
            .build());

        buffer.clear();
        position += bytesRead;
        partNumber++;
    }
} catch (IOException e) {
    e.printStackTrace();
}

3.4. Complete the Multipart Upload

Once the file upload is complete, we instantiate a CreateMultipartUploadResponse object with the list of completed parts. The CompletedMultipartUpload object then is passed to CompleteMultipartUploadRequest object with the bucket name, key name, and upload ID. The s3.completeMultipartUpload() method is called to complete the multipart upload and returns a CompleteMultipartUploadResponse object.

// Complete the multipart upload
CompletedMultipartUpload completedUpload = CompletedMultipartUpload.builder()
    .parts(completedParts)
    .build();

CompleteMultipartUploadRequest completeRequest = CompleteMultipartUploadRequest.builder()
    .bucket(existingBucketName)
    .key(keyName)
    .uploadId(uploadId)
    .multipartUpload(completedUpload)
    .build();

CompleteMultipartUploadResponse completeResponse = s3.completeMultipartUpload(completeRequest);

3.5. Verify Upload

Now that we uploded the file, let’s verify it’s existence:

String objectUrl = s3.utilities().getUrl(GetUrlRequest.builder()
        .bucket(existingBucketName)
        .key(keyName)
        .build())
    .toExternalForm();

System.out.println("Uploaded object URL: " + objectUrl);

4. Conclusion

In this quick article, we learned how to perform multipart uploads using AWS SDK for Java.

As always, the complete code of this article is available over on GitHub.

Persistence

REST

Security