On this page

Object Storage

1 Introduction

Micronaut Object Storage provides a uniform API to create, read, delete and generate pre-authorized requests for objects in the major cloud providers:

There is also a local storage implementation for testing purposes.

Using this API enables the creation of truly multi-cloud, portable applications.

2 Release History

For this project, you can find a list of releases (with release notes) here:

https://github.com/micronaut-projects/micronaut-object-storage/releases

3 Quick Start

To get started, you need to declare a dependency for the actual cloud provider you are using. See the actual cloud provider documentation for more details:

Amazon S3.
Azure Blob Storage.
Google Cloud Storage.
Oracle Cloud Infrastructure (OCI) Object Storage.

Then, you can inject in your controllers/services/etc. a bean of type ObjectStorageOperations, the parent interface that allows you to use the API in a generic way for all cloud providers:

@Singleton
public class ProfileService {

    private static final Logger LOG = LoggerFactory.getLogger(ProfileService.class);

    private final ObjectStorageOperations<?, ?, ?> objectStorage;

    public ProfileService(ObjectStorageOperations<?, ?, ?> objectStorage) {
        this.objectStorage = objectStorage;
    }

}

@Singleton
open class ProfileService(private val objectStorage: ObjectStorageOperations<*, *, *>) {

}

@Singleton
class ProfileService {

    final ObjectStorageOperations<?, ?, ?> objectStorage

    ProfileService(ObjectStorageOperations<?, ?, ?> objectStorage) {
        this.objectStorage = objectStorage
    }

}

If your application is not multi-cloud, and/or you need cloud-specific details, you can use a concrete implementation. For example, for AWS S3:

@Controller
public class UploadController {

    private final AwsS3Operations objectStorage;

    public UploadController(AwsS3Operations objectStorage) {
        this.objectStorage = objectStorage;
    }

}

@Controller
open class UploadController(private val objectStorage: AwsS3Operations) {

}

@Controller
class UploadController {

    final AwsS3Operations objectStorage

    UploadController(AwsS3Operations objectStorage) {
        this.objectStorage = objectStorage
    }

}

If you have multiple object storages configured, it is possible to select which one to work with via bean qualifiers.

For example, given the following configuration:

src/main/resources/application-ec2.yml

micronaut.object-storage.aws.pictures.bucket=pictures-bucket
micronaut.object-storage.aws.logos.bucket=logos-bucket

micronaut:
  object-storage:
    aws:
      pictures:
        bucket: pictures-bucket
      logos:
        bucket: logos-bucket

[micronaut.object-storage.aws.pictures]
bucket = "pictures-bucket"

[micronaut.object-storage.aws.logos]
bucket = "logos-bucket"

micronaut {
  objectStorage {
    aws {
      pictures {
        bucket = 'pictures-bucket'
      }
      logos {
        bucket = 'logos-bucket'
      }
    }
  }
}

{
  micronaut = {
    object-storage = {
      aws = {
        pictures = {
          bucket = "pictures-bucket"
        }
        logos = {
          bucket = "logos-bucket"
        }
      }
    }
  }
}

{
  "micronaut": {
    "object-storage": {
      "aws": {
        "pictures": {
          "bucket": "pictures-bucket"
        },
        "logos": {
          "bucket": "logos-bucket"
        }
      }
    }
  }
}

You then need to use @Named("pictures") or @Named("logos") to specify which of the object storages you want to use.

Uploading files

public String saveProfilePicture(String userId, Path path) {
    UploadRequest request = UploadRequest.fromPath(path, userId); // 
    UploadResponse<?> response = objectStorage.upload(request); // 
    return response.getKey(); // 
}

open fun saveProfilePicture(userId: String, path: Path): String? {
    val request = UploadRequest.fromPath(path, userId) // 
    val response = objectStorage.upload(request) // 
    return response.key // 
}

String saveProfilePicture(String userId, Path path) {
    UploadRequest request = UploadRequest.fromPath(path, userId) // 
    UploadResponse response = objectStorage.upload(request) // 
    response.key // 
}

You can use any of the UploadRequest static methods to build an upload request.
The upload operation returns an UploadResponse, which wraps the cloud-specific SDK response object.
The response object contains some common properties for all cloud vendor, and a getNativeResponse() method that can be used for accessing the vendor-specific response object.

In case you want to have better control of the upload options used, you can use the method upload(UploadRequest, Consumer) of ObjectStorageOperations, which will give you access to the cloud vendor-specific request class or builder.

For example, for AWS S3:

UploadResponse<PutObjectResponse> response = objectStorage.upload(objectStorageUpload, builder -> {
    builder.acl(ObjectCannedACL.PUBLIC_READ);
});

val response = objectStorage.upload(objectStorageUpload) { builder: PutObjectRequest.Builder ->
    builder.acl(ObjectCannedACL.PUBLIC_READ)
}

UploadResponse<PutObjectResponse> response = objectStorage.upload(objectStorageUpload, { builder ->
    builder.acl(ObjectCannedACL.PUBLIC_READ)
})

Retrieving files

public Optional<Path> retrieveProfilePicture(String userId, String fileName) {
    Path destination = null;
    try {
        String key = userId + "/" + fileName;
        Optional<InputStream> stream = objectStorage.retrieve(key) // 
            .map(ObjectStorageEntry::getInputStream);
        if (stream.isPresent()) {
            destination = File.createTempFile(userId, "temp").toPath();
            Files.copy(stream.get(), destination, StandardCopyOption.REPLACE_EXISTING);
            return Optional.of(destination);
        } else {
            return Optional.empty();
        }
    } catch (IOException e) {
        LOG.error("Error while trying to save profile picture to the local file [{}]: {}", destination, e.getMessage());
        return Optional.empty();
    }
}

open fun retrieveProfilePicture(userId: String, fileName: String): Path? {
    val key = "$userId/$fileName"
    val stream = objectStorage.retrieve<ObjectStorageEntry<*>>(key) // 
        .map { obj: ObjectStorageEntry<*> -> obj.inputStream }

    return if (stream.isPresent) {
        val destination = File.createTempFile(userId, "temp").toPath()
        Files.copy(stream.get(), destination, StandardCopyOption.REPLACE_EXISTING)
        destination
    } else {
        null
    }
}

Optional<Path> retrieveProfilePicture(String userId, String fileName) {
    String key = "${userId}/${fileName}"
    Optional<InputStream> stream = objectStorage.retrieve(key) // 
            .map(ObjectStorageEntry::getInputStream)
    if (stream.isPresent()) {
        Path destination = File.createTempFile(userId, "temp").toPath()
        Files.copy(stream.get(), destination, StandardCopyOption.REPLACE_EXISTING)
        return Optional.of(destination)
    } else {
        return Optional.empty()
    }
}

The retrieve operation returns an ObjectStorageEntry, from which you can get an InputStream. There is also a getNativeEntry() method that gives you access to the cloud vendor-specific response object.

Deleting files

public void deleteProfilePicture(String userId, String fileName) {
    String key = userId + "/" + fileName;
    objectStorage.delete(key); // 
}

open fun deleteProfilePicture(userId: String, fileName: String) {
    val key = "$userId/$fileName"
    objectStorage.delete(key) // 
}

void deleteProfilePicture(String userId, String fileName) {
    String key = "${userId}/${fileName}"
    objectStorage.delete(key) // 
}

The delete operation returns the cloud vendor-specific delete response object in case you need it.

Pre-authorized requests

Every cloud supports the creation of URLs that act as bearer tokens: anyone with the URL can upload or download from that object using only the URL and no other credentials. Micronaut abstracts this capability behind a uniform interface.

public String generateDownloadUrl(String userId, String fileName) {
    String key = userId + "/" + fileName;
    PresignRequest request = PresignRequest.builder(key, PresignRequest.Operation.DOWNLOAD)
        .expiresIn(Duration.ofMinutes(15)) // 
        .build();
    PresignResponse response = objectStorage.presign(request); // 
    return response.url().toString(); // 
}

open fun generateDownloadUrl(userId: String, fileName: String): String {
    val key = "$userId/$fileName"
    val request = PresignRequest.builder(key, PresignRequest.Operation.DOWNLOAD)
        .expiresIn(Duration.ofMinutes(15)) // 
        .build()
    val response: PresignResponse = objectStorage.presign(request) // 
    return response.url().toString() // 
}

String generateDownloadUrl(String userId, String fileName) {
    String key = "${userId}/${fileName}"
    PresignRequest request = PresignRequest.builder(key, PresignRequest.Operation.DOWNLOAD)
            .expiresIn(Duration.ofMinutes(15)) // 
            .build()
    PresignResponse response = objectStorage.presign(request) // 
    response.url().toString() // 
}

Build a PresignRequest with the object key and desired operation (DOWNLOAD or UPLOAD). You can optionally specify a custom expiration via expiresIn. For uploads, you may also provide contentLength and contentType hints.
The method presign(PresignRequest) of ObjectStorageOperations returns an PresignResponse containing the generated URL and its expiration time.
Use url() on the response to obtain the URL.

public void invalidatePresignedUrl(PresignResponse presignResponse) {
    objectStorage.invalidatePresignedRequest(presignResponse); // 
}

open fun invalidatePresignedUrl(presignResponse: PresignResponse) {
    objectStorage.invalidatePresignedRequest(presignResponse) // 
}

void invalidatePresignedUrl(PresignResponse presignResponse) {
    objectStorage.invalidatePresignedRequest(presignResponse) // 
}

To proactively revoke a previously generated URL, use invalidatePresignedRequest(presignResponse) of ObjectStorageOperations. Only a few providers support explicit invalidation (e.g Oracle); in those other cases, an UnsupportedOperationException will be thrown and you should rely on expiration instead.

4 Amazon S3

To use Amazon S3, you need the following dependency:

implementation("io.micronaut.objectstorage:micronaut-object-storage-aws")

<dependency>
    <groupId>io.micronaut.objectstorage</groupId>
    <artifactId>micronaut-object-storage-aws</artifactId>
</dependency>

Refer to the Micronaut AWS documentation for more information about credentials and region configuration.

The object storage specific configuration options available are:

For example:

src/main/resources/application-ec2.yml

micronaut.object-storage.aws.default.bucket=profile-pictures-bucket

micronaut:
  object-storage:
    aws:
      default:
        bucket: profile-pictures-bucket

[micronaut.object-storage.aws.default]
bucket = "profile-pictures-bucket"

micronaut {
  objectStorage {
    aws {
      'default' {
        bucket = 'profile-pictures-bucket'
      }
    }
  }
}

{
  micronaut = {
    object-storage = {
      aws = {
        default = {
          bucket = "profile-pictures-bucket"
        }
      }
    }
  }
}

{
  "micronaut": {
    "object-storage": {
      "aws": {
        "default": {
          "bucket": "profile-pictures-bucket"
        }
      }
    }
  }
}

The concrete implementation of ObjectStorageOperations is AwsS3Operations.

Advanced configuration

For configuration properties other than the specified above, you can add bean to your application that implements BeanCreatedEventListener. For example:

@Singleton
public class S3ClientBuilderCustomizer implements BeanCreatedEventListener<S3ClientBuilder> {

    @Override
    public S3ClientBuilder onCreated(@NonNull BeanCreatedEvent<S3ClientBuilder> event) {
        return event.getBean()
            .overrideConfiguration(c -> c.apiCallTimeout(Duration.of(60, ChronoUnit.SECONDS)));
    }
}

Tip	See the guide for Use the Micronaut Object Storage API to Store Files in Amazon S3 to learn more.

5 Azure Blob Storage

To use Azure Blob Storage, you need the following dependency:

implementation("io.micronaut.objectstorage:micronaut-object-storage-azure")

<dependency>
    <groupId>io.micronaut.objectstorage</groupId>
    <artifactId>micronaut-object-storage-azure</artifactId>
</dependency>

Refer to the Micronaut Azure documentation for more information about authentication options.

The object storage specific configuration options available are:

For example:

src/main/resources/application-azure.yml

azure.credential.client-secret.client-id=<client-id>
azure.credential.client-secret.tenant-id=<tenant-id>
azure.credential.client-secret.secret=<secret>
micronaut.object-storage.azure.default.container=profile-pictures-container
micronaut.object-storage.azure.default.endpoint=https://my-account.blob.core.windows.net

azure:
  credential:
    client-secret:
      client-id: <client-id>
      tenant-id: <tenant-id>
      secret: <secret>

micronaut:
  object-storage:
    azure:
      default:
        container: profile-pictures-container
        endpoint: https://my-account.blob.core.windows.net

[azure.credential.client-secret]
client-id = "<client-id>"
tenant-id = "<tenant-id>"
secret = "<secret>"

[micronaut.object-storage.azure.default]
container = "profile-pictures-container"
endpoint = "https://my-account.blob.core.windows.net"

azure {
  credential {
    clientSecret {
      clientId = '<client-id>'
      tenantId = '<tenant-id>'
      secret = '<secret>'
    }
  }
}
micronaut {
  objectStorage {
    azure {
      'default' {
        container = 'profile-pictures-container'
        endpoint = 'https://my-account.blob.core.windows.net'
      }
    }
  }
}

{
  azure = {
    credential = {
      client-secret = {
        client-id = "<client-id>"
        tenant-id = "<tenant-id>"
        secret = "<secret>"
      }
    }
  }
  micronaut = {
    object-storage = {
      azure = {
        default = {
          container = "profile-pictures-container"
          endpoint = "https://my-account.blob.core.windows.net"
        }
      }
    }
  }
}

{
  "azure": {
    "credential": {
      "client-secret": {
        "client-id": "<client-id>",
        "tenant-id": "<tenant-id>",
        "secret": "<secret>"
      }
    }
  },
  "micronaut": {
    "object-storage": {
      "azure": {
        "default": {
          "container": "profile-pictures-container",
          "endpoint": "https://my-account.blob.core.windows.net"
        }
      }
    }
  }
}

The concrete implementation of ObjectStorageOperations is AzureBlobStorageOperations.

Advanced configuration

For configuration properties other than the specified above, you can add bean to your application that implements BeanCreatedEventListener. For example:

@Singleton
public class BlobServiceClientBuilderCustomizer implements BeanCreatedEventListener<BlobServiceClientBuilder> {

    @Override
    public BlobServiceClientBuilder onCreated(@NonNull BeanCreatedEvent<BlobServiceClientBuilder> event) {
        HttpPipelinePolicy noOp = (context, next) -> next.process();
        return event.getBean().addPolicy(noOp);
    }
}

6 Google Cloud Storage

To use Google Cloud Storage, you need the following dependency:

implementation("io.micronaut.objectstorage:micronaut-object-storage-gcp")

<dependency>
    <groupId>io.micronaut.objectstorage</groupId>
    <artifactId>micronaut-object-storage-gcp</artifactId>
</dependency>

Refer to the Micronaut GCP documentation for more information about configuring your GCP project.

The object storage specific configuration options available are:

For example:

src/main/resources/application-gcp.yml

gcp.project-id=my-gcp-project
micronaut.object-storage.gcp.default.bucket=profile-pictures-bucket

gcp:
  project-id: my-gcp-project

micronaut:
  object-storage:
    gcp:
      default:
        bucket: profile-pictures-bucket

[gcp]
project-id = "my-gcp-project"

[micronaut.object-storage.gcp.default]
bucket = "profile-pictures-bucket"

gcp {
  projectId = 'my-gcp-project'
}
micronaut {
  objectStorage {
    gcp {
      'default' {
        bucket = 'profile-pictures-bucket'
      }
    }
  }
}

{
  gcp = {
    project-id = "my-gcp-project"
  }
  micronaut = {
    object-storage = {
      gcp = {
        default = {
          bucket = "profile-pictures-bucket"
        }
      }
    }
  }
}

{
  "gcp": {
    "project-id": "my-gcp-project"
  },
  "micronaut": {
    "object-storage": {
      "gcp": {
        "default": {
          "bucket": "profile-pictures-bucket"
        }
      }
    }
  }
}

The concrete implementation of ObjectStorageOperations is GoogleCloudStorageOperations.

Advanced configuration

For configuration properties other than the specified above, you can add bean to your application that implements BeanCreatedEventListener. For example:

@Singleton
public class StorageOptionsBuilderCustomizer implements BeanCreatedEventListener<StorageOptions.Builder> {

    @Override
    public StorageOptions.Builder onCreated(@NonNull BeanCreatedEvent<StorageOptions.Builder> event) {
        return event.getBean()
            .setTransportOptions(HttpTransportOptions.newBuilder().setConnectTimeout(60_000).build());
    }
}

Tip	See the guide for Use the Micronaut Object Storage API to Store Files in Google Cloud Storage to learn more.

7 Oracle Cloud Infrastructure (OCI) Object Storage

To use Oracle Cloud Infrastructure (OCI) Object Storage, you need the following dependency:

implementation("io.micronaut.objectstorage:micronaut-object-storage-oracle-cloud")

<dependency>
    <groupId>io.micronaut.objectstorage</groupId>
    <artifactId>micronaut-object-storage-oracle-cloud</artifactId>
</dependency>

Refer to the Micronaut Oracle Cloud documentation for more information about authentication options.

The object storage specific configuration options available are:

For example:

src/main/resources/application-oraclecloud.yml

oci.config.profile=DEFAULT
micronaut.object-storage.oracle-cloud.default.bucket=profile-pictures-bucket
micronaut.object-storage.oracle-cloud.default.namespace=MyNamespace

oci:
  config:
    profile: DEFAULT

micronaut:
  object-storage:
    oracle-cloud:
      default:
        bucket: profile-pictures-bucket
        namespace: MyNamespace

[oci.config]
profile = "DEFAULT"

[micronaut.object-storage.oracle-cloud.default]
bucket = "profile-pictures-bucket"
namespace = "MyNamespace"

oci {
  config {
    profile = 'DEFAULT'
  }
}
micronaut {
  objectStorage {
    oracleCloud {
      'default' {
        bucket = 'profile-pictures-bucket'
        namespace = 'MyNamespace'
      }
    }
  }
}

{
  oci = {
    config = {
      profile = "DEFAULT"
    }
  }
  micronaut = {
    object-storage = {
      oracle-cloud = {
        default = {
          bucket = "profile-pictures-bucket"
          namespace = "MyNamespace"
        }
      }
    }
  }
}

{
  "oci": {
    "config": {
      "profile": "DEFAULT"
    }
  },
  "micronaut": {
    "object-storage": {
      "oracle-cloud": {
        "default": {
          "bucket": "profile-pictures-bucket",
          "namespace": "MyNamespace"
        }
      }
    }
  }
}

The concrete implementation of ObjectStorageOperations is OracleCloudStorageOperations

Advanced configuration

For configuration properties other than the specified above, you can add bean to your application that implements BeanCreatedEventListener. For example:

//See https://github.com/oracle/oci-java-sdk/blob/master/bmc-examples/src/main/java/ClientConfigurationTimeoutExample.java
@Singleton
public class ObjectStorageClientBuilderCustomizer implements BeanCreatedEventListener<ObjectStorageClient.Builder> {

    public static final int CONNECTION_TIMEOUT_IN_MILLISECONDS = 25000;
    public static final int READ_TIMEOUT_IN_MILLISECONDS = 35000;

    @Override
    public ObjectStorageClient.Builder onCreated(@NonNull BeanCreatedEvent<ObjectStorageClient.Builder> event) {
        ClientConfiguration clientConfiguration =
            ClientConfiguration.builder()
                .connectionTimeoutMillis(CONNECTION_TIMEOUT_IN_MILLISECONDS)
                .readTimeoutMillis(READ_TIMEOUT_IN_MILLISECONDS)
                .build();


        return event.getBean()
            .configuration(clientConfiguration);
    }
}

Tip	See the guide for Use the Micronaut Object Storage API to Store Files in Oracle Cloud Infrastructure (OCI) Object Storage to learn more.

8 Local Storage

To use the local storage implementation (useful for tests), you need the following dependency:

testImplementation("io.micronaut.objectstorage:micronaut-object-storage-local")

<dependency>
    <groupId>io.micronaut.objectstorage</groupId>
    <artifactId>micronaut-object-storage-local</artifactId>
    <scope>test</scope>
</dependency>

Then, simply define a local storage:

micronaut.object-storage.local.default.enabled=true

micronaut:
  object-storage:
    local:
      default:
        enabled: true

[micronaut.object-storage.local.default]
enabled = true

micronaut {
  objectStorage {
    local {
      'default' {
        enabled = true
      }
    }
  }
}

{
  micronaut = {
    object-storage = {
      local = {
        default = {
          enabled = true
        }
      }
    }
  }
}

{
  "micronaut": {
    "object-storage": {
      "local": {
        "default": {
          "enabled": true
        }
      }
    }
  }
}

Note	When added to the classpath, LocalStorageOperations becomes the primary implementation of ObjectStorageOperations.

By default, it will create a temporary folder to store the files, but you can configure it to use a specific folder:

For example:

src/main/resources/application-test.yml

micronaut.object-storage.local.default.path=/tmp/my-object-storage

micronaut:
  object-storage:
    local:
      default:
        path: /tmp/my-object-storage

[micronaut.object-storage.local.default]
path = "/tmp/my-object-storage"

micronaut {
  objectStorage {
    local {
      'default' {
        path = '/tmp/my-object-storage'
      }
    }
  }
}

{
  micronaut = {
    object-storage = {
      local = {
        default = {
          path = "/tmp/my-object-storage"
        }
      }
    }
  }
}

{
  "micronaut": {
    "object-storage": {
      "local": {
        "default": {
          "path": "/tmp/my-object-storage"
        }
      }
    }
  }
}

The concrete implementation of ObjectStorageOperations is LocalStorageOperations.

9 Guides

See the following list of guides to learn more about working with Object Storage in the Micronaut Framework:

https://guides.micronaut.io/latest/tag-object_storage.html

10 Repository

You can find the source code of this project in this repository:

https://github.com/micronaut-projects/micronaut-object-storage