On this page

CRaC

1 Introduction

This Micronaut module adds support for CRaC (Coordinated Restore at Checkpoint) within the framework.

2 Release History

For this project, you can find a list of releases (with release notes) here:

https://github.com/micronaut-projects/micronaut-crac/releases

3 Installation

Add the following the dependency to your build:

implementation("io.micronaut.crac:micronaut-crac")

<dependency>
    <groupId>io.micronaut.crac</groupId>
    <artifactId>micronaut-crac</artifactId>
</dependency>

4 CRaC Resources

4.1 DataSources

We currently support Hikari DataSources that are configured to allow suspension with:

datasources.default.allow-pool-suspension=true

datasources:
  default:
    allow-pool-suspension: true

[datasources.default]
allow-pool-suspension = true

datasources {
  'default' {
    allowPoolSuspension = true
  }
}

{
  datasources = {
    default = {
      allow-pool-suspension = true
    }
  }
}

{
  "datasources": {
    "default": {
      "allow-pool-suspension": true
    }
  }
}

When a checkpoint is taken, the pool is suspended and the connections are closed. When the checkpoint is restored, the pool is resumed and the connections are re-established.

4.2 Redis

Since v1.2.1 of this library, we also support Redis connections.

When the checkpoint is taken, we destroy RedisClient, RedisCache and StatefulRedisConnection beans that exist in the application.

These will be re-created once the checkpoint is restored, however you will need to add your beans to the refresh scope so that these new beans are used.

The resources for each of the above can be disabled via configuration:

crac.redis.enabled=false
crac.redis.client-enabled=false
crac.redis.cache-enabled=false
crac.redis.connection-enabled=false

crac:
    redis:
        enabled: false            # disable all redis support
        client-enabled: false     # disable RedisClient support
        cache-enabled: false      # disable RedisCache support
        connection-enabled: false # disable StatefulRedisConnection support

[crac.redis]
enabled = false
client-enabled = false
cache-enabled = false
connection-enabled = false

crac {
  redis {
    enabled = false
    clientEnabled = false
    cacheEnabled = false
    connectionEnabled = false
  }
}

{
  crac = {
    redis = {
      enabled = false
      client-enabled = false
      cache-enabled = false
      connection-enabled = false
    }
  }
}

{
  "crac": {
    "redis": {
      "enabled": false,
      "client-enabled": false,
      "cache-enabled": false,
      "connection-enabled": false
    }
  }
}

4.3 Custom CRaC Resources

To provide custom CRaC resources, create beans of type OrderedResource.

Micronaut CRaC registers resources for you into the CRaC Context. You just focus on providing implementations for OrderedResource::beforeCheckpoint and OrderedResource::afterRestore in your resources.

Micronaut CRaC registers resources in order. You can control the order by overriding OrderedResource::getOrder.

5 Refresh scope

Prior to a checkpoint being taken, a RefreshEvent will be published to invalidate all beans in the @Refreshable scope.

This behaviour can be disabled by setting the crac.refresh-beans property to false in the application config.

6 Events

To notify external components when the default Resource handlers execute, there are two events; BeforeCheckpointEvent and AfterRestoreEvent. These events contain the java.time.Instant that the action completed, the length of time it took to execute the action in nanoseconds, and the Resource that was acted upon.

Please see the Micronaut Framework guide for information on how to listen for these events.

7 Context Provider

GlobalCracContextProvider, Micronaut CRaC’s default implementation of CracContextProvider, returns the global context. You can provide a replacement for CracContextProvider and provide your custom Context.

8 Docker Support

Support for building CRaC-enabled Docker images is provided by the Micronaut Gradle or Maven plugin. Either one is capable of generating a docker image containing a CRaC enabled JDK and a pre-warmed, checkpointed application.

9 Testing

When you create a Micronaut Framework application (either via https://launch.micronaut.io or the command line application), it creates a ContextConfigurer with enables eager singleton initialization.

As tests annotated with @MicronautTest are implicitly in the Singleton scope, this can cause problems injecting some beans (for example an HttpClient) into your test class.

To avoid this, you can either disable eager singleton initialization for your tests, or you will need to manually get an instance of the bean you would normally inject. As an example, to get an HttpClient you could do:

Lazily get an HttpClient in a test

@Inject
EmbeddedServer server; // 

Supplier<HttpClient> clientSupplier = SupplierUtil.memoizedNonEmpty(() -> // 
    server.getApplicationContext().createBean(HttpClient.class, server.getURL())
);

@Test
void testClient() {
    assertEquals("ok", clientSupplier.get().toBlocking().retrieve("/eager")); // 
}

@field:Inject
lateinit var server: EmbeddedServer // 

val client by lazy {
    server.applicationContext.createBean(HttpClient::class.java, server.url) // 
}

@Test
fun testClient() {
    assertEquals("ok", client.toBlocking().retrieve("/eager")) // 
}

@Inject
EmbeddedServer server // 

@Memoized // 
HttpClient clientSupplier() {
    server.applicationContext.createBean(HttpClient, server.URL)
}

void 'test client'() {
    expect:
    clientSupplier().toBlocking().retrieve("/eager") == "ok" // 
}

Inject the EmbeddedServer as normal
Lazily create a HttpClient when it is first called
Get the HttpClient and make the request

9.1 Disable Eager Singleton Initialization

When you create a Micronaut Framework application (either via https://launch.micronaut.io or the command line application), it creates a ContextConfigurer with enables eager singleton initialization.

As tests annotated with @MicronautTest are implicitly in the Singleton scope, this can cause problems injecting some beans (for example an HttpClient) into your test class.

Lazily get an HttpClient in a test

@Inject
EmbeddedServer server; // 

Supplier<HttpClient> clientSupplier = SupplierUtil.memoizedNonEmpty(() -> // 
    server.getApplicationContext().createBean(HttpClient.class, server.getURL())
);

@Test
void testClient() {
    assertEquals("ok", clientSupplier.get().toBlocking().retrieve("/eager")); // 
}

@field:Inject
lateinit var server: EmbeddedServer // 

val client by lazy {
    server.applicationContext.createBean(HttpClient::class.java, server.url) // 
}

@Test
fun testClient() {
    assertEquals("ok", client.toBlocking().retrieve("/eager")) // 
}

@Inject
EmbeddedServer server // 

@Memoized // 
HttpClient clientSupplier() {
    server.applicationContext.createBean(HttpClient, server.URL)
}

void 'test client'() {
    expect:
    clientSupplier().toBlocking().retrieve("/eager") == "ok" // 
}

Inject the EmbeddedServer as normal
Lazily create a HttpClient when it is first called
Get the HttpClient and make the request

9.2 Using a CheckpointSimulator

With CRaC, you run your application to a point and then "checkpoint" it. This calls the app to close all its sockets and file handles, and then dumps the memory to disk. When it restarts from this snapshot, it calls the app again to say it’s been restored, and one can re-open files and network connections.

The simulator allows you to synthesise these two calls (before checkpoint and after restore), so that under a test you can check your service works again after it was closed and recreated.

The following example demonstrates how to write and test custom OrderedResource implementations. Here it is doing nothing for sake of the example, but in reality it would have a socket or file that needs to be closed, or something along those lines.

A resource bean

@Singleton
class ResourceBean {

    private boolean running = true; // 

    boolean isRunning() {
        return running;
    }

    void stop() {
        this.running = false;
    }

    void start() {
        this.running = true;
    }
}

@Singleton
class ResourceBean {

    var isRunning = true // 
        private set

    fun stop() {
        isRunning = false
    }

    fun start() {
        isRunning = true
    }
}

@Singleton
 class ResourceBean {

    private boolean running = true // 

    boolean isRunning() {
        return running
    }

    void stop() {
        this.running = false
    }

    void start() {
        this.running = true
    }
}

A simulated state for the bean

An OrderedResource which will stop the ResourceBean before a checkpoint and start it again after a restore.

An OrderedResouce for the bean

@Singleton
class ResourceBeanResource implements OrderedResource { // 

    private final ResourceBean resourceBean;

    ResourceBeanResource(ResourceBean resourceBean) {
        this.resourceBean = resourceBean;
    }

    @Override
    public void beforeCheckpoint(Context<? extends Resource> context) throws Exception { // 
        resourceBean.stop();
    }

    @Override
    public void afterRestore(Context<? extends Resource> context) throws Exception { // 
        resourceBean.start();
    }
}

@Singleton
class ResourceBeanResource(private val resourceBean: ResourceBean) : OrderedResource { // 

    @Throws(Exception::class)
    override fun beforeCheckpoint(context: Context<out Resource?>?) { // 
        resourceBean.stop()
    }

    @Throws(Exception::class)
    override fun afterRestore(context: Context<out Resource?>?) { // 
        resourceBean.start()
    }
}

@Singleton
 class ResourceBeanResource implements OrderedResource { // 

    private final ResourceBean resourceBean

    ResourceBeanResource(ResourceBean resourceBean) {
        this.resourceBean = resourceBean
    }

    @Override
    void beforeCheckpoint(Context<? extends Resource> context) throws Exception { // 
        resourceBean.stop()
    }

    @Override
    void afterRestore(Context<? extends Resource> context) throws Exception { // 
        resourceBean.start()
    }
}

Implement the OrderedResource interface
Override to stop the resource before a checkpoint
Override to restart it again after a restore

Using CheckpointSimulator in a test

@Inject
BeanContext ctx;

@Test
void testCustomOrderedResourceUsingCheckpointSimulator() {
    ResourceBean myBean = ctx.getBean(ResourceBean.class);
    CheckpointSimulator checkpointSimulator = ctx.getBean(CheckpointSimulator.class); // 
    assertTrue(myBean.isRunning());

    checkpointSimulator.runBeforeCheckpoint();  // 
    assertFalse(myBean.isRunning());

    checkpointSimulator.runAfterRestore(); // 
    assertTrue(myBean.isRunning());
}

@field:Inject
lateinit var ctx: BeanContext

@Test
fun testCustomOrderedResourceUsingCheckpointSimulator() {
    val myBean = ctx.getBean(ResourceBean::class.java)
    val checkpointSimulator = ctx.getBean(CheckpointSimulator::class.java) // 
    Assertions.assertTrue(myBean.isRunning)

    checkpointSimulator.runBeforeCheckpoint() // 
    Assertions.assertFalse(myBean.isRunning)

    checkpointSimulator.runAfterRestore() // 
    Assertions.assertTrue(myBean.isRunning)
}

@Inject
BeanContext ctx

void "test custom OrderedResource implementation using the CheckpointSimulator"() {
    given:
    ResourceBean myBean = ctx.getBean(ResourceBean)
    CheckpointSimulator checkpointSimulator = ctx.getBean(CheckpointSimulator) // 
    expect:
    myBean.running

    when:
    checkpointSimulator.runBeforeCheckpoint()  // 
    then:
    !myBean.running

    when:
    checkpointSimulator.runAfterRestore() // 

    then:
    myBean.running
}

Obtain the CheckpointSimulator from the running environment
Emulate the checkpoint and assert the bean is no longer running
Emulate the restore and assert the bean is running again

10 Info sources

If the info endpoint is enabled, then a crac section will be automatically added which shows the restore time and uptime since restore, both in milliseconds and taken from the CRaCMXBean provided by the CRaC API.

This can be disabled by setting endpoints.info.crac.enabled configuration to false.

11 Guides

See the following list of guides to learn more about working with CRaC in the Micronaut Framework:

https://guides.micronaut.io/latest/tag-crac.html

12 Repository

You can find the source code of this project in this repository:

https://github.com/micronaut-projects/micronaut-crac