Chronicle Map：基于键值的存储

最后修改:2024年3月17日

by Tino Mulanchira Thomas

1. Overview

In this tutorial, we’re going to see how we can use the Chronicle Map for storing key-value pairs. We’ll also be creating short examples to demonstrate its behavior and usage.

2. What Is a Chronicle Map?

Following the documentation, “Chronicle Map is a super-fast, in-memory, non-blocking, key-value store, designed for low-latency, and/or multi-process applications”.

In a nutshell, it’s an off-heap key-value store. The map doesn’t require a large amount of RAM for it to function properly. It can grow based on the available disk capacity. Furthermore, it supports the replication of the data in a multi-master server setup.

Let’s now see how we can set up and work with it.

3. Maven Dependency

To get started, we’ll need to add the chronicle-map dependency to our project:

<dependency>
    <groupId>net.openhft</groupId>
    <artifactId>chronicle-map</artifactId>
    <version>3.24ea1</version>
</dependency>

If you are running chronicle-map with JDK 11 and above, you will need to pass additional arguments to the JVM, as detailed in the official documentation.

In our project, we’ll add these to the Maven surefire plugin configuration so that our tests can run correctly:

<build>
    <plugins>
        <plugin>
            <groupId>org.apache.maven.plugins</groupId>
            <artifactId>maven-surefire-plugin</artifactId>
            <version>${maven-surefire-plugin.version}</version>
            <configuration>
                <argLine>--add-exports=java.base/jdk.internal.ref=ALL-UNNAMED 
                  --add-exports=java.base/sun.nio.ch=ALL-UNNAMED 
                  --add-exports=jdk.unsupported/sun.misc=ALL-UNNAMED 
                  --add-exports=jdk.compiler/com.sun.tools.javac.file=ALL-UNNAMED 
                  --add-opens=jdk.compiler/com.sun.tools.javac=ALL-UNNAMED 
                  --add-opens=java.base/java.lang=ALL-UNNAMED 
                  --add-opens=java.base/java.lang.reflect=ALL-UNNAMED 
                  --add-opens=java.base/java.io=ALL-UNNAMED 
                  --add-opens=java.base/java.util=ALL-UNNAMED</argLine>
            </configuration>
        </plugin>
    </plugins>
</build>

4. Types of Chronicle Map

We can create a map in two ways: either as an in-memory map or as a persisted map.

Let’s see both of these in detail.

4.1. In-Memory Map

An in-memory Chronicle Map is a map store that is created within the physical memory of the server. This means it’s accessible only within the JVM process in which the map store is created.

Let’s see a quick example:

ChronicleMap<LongValue, CharSequence> inMemoryCountryMap = ChronicleMap
  .of(LongValue.class, CharSequence.class)
  .name("country-map")
  .entries(50)
  .averageValue("America")
  .create();

For the sake of simplicity, we’re creating a map that stores 50 country ids and their names. As we can see in the code snippet, the creation is pretty straightforward except for the averageValue() configuration. This tells the map to configure the average number of bytes taken by map entry values.

In other words, when creating the map, the Chronicle Map determines the average number of bytes taken by the serialized form of values. It does this by serializing the given average value using the configured value marshallers. It will then allocate the determined number of bytes for the value of each map entry.

One thing we have to note when it comes to the in-memory map is that the data is accessible only when the JVM process is alive. The library will clear the data when the process terminates.

4.2. Persisted Map

Unlike an in-memory map, the implementation will save a persisted map to disk. Let’s now see how we can create a persisted map:

ChronicleMap<LongValue, CharSequence> persistedCountryMap = ChronicleMap
  .of(LongValue.class, CharSequence.class)
  .name("country-map")
  .entries(50)
  .averageValue("America")
  .createPersistedTo(new File(System.getProperty("user.home") + "/country-details.dat"));

This will create a file called country-details.dat in the folder specified. If this file is already available in the specified path, then the builder implementation will open a link to the existing data store from this JVM process.

We can make use of the persisted map in cases where we want it to:

  • survive beyond the creator process; for example, to support hot application redeployment
  • make it global in a server; for example, to support multiple concurrent process access
  • act as a data store that we’ll save to the disk

5. Size Configuration

It’s mandatory to configure the average value and average key while creating a Chronicle Map, except in the case where our key/value type is either a boxed primitive or a value interface. In our example, we’re not configuring the average key since the key type LongValue is a value interface.

Now, let’s see what the options are for configuring the average number of key/value bytes:

  • averageValue() – The value from which the average number of bytes to be allocated for the value of a map entry is determined
  • averageValueSize() – The average number of bytes to be allocated for the value of a map entry
  • constantValueSizeBySample() – The number of bytes to be allocated for the value of a map entry when the size of the value is always the same
  • averageKey() – The key from which the average number of bytes to be allocated for the key of a map entry is determined
  • averageKeySize() – The average number of bytes to be allocated for the key of a map entry
  • constantKeySizeBySample() – The number of bytes to be allocated for the key of a map entry when the size of the key is always the same

6. Key and Value Types

There are certain standards that we need to follow when creating a Chronicle Map, especially when defining the key and value. The map works best when we create the key and value using the recommended types.

Here are some of the recommended types:

  • Value interfaces
  • Any class implementing Byteable interface from Chronicle Bytes
  • Any class implementing BytesMarshallable interface from Chronicle Bytes; the implementation class should have a public no-arg constructor
  • byte[] and ByteBuffer
  • CharSequence, String, and StringBuilder
  • Integer, Long, and Double
  • Any class implementing java.io.Externalizable; the implementation class should have a public no-arg constructor
  • Any type implementing java.io.Serializable, including boxed primitive types (except those listed above) and array types
  • Any other type, if custom serializers are provided

7. Querying a Chronicle Map

Chronicle Map supports single-key queries as well as multi-key queries.

7.1. Single-Key Queries

Single-key queries are the operations that deal with a single key. ChronicleMap supports all the operations from the Java Map interface and ConcurrentMap interface:

LongValue qatarKey = Values.newHeapInstance(LongValue.class);
qatarKey.setValue(1);
inMemoryCountryMap.put(qatarKey, "Qatar");

//...

CharSequence country = inMemoryCountryMap.get(key);

In addition to the normal get and put operations, ChronicleMap adds a special operation, getUsing(), that reduces the memory footprint while retrieving and processing an entry. Let’s see this in action:

LongValue key = Values.newHeapInstance(LongValue.class);
StringBuilder country = new StringBuilder();
key.setValue(1);
persistedCountryMap.getUsing(key, country);
assertThat(country.toString(), is(equalTo("Romania")));

key.setValue(2);
persistedCountryMap.getUsing(key, country);
assertThat(country.toString(), is(equalTo("India")));

Here we’ve used the same StringBuilder object for retrieving values of different keys by passing it to the getUsing() method. It basically reuses the same object for retrieving different entries. In our case, the getUsing() method is equivalent to:

country.setLength(0);
country.append(persistedCountryMap.get(key));

7.2. Multi-Key Queries

There may be use cases where we need to deal with multiple keys at the same time. For this, we can use the queryContext() functionality. The queryContext() method will create a context for working with a map entry.

Let’s first create a multimap and add some values to it:

Set<Integer> averageValue = IntStream.of(1, 2).boxed().collect(Collectors.toSet());
ChronicleMap<Integer, Set<Integer>> multiMap = ChronicleMap
  .of(Integer.class, (Class<Set<Integer>>) (Class) Set.class)
  .name("multi-map")
  .entries(50)
  .averageValue(averageValue)
  .create();

Set<Integer> set1 = new HashSet<>();
set1.add(1);
set1.add(2);
multiMap.put(1, set1);

Set<Integer> set2 = new HashSet<>();
set2.add(3);
multiMap.put(2, set2);

To work with multiple entries, we have to lock those entries to prevent inconsistency that may occur due to a concurrent update:

try (ExternalMapQueryContext<Integer, Set<Integer>, ?> fistContext = multiMap.queryContext(1)) {
    try (ExternalMapQueryContext<Integer, Set<Integer>, ?> secondContext = multiMap.queryContext(2)) {
        fistContext.updateLock().lock();
        secondContext.updateLock().lock();

        MapEntry<Integer, Set<Integer>> firstEntry = fistContext.entry();
        Set<Integer> firstSet = firstEntry.value().get();
        firstSet.remove(2);

        MapEntry<Integer, Set<Integer>> secondEntry = secondContext.entry();
        Set<Integer> secondSet = secondEntry.value().get();
        secondSet.add(4);

        firstEntry.doReplaceValue(fistContext.wrapValueAsData(firstSet));
        secondEntry.doReplaceValue(secondContext.wrapValueAsData(secondSet));
    }
} finally {
    assertThat(multiMap.get(1).size(), is(equalTo(1)));
    assertThat(multiMap.get(2).size(), is(equalTo(2)));
}

8. Closing the Chronicle Map

Now that we’ve finished working with our maps, let’s call the close() method on our map objects to release the off-heap memory and the resources associated with it:

persistedCountryMap.close();
inMemoryCountryMap.close();
multiMap.close();

One thing to keep in mind here is that all the map operations must be completed before closing the map. Otherwise, the JVM might crash unexpectedly.

9. Conclusion

In this tutorial, we’ve learned how to use a Chronicle Map to store and retrieve key-value pairs. Even though the community version is available with most of the core functionalities, the commercial version has some advanced features like data replication across multiple servers and remote calls.

All the examples we’ve discussed here can be found over the Github project.

« 上一篇： 从嵌套循环中解脱出来
» 下一篇： Serialize Enum in Retrofit With Kotlin on Kotlin