1. Introduction
In this tutorial, we’ll go over Redis and how we can use it within Scala projects.
Redis stands for Remote Dictionary Server and was created by Salvatore Sanfilippo in 2009. Without a doubt, with billions of docker pulls and thousands of GitHub repository stars and forks, we can say that it is probably the most popular in-memory data store. Also, Redis is written in ANSI C and is officially supported only for Linux and macOS.
2. Redis Overview
Before we dive into more technical details about Redis, it’s essential to mention some of the key features that have made it so popular. Redis is persistent, gives us the ability to keep snapshots, and has multiple data structures such as Sets, Hashes, Geospatial Indexes, HyperLogLogs, and more. Projects utilize Redis for a wide variety of use cases including – but not limited to – caching, real-time analytics, and publish/subscribe.
It is equally important to mention some examples where other storage options might be more suitable choices than Redis:
- Complex queries with multiple parameters – Ultimately, we can’t replicate SQL-like queries on a Redis and expect it to be a practical solution.
- Relational Data or High Integrity Guarantees – In other words, it’s not optimal to implement joins over Redis or try to keep data integrity on unrelated data structures.
- Very large datasets or not enough RAM – Specifically, if the dataset size is larger or close to the size of the available RAM, that’s a good indication that Redis might not the best choice.
3. Clients
There are a lot of Redis clients in many languages, and a bunch of them can be used in Scala projects. A Scala engineer can choose from the Scala or the Java list of clients.
For this article, we’ll demonstrate the most popular client someone can use in Scala projects: the Jedis client. Jedis is a Java client for Redis, hence the initial ‘J’ in the name. Its primary focus is performance and ease of use. So, let’s import Jedis to our project:
lazy val redisClients = "redis.clients" % "jedis" % "5.1.2"
libraryDependencies += redisClients
4. Jedis Client Examples
Now that we’ve covered the basics and how and when to use Redis, it’s time to use it!
Let’s go through some of the most common Redis use cases.
4.1. Caching
Obviously, caching is the first thing that comes to mind when discussing Redis, so let’s do just that.
Let’s implement an object that acts as a database, which will play the role of the slow part that can benefit from caching:
object BooksDB {
private val author1 = Author(1L, "James Brown")
private val author2 = Author(2L, "Bob White")
private val author3 = Author(3L, "Hannah Robertson")
def authors: List[Author] = {
List(
author1,
author2,
author3,
)
}
def books: List[Book] = {
List(
Book(1L, author1.id, "Title 1"),
Book(2L, author2.id, "Title 2"),
Book(3L, author3.id, "Title 3"),
Book(4L, author1.id, "Title 5"),
Book(5L, author2.id, "Title 6"),
Book(6L, author3.id, "Title 7"),
Book(7L, author1.id, "Title 9"),
Book(8L, author2.id, "Title 10")
)
}
}
For the cache itself, let’s implement a cache-through interface using the Jedis client:
class CacheThrough(jedis: Jedis, db: VirtualDatabase) {
def books(): List[Book] = cachedLookup("books", () => db.books(), deserializeList[Book])
def authors(): List[Author] = cachedLookup("authors", () => db.authors(), deserializeList[Author])
private val mapper = JsonMapper.builder().addModule(DefaultScalaModule).build() :: ClassTagExtensions
private def serialize(obj: AnyRef): String = {
mapper.writeValueAsString(obj)
}
private def deserialize[A](obj: String)(implicit manifest: Manifest[A]): A = {
mapper.readValue[A](obj)
}
private def deserializeList[A](obj: String)(implicit manifest: Manifest[A]): List[A] = {
mapper.readValue[List[A]](obj)
}
private def cachedLookup[A <: AnyRef](key: String, backup: () => A, deserializer: String => A)(implicit manifest: Manifest[A]): A = {
jedis.get(key) match {
case null =>
val backedUp = backup()
jedis.set(key, serialize(backedUp))
backedUp
case a =>
deserializer(a)
}
}
}
Now, let’s verify the CacheThrough implementation with a simple test scenario:
"CacheThrough" should "fetch from DB only the first time" in {
val mockDb = mock[VirtualDatabase]
when(mockDb.books()).thenReturn(BooksDB.books)
val cacheThrough = new CacheThrough(getJedis(), mockDb)
cacheThrough.books()
cacheThrough.books()
verify(mockDb, times(1)).books()
}
4.2. Rate Limiting
Request rate limiting and similar application needs can be handled by using the key expiration features of Redis.
In other words, by setting expiration on keys, the decision on whether a certain duration has passed can be made by checking if the key exists in the dataset. Let’s demonstrate that with the RateLimit class:
class RateLimit(jedis: Jedis) {
private final val existenceValue = "Exists"
def setLimit(key: String, duration: Duration): Unit = {
jedis.setex(key, duration.toSeconds, existenceValue)
}
def isLimited(key: String): Boolean = {
jedis.exists(key)
}
}
Of course, there’s a test case that checks the limiting behavior:
"RateLimit" should "respond with not allowed for the given duration" in {
val limitedKey = "limited-key"
val limitDuration = 5 seconds
val rateLimit = new RateLimit(getJedis())
rateLimit.setLimit(limitedKey, limitDuration)
assert(rateLimit.isLimited(limitedKey) === true)
Thread.sleep(limitDuration.toMillis)
assert(rateLimit.isLimited(limitedKey) === false)
}
4.3. Publish/Subscribe
Even though Redis is not the first thing that comes to mind when considering event-based solutions, Redis with PUBSUB can be used as a message broker.
Let’s showcase how simple it is to write a Redis subscriber:
val channel = "channel1"
getJedis().subscribe(new JedisPubSub {
override def onMessage(channel: String, message: String): Unit = {
println(s"Got message: $message on channel: $channel")
}
}, channel)
Now, we can publish messages to the same channel with a publisher:
class Publisher(jedis: Jedis) {
def publish(channel: String, message: String): Unit = {
jedis.publish(channel, message)
}
}
// main
val channel = "channel1"
val publisher = new Publisher(getJedis())
publisher.publish(channel, "my message")
4.4. Geospatial Queries
In recent years, the need for geospatial queries and spatial data capabilities in applications has increased. Our article Architecture of a Geospatial Application with Java further describes the geospatial application architecture.
Interestingly enough, Redis provides a way to handle and query geospatial data, so let’s define our geospatial model with some reference data points:
case class GeoPoint(name: String, latLon: LatLon)
case class LatLon(lon: Double, lat: Double)
object LatLon{
def toGeoCoordinate(latLon: LatLon): GeoCoordinate = {
new GeoCoordinate(latLon.lon, latLon.lat)
}
}
object GeoPoints {
val Methana = GeoPoint("Methana", LatLon(37.58303564998219, 23.387115029515513))
val Vromolimni = GeoPoint("Vromolimni", LatLon(37.589273863460754, 23.38577712741729))
val Dritseika = GeoPoint("Dritseika", LatLon(37.58546506226565, 23.38088477825137))
val Kipseli = GeoPoint("Kipseli", LatLon(37.61021878600531, 23.396505963307447))
val MegaloPotami = GeoPoint("MegaloPotami", LatLon(37.597927893434225, 23.35013310734787))
val Makriloggos = GeoPoint("Makriloggos", LatLon(37.621657733616175, 23.35880200674712))
val MethanaIndex = "MethanaIndex"
}
Afterward, we can extract a subset of the Jedis client geospatial functions so we can demonstrate some of their capabilities. Let’s see the Geospatial interface:
class Geospatial(jedis: Jedis) {
def add(geoIndexName: String)(memberName: String)(latLon: LatLon): Long = {
jedis.geoadd(geoIndexName, latLon.lon, latLon.lat, memberName)
}
def dist(geoIndexName: String)(member1: String, member2: String): Double = {
jedis.geodist(geoIndexName, member1, member2, GeoUnit.KM)
}
def searchRadius(geoIndexName: String)(latLon: LatLon)(radius: Double): List[GeoRadiusResponse] = {
jedis.geosearch(geoIndexName, toGeoCoordinate(latLon), radius, GeoUnit.KM).asScala.toList
}
def searchBox(geoIndexName: String)(latLon: LatLon)(width: Double, height: Double): List[GeoRadiusResponse] = {
jedis.geosearch(geoIndexName, toGeoCoordinate(latLon), width, height, GeoUnit.KM).asScala.toList
}
}
Let’s write a few test cases to verify our implementation by checking point-to-point distance, searching by radius, and searching by box:
"Geospatial#dist" should "return correct distance for geo points - Vromolimni is closer to Mathana than Kipseli" in {
val geoSpatial = new Geospatial(getJedis())
loadGeopoints(geoSpatial)
val methanaToVromolimni = geoSpatial.dist(MethanaIndex)(Methana.name, Vromolimni.name)
val methanaToKipseli = geoSpatial.dist(MethanaIndex)(Methana.name, Kipseli.name)
assert(methanaToVromolimni < methanaToKipseli)
}
"Geospatial#searchRadius" should "return only Dritseika, Vromolimni and Methana for 2km radius from Methana" in {
val geoSpatial = new Geospatial(getJedis())
loadGeopoints(geoSpatial)
val result = geoSpatial.searchRadius(MethanaIndex)(Methana.latLon)(2.0)
assert(result.size === 3)
assert(result.exists(_.getMemberByString == Vromolimni.name))
assert(result.exists(_.getMemberByString == Dritseika.name))
assert(result.exists(_.getMemberByString == Methana.name))
}
"Geospatial#searchBox" should "return only MegaloPotami, Makriloggos and Kipseli for 4 x 12 km box from Kipseli" in {
val geoSpatial = new Geospatial(getJedis())
loadGeopoints(geoSpatial)
val result = geoSpatial.searchBox(MethanaIndex)(Kipseli.latLon)(4.0, 12.0)
assert(result.size === 3)
assert(result.exists(_.getMemberByString == MegaloPotami.name))
assert(result.exists(_.getMemberByString == Makriloggos.name))
assert(result.exists(_.getMemberByString == Kipseli.name))
}
4.5. Leaderboards
We can leverage Redis data structures to produce leaderboards and real-time analytics. With HyperLogLogs, counting – which is one of the most frequent tasks – becomes way less expensive compared to its SQL alternatives.
It’s important to note that HyperLogLog is an algorithm that estimates the cardinality of very large datasets.
Let’s demonstrate a simple leaderboard that counts on three separate levels:
class LeaderBoard(hllKey: String, jedis: Jedis) {
def plusOne(key: LeaderboardKey): Unit = {
val randValue = random()
jedis.pfadd(s"$hllKey:${key.firstLevelKey}", randValue)
jedis.pfadd(s"$hllKey:${key.secondLevelKey}", randValue)
jedis.pfadd(s"$hllKey:${key.thirdLevelKey}", randValue)
}
def plusN(key: LeaderboardKey, n: Int): Unit = {
val randValues = (0 until n) map (_ => random())
jedis.pfadd(s"$hllKey:${key.firstLevelKey}", randValues: _*)
jedis.pfadd(s"$hllKey:${key.secondLevelKey}", randValues: _*)
jedis.pfadd(s"$hllKey:${key.thirdLevelKey}", randValues: _*)
}
def count(key: String): Long = {
jedis.pfcount(s"$hllKey:$key")
}
private def random(): String = UUID.randomUUID().toString
}
trait LeaderboardKey {
def firstLevelKey: String
def secondLevelKey: String
def thirdLevelKey: String
}
Also, let’s add test cases to verify that counts are correct on all levels:
"Leaderboard#count" should "return correct counts for Employees after plus one calls" in {
val commitLeaderboardKey = "commits"
val commitLeaderboard = new LeaderBoard(commitLeaderboardKey, getJedis())
commitLeaderboard.plusOne(EmployDB.Emp1)
commitLeaderboard.plusOne(EmployDB.Emp2)
commitLeaderboard.plusOne(EmployDB.Emp3)
commitLeaderboard.plusOne(EmployDB.Emp4)
assert(commitLeaderboard.count(EmployDB.Emp1.firstLevelKey) === 3)
assert(commitLeaderboard.count(EmployDB.Emp1.secondLevelKey) === 2)
assert(commitLeaderboard.count(EmployDB.Emp1.thirdLevelKey) === 1)
}
"Leaderboard#count" should "return correct counts for Employees after plus N calls" in {
val commitLeaderboardKey = "commits"
val commitLeaderboard = new LeaderBoard(commitLeaderboardKey, getJedis())
commitLeaderboard.plusN(EmployDB.Emp1, 3)
commitLeaderboard.plusN(EmployDB.Emp2, 2)
commitLeaderboard.plusN(EmployDB.Emp3, 12)
commitLeaderboard.plusN(EmployDB.Emp5, 4)
assert(commitLeaderboard.count(EmployDB.Emp1.firstLevelKey) === 17)
assert(commitLeaderboard.count(EmployDB.Emp1.secondLevelKey) === 3)
assert(commitLeaderboard.count(EmployDB.Emp2.secondLevelKey) === 6)
assert(commitLeaderboard.count(EmployDB.Emp1.thirdLevelKey) === 3)
assert(commitLeaderboard.count(EmployDB.Emp3.secondLevelKey) === 12)
}
As can be seen in the test cases above, the LeaderBoard count method is a convenient way to get every metric we gather. To put it simply, in our example, firstLevelKey represents every employee within a department, secondLevelKey represents every employee in a project, and thirdLevelKey represents a single employee.
To sum up, we can group and count anything we want as long as we specify the right LeaderBoard keys.
5. Conclusion
In this article, we discussed Redis in general and its usage in Scala projects.
As always, the code of the above examples is available over on GitHub.