1. Introduction
Working with software often involves using configuration files to manage the application. Therefore, we’ll find different libraries that do config management in any programming language. Scala is no exception.
In a previous article, we looked at PureConfig to avoid boilerplate code for configurations. However, sometimes we might run into problems if we miss out on some configuration values. These issues might only be visible during runtime and can cause unintended situations in a production environment.
In this tutorial, we’ll take a look at the ClearConfig library to make the configuration management safer and clearer to avoid the problems at runtime.
2. Common Configuration Problems
Even though PureConfig provides type safety and boilerplate reduction to configuration management, there are still some problems it doesn’t solve.
2.1. Configuration Mistakes in Different Environments
A common practice is to have separate configuration files for each environment. At times, there can be some mismatch between the configurations of different environments. These mistakes are sometimes identified very late and can cause downtime in production systems. Making it very difficult to manually verify every configuration file before deployment.
2.2. Unused Configurations
Sometimes we may add new configurations at the initial stages and later decide against them. It can be difficult to manually check if all the configurations in the file are used or not.
2.3. Difficulty to Override Configuration Values
We may also need to override some of the configuration values at runtime. It’s also useful if we can prioritize the configurations from different files or sources.
3. Advantages of ClearConfig
ClearConfig is a simple library that helps to overcome the problems mentioned above with ease. It provides valuable information regarding the configurations, such as:
- Where each configuration value is coming from
- The configuration sources that are available
- Configurations that are being overridden
- Checks for missing configurations in any environment
- List of used and unused configurations
- Early detection of configuration mistakes
4. Disadvantages
Unfortunately, ClearConfig only supports configuration as flat properties. Therefore, other formats, such as the popular HOCON format of configurations are not supported, which can make it difficult to migrate existing projects that are configured in other formats.
5. Setup
Let’s start by adding the ClearConfig library to the build.sbt:
libraryDependencies += "com.github.japgolly.clearconfig" %% "core" % "3.0.0"
Note that ClearConfig v3.0.0 only supports Scala 2.13 or above. Therefore, in this tutorial, we’ll be using Scala 3.
6. Usage
Let’s see how we can process the configurations using ClearConfig. We’ll be using the same configuration keys used in the PureConfig tutorial. However, since the HOCON format is not supported, we’ll need to re-write some of the configurations.
6.1. Single File Config Source
Let’s start with a simple configuration in notification.conf:
notificationUrl = http://mynotificationservice.com/push
params = status=completed
intervalInMin = 2
Next, we can create a case class corresponding to the config file:
final case class CCNotificationConfig(
notificationUrl: String,
params: String,
intervalInMin: Option[Int]
)
ClearConfig, we need to add the necessary imports as below to proceed with reading the configurations:
import japgolly.clearconfig._
import cats.implicits._
Since ClearConfig is built on top of the Cats library, we’re including cat.implicits._ in the imports. Next, we need to create a mapping between the configurations and case class fields. We can create this mapping within the companion object of the configuration case class:
def notificationConfig: ConfigDef[CCNotificationConfig] =
(
ConfigDef.need[String]("notificationUrl"),
ConfigDef.need[String]("params"),
ConfigDef.get[Int]("intervalInMin")
).mapN(apply)
For the sake of simplicity, we’re using cats.Id rather than an effect type such as IO. *For reading required fields, we can use ConfigDef.need(). If the property key is optional, we can use the method ConfigDef.get(). This will lift the value in Option.* Now, we need to define the ConfigSource, which essentially maps the above-defined ConfigDef to the configuration file:
def configSources: ConfigSources[Id] =
ConfigSource.propFileOnClasspath[Id]("/notification.conf", optional = false)
We are now ready to read the file and map the configurations into the case class:
val notificationConfig: CCNotificationConfig =
CCNotificationConfig.notificationConfig
.run(CCNotificationConfig.configSources)
.getOrDie()
6.2. Multi File Config Sources
ClearConfig also has the ability to read multiple files. Let’s assume that we have configuration files for each environment. For example, let’s create notification-prod.conf file with the same keys as before:
notificationUrl = http://live.notification.com/push
params = status=completed
intervalInMin = 5
Now, let’s define a config source for this new file as and combine it with the previous source using > combinator:
def configSources: ConfigSources[Id] =
ConfigSource.propFileOnClasspath[Id]("/notification-prod.conf", optional = true) >
ConfigSource.propFileOnClasspath[Id]("/notification.conf", optional = false)
The sources are defined in the order of priority. Here, notification-prod.conf has more priority than notification.conf file. Hence when we run the config source, it will use the values from the notification-prod.conf in the case class. If we remove the intervalInMin key from notification-prod.conf, then the value for it will be taken from notification.conf file. Additionally, while defining the config sources, we can provide the optional flag. If the flag is set as false, then the application will throw an exception if the file is not found. This way, we can build a hierarchy of configuration sources based on the priority. If there is a mistake in the entire hierarchy, the application will fail to start. This will ensure that we’ll not have any unexpected configuration errors in the production environment.
6.3. Using Environment Variables as Source
In the above section, we used different configuration files as config sources. However, it is also possible to read the config values from environment variables. This is extremely useful for defining some of the properties at the system level instead of each file. For this, we need to use the environment config source:
def configSources: ConfigSources[Id] = ConfigSource.environment[Id] >
ConfigSource.propFileOnClasspath[Id]("/notification-prod.conf", optional = true) >
ConfigSource.propFileOnClasspath[Id]("/notification.conf", optional = false)
Now, it will use the values from the environment variable if available. Otherwise, it will take the values from the files.
6.4. Using JVM Properties
Similar to environment variables, we can also read config values from Java properties provided using the –D flag. We can use ConfigSource.system as:
ConfigSource.propFileOnClasspath[Id]("/notification.conf", optional = false) >
ConfigSource.system[Id]
Now, we can pass the config values while starting the JVM using the -D flag, which will be used in the application.
6.5. Using Config Files from Outside Classpath
ClearConfig can also use configuration files outside the classpath as a config source. This is useful in having a fixed path for some of the configuration files. We can use the method propFile instead of propFileOnClasspath:
ConfigSource.propFile[Id](sysFileFullPath, optional = true)
6.6. Using Prefix for Grouping Configurations
As mentioned, ClearConfig can only read flattened keys for configurations. However, we can group the config properties using the dot notation like a package path for better readability. ClearConfig provides a way to use prefix values and read the properties without providing the full path each time. Let’s re-write the configuration HOCON values we used in PureConfig as kafka.conf:
kafka.port = 8090
kafka.bootstrap-server = kafka.mydomain.com
kafka.protocol = https
kafka.timeout = 2s
Note that the config keys are prefixed with kafka. Instead of the HOCON format. Now, we can build the config sources as:
val kafkaConfigSource: ConfigSources[Id] =
ConfigSource.propFileOnClasspath[Id]("kafka.conf", optional = false)
val kafkaConfig: KafkaConfig = KafkaConfig.kafkaConfigDef
.withPrefix("kafka.")
.run(kafkaConfigSource)
.getOrDie()
Before running the source, we used withPrefix(“kafka.”). This will automatically append the key kafka. to each property key of the config.
6.7. Case Insensitive Configurations
So far, we have used case-sensitive configurations. Sometimes, we need to ignore the case while reading configurations. For example, all the environment variables are used in an upper-case format as a standard practice. If we want to handle such cases, we can apply the method caseInsensitive to the source. For instance, let’s apply case-insensitivity to the environment source:
def configSources: ConfigSources[Id] = ConfigSource.environment[Id].caseInsensitive
6.8. Using Custom Types in Configuration
By default, ClearConfig can read all the basic data types in Scala, such as Int, String, Long, Boolean, FiniteDuration, etc. Additionally, we might need to use custom types. We can do that by providing the ConfigValueParser implicit. Let’s define an ADT using a sealed trait for handling protocol:
sealed trait Protocol
object Protocol {
case object Http extends Protocol
case object Https extends Protocol
}
To use this type, we need to define the parser:
given protocolParser: ConfigValueParser[Protocol] = ConfigValueParser
.oneOf[Protocol]("http" -> Protocol.Http, "https" -> Protocol.Https)
.preprocessValue(_.toLowerCase)
Now, we can use the type directly while reading the property as:
ConfigDef.need[Protocol]("protocol")
7. Configuration Report
One of the most powerful features of ClearConfig is its ability to generate a very comprehensive configuration usage report. This tabular report gives complete clarity on which configs are used or unused and in which order they are considered. Let’s generate a report for notification configurations. For this, we need to use the method withReport before running the source:
val (notificationConfigResult, notificationReport) = CCNotificationConfig.notificationConfig
.withReport.run(CCNotificationConfig.configSources).getOrDie()
As a result, we can now use notificationReport to print a report. To generate the completed report, we can invoke the method full() on notificationReport:
println(notificationReport.full)
This will print a tabular report with all the configurations. If we want to avoid some of the sources from the report, we can ignore them by:
val reducedReport = notificationReport.mapUnused(_.withoutSources(ConfigSourceName.environment, ConfigSourceName.system)).full
println(reducedReport)
As a result, this will generate a report without environment and system sources:
8. Conclusion
In this article, we looked at ClearConfig and how it can make configuration handling easier and safer.
As always, the sample code used here is available over on GitHub.