Use Liquibase to Safely Evolve Your Database Schema

1. Overview

In this quick tutorial, we’ll make use of Liquibase to evolve the database schema of a Java web application.

We’ll look at a general Java app first, and we’re also going to take a focused look at some interesting options available for Spring and Hibernate.

Very briefly, the core of using Liquibase is the changelog file, an XML file that describes changes that need for our DB, at a time.

We can have many changelog files for our DB changes throughout the time.

All these are referenced in one root changelog file master.xml that is executing when Liquibase is running.

We can use the include and includeAll attributes for nesting other changelog files under this root changelog.

Let’s start with the Maven dependency (v.3.6.2) we need to add to our pom.xml:

<dependency>
    <groupId>org.liquibase</groupId>
    <artifactId>liquibase-core</artifactId>
    <version>3.6.2</version>
</dependency>

We can also check if there’s a newer version of liquibase-core here.

2. The Database Change Log

Now let’s take a look at a simple changelog file, with name 20170503041524_added_entity_Car.xml

This one only adds a new  table – car – with id, make, brand and price columns:

<databaseChangeLog xmlns="http://www.liquibase.org/xml/ns/dbchangelog" xmlns:ext="http://www.liquibase.org/xml/ns/dbchangelog-ext" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://www.liquibase.org/xml/ns/dbchangelog http://www.liquibase.org/xml/ns/dbchangelog/dbchangelog-3.4.xsd http://www.liquibase.org/xml/ns/dbchangelog-ext http://www.liquibase.org/xml/ns/dbchangelog/dbchangelog-ext.xsd">
   ...
   <!--Added the entity Car-->
   <changeSet id="20170503041524-1" author="user">
      <createTable tableName="car">
         <column name="id" type="bigint" autoIncrement="${autoIncrement}">
            <constraints primaryKey="true" nullable="false" />
         </column>
         <column name="make" type="varchar(255)">
            <constraints nullable="true" />
         </column>
         <column name="brand" type="varchar(255)">
            <constraints nullable="true" />
         </column>
         <column name="price" type="double">
            <constraints nullable="true" />
         </column>
      </createTable>
   </changeSet>
</databaseChangeLog>

Note how the change set is identified by an id and an author to make sure it can be uniquely identified and only applied once.

Here is an example of the master.xml, which also contains the above changelog file:

<databaseChangeLog xmlns="http://www.liquibase.org/xml/ns/dbchangelog" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://www.liquibase.org/xml/ns/dbchangelog http://www.liquibase.org/xml/ns/dbchangelog/dbchangelog-3.4.xsd">
   
    <include file="classpath:config/liquibase/changelog/00000000000000_initial_schema.xml" relativeToChangelogFile="false" />
   
    <include file="classpath:config/liquibase/changelog/20170503041524_added_entity_Car.xml" relativeToChangelogFile="false" />

</databaseChangeLog>

Notice the timestamps in changelogs’ references. The master.xml file is used by Liquibase to track the changesets that have already been applied to the database. The date timestamp allows Liquibase to determine which changesets have already been applied and which still need to be applied.

Once the changesets are applied, normally we cannot edit the same ones further. It’s best to make a new changeset always. That’s how Liquibase ensures consistency of our database versioning.

Let’s now see how to wire this into our application and make sure that it runs when the application starts up.

3. Run Liquibase With a Spring Bean

Our first option to run the changes on application startup is via a Spring bean.

There are, of course, many other ways, but this is a good, simple way to go if we’re dealing with a Spring application:

@Bean
public SpringLiquibase liquibase() {
    SpringLiquibase liquibase = new SpringLiquibase();
    liquibase.setChangeLog("classpath:config/liquibase/master.xml");
    liquibase.setDataSource(dataSource());
    return liquibase;
}

For our GitHub example, we apply a more advanced way:

@Bean
public SpringLiquibase liquibase(@Qualifier("taskExecutor") TaskExecutor taskExecutor,
        DataSource dataSource, LiquibaseProperties liquibaseProperties) {

    // Use liquibase.integration.spring.SpringLiquibase if you don't want Liquibase to start asynchronously
    SpringLiquibase liquibase = new AsyncSpringLiquibase(taskExecutor, env);
    liquibase.setDataSource(dataSource);
    liquibase.setChangeLog("classpath:config/liquibase/master.xml");
    liquibase.setContexts(liquibaseProperties.getContexts());
    liquibase.setDefaultSchema(liquibaseProperties.getDefaultSchema());
    liquibase.setDropFirst(liquibaseProperties.isDropFirst());
    if (env.acceptsProfiles(JHipsterConstants.SPRING_PROFILE_NO_LIQUIBASE)) {
        liquibase.setShouldRun(false);
    } else {
        liquibase.setShouldRun(liquibaseProperties.isEnabled());
        log.debug("Configuring Liquibase");
    }
    return liquibase;
}

Note how we’re pointing it to a valid master.xml file that needs to exist on the classpath.

4. Use Liquibase With Spring Boot

If we’re using Spring Boot, there is no need to define a bean for Liquibase, but we still need to make sure we add the liquibase-core dependency.

Then all we need is to put our change log in src/main/resources/config/liquibase/master.xml, and Liquibase migrations will run automatically on startup.

We can also change the default changelog file using the liquibase.change-log property:

liquibase.change-log=classpath:liquibase-changeLog.xml

5. Disable Liquibase in Spring Boot

Sometimes, we may need to disable Liquibase migration’s execution on startup.

The simplest option we have is to use a spring.liquibase.enabled property. This way, all the remaining Liquibase configuration stays untouched.

Here’s the example for Spring Boot 2:

spring.liquibase.enabled=false

For Spring Boot 1.x, we need to use a liquibase.enabled property:

liquibase.enabled=false

6. Generate the changelog With a Maven Plugin

Instead of writing the changelog file manually, we can use the Liquibase Maven plugin to generate one from an existing database and save ourselves a lot of work.

6.1. Plugin Configuration

Here are the changes to our pom.xml:

<plugin>
   <groupId>org.liquibase</groupId>
   <artifactId>liquibase-maven-plugin</artifactId>
   <version>${liquibase.version}</version>
    ...
   <configuration>
      ...
      <driver>org.h2.Driver</driver>
      <url>jdbc:h2:file:./target/h2db/db/carapp</url>
      <username>carapp</username>
      <password />
      <outputChangeLogFile>src/main/resources/liquibase-outputChangeLog.xml</outputChangeLogFile>
   </configuration>
</plugin>

6.2. Generate a Changelog From an Existing Database

We can use the plugin to generate a changelog from an existing database:

mvn liquibase:generateChangeLog

The end result is a changelog file that we can use either to create an initial DB schema or to populate data.

Here’s how that would look:

<databaseChangeLog ...>
   <changeSet id="00000000000001" author="jhipster">
      <createTable tableName="jhi_persistent_audit_event">
         <column name="event_id" type="bigint" autoIncrement="${autoIncrement}">
            <constraints primaryKey="true" nullable="false" />
         </column>
         <column name="principal" type="varchar(50)">
            <constraints nullable="false" />
         </column>
         <column name="event_date" type="timestamp" />
         <column name="event_type" type="varchar(255)" />
      </createTable>
      <createTable tableName="jhi_persistent_audit_evt_data">
         <column name="event_id" type="bigint">
            <constraints nullable="false" />
         </column>
         <column name="name" type="varchar(150)">
            <constraints nullable="false" />
         </column>
         <column name="value" type="varchar(255)" />
      </createTable>
      <addPrimaryKey columnNames="event_id, name" tableName="jhi_persistent_audit_evt_data" />
      <createIndex indexName="idx_persistent_audit_event" tableName="jhi_persistent_audit_event" unique="false">
         <column name="principal" type="varchar(50)" />
         <column name="event_date" type="timestamp" />
      </createIndex>
      <createIndex indexName="idx_persistent_audit_evt_data" tableName="jhi_persistent_audit_evt_data" unique="false">
         <column name="event_id" type="bigint" />
      </createIndex>
      <addForeignKeyConstraint baseColumnNames="event_id" baseTableName="jhi_persistent_audit_evt_data" constraintName="fk_evt_pers_audit_evt_data" referencedColumnNames="event_id" referencedTableName="jhi_persistent_audit_event" />
   </changeSet>

   <changeSet id="20170503041524-1" author="jhipster">
      <createTable tableName="car">
         <column name="id" type="bigint" autoIncrement="${autoIncrement}">
            <constraints primaryKey="true" nullable="false" />
         </column>
         <column name="make" type="varchar(255)">
            <constraints nullable="true" />
         </column>
         <column name="brand" type="varchar(255)">
            <constraints nullable="true" />
         </column>
         <column name="price" type="double">
            <constraints nullable="true" />
         </column>
         <!-- jhipster-needle-liquibase-add-column - JHipster will add columns here, do not remove-->
      </createTable>
   </changeSet>
</databaseChangeLog>

6.3. Generate a ChangeLog From Diffs Between Two Databases

We can use the plugin to generate a changeLog file from the differences between two existing databases (for example, development and production):

mvn liquibase:diff

Here are the properties:

changeLogFile=src/main/resources/config/liquibase/master.xml
url=jdbc:h2:file:./target/h2db/db/carapp 
username=carapp 
password= 
driver=com.mysql.jdbc.Driver 
referenceUrl=jdbc:h2:file:./target/h2db/db/carapp2 
referenceDriver=org.h2.Driver 
referenceUsername=tutorialuser2 
referencePassword=tutorialmy5ql2

And here’s an example snippet of the generated changeLog:

<databaseChangeLog ...>
    <changeSet author="John" id="1439227853089-1">
        <dropColumn columnName="brand" tableName="car"/>
    </changeSet>
</databaseChangeLog>

This is a super powerful way to evolve our DB by, for example, allowing Hibernate to auto-generate a new schema for development and then using that as a reference point against the old schema.

7. Use the Liquibase Hibernate Plugin

In case our application uses Hibernate, we’re going to take a look at a very useful way of generating the changeLog, which is the liquibase-hibernate plugin.

7.1. Plugin Configuration

First, let’s get the new plugin configured and use the right dependencies:

<plugin>
   <groupId>org.liquibase</groupId>
   <artifactId>liquibase-maven-plugin</artifactId>
   <version>3.6.2</version>
   <dependencies>
      ...
      <dependency>
         <groupId>org.liquibase.ext</groupId>
         <artifactId>liquibase-hibernate5</artifactId>
         <version>${liquibase-hibernate5.version}</version>
      </dependency>
     ...
   </dependencies>
   ...
</plugin>

7.2. Generate a changelog From Diffs Between a Database and Persistence Entities

We can use this plugin to generate a changelog file from the differences between an existing database (for example, production) and our new persistence entities.

So, to make things simple, once an entity is modified, we can simply generate the changes against the old DB schema, getting a clean, powerful way to evolve our schema in production.

Here are the liquibase properties in our plugin configuration inside the pom.xml:

<configuration>
   <changeLogFile>src/main/resources/config/liquibase/master.xml</changeLogFile>
   <diffChangeLogFile>src/main/resources/config/liquibase/changelog/${maven.build.timestamp}_changelog.xml</diffChangeLogFile>
   <driver>org.h2.Driver</driver>
   <url>jdbc:h2:file:./target/h2db/db/carapp</url>
   <defaultSchemaName />
   <username>carapp</username>
   <password />
   <referenceUrl>hibernate:spring:com.car.app.domain?dialect=org.hibernate.dialect.H2Dialect
    &hibernate.physical_naming_strategy=org.springframework.boot.orm.jpa.hibernate.SpringPhysicalNamingStrategy
    &hibernate.implicit_naming_strategy=org.springframework.boot.orm.jpa.hibernate.SpringImplicitNamingStrategy
   </referenceUrl>
   <verbose>true</verbose>
   <logging>debug</logging>
</configuration>

Note that the referenceUrl is using package scan, so the dialect parameter is required.

8. Generate the changelog in IntelliJ IDEA Using the JPA Buddy Plugin

If we’re using a non-Hibernate ORM (e.g., EclipseLink or OpenJPA) or we don’t want to add extra dependencies like the liquibase-hibernate plugin, we can use JPA Buddy. This IntelliJ IDEA plugin integrates useful features of Liquibase into the IDE.

To generate a differential changelog, we simply install the plugin and then call the action from the JPA Structure panel. We select what source we want to compare (database, JPA entities or Liquibase snapshot) with what target (database or Liquibase snapshot).

JPA Buddy will generate the changeLog as shown in the animation below:

Another advantage of JPA Buddy over the liquibase-hibernate plugin is the ability to override default mappings between Java and database types. Also, it works correctly with Hibernate custom types and JPA converters.

9. Conclusion

In this article, we illustrated several ways to use Liquibase and get to a safe and mature way of evolving and refactoring the DB schema of a Java app.

The implementation of all these examples and code snippets is available on GitHub.