Notes on Upgrading Datomic On-Prem to v1.0.6726

· Tagged Datomic · 7 minute read

After the news from the Clojure/conj that Datomic is now free, I was excited to get home and upgrade and access newer features and high-availability. Herein are notes from how that upgrade process went that I hope will be helpful to anyone else upgrading, including difficulties I had running the official Datomic AMIs.

The system I was running was using the very old Datomic On-Prem 0.9.5544 version. Your mileage may vary upgrading from different versions, but I was impressed that the large version jump didn’t seem to cause any issues.

Disclaimer: This post is for informational purposes only; I can’t guarantee these details are correct / harmless for your configuration, and disclaim any responsibility for damage they may cause. I’m only detailing my personal experience in hopes the notes will be helpful for others.

First, before doing anything to your setup, make sure you have a current backup of your database (instructions here). I didn’t run into any issues that required me to restore a backup, but safety first.

Peer Library

Upgrading the peer library was quite trivial and pleasant. Peer libraries are backwards compatible past version 0.9.4532 (released 2014-02-07). You need only pull the latest version of the peer library into your project via your deps.edn (or project.clj) file, and the peer should have no trouble interacting with your existing transactor(s).

N.B. the Maven artifact ID has changed in this release as the library is now available via public sources (i.e. Maven Central):

com.datomic/peer {:mvn/version "1.0.6726"}

With this change, you can now remove configuration of the my.datomic.com Maven repository from your deps.edn or ~.m2/settings.xml files (if you don’t use those repositories for other dependencies). You may also have Datomic download credentials configured in your CI environment that you can now remove.

Transactors (on AWS)

Upgrading my transactor was where I ran into some trouble, and where I expect these notes to be most helpful.

I’m assuming you’re now running updated (v1.0.6726) peers, but transactors should also be backwards compatible back to version 0.9.4532, so you’re likely safe either way. That said, I haven’t tested that configuration.

Upgraded transactors can run alongside older transactors, even if you were previously running a version of Datomic that did not support “high availability” (i.e. multiple transactors), so my recommended approach here is to spin up a new CloudFormation stack with one or more upgraded transactors alongside your existing CloudFormation stack running the old transactor(s). The new transactor(s) will not be used until the old transactor(s) have failed to communicate a heartbeat to storage, so nothing will immediately change with respect to your transactions when the new stack starts running.

If you have existing cf.properties and transactor.properties files, you can simply update these and use the bin/datomic create-cf-template command to generate a new cf.json file. I found my old notes on updating Datomic to be very helpful in navigating this process.

Specifically, update:

  • cf.properties
    • datomic-version to 1.0.6726
    • aws-autoscaling-group-size: if you were previously limited by your license to running a single transactor, you may now want to increase this value to 2 to take advantage of high availability.
  • transactor.properties
    • license-key: you can now remove this value entirely. Leaving it in doesn’t hurt anything either.
    • aws-cloudwatch-dimension-value: you may consider using a new name here; I found this helpful in ensuring my new transactors were running properly. I assume below that you have used a new dimension name.

With this approach, your existing AWS IAM Roles and SecurityGroups will be used for the new transactor EC2 instances, so all of the access restrictions and capabilities should remain the same as before.

Now you’re ready to generate your new CloudFormation configuration:

bin/datomic create-cf-template my-transactor.properties my-cf.properties my-cf.json

Make sure you’re running this command from the directory where you’ve downloaded and unpacked the 1.0.6727 version of Datomic.

At this point, you should be able to use the newly generated my-cf.json file to run your new transactor stack—but this was not the case for me; my transactors kept restarting shortly after launch, and it took several hours of work to make sense of what was going on and create a solution.

By my read, the official Datomic AMIs are running Java 8, whereas Datomic requires Java 11 since 2023-03-31. Specifically, java -version reported openjdk version "1.8.0_362" in the ami-cca215b4 AMI that bin/datomic injected into my cf.json for the us-west-2 region.

Short of building your own AMI, you can fix this issue by upgrading java via the UserData script embedded in the cf.json file.

Add the five lines between the blank lines here to your cf.json file (the blank lines are not necessary and only added for clarity):

        {"Fn::Join":
         ["=", ["export DATOMIC_VERSION", {"Ref":"DatomicVersion"}]]},

        "curl -LO https://corretto.aws/downloads/latest/amazon-corretto-11-x64-linux-jdk.tar.gz",
        "tar -xvzf amazon-corretto-11-x64-linux-jdk.tar.gz -C /usr/lib/jvm/",
        "JVM_DIR=`ls -d /usr/lib/jvm/amazon-corretto-*-linux-x64/bin`",
        "alternatives --install /usr/bin/java java $JVM_DIR/java 20000 --slave /usr/bin/javac javac $JVM_DIR/javac",
        "alternatives --set java $JVM_DIR/java",

        "cd \/datomic", "cat <<EOF >aws.properties",

Now that you have a corrected cf.json file, you can spin up the CloudFormation stack with:

bin/datomic create-cf-stack us-west-2 DatomicTransactors my-cf.json

Make sure to…

  • Replace us-west-2 with your preferred AWS region.
  • Replace DatomicTransactors with the name you’d like to use for your CloudFormation stack. You’ll need to use a different name from your pre-existing CloudFormation stack.

If you now open CloudFormation on the AWS Management Console, you’ll see your new stack, which will be created very quickly. Navigate to EC2 > Instances, and you should see your new transactor instance(s) spinning up. The Name field will match the name you set for the CloudFormation stack.

Wait a few minutes for the Instance state to change to Running and for the Status check field to say checks have passed.

Now navigate to CloudWatch > Metrics > All Metrics, select Datomic, then Transactor. If you used a new name for aws-cloudwatch-dimension-value, as suggested above, the presence of metrics with this new dimension name means the transactor(s) are running and reporting metrics.

At this point you’re ready to make one of your new transactors be the primary transactor. All you need to do is knock your old transactors offline briefly and the (still) running transactor(s) will take over. Obviously you’ll want to do this at a time when it is OK (or most OK) for your database to have downtime in case something goes wrong; that said, if things are configured properly, it shouldn’t take more than a few seconds before the peers start communicating with the new transactor(s).

If you were previously only running one transactor, simply go back to the EC2 > Instances page, select the old transactor, click Instance state and Reboot instance. After a few seconds, you can confirm that things are working properly by triggering a transaction from any application you have talking to the database and / or by going back to CloudWatch and checking the RemotePeers metric for the new transactor dimension—this metric represents the number of peers which are currently connected to the transactor(s).

If something has gone wrong, simply go to the CloudFormation page and delete the new transactor stack. The old transactor(s) will take over again after rebooting when there are no other transactors running.

To continue the wind down of the old transactors, or if you were running multiple transactors to begin with, go to EC2 > Autoscaling > Autoscaling Groups, select the old transactor group, click Edit on Group details, and change the Desired capacity and Minimum capacity (both) to 0. This will result in your old transactor(s) instances being terminated. You can watch this process play out on the EC2 > Instances page. Rebuilding these instances from scratch will take longer than rebooting an instance, so the reboot approach is preferable.

Once the old transactor instances are terminated, assuming everything is functioning properly, the only thing left to do is clean up:

Open the CloudFormation page and select the old transactor stack. If you’re cautious, you may want to download the JSON configuration for the old stack just in case you need to restore it.

When you’re ready, use the Delete button to delete the stack. You’re no longer running any instances, so you’re just removing the (now unused) launch configuration and autoscaling group.

Last but not least, you may want to commit your new cf.properties, transactor.properties, and cf.json files to revision control to make sure you don’t lose them.

Wrap-Up

You’re done! Enjoy Datomic v1.0.6727!

Many thanks to the fine people at Nubank for making Datomic free! I’m optimistic that this change will increase adoption of Datomic and make it a more viable option for hobby projects where the cost of Datomic may have previously made it untenable.

Was this post helpful or interesting? Please consider supporting my (independent) writing with a Brave or Lightning tip, and follow me on Mastodon for additional posts and content.

© 2024. All rights reserved.