Notes on Upgrading Datomic On-Prem to v1.0.6726
After the exciting news from the Clojure/conj that Datomic is now free, I was excited to get home and upgrade and access newer features and high-availability. Herein are notes from how that upgrade process went that I help will be helpful to anyone else upgrading, including difficulties I had running the official Datomic AMIs.
The system I was running was using the very old Datomic On-Prem 0.9.5544 version. Your mileage may vary upgrading from different versions, but I was impressed that the large version jump didn’t seem to cause any issues.
Disclaimer: This post is for informational purposes only; I can’t guarantee these details are correct / harmless for your configuration, and disclaim any responsibility for damage they may cause. I’m only detailing my personal experience in hopes the notes will be helpful for others.
First, before doing anything to your setup, make sure you have a current backup of your database (instructions here). I didn’t run into any issues that required me to restore a backup, but safety first.
Peer Library
Upgrading the peer library was quite trivial and pleasant. Peer
libraries are backwards compatible past version 0.9.4532 (released
2014-02-07). You need only pull the latest version of the peer
library into your project via your deps.edn
(or project.clj
) file,
and the peer should have no trouble interacting with your existing
transactor(s).
N.B. the Maven artifact ID has changed in this release as the library is now available via public sources (i.e. Maven Central):
com.datomic/peer {:mvn/version "1.0.6726"}
With this change, you can now remove configuration of the
my.datomic.com
Maven repository from your deps.edn
or
~.m2/settings.xml
files (if you don’t use those repositories for
other dependencies). You may also have Datomic download credentials
configured in your CI environment that you can now remove.
Transactors (on AWS)
Upgrading my transactor was where I ran into some trouble, and where I expect these notes to be most helpful.
I’m assuming you’re now running updated (v1.0.6726) peers, but transactors should also be backwards compatible back to version 0.9.4532, so you’re likely safe either way. That said, I haven’t tested that configuration.
Upgraded transactors can run alongside older transactors, even if you were previously running a version of Datomic that did not support “high availability” (i.e. multiple transactors), so my recommended approach here is to spin up a new CloudFormation stack with one or more upgraded transactors alongside your existing CloudFormation stack running the old transactor(s). The new transactor(s) will not be used until the old transactor(s) have failed to communicate a heartbeat to storage, so nothing will immediately change with respect to your transactions when the new stack starts running.
If you have existing cf.properties
and transactor.properties
files, you can simply update these and use the bin/datomic create-cf-template
command to generate a new cf.json
file. I found
my old notes on updating Datomic to be very
helpful in navigating this process.
Specifically, update:
cf.properties
datomic-version
to1.0.6726
aws-autoscaling-group-size
: if you were previously limited by your license to running a single transactor, you may now want to increase this value to2
to take advantage of high availability.
transactor.properties
license-key
: you can now remove this value entirely. Leaving it in doesn’t hurt anything either.aws-cloudwatch-dimension-value
: you may consider using a new name here; I found this helpful in ensuring my new transactors were running properly. I assume below that you have used a new dimension name.
With this approach, your existing AWS IAM Roles and SecurityGroups will be used for the new transactor EC2 instances, so all of the access restrictions and capabilities should remain the same as before.
Now you’re ready to generate your new CloudFormation configuration:
bin/datomic create-cf-template my-transactor.properties my-cf.properties my-cf.json
Make sure you’re running this command from the directory where you’ve downloaded and unpacked the 1.0.6727 version of Datomic.
At this point, you should be able to use the newly generated
my-cf.json
file to run your new transactor stack—but this was not
the case for me; my transactors kept restarting shortly after launch,
and it took several hours of work to make sense of what was going on
and create a solution.
By my read, the official Datomic AMIs are running Java 8, whereas
Datomic requires Java 11 since 2023-03-31.
Specifically, java -version
reported openjdk version "1.8.0_362"
in the ami-cca215b4
AMI that bin/datomic
injected into my
cf.json
for the us-west-2
region.
Short of building your own AMI, you can fix this issue by upgrading
java
via the UserData
script embedded in the cf.json
file.
Add the five lines between the blank lines here to your cf.json
file
(the blank lines are not necessary and only added for clarity):
{"Fn::Join":
["=", ["export DATOMIC_VERSION", {"Ref":"DatomicVersion"}]]},
"curl -LO https://corretto.aws/downloads/latest/amazon-corretto-11-x64-linux-jdk.tar.gz",
"tar -xvzf amazon-corretto-11-x64-linux-jdk.tar.gz -C /usr/lib/jvm/",
"JVM_DIR=`ls -d /usr/lib/jvm/amazon-corretto-*-linux-x64/bin`",
"alternatives --install /usr/bin/java java $JVM_DIR/java 20000 --slave /usr/bin/javac javac $JVM_DIR/javac",
"alternatives --set java $JVM_DIR/java",
"cd \/datomic", "cat <<EOF >aws.properties",
Now that you have a corrected cf.json
file, you can spin up the
CloudFormation stack with:
bin/datomic create-cf-stack us-west-2 DatomicTransactors my-cf.json
Make sure to…
- Replace
us-west-2
with your preferred AWS region. - Replace
DatomicTransactors
with the name you’d like to use for your CloudFormation stack. You’ll need to use a different name from your pre-existing CloudFormation stack.
If you now open CloudFormation on the AWS Management Console, you’ll
see your new stack, which will be created very quickly. Navigate to
EC2 > Instances, and you should see your new transactor instance(s)
spinning up. The Name
field will match the name you set for the
CloudFormation stack.
Wait a few minutes for the Instance state
to change to Running
and
for the Status check
field to say checks have passed.
Now navigate to CloudWatch > Metrics > All Metrics, select Datomic
,
then Transactor
. If you used a new name for
aws-cloudwatch-dimension-value
, as suggested above, the presence of
metrics with this new dimension name means the transactor(s) are
running and reporting metrics.
At this point you’re ready to make one of your new transactors be the primary transactor. All you need to do is knock your old transactors offline briefly and the (still) running transactor(s) will take over. Obviously you’ll want to do this at a time when it is OK (or most OK) for your database to have downtime in case something goes wrong; that said, if things are configured properly, it shouldn’t take more than a few seconds before the peers start communicating with the new transactor(s).
If you were previously only running one transactor, simply go back to
the EC2 > Instances page, select the old transactor, click Instance state
and Reboot instance
. After a few seconds, you can confirm
that things are working properly by triggering a transaction from any
application you have talking to the database and / or by going back to
CloudWatch and checking the RemotePeers
metric for the new
transactor dimension—this metric represents the number of peers
which are currently connected to the transactor(s).
If something has gone wrong, simply go to the CloudFormation page and delete the new transactor stack. The old transactor(s) will take over again after rebooting when there are no other transactors running.
To continue the wind down of the old transactors, or if you were
running multiple transactors to begin with, go to EC2 > Autoscaling >
Autoscaling Groups, select the old transactor group, click Edit
on
Group details
, and change the Desired capacity
and Minimum capacity
(both) to 0. This will result in your old transactor(s)
instances being terminated. You can watch this process play out on
the EC2 > Instances page. Rebuilding these instances from scratch
will take longer than rebooting an instance, so the reboot approach is
preferable.
Once the old transactor instances are terminated, assuming everything is functioning properly, the only thing left to do is clean up:
Open the CloudFormation page and select the old transactor stack. If you’re cautious, you may want to download the JSON configuration for the old stack just in case you need to restore it.
When you’re ready, use the Delete
button to delete the stack.
You’re no longer running any instances, so you’re just removing the
(now unused) launch configuration and autoscaling group.
Last but not least, you may want to commit your new cf.properties
,
transactor.properties
, and cf.json
files to revision control to
make sure you don’t lose them.
Wrap-Up
You’re done! Enjoy Datomic v1.0.6727!
Many thanks to the fine people at Nubank for making Datomic free! I’m optimistic that this change will increase adoption of Datomic and make it a more viable option for hobby projects where the cost of Datomic may have previously made it untenable.