Dealing with difficult library upgrades has been a recurring task in my career as a software developer. This week I got a new tool into my toolbox for handling dependency conflicts.
I was working on a Scala project. It heavily uses a library that is no longer maintained. We would like to migrate to a newer library.
It would nice to migrate piece-by-piece: that would make both development and testing much easier. However, both the libaries depend on the different version of the same library. Our dependency graph looks something like this:
In our case, there’s no need to pass data structures between
new-lib, so this could work if v1 and v2 were somewhat compatible.
Unfortunately that was not the case: the changes were so severe that our app
would not even start if the newer version of
base-lib was added to the
This is a variation of the diamond dependency problem. It’s an
annoying problem in general, but in this case we were able to solve it easily by
old-lib. This means that we created our own version of
old-lib that bundles
base-lib with all the
base-lib classes renamed
and all the usage sites in
old-lib changed to use the new names.
net/quanttype/base_lib/ExampleClass.class would be renamed to
shaded/net/quanttype/base_lib/ExampleClass.class in the resulting
Our dependency graph now looks like this:
shaded-old-lib does not explicitly depend on
base-lib, since it already includes shaded versions of them.
lazy val shadedOldLibRoot = project .settings( libraryDependencies ++= Seq( // We depend on old-lib, which pulls in the correct version of base-lib "net.quanttype" %% "old-lib" % "1.0.0" ), assemblyShadeRules in assembly := Seq( // You can add more rules here if needed ShadeRule.rename("net.quanttype.base_lib.**" -> "shaded.net.quanttype.base_lib.@1").inAll, ), assemblyOption in assembly := (assemblyOption in assembly).value.copy(includeScala = false), skip in publish := true )
The second project publishes the first one without the dependency information:
lazy val shadedOldLib = project .settings( name := "shaded-old-lib", version in ThisBuild := "1.0.0", organization in ThisBuild := "net.quanttype" packageBin in Compile := (assembly in (shadedOldLibRoot, Compile)).value )
The resulting package can be installed to the local Ivy repository with
sbt shadedOldLib/publishLocal. You can do full-fledged
publish if you want – for
us the local version was enough.
Add the new package to the dependencies of your main project:
libraryDependencies += "net.quanttype" %% "shaded-old-lib" % "1.0.0"
Finally, if you have used
base-lib directly in your app, you’ll have rewrite
any imports in your codebase. Like this:
find . -name "*.scala*" | \ xargs sed -i .bak 's/net\.quanttype\.base_lib/shaded.\0/g'
Now you can add
new-lib as a dependency and everything should just work.
When to use this?
There are some drawbacks to this approach.
The biggest one is that if
base-lib have any other dependencies,
they get included in
shaded-old-lib. If your app also depends on them, there
may be new conflicts and they’re more confusing than before, since the conflict
is not directly visible in the dependency graph. You can manually exclude those
dependencies - see sbt-assembly’s instructions.
For us, shading solved the diamond problem neatly. Since we’re trying to migrate away from the old library, the drawbacks were acceptable.