Solving the diamond problem with shading

Dealing with difficult library upgrades has been a recurring task in my career as a software developer. This week I got a new tool into my toolbox for handling dependency conflicts.

I was working on a Scala project. It heavily uses a library that is no longer maintained. We would like to migrate to a newer library.

It would nice to migrate piece-by-piece: that would make both development and testing much easier. However, both the libaries depend on the different version of the same library. Our dependency graph looks something like this:

In our case, there’s no need to pass data structures between old-lib and new-lib, so this could work if v1 and v2 were somewhat compatible. Unfortunately that was not the case: the changes were so severe that our app would not even start if the newer version of base-lib was added to the classpath.

This is a variation of the diamond dependency problem. It’s an annoying problem in general, but in this case we were able to solve it easily by shading base-lib in old-lib. This means that we created our own version of old-lib that bundles base-lib with all the base-lib classes renamed and all the usage sites in old-lib changed to use the new names.

For example, net/quanttype/base_lib/ExampleClass.class would be renamed to shaded/net/quanttype/base_lib/ExampleClass.class in the resulting .jar file. Our dependency graph now looks like this:

Note that shaded-old-lib does not explicitly depend on old-lib or base-lib, since it already includes shaded versions of them.

Shading sounds difficult, but it was easy in practice using sbt-assembly’s built-in shading support. To do this, we need to create two sbt projects. The first one is the one that does shading:

lazy val shadedOldLibRoot = project
  .settings(
    libraryDependencies ++= Seq(
      // We depend on old-lib, which pulls in the correct version of base-lib
      "net.quanttype" %% "old-lib" % "1.0.0"
    ),
    assemblyShadeRules in assembly := Seq(
      // You can add more rules here if needed
      ShadeRule.rename("net.quanttype.base_lib.**" -> "shaded.net.quanttype.base_lib.@1").inAll,
    ),
    assemblyOption in assembly := (assemblyOption in assembly).value.copy(includeScala = false),
    skip in publish := true
  )

The second project publishes the first one without the dependency information:

lazy val shadedOldLib = project
  .settings(
    name := "shaded-old-lib",
    version in ThisBuild := "1.0.0",
    organization in ThisBuild := "net.quanttype"
    packageBin in Compile := (assembly in (shadedOldLibRoot, Compile)).value
  )

The resulting package can be installed to the local Ivy repository with sbt shadedOldLib/publishLocal. You can do full-fledged publish if you want – for us the local version was enough.

Add the new package to the dependencies of your main project:

libraryDependencies += "net.quanttype" %% "shaded-old-lib" % "1.0.0"

Finally, if you have used base-lib directly in your app, you’ll have rewrite any imports in your codebase. Like this:

find . -name "*.scala*" | \
  xargs sed -i .bak 's/net\.quanttype\.base_lib/shaded.\0/g'

Now you can add new-lib as a dependency and everything should just work.

When to use this?

There are some drawbacks to this approach.

The biggest one is that if old-lib or base-lib have any other dependencies, they get included in shaded-old-lib. If your app also depends on them, there may be new conflicts and they’re more confusing than before, since the conflict is not directly visible in the dependency graph. You can manually exclude those dependencies - see sbt-assembly’s instructions.

For us, shading solved the diamond problem neatly. Since we’re trying to migrate away from the old library, the drawbacks were acceptable.