How to debug dependency conflicts in Maven and Gradle

Dependency hell

I bet there is no one among you who is unfamiliar with the concept “Dependency hell”.

This problem exists in all technologies and languages. People feel distracted and helpless in the face of this problem. I personally hate dependency conflicts in JavaScript (npm), Ruby, Python. I just can do nothing to solve them.

But fortunately, I learnt how to solve dependency conflicts in Java. At least in Maven and Gradle – two most popular build tools in Java world. Let me share my knowledge with you.

Background

As an author of popular open-source library Selenide, I often have to investigates issues caused by Selenide dependencies, not Selenide itself. Selenium, Guava, WebDriverManager, Netty to name a few.

Let me show some examples.

Sample project

Let’s assume that you have a project using Selenide for automated UI tests. You can refer to todo mvc tests if you wish.

While you have only a dependency on Selenide, everything works fine.

build.gradle:

dependencies {
  testImplementation("com.codeborne:selenide:5.14.0")
}

or pom.xml:

 <dependencies>
  <dependency>
    <groupId>com.codeborne</groupId>
    <artifactId>selenide</artifactId>
    <version>5.14.0</version>
    <scope>test</scope>
  </dependency>
</dependencies>

First problem

Now let’s assume you want to use TestNG in your project. Then you add a dependency:

build.gradle:

dependencies {
  testImplementation("org.testng:testng:7.3.0")
  testImplementation("com.codeborne:selenide:5.14.0")
}

or pom.xml:

<dependency>
  <groupId>org.testng</groupId>
  <artifactId>testng</artifactId>
  <version>7.3.0</version>
  <scope>test</scope>
</dependency>

This is the road to dependency hell. 🙂

Let’s run the tests

If you run mvn test or gradle test, your tests fail. Just because you added testng*.jar – even if you don’t use it yet.

Let’s look at the test report

Your tests have failed with the following error:

java.lang.NoSuchMethodError: 'java.util.stream.Collector
      com.google.common.collect.ImmutableList.toImmutableList()'
  at org.openqa.selenium.chrome.ChromeOptions.asMap()
  at org.openqa.selenium.MutableCapabilities.merge()
  at com.codeborne.selenide.webdriver.MergeableCapabilities

Errors like “NoSuchMethod” clearly indicate that you have some mismatching versions of dependencies.

In this case, dependency selenium-chrome*.jar (which contains ChromeOptions) was built with one version of guava-*jar (which contains class ImmutableList), but is being run with another. We are stuck. Too many people don’t know what to do with this problem.

The cure

Luckily, both Gradle and Maven have a simple command that prints out a dependency tree of your project. Let’s try it.

Maven

run command mvn dependency:tree in terminal.

org.selenide.examples:todomvc:jar:1.0-SNAPSHOT
+- org.testng:testng:jar:7.3.0:test
| +- com.google.inject:guice:jar:no_aop:4.2.2:test
| | \- com.google.guava:guava:jar:25.1-android:test
+- com.codeborne:selenide:jar:5.14.0:test
| +- org.seleniumhq.selenium:selenium-java:jar:3.141.59:test
| | +- org.seleniumhq.selenium:selenium-chrome-driver:jar:3.141.59:test

Gradle

run command gradle dependencies in terminal.

testRuntimeClasspath - Runtime classpath of source set 'test'.
+--- org.testng:testng:7.3.0
| +--- com.google.inject:guice:4.2.2
| | +--- javax.inject:javax.inject:1
| | \--- com.google.guava:guava:25.1-android
+--- com.codeborne:selenide:5.14.0
| +--- org.seleniumhq.selenium:selenium-java:3.141.59
| | +--- org.seleniumhq.selenium:selenium-api:3.141.59
| | +--- org.seleniumhq.selenium:selenium-chrome-driver:3.141.59
| | | +--- org.seleniumhq.selenium:selenium-remote-driver:3.141.59
| | | | +--- com.google.guava:guava:25.0-jre -> 25.1-android

You see a similar output: both Maven and Gradle decided to use version guava:25.1-android instead of guava:25.0-jre – and it’s the wrong version.

Actually, the latest version of Guava is 30.0-jre nowadays (you can check using “package search” site). But none of Selenide dependencies use the latest version. Almost all of them depend on guava:25.0-jre, and only one depends on guava:25.1-android. It’s testng:7.3.0 -> guice:4.2.2 -> guava:25.1-android.

It’s a bug of guice:4.2.2, if you ask me. By the way, it was fixed in guice:4.2.3 which depends on guava:27.1-jre.

How to fix it?

Now we know that the problem is with guava version. But how to fix this problem?

There is no single correct answer. It depends on your needs.

Most probably you don’t actually need Guice to run your tests with TestNG. I would probably suggest to exclude all transitive dependencies of TestNG in this case.

build.gradle:

dependencies {
  testImplementation("org.testng:testng:7.3.0") { 
    transitive = false 
  }
  testImplementation("com.codeborne:selenide:5.14.0")
}

or pom.xml:

<dependency>
  <groupId>org.testng</groupId>
  <artifactId>testng</artifactId>
  <version>7.3.0</version>
  <scope>test</scope>
  <exclusions>
    <exclusion>
      <artifactId>com.google.inject</artifactId>
      <groupId>guice</groupId>
    </exclusion>
  </exclusions>
</dependency>

It’s not ideal, but it works.

See detailed case description here.

Now let’s look at another example.

SECOND PROBLEM

Selenide has method $("input").download() for  downloading files. It may use different ways to fetch the file, one of which is using embedded proxy server.

If you want this way, you are going to add a proxy dependency:

dependencies {
  testImplementation("com.codeborne:selenide:5.14.0")
  testRuntimeOnly("com.browserup:browserup-proxy-core:2.1.1")
}

and write a test, something like that:

open("https://the-internet.herokuapp.com/download");
File file = $(byText("some-file.txt")).download();
assertThat(file.getName()).isEqualTo("some-file.txt");

But your test fails with a strange error:

org.openqa.selenium.WebDriverException: unknown error: net::ERR_TUNNEL_CONNECTION_FAILED
   ...
   at com.codeborne.selenide.Selenide.open(Selenide.java:49)
   at org.selenide.selenoid.FileDownloadTest.download(FileDownloadTest.java:45)

Too many people have been complaining about that… 🙁

Most people start a panic here. But all you need to do is just read the log carefully. You will see the error:

[LittleProxy-0-ProxyToServerWorker-1] ERROR org.littleshoot.proxy.impl.ProxyToServerConnection
- (HANDSHAKING) [id: 0xc05a41d5, L:/10.10.10.145:56103
- R:the-internet.herokuapp.com/52.1.16.137:443]
: Caught an exception on ProxyToServerConnection
java.lang.NoSuchMethodError: 'int io.netty.buffer.ByteBuf.maxFastWritableBytes()'
at io.netty.handler.codec.ByteToMessageDecoder$1.cumulate(ByteToMessageDecoder.java:86)

Gotcha! We see NoSuchMethodError which usually signals a versions conflict between your dependencies.

This time dependency browserup-proxy was compiled with one version of Netty, but is being run with another.

Let’s run our familiar command gradle dependencies or mvn dependency:tree:

\--- com.browserup:browserup-proxy-core:2.1.1
+--- io.netty:netty-codec:4.1.44.Final
+--- xyz.rogfam:littleproxy:2.0.0-beta-5
| +--- io.netty:netty-all:4.1.34.Final

We have two jars with different versions: netty-codec:4.1.44.Final and netty-all:4.1.34.Final.

It seems to be weird: library browserup-proxy-core uses netty-codec (and excludes netty-all, by the way). But its dependency littleproxy does use netty-all (which presumably contains netty-codec and some other netty sub-jars, hugh?). Obviously, they do conflict with each other.

How to fix it?

Again, there is no single correct answer.

There are many ways to fix the problem. Probably the easiest one is to declare Netty versions explicitly in build.gradle or pom.xml:

testRuntimeOnly("io.netty:netty-all:4.1.54.Final")
testRuntimeOnly("io.netty:netty-codec:4.1.54.Final")

Now command gradle dependencies shows that Netty versions are matching:

\--- com.browserup:browserup-proxy-core:2.1.1
+--- io.netty:netty-codec:4.1.44.Final -> 4.1.54.Final
+--- xyz.rogfam:littleproxy:2.0.0-beta-5
| +--- io.netty:netty-all:4.1.34.Final -> 4.1.54.Final

The test runs, proxy works, file is being downloaded. Everyone is happy. See detailed case description here.

Summary

I showed two examples of real-life problems caused by dependency conflicts. We saw how to investigate them using Maven or Gradle built-in tools. Now you know that it’s not impossible. You can do it. 🙂

Moral

Know your tools. Read your logs. Don’t be afraid.

Exclude as many dependencies as you can.

And feel like in dependency heaven.