查看: 353|回复: 0

[Java学习] Hbase、elasticsearch整合中jar包冲突的问题解决

发表于 2018-1-5 08:00:00

问题背景

再数据平台中,项目搭建需要使用es和HBASE搭建数据查询接口,整合的过程中出现jar包冲突的bug :com.google.common.base.Stopwatch.()V from class org.apache.hadoop.hbase.zookeeper.MetaTableLocator

  1. org.apache.hadoop.hbase.DoNotRetryIOException: java.lang.IllegalAccessError: tried to access method com.google.common.base.Stopwatch.<init>()V from class org.apache.hadoop.hbase.zookeeper.MetaTableLocator
  2. at org.apache.hadoop.hbase.client.RpcRetryingCaller.translateException(RpcRetryingCaller.java:239)
  3. at org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithRetries(RpcRetryingCaller.java:150)
  4. ...
  5. at java.lang.Thread.run(Thread.java:745)
  6. Caused by: java.lang.IllegalAccessError: tried to access method com.google.common.base.Stopwatch.<init>()V from class org.apache.hadoop.hbase.zookeeper.MetaTableLocator
  7. at org.apache.hadoop.hbase.zookeeper.MetaTableLocator.blockUntil
  8. ...
  9. at org.apache.hadoop.hbase.client.RegionServerCallable.prepare(RegionServerCallable.java:75)
  10. at org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithRetries(RpcRetryingCaller.java:134)
  11. ... 45 more
复制代码

解决办法

经排查,确认是因为 com.google.guava 这个包引起的冲突。es依赖于18及以上版本,而HBASE只支持16及以上版本。而guava这个包从17开始内部发生变化,方法发生改变,所以18并不会向下兼容16版本。在项目运行的过程中,如果同时引入16、18版本,es及hbase的调用过程会发生混乱。那么接下来就好办了,我们可以重新打包,将guava18打入es内部,再在pom文件中显示引用guava16版本的包。这样,es调用包内部打入的guava18,而hbase调用外部的guava16。

重新打包

新建maven工程,在pom文件中进行如下配置:

  1. <?xml version="1.0" encoding="UTF-8"?>
  2. <project xmlns="http://maven.apache.org/POM/4.0.0"
  3. xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
  4. xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
  5. <modelVersion>4.0.0</modelVersion>
  6. <!--
  7. this maven project is aim to fix the jar comflic in hbase & elasticsearch.
  8. in es, required guava 18+ or up. but in hbase you should use guava 16- or blow.
  9. cd the project directory and run 'sh ./cleanbuild.sh',you will get a new-self jar.
  10. then, adjust your project pom.xml whth:
  11. <dependency>
  12. <groupId>douguo.shaded.elasticsearch</groupId>
  13. <artifactId>douguo_shaded_elasticsearch</artifactId>
  14. <version>1.0-SNAPSHOT</version>
  15. <exclusions>
  16. <exclusion>
  17. <groupId>org.elasticsearch</groupId>
  18. <artifactId>elasticsearch</artifactId>
  19. </exclusion>
  20. </exclusions>
  21. </dependency>
  22. finally, your jar comflic will be fixed.
  23. @date:2017-11-30
  24. @author:zhangjianfei
  25. @since:douguo.shaded.elasticsearch-1.0.0
  26. -->
  27. <groupId>douguo.shaded.elasticsearch</groupId>
  28. <artifactId>douguo_shaded_elasticsearch</artifactId>
  29. <version>1.0-SNAPSHOT</version>
  30. <properties>
  31. <elasticsearch.version>2.4.1</elasticsearch.version>
  32. </properties>
  33. <dependencies>
  34. <dependency>
  35. <groupId>org.elasticsearch</groupId>
  36. <artifactId>elasticsearch</artifactId>
  37. <version>${elasticsearch.version}</version>
  38. </dependency>
  39. <dependency>
  40. <groupId>org.elasticsearch.plugin</groupId>
  41. <artifactId>shield</artifactId>
  42. <version>${elasticsearch.version}</version>
  43. </dependency>
  44. </dependencies>
  45. <build>
  46. <plugins>
  47. <plugin>
  48. <groupId>org.apache.maven.plugins</groupId>
  49. <artifactId>maven-shade-plugin</artifactId>
  50. <version>2.4.1</version>
  51. <configuration>
  52. <createDependencyReducedPom>false</createDependencyReducedPom>
  53. </configuration>
  54. <executions>
  55. <execution>
  56. <phase>package</phase>
  57. <goals>
  58. <goal>shade</goal>
  59. </goals>
  60. <configuration>
  61. <relocations>
  62. <relocation>
  63. <pattern>com.google.guava</pattern>
  64. <shadedPattern>douguo.shaded.elasticsearch.guava</shadedPattern>
  65. </relocation>
  66. <relocation>
  67. <pattern>org.joda</pattern>
  68. <shadedPattern>douguo.shaded.elasticsearch.joda</shadedPattern>
  69. </relocation>
  70. <relocation>
  71. <pattern>com.google.common</pattern>
  72. <shadedPattern>douguo.shaded.elasticsearch.common</shadedPattern>
  73. </relocation>
  74. <relocation>
  75. <pattern>com.google.thirdparty</pattern>
  76. <shadedPattern>douguo.shaded.elasticsearch.thirdparty</shadedPattern>
  77. </relocation>
  78. </relocations>
  79. <transformers>
  80. <transformer implementation="org.apache.maven.plugins.shade.resource.ManifestResourceTransformer" />
  81. </transformers>
  82. </configuration>
  83. </execution>
  84. </executions>
  85. </plugin>
  86. </plugins>
  87. </build>
  88. <repositories>
  89. <repository>
  90. <id>elasticsearch-releases</id>
  91. <url>http://maven.elasticsearch.org/releases</url>
  92. <releases>
  93. <enabled>true</enabled>
  94. <updatePolicy>daily</updatePolicy>
  95. </releases>
  96. <snapshots>
  97. <enabled>false</enabled>
  98. </snapshots>
  99. </repository>
  100. </repositories>
  101. </project>
复制代码

将org.joda等4个可能有冲突的jar包通过maven-shade-plugin插件迁移后重新打个jar包从而使得在引入这个jar包时能够使用该jar包自己的依赖而不是使用外部依赖。这里需要注意的是,需要将com.google.common等4个包全部重新迁移,否则会出现java.lang.IllegalAccessError: tried to access method com.google.common.base的错误

项目打包

  1. mvn clean install
复制代码

新的依赖包会在.m2 maven仓库中,如果公司搭建了仓库的话,需要上传jar包。如果直接运行jar包的话,记得重新编译项目,并替换lib目录

项目载入新包

只需要在pom文件中配置:

  1. <!-- douguo.shaded.elasticsearch -->
  2. <dependency>
  3. <groupId>douguo.shaded.elasticsearch</groupId>
  4. <artifactId>douguo_shaded_elasticsearch</artifactId>
  5. <version>1.0-SNAPSHOT</version>
  6. <exclusions>
  7. <exclusion>
  8. <groupId>org.elasticsearch</groupId>
  9. <artifactId>elasticsearch</artifactId>
  10. </exclusion>
  11. </exclusions>
  12. </dependency>
  13. <!-- https://mvnrepository.com/artifact/com.google.guava/guava -->
  14. <!-- this guava is only used in habse
  15. in es, 18.0+ is required, but hbase only supported 16.0 or blow.
  16. clean as install douguo.shaded.elasticsearch -->
  17. <dependency>
  18. <groupId>com.google.guava</groupId>
  19. <artifactId>guava</artifactId>
  20. <version>16.0</version>
  21. </dependency>
复制代码

这样,guava18包就在douguo.shaded.elasticsearch下,es会优先调用。而外部配置的guava16就会被HBASE调用。2个版本的jar包互相独立存在!

到此,问题就解决了!

以上就是本文的全部内容,希望对大家的学习有所帮助,也希望大家多多支持程序员之家。



回复

使用道具 举报