Caused by: com.fasterxml.jackson.databind.JsonMappingException: Incompatible Jackson version: 2.9.10-4
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 | Exception in thread "main" java.lang.ExceptionInInitializerError at org.apache.spark.sql.execution.SparkPlan.executeQuery(SparkPlan.scala:135) at org.apache.spark.sql.execution.SparkPlan.execute(SparkPlan.scala:116) at org.apache.spark.sql.execution.SparkPlan.getByteArrayRdd(SparkPlan.scala:228) at org.apache.spark.sql.execution.SparkPlan.executeTake(SparkPlan.scala:311) at org.apache.spark.sql.execution.CollectLimitExec.executeCollect(limit.scala:38) at org.apache.spark.sql.Dataset.org$apache$spark$sql$Dataset$$collectFromPlan(Dataset.scala:2853) at org.apache.spark.sql.Dataset$$anonfun$head$1.apply(Dataset.scala:2153) at org.apache.spark.sql.Dataset$$anonfun$head$1.apply(Dataset.scala:2153) at org.apache.spark.sql.Dataset$$anonfun$55.apply(Dataset.scala:2837) at org.apache.spark.sql.execution.SQLExecution$.withNewExecutionId(SQLExecution.scala:65) at org.apache.spark.sql.Dataset.withAction(Dataset.scala:2836) at org.apache.spark.sql.Dataset.head(Dataset.scala:2153) at org.apache.spark.sql.Dataset.take(Dataset.scala:2366) at org.apache.spark.sql.Dataset.showString(Dataset.scala:245) at org.apache.spark.sql.Dataset.show(Dataset.scala:644) at org.apache.spark.sql.Dataset.show(Dataset.scala:603) at org.apache.spark.sql.Dataset.show(Dataset.scala:612) at Demo01$.main(Demo01.scala:23) at Demo01.main(Demo01.scala) Caused by: com.fasterxml.jackson.databind.JsonMappingException: Incompatible Jackson version: 2.9.10-4 at com.fasterxml.jackson.module.scala.JacksonModule$class.setupModule(JacksonModule.scala:64) at com.fasterxml.jackson.module.scala.DefaultScalaModule.setupModule(DefaultScalaModule.scala:19) at com.fasterxml.jackson.databind.ObjectMapper.registerModule(ObjectMapper.java:751) at org.apache.spark.rdd.RDDOperationScope$.<init>(RDDOperationScope.scala:82) at org.apache.spark.rdd.RDDOperationScope$.<clinit>(RDDOperationScope.scala) ... 19 more |
本地Scala版本是2.11.8,给项目pom单独调整jackson-databind包的版本:
java.io.FileNotFoundException: HADOOP_HOME and hadoop.home.dir are unset.
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 | java.lang.RuntimeException: java.io.FileNotFoundException: java.io.FileNotFoundException: HADOOP_HOME and hadoop.home.dir are unset. -see https://wiki.apache.org/hadoop/WindowsProblems at org.apache.hadoop.util.Shell.getWinUtilsPath(Shell.java:737) at org.apache.hadoop.util.Shell.getSetPermissionCommand(Shell.java:272) at org.apache.hadoop.util.Shell.getSetPermissionCommand(Shell.java:288) at org.apache.hadoop.fs.RawLocalFileSystem.setPermission(RawLocalFileSystem.java:840) at org.apache.hadoop.fs.RawLocalFileSystem$LocalFSFileOutputStream.<init>(RawLocalFileSystem.java:239) at org.apache.hadoop.fs.RawLocalFileSystem$LocalFSFileOutputStream.<init>(RawLocalFileSystem.java:219) at org.apache.hadoop.fs.RawLocalFileSystem.createOutputStreamWithMode(RawLocalFileSystem.java:318) at org.apache.hadoop.fs.RawLocalFileSystem.create(RawLocalFileSystem.java:307) at org.apache.hadoop.fs.RawLocalFileSystem.create(RawLocalFileSystem.java:338) at org.apache.hadoop.fs.ChecksumFileSystem$ChecksumFSOutputSummer.<init>(ChecksumFileSystem.java:401) at org.apache.hadoop.fs.ChecksumFileSystem.create(ChecksumFileSystem.java:464) at org.apache.hadoop.fs.ChecksumFileSystem.create(ChecksumFileSystem.java:443) at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:1118) at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:1098) at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:987) at org.apache.hadoop.fs.FileUtil.copy(FileUtil.java:414) at org.apache.hadoop.fs.FileUtil.copy(FileUtil.java:387) at org.apache.hadoop.fs.FileUtil.copy(FileUtil.java:337) at org.apache.hadoop.fs.FileSystem.copyToLocalFile(FileSystem.java:2434) at org.apache.hadoop.fs.FileSystem.copyToLocalFile(FileSystem.java:2403) at org.apache.hadoop.fs.FileSystem.copyToLocalFile(FileSystem.java:2379) at cn.itcast.hdfs.HDFSClientTest.getFile2Local(HDFSClientTest.java:71) at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.base/java.lang.reflect.Method.invoke(Method.java:564) at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:59) at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:56) at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17) at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27) at org.junit.runners.ParentRunner$3.evaluate(ParentRunner.java:306) at org.junit.runners.BlockJUnit4ClassRunner$1.evaluate(BlockJUnit4ClassRunner.java:100) at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:366) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:103) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:63) at org.junit.runners.ParentRunner$4.run(ParentRunner.java:331) at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:79) at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:329) at org.junit.runners.ParentRunner.access$100(ParentRunner.java:66) at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:293) at org.junit.runners.ParentRunner$3.evaluate(ParentRunner.java:306) at org.junit.runners.ParentRunner.run(ParentRunner.java:413) at org.junit.runner.JUnitCore.run(JUnitCore.java:137) at com.intellij.junit4.JUnit4IdeaTestRunner.startRunnerWithArgs(JUnit4IdeaTestRunner.java:68) at com.intellij.rt.junit.IdeaTestRunner$Repeater.startRunnerWithArgs(IdeaTestRunner.java:33) at com.intellij.rt.junit.JUnitStarter.prepareStreamsAndStart(JUnitStarter.java:230) at com.intellij.rt.junit.JUnitStarter.main(JUnitStarter.java:58) Caused by: java.io.FileNotFoundException: java.io.FileNotFoundException: HADOOP_HOME and hadoop.home.dir are unset. -see https://wiki.apache.org/hadoop/WindowsProblems at org.apache.hadoop.util.Shell.fileNotFoundException(Shell.java:549) at org.apache.hadoop.util.Shell.getHadoopHomeDir(Shell.java:570) at org.apache.hadoop.util.Shell.getQualifiedBin(Shell.java:593) at org.apache.hadoop.util.Shell.<clinit>(Shell.java:690) at org.apache.hadoop.util.StringUtils.<clinit>(StringUtils.java:78) at org.apache.hadoop.fs.FileSystem$Cache$Key.<init>(FileSystem.java:3482) at org.apache.hadoop.fs.FileSystem$Cache$Key.<init>(FileSystem.java:3477) at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:3319) at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:479) at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:227) at cn.itcast.hdfs.HDFSClientTest.connect2HDFS(HDFSClientTest.java:31) at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.base/java.lang.reflect.Method.invoke(Method.java:564) at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:59) at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:56) at org.junit.internal.runners.statements.RunBefores.invokeMethod(RunBefores.java:33) at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:24) ... 18 more Caused by: java.io.FileNotFoundException: HADOOP_HOME and hadoop.home.dir are unset. at org.apache.hadoop.util.Shell.checkHadoopHomeInner(Shell.java:469) at org.apache.hadoop.util.Shell.checkHadoopHome(Shell.java:440) at org.apache.hadoop.util.Shell.<clinit>(Shell.java:517) ... 34 more |
在windows本地运行spark程序时,需要有本地hadoop环境和winutils.exe插件,该插件用以支持本地运行:
Hadoop下载:Index of /apache/hadoop/common (tsinghua.edu.cn)
winutils.exe下载:GitHub – cdarlint/winutils: winutils.exe hadoop.dll and hdfs.dll binaries for hadoop windows
解压hadoop配置环境变量,解压winutils拷贝覆盖${HADOOP_HOME}/bin目录,并拷贝winutils.exe文件到C:\Windows\System32下
java.lang.NoSuchMethodError: com.google.common.base.Preconditions.checkArgument
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 | Exception in thread "main" java.lang.NoSuchMethodError: com.google.common.base.Preconditions.checkArgument(ZLjava/lang/String;Ljava/lang/Object;)V at org.apache.hadoop.conf.Configuration.set(Configuration.java:1380) at org.apache.hadoop.conf.Configuration.set(Configuration.java:1361) at org.apache.hadoop.mapred.JobConf.setJar(JobConf.java:536) at org.apache.hadoop.mapred.JobConf.setJarByClass(JobConf.java:554) at org.apache.hadoop.mapred.JobConf.<init>(JobConf.java:448) at org.apache.hadoop.hive.conf.HiveConf.initialize(HiveConf.java:5141) at org.apache.hadoop.hive.conf.HiveConf.<init>(HiveConf.java:5099) at org.apache.hadoop.hive.common.LogUtils.initHiveLog4jCommon(LogUtils.java:97) at org.apache.hadoop.hive.common.LogUtils.initHiveLog4j(LogUtils.java:81) at org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:699) at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:683) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.hadoop.util.RunJar.run(RunJar.java:323) at org.apache.hadoop.util.RunJar.main(RunJar.java:236) |
部署Hive时,执行hive命令报错,属于hadoop和hive之间的guava版本不一致,用hadoop的覆盖hive:
cp $HADOOP_HOME/share/hadoop/common/lib/guava-27.0-jre.jar $HIVE_HOME/lib/
rm -rf $HIVE_HOME/lib/guava-19.0.jar
You need either to explicitly disable SSL by setting useSSL=false, or set useSSL=true
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 | Thu Dec 16 22:52:53 EST 2021 WARN: Establishing SSL connection without server's identity verification is not recommended. According to MySQL 5.5.45+, 5.6.26+ and 5.7.6+ requirements SSL connection must be established by default if explicit option isn't set. For compliance with existing applications not using SSL the verifyServerCertificate property is set to 'false'. You need either to explicitly disable SSL by setting useSSL=false, or set useSSL=true and provide truststore for server certificate verification. com.mysql.jdbc.exceptions.jdbc4.CommunicationsException: Communications link failure The last packet successfully received from the server was 252 milliseconds ago. The last packet sent successfully to the server was 241 milliseconds ago. at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at com.mysql.jdbc.Util.handleNewInstance(Util.java:425) at com.mysql.jdbc.SQLError.createCommunicationsException(SQLError.java:990) at com.mysql.jdbc.ExportControlled.transformSocketToSSLSocket(ExportControlled.java:201) at com.mysql.jdbc.MysqlIO.negotiateSSLConnection(MysqlIO.java:4912) at com.mysql.jdbc.MysqlIO.proceedHandshakeWithPluggableAuthentication(MysqlIO.java:1663) at com.mysql.jdbc.MysqlIO.doHandshake(MysqlIO.java:1224) at com.mysql.jdbc.ConnectionImpl.coreConnect(ConnectionImpl.java:2190) at com.mysql.jdbc.ConnectionImpl.connectOneTryOnly(ConnectionImpl.java:2221) at com.mysql.jdbc.ConnectionImpl.createNewIO(ConnectionImpl.java:2016) at com.mysql.jdbc.ConnectionImpl.<init>(ConnectionImpl.java:776) at com.mysql.jdbc.JDBC4Connection.<init>(JDBC4Connection.java:47) at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at com.mysql.jdbc.Util.handleNewInstance(Util.java:425) at com.mysql.jdbc.ConnectionImpl.getInstance(ConnectionImpl.java:386) at com.mysql.jdbc.NonRegisteringDriver.connect(NonRegisteringDriver.java:330) at org.apache.spark.sql.execution.datasources.jdbc.JdbcUtils$$anonfun$createConnectionFactory$1.apply(JdbcUtils.scala:61) at org.apache.spark.sql.execution.datasources.jdbc.JdbcUtils$$anonfun$createConnectionFactory$1.apply(JdbcUtils.scala:52) at org.apache.spark.sql.execution.datasources.jdbc.JDBCRDD$.resolveTable(JDBCRDD.scala:58) at org.apache.spark.sql.execution.datasources.jdbc.JDBCRelation.<init>(JDBCRelation.scala:113) at org.apache.spark.sql.execution.datasources.jdbc.JdbcRelationProvider.createRelation(JdbcRelationProvider.scala:47) at org.apache.spark.sql.execution.datasources.DataSource.resolveRelation(DataSource.scala:306) at org.apache.spark.sql.DataFrameReader.load(DataFrameReader.scala:178) at org.apache.spark.sql.DataFrameReader.load(DataFrameReader.scala:146) at org.apache.spark.sql.DataFrameReader.jdbc(DataFrameReader.scala:193) ... 54 elided Caused by: javax.net.ssl.SSLHandshakeException: No appropriate protocol (protocol is disabled or cipher suites are inappropriate) at sun.security.ssl.HandshakeContext.<init>(HandshakeContext.java:171) at sun.security.ssl.ClientHandshakeContext.<init>(ClientHandshakeContext.java:106) at sun.security.ssl.TransportContext.kickstart(TransportContext.java:238) at sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:394) at sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:373) at com.mysql.jdbc.ExportControlled.transformSocketToSSLSocket(ExportControlled.java:186) ... 78 more |
Spark连接MySQL时,Url需要在后面加useSSL参数:
val url = “jdbc:mysql://localhost:3306/ccpc?useUnicode=true&characterEncoding=UTF-8&serverTimezone=UTC&useSSL=false”
java.io.IOException: Not a file: hdfs
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 | java.io.IOException: Not a file: hdfs://sandbox-hdp.hortonworks.com:8020/warehouse/tablespace/managed/hive/xxx.db/xxxxx/delta_0000001_0000001_0000 at org.apache.hadoop.mapred.FileInputFormat.getSplits(FileInputFormat.java:337) at org.apache.spark.rdd.HadoopRDD.getPartitions(HadoopRDD.scala:200) at org.apache.spark.rdd.RDD$$anonfun$partitions$2.apply(RDD.scala:253) at org.apache.spark.rdd.RDD$$anonfun$partitions$2.apply(RDD.scala:251) at scala.Option.getOrElse(Option.scala:121) at org.apache.spark.rdd.RDD.partitions(RDD.scala:251) at org.apache.spark.rdd.MapPartitionsRDD.getPartitions(MapPartitionsRDD.scala:35) at org.apache.spark.rdd.RDD$$anonfun$partitions$2.apply(RDD.scala:253) at org.apache.spark.rdd.RDD$$anonfun$partitions$2.apply(RDD.scala:251) at scala.Option.getOrElse(Option.scala:121) at org.apache.spark.rdd.RDD.partitions(RDD.scala:251) at org.apache.spark.rdd.MapPartitionsRDD.getPartitions(MapPartitionsRDD.scala:35) at org.apache.spark.rdd.RDD$$anonfun$partitions$2.apply(RDD.scala:253) at org.apache.spark.rdd.RDD$$anonfun$partitions$2.apply(RDD.scala:251) at scala.Option.getOrElse(Option.scala:121) at org.apache.spark.rdd.RDD.partitions(RDD.scala:251) at org.apache.spark.rdd.MapPartitionsRDD.getPartitions(MapPartitionsRDD.scala:35) at org.apache.spark.rdd.RDD$$anonfun$partitions$2.apply(RDD.scala:253) at org.apache.spark.rdd.RDD$$anonfun$partitions$2.apply(RDD.scala:251) at scala.Option.getOrElse(Option.scala:121) at org.apache.spark.rdd.RDD.partitions(RDD.scala:251) at org.apache.spark.rdd.MapPartitionsRDD.getPartitions(MapPartitionsRDD.scala:35) at org.apache.spark.rdd.RDD$$anonfun$partitions$2.apply(RDD.scala:253) at org.apache.spark.rdd.RDD$$anonfun$partitions$2.apply(RDD.scala:251) at scala.Option.getOrElse(Option.scala:121) at org.apache.spark.rdd.RDD.partitions(RDD.scala:251) at org.apache.spark.rdd.MapPartitionsRDD.getPartitions(MapPartitionsRDD.scala:35) at org.apache.spark.rdd.RDD$$anonfun$partitions$2.apply(RDD.scala:253) at org.apache.spark.rdd.RDD$$anonfun$partitions$2.apply(RDD.scala:251) at scala.Option.getOrElse(Option.scala:121) at org.apache.spark.rdd.RDD.partitions(RDD.scala:251) at org.apache.spark.sql.execution.SparkPlan.executeTake(SparkPlan.scala:340) at org.apache.spark.sql.execution.CollectLimitExec.executeCollect(limit.scala:38) at org.apache.spark.sql.Dataset.org$apache$spark$sql$Dataset$$collectFromPlan(Dataset.scala:3278) at org.apache.spark.sql.Dataset$$anonfun$head$1.apply(Dataset.scala:2489) at org.apache.spark.sql.Dataset$$anonfun$head$1.apply(Dataset.scala:2489) at org.apache.spark.sql.Dataset$$anonfun$52.apply(Dataset.scala:3259) at org.apache.spark.sql.execution.SQLExecution$.withNewExecutionId(SQLExecution.scala:77) at org.apache.spark.sql.Dataset.withAction(Dataset.scala:3258) at org.apache.spark.sql.Dataset.head(Dataset.scala:2489) at org.apache.spark.sql.Dataset.take(Dataset.scala:2703) at org.apache.spark.sql.Dataset.showString(Dataset.scala:254) at org.apache.spark.sql.Dataset.show(Dataset.scala:723) at org.apache.spark.sql.Dataset.show(Dataset.scala:682) at org.apache.spark.sql.Dataset.show(Dataset.scala:691) ... 55 elided |
Spark执行Hive的DQL时,找不到实际存在的文件,权限问题:
hdfs dfs -chmod -R 777 /warehouse
zookeeeper is not a recognized option
1 2 3 4 5 6 7 8 | Exception in thread "main" joptsimple.UnrecognizedOptionException: zookeeeper is not a recognized option at joptsimple.OptionException.unrecognizedOption(OptionException.java:108) at joptsimple.OptionParser.handleLongOptionToken(OptionParser.java:510) at joptsimple.OptionParserState$2.handleArgument(OptionParserState.java:56) at joptsimple.OptionParser.parse(OptionParser.java:396) at kafka.admin.TopicCommand$TopicCommandOptions.<init>(TopicCommand.scala:517) at kafka.admin.TopicCommand$.main(TopicCommand.scala:47) at kafka.admin.TopicCommand.main(TopicCommand.scala) |
Kafka命令行执行报错,版本太高,Kafka2.8.x之后舍弃了Zk的依赖,重新安装低版本
https://mirrors.tuna.tsinghua.edu.cn/apache/kafka/2.6.3/kafka_2.13-2.6.3.tgz
gzip: stdin: not in gzip format
1 2 3 4 5 | gzip: stdin: not in gzip format tar: Child returned status 1 tar: Error is not recoverable: exiting now |
tar解压的时候报错,file xxx 查看一下文件实际格式
大概率因为文件不是tar.gz压缩包,是http或者其他格式
Unexpected character encountered while parsing value: . Path ”, line 0, position 0.
1 | {"errors":{"":["Unexpected character encountered while parsing value: P. Path '', line 0, position 0."]},"type":"https://tools.ietf.org/html/rfc7231#section-6.5.1","title":"One or more validation errors occurred.","status":400,"traceId":"|a21e453e-49b9544f8166da99."} |
在使用HttpPost拉取post接口时,错误地把body参数传入url造成。
StringEntity requestBody = new StringEntity(JSON.toJSONString(参数));
httpPost.setEntity(requestBody);
Caused by: java.lang.reflect.InvocationTargetException: null
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 | 15:22:57.060 [Source: apireader -> Sink: hivewriter (1/1)] WARN org.apache.flink.runtime.taskmanager.Task - Source: apireader -> Sink: hivewriter (1/1) (2d0dbbd7a5df8568ffa988c8352566bb) switched from RUNNING to FAILED. java.io.IOException: 方法调用失败,内部抛出异常 com.guerdan.gdmp.mock.jar.ApiExt#queryList at com.dtstack.flinkx.api.handler.JarHandler.exchange(JarHandler.java:82) at com.dtstack.flinkx.api.format.ApiInputFormat.reachedEnd(ApiInputFormat.java:211) at com.dtstack.flinkx.streaming.api.functions.source.DtInputFormatSourceFunction.run(DtInputFormatSourceFunction.java:127) at org.apache.flink.streaming.api.operators.StreamSource.run(StreamSource.java:100) at org.apache.flink.streaming.api.operators.StreamSource.run(StreamSource.java:63) at org.apache.flink.streaming.runtime.tasks.SourceStreamTask$LegacySourceFunctionThread.run(SourceStreamTask.java:213) Caused by: java.lang.reflect.InvocationTargetException: null at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at com.dtstack.flinkx.api.handler.JarHandler.exchange(JarHandler.java:76) ... 5 common frames omitted Caused by: java.lang.Exception: 查询失败 url: https://apiopenvoi.nike.net/Black/GetPaging token: 23AE6127-D2C4-4E9C-BBF5-629DE9379888 payload: {CompanyName=云南公司, pageIndex=5, companyNameList=云南有限公司, pageSize=20} response: {"Code":400,"Msg":"接口异常"} at com.guerdan.gdmp.mock.jar.nike.nikeClient.post(nikeClient.java:995) at com.guerdan.gdmp.mock.jar.ApiExt.queryList(ApiExt.java:62) ... 10 common frames omitted Caused by: org.apache.http.client.HttpResponseException: {"Code":400,"Msg":"接口异常"} at com.guerdan.gdmp.mock.jar.nike.nikeClient.lambda$post$0(nikeClient.java:990) at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:223) at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:165) at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:140) at com.guerdan.gdmp.mock.jar.nike.nikeClient.post(nikeClient.java:979) ... 11 common frames omitted |
Api请求的返回值为空,服务器问题
java.lang.IllegalArgumentException: open() failed.ORA-01843: not a valid month
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 | java.lang.IllegalArgumentException: open() failed.ORA-01843: not a valid month at com.dtstack.flinkx.rdb.inputformat.JdbcInputFormat.openInternal(JdbcInputFormat.java:151) at com.dtstack.flinkx.inputformat.BaseRichInputFormat.open(BaseRichInputFormat.java:172) at com.dtstack.flinkx.streaming.api.functions.source.DtInputFormatSourceFunction.run(DtInputFormatSourceFunction.java:122) at org.apache.flink.streaming.api.operators.StreamSource.run(StreamSource.java:100) at org.apache.flink.streaming.api.operators.StreamSource.run(StreamSource.java:63) at org.apache.flink.streaming.runtime.tasks.SourceStreamTask$LegacySourceFunctionThread.run(SourceStreamTask.java:213) Caused by: java.sql.SQLDataException: ORA-01843: not a valid month at oracle.jdbc.driver.T4CTTIoer11.processError(T4CTTIoer11.java:494) at oracle.jdbc.driver.T4CTTIoer11.processError(T4CTTIoer11.java:446) at oracle.jdbc.driver.T4C8Oall.processError(T4C8Oall.java:1054) at oracle.jdbc.driver.T4CTTIfun.receive(T4CTTIfun.java:623) at oracle.jdbc.driver.T4CTTIfun.doRPC(T4CTTIfun.java:252) at oracle.jdbc.driver.T4C8Oall.doOALL(T4C8Oall.java:612) at oracle.jdbc.driver.T4CStatement.doOall8(T4CStatement.java:213) at oracle.jdbc.driver.T4CStatement.doOall8(T4CStatement.java:37) at oracle.jdbc.driver.T4CStatement.executeForRows(T4CStatement.java:896) at oracle.jdbc.driver.OracleStatement.executeMaybeDescribe(OracleStatement.java:933) at oracle.jdbc.driver.OracleStatement.doExecuteWithTimeout(OracleStatement.java:1082) at oracle.jdbc.driver.OracleStatement.executeQuery(OracleStatement.java:1276) at oracle.jdbc.driver.OracleStatementWrapper.executeQuery(OracleStatementWrapper.java:366) at com.dtstack.flinkx.rdb.inputformat.JdbcInputFormat.executeQuery(JdbcInputFormat.java:783) at com.dtstack.flinkx.rdb.inputformat.JdbcInputFormat.openInternal(JdbcInputFormat.java:148) ... 5 common frames omitted Caused by: oracle.jdbc.OracleDatabaseException: ORA-01843: not a valid month at oracle.jdbc.driver.T4CTTIoer11.processError(T4CTTIoer11.java:498) ... 19 common frames omitted |
Oracle格式踩坑,做增量集成时,直接用to_date(yyyy-mm-dd)过滤增量日期字段报错。
是因为中文环境日期显示为【31-1月-21】,不能直接过滤,需要转成时间戳再转成格式化date:
select to_char(cast(cast(“modify_time” as timestamp) as date),’yyyy-mm-dd’) from table;
PKIX path building failed: unable to find valid certification path to requested target
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 | javax.net.ssl.SSLHandshakeException: sun.security.validator.ValidatorException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target at sun.security.ssl.Alert.createSSLException(Alert.java:131) at sun.security.ssl.TransportContext.fatal(TransportContext.java:370) at sun.security.ssl.TransportContext.fatal(TransportContext.java:313) at sun.security.ssl.TransportContext.fatal(TransportContext.java:308) at sun.security.ssl.CertificateMessage$T12CertificateConsumer.checkServerCerts(CertificateMessage.java:652) at sun.security.ssl.CertificateMessage$T12CertificateConsumer.onCertificate(CertificateMessage.java:471) at sun.security.ssl.CertificateMessage$T12CertificateConsumer.consume(CertificateMessage.java:367) at sun.security.ssl.SSLHandshake.consume(SSLHandshake.java:376) at sun.security.ssl.HandshakeContext.dispatch(HandshakeContext.java:479) at sun.security.ssl.HandshakeContext.dispatch(HandshakeContext.java:457) at sun.security.ssl.TransportContext.dispatch(TransportContext.java:200) at sun.security.ssl.SSLTransport.decode(SSLTransport.java:154) at sun.security.ssl.SSLSocketImpl.decode(SSLSocketImpl.java:1290) at sun.security.ssl.SSLSocketImpl.readHandshakeRecord(SSLSocketImpl.java:1199) at sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:401) at sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:373) at sun.net.www.protocol.https.HttpsClient.afterConnect(HttpsClient.java:587) at sun.net.www.protocol.https.AbstractDelegateHttpsURLConnection.connect(AbstractDelegateHttpsURLConnection.java:197) at sun.net.www.protocol.http.HttpURLConnection.getInputStream0(HttpURLConnection.java:1570) at sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1498) at sun.net.www.protocol.https.HttpsURLConnectionImpl.getInputStream(HttpsURLConnectionImpl.java:268) at tools.HttpRequests.GetRequest(HttpRequests.java:37) at bcegc.creditChina.MainTest.main(MainTest.java:25) Caused by: sun.security.validator.ValidatorException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target at sun.security.validator.PKIXValidator.doBuild(PKIXValidator.java:439) at sun.security.validator.PKIXValidator.engineValidate(PKIXValidator.java:306) at sun.security.validator.Validator.validate(Validator.java:271) at sun.security.ssl.X509TrustManagerImpl.validate(X509TrustManagerImpl.java:312) at sun.security.ssl.X509TrustManagerImpl.checkTrusted(X509TrustManagerImpl.java:221) at sun.security.ssl.X509TrustManagerImpl.checkServerTrusted(X509TrustManagerImpl.java:128) at sun.security.ssl.CertificateMessage$T12CertificateConsumer.checkServerCerts(CertificateMessage.java:636) ... 18 more Caused by: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target at sun.security.provider.certpath.SunCertPathBuilder.build(SunCertPathBuilder.java:141) at sun.security.provider.certpath.SunCertPathBuilder.engineBuild(SunCertPathBuilder.java:126) at java.security.cert.CertPathBuilder.build(CertPathBuilder.java:280) at sun.security.validator.PKIXValidator.doBuild(PKIXValidator.java:434) ... 24 more Exception in thread "main" java.lang.NullPointerException at tools.HttpRequests.GetRequest(HttpRequests.java:49) at bcegc.creditChina.MainTest.main(MainTest.java:25) |
拉取网站接口时出现认证问题,需要安装认证或者绕过认证
1 2 3 4 5 | 方法1:下载证书后安装 cd jre/bin keytool -import -alias abc -keystore cacerts -file D:\Java\jdk1.8.0_291\jre\lib\security\xyzg.cer keytool -list -keystore cacerts -alias abc |
1 2 3 4 5 6 7 8 9 10 11 12 | 方法2:伪装请求头绕过认证 public String getRequest(String url, int timeOut) throws Exception { URL u = new URL(url); if ("https".equalsIgnoreCase(u.getProtocol())) { SslUtils.ignoreSsl(); } URLConnection conn = u.openConnection(); conn.setConnectTimeout(timeOut); conn.setReadTimeout(timeOut); conn.setRequestProperty("User-agent", "Mozilla/4.0"); return IOUtils.toString(conn.getInputStream()); } |
SQL 错误[1064][42000]: get hive table from meta storefailed: Invalid method name: ‘get_table’
Hive 2 StarRocks 映射外部表数据,创建映射关系时报出异常
翻阅StarRocks官方文献:外部表 @ External_table @ StarRocks Docs
1 2 3 | fe 配置文件路径为 $FE_HOME/conf,如果需要自定义 hadoop 集群的配置可以在该目录下添加配置文件,例如:hdfs 集群采用了高可用的 nameservice,需要将 hadoop 集群中的 hdfs-site.xml 放到该目录下,如果 hdfs 配置了 viewfs,需要将 core-site.xml 放到该目录下。 be 配置文件路径为 $BE_HOME/conf,如果需要自定义 hadoop 集群的配置可以在该目录下添加配置文件,例如:hdfs 集群采用了高可用的 nameservice,需要将 hadoop 集群中的 hdfs-site.xml 放到该目录下,如果 hdfs 配置了 viewfs,需要将 core-site.xml 放到该目录下。 |
拷贝$HADOOP_HOME/etc/hadoop/core-site.xml 到指定目录即可
…