当前位置:   article > 正文

【大数据-调度工具】dolphinscheduler安装和遇到的问题_安装dolphinscheduler报错caused by: java.lang.illegalst

安装dolphinscheduler报错caused by: java.lang.illegalstateexception: cannot l

1.安装

安装步骤按照官网安装即可

官网:DolphinScheduler | 文档中心 (apache.org)

版本:3.1.5

2.踩坑记录

Q1.大文件无法上传

Q1问题描述:

在资源中心中上传文件选择完大文件夹之后,选择确认之后确认按钮转了几圈圈之后就没反应了,对应服务器上使用ds登录的用户资源目录也没有对应的文件。

Q1问题解决:

通过排查发现每到15s。网络请求就会断开。

利用浏览器工具找到请求的js文件。找到请求url,baseURL中有个参数是timeout 为 15e3(15000ms即15s),找到对应ds安装目录下的ui/asserts/ 对应js脚本文件,时间改大一点。

Q2.Flink程序无法提交到yarn

Q2问题描述:

创建完Flink-Stream 之后,命令总提交失败,分析日志DS生成的提交命令为:

服务器上执行,发现有以下报错,主要是因为Flink的总内存默认为1.5G。

org.apache.flink.client.program.ProgramInvocationException: The main method caused an error: TaskManager memory configuration failed: Sum of configured Framework Heap Memory (128.000mb (134217728 bytes)), Framework Off-Heap Memory (512.000mb (536870912 bytes)), Task Off-Heap Memory (512.000mb (536870912 bytes)), Managed Memory (1024.000mb (1073741824 bytes)) and Network Memory (158.720mb (166429984 bytes)) exceed configured Total Flink Memory (1.550gb (1664299824 bytes))

Q2问题解决:

对应的参数调整成以下配置项即可:(应该有其他更好的解法,暂时没找到......)

对应JobManager内存数 和 taskManager内存数分别用以下两个参数代替,原有选项置空。

-Djobmanager.memory.process.size=1024mb -Dtaskmanager.memory.process.size=6144mb

 Q3.提交完flinkStream 之后,在ResourceManagerWebUi上一直处于create状态

Q3问题描述:

使用root用户在对应节点的服务器提交任务,任务可以成功被提交,并且可以running起来。

但是切换完ds租户提交任务之后,也可以成功提交,但是任务一直处于created 的状态,并且在一段时间之后会失败并报错

  1. org.apache.flink.runtime.jobmaster.slotpool.PhysicalSlotRequestBulkCheckerImpl.lambda$schedulePendingRequestBulkWithTimestampCheck$0(PhysicalSlotRequestBulkCheckerImpl.java:91)
  2. at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
  3. at java.util.concurrent.FutureTask.run(FutureTask.java:266)
  4. at org.apache.flink.runtime.rpc.akka.AkkaRpcActor.handleRunAsync(AkkaRpcActor.java:440)
  5. at org.apache.flink.runtime.rpc.akka.AkkaRpcActor.handleRpcMessage(AkkaRpcActor.java:208)
  6. at org.apache.flink.runtime.rpc.akka.FencedAkkaRpcActor.handleRpcMessage(FencedAkkaRpcActor.java:77)
  7. at org.apache.flink.runtime.rpc.akka.AkkaRpcActor.handleMessage(AkkaRpcActor.java:158)
  8. at akka.japi.pf.UnitCaseStatement.apply(CaseStatements.scala:26)
  9. at akka.japi.pf.UnitCaseStatement.apply(CaseStatements.scala:21)
  10. at scala.PartialFunction$class.applyOrElse(PartialFunction.scala:123)
  11. at akka.japi.pf.UnitCaseStatement.applyOrElse(CaseStatements.scala:21)
  12. at scala.PartialFunction$OrElse.applyOrElse(PartialFunction.scala:170)
  13. at scala.PartialFunction$OrElse.applyOrElse(PartialFunction.scala:171)
  14. at scala.PartialFunction$OrElse.applyOrElse(PartialFunction.scala:171)
  15. at akka.actor.Actor$class.aroundReceive(Actor.scala:517)
  16. at akka.actor.AbstractActor.aroundReceive(AbstractActor.scala:225)
  17. at akka.actor.ActorCell.receiveMessage(ActorCell.scala:592)
  18. at akka.actor.ActorCell.invoke(ActorCell.scala:561)
  19. at akka.dispatch.Mailbox.processMailbox(Mailbox.scala:258)
  20. at akka.dispatch.Mailbox.run(Mailbox.scala:225)
  21. at akka.dispatch.Mailbox.exec(Mailbox.scala:235)
  22. at akka.dispatch.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260)
  23. at akka.dispatch.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339)
  24. at akka.dispatch.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
  25. at akka.dispatch.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
  26. Caused by: java.util.concurrent.CompletionException: org.apache.flink.runtime.jobmanager.scheduler.NoResourceAvailableException: Slot request bulk is not fulfillable! Could not allocate the required slot within slot request timeout
  27. at java.util.concurrent.CompletableFuture.encodeThrowable(CompletableFuture.java:292)
  28. at java.util.concurrent.CompletableFuture.completeThrowable(CompletableFuture.java:308)
  29. at java.util.concurrent.CompletableFuture.uniApply(CompletableFuture.java:593)
  30. at java.util.concurrent.CompletableFuture$UniApply.tryFire(CompletableFuture.java:577)
  31. ... 31 more
  32. Caused by: org.apache.flink.runtime.jobmanager.scheduler.NoResourceAvailableException: Slot request bulk is not fulfillable! Could not allocate the required slot within slot request timeout
  33. at org.apache.flink.runtime.jobmaster.slotpool.PhysicalSlotRequestBulkCheckerImpl.lambda$schedulePendingRequestBulkWithTimestampCheck$0(PhysicalSlotRequestBulkCheckerImpl.java:86)
  34. ... 24 more
  35. Caused by: java.util.concurrent.TimeoutException: Timeout has occurred: 300000 ms
  36. ... 25 more
  37. Exception in thread "Thread-5" java.lang.IllegalStateException: Trying to access closed classloader. Please check if you store classloaders directly or indirectly in static fields. If the stacktrace suggests that the leak occurs in a third party library and cannot be fixed immediately, you can disable this check with the configuration 'classloader.check-leaked-classloader'.
  38. at org.apache.flink.runtime.execution.librarycache.FlinkUserCodeClassLoaders$SafetyNetWrapperClassLoader.ensureInner(FlinkUserCodeClassLoaders.java:164)
  39. at org.apache.flink.runtime.execution.librarycache.FlinkUserCodeClassLoaders$SafetyNetWrapperClassLoader.getResource(FlinkUserCodeClassLoaders.java:183)
  40. at org.apache.hadoop.conf.Configuration.getResource(Configuration.java:2647)
  41. at org.apache.hadoop.conf.Configuration.getStreamReader(Configuration.java:2905)
  42. at org.apache.hadoop.conf.Configuration.loadResource(Configuration.java:2864)
  43. at org.apache.hadoop.conf.Configuration.loadResources(Configuration.java:2838)
  44. at org.apache.hadoop.conf.Configuration.getProps(Configuration.java:2715)
  45. at org.apache.hadoop.conf.Configuration.get(Configuration.java:1186)
  46. at org.apache.hadoop.conf.Configuration.getTimeDuration(Configuration.java:1774)
  47. at org.apache.hadoop.util.ShutdownHookManager.getShutdownTimeout(ShutdownHookManager.java:183)
  48. at org.apache.hadoop.util.ShutdownHookManager.shutdownExecutor(ShutdownHookManager.java:145)
  49. at org.apache.hadoop.util.ShutdownHookManager.access$300(ShutdownHookManager.java:65)
  50. at org.apache.hadoop.util.ShutdownHookManager$1.run(ShutdownHookManager.java:102)

主要看  Caused by: java.util.concurrent.CompletionException: org.apache.flink.runtime.jobmanager.scheduler.NoResourceAvailableException: Slot request bulk is not fulfillable! Could not allocate the required slot within slot request timeout

Q3问题解决:

找到对应node节点的flink安装目录,在对应 flink 启动脚本第一行添加一行代码

export HADOOP_CLASSPATH=`hadoop classpath`

其含义为运行shell命令hadoop classpath 并将输出的值赋予给HADOOP_CLASSPATH环境变量。

然后重新提交flink任务即可在ResourceManagerWebUi看到对应DS租户提交的Flink任务。

 Q4.关于一些僵尸任务前端不能删除

Q4问题描述:

1.对于DS的任务管理中任务定义没有删除的操作。已经删除了的工作流中的任务定义依然存在,也不能通过前端删除。

2.在任务示例中实时任务也没有删除选项,但会存在僵尸任务示例。

Q4问题解决:

对于配置了mysql环境的DS。

1.任务定义表为t_ds_process_instance,根据名称(字段name)去删除对应任务定义即可。

2.任务实例表为t_ds_task_instance,根据需要条件删除任务实例即可。

声明:本文内容由网友自发贡献,不代表【wpsshop博客】立场,版权归原作者所有,本站不承担相应法律责任。如您发现有侵权的内容,请联系我们。转载请注明出处:https://www.wpsshop.cn/w/笔触狂放9/article/detail/946536
推荐阅读
相关标签
  

闽ICP备14008679号