赞
踩
最近服务器状态不佳,oracle有时性能较差,运行sparksql向oracle中存数据时运行超级缓慢,开始还特意排查代码找错,最终换了一个oracle数据库做实验发现整个流程在几秒内非常顺利的完成了,可见oracle性能对sparksql执行速度影响巨大,特此记录此次事件。
以下是运行时部分日志,可以看到每执行1/200的进度要30000ms,整个预计要200*30=6000秒才能完毕。。。。。。
- 2019-03-15 15:11:51 INFO TaskSetManager:54 - Finished task 2.0 in stage 1.0 (TID 3) in 30618 ms on localhost (executor driver) (3/200)
- 2019-03-15 15:11:51 INFO ShuffleBlockFetcherIterator:54 - Getting 0 non-empty blocks out of 1 blocks
- 2019-03-15 15:11:51 INFO ShuffleBlockFetcherIterator:54 - Started 0 remote fetches in 1 ms
- 2019-03-15 15:12:22 WARN JdbcUtils:66 - Requested isolation level 1 is not supported; falling back to default isolation level 2
- 2019-03-15 15:12:22 INFO Executor:54 - Finished task 3.0 in stage 1.0 (TID 4). 2524 bytes result sent to driver
- 2019-03-15 15:12:22 INFO TaskSetManager:54 - Starting task 4.0 in stage 1.0 (TID 5, localhost, executor driver, partition 4, PROCESS_LOCAL, 7754 bytes)
- 2019-03-15 15:12:22 INFO Executor:54 - Running task 4.0 in stage 1.0 (TID 5)
Copyright © 2003-2013 www.wpsshop.cn 版权所有,并保留所有权利。