张坤的个人博客

  • 首页
  • 分类
  • 标签
  • 日志

  • 搜索
Jenkins RabbitMQ Zookeeper IDEA Logstash Kibana ELK NIO Netty Spring Cloud Golang DataX Elasticsearch React Native Mysql H2 Socket Spring Boot Kafka Mybatis Sqlmap Vue Postgresql Docker Vert.x Flutter Flink Redis

Flink安装排坑指南

发表于 2020-05-29 | 分类于 Flink | 0 | 阅读次数 29

Flink安装排坑指南

Flink依赖与java环境,请先安装jdk,最好1.8以上,配置好JAVA_HOME环境变量

首先我是在window上安装flink的,下载flink

下载之后解压到任意目录,进入bin目录运行start-cluster.bat脚本启动flink服务,但是Task Slots却为0,查看log发现是TaskManager没有初始化成功

image.png

2020-03-14 18:28:37,888 ERROR org.apache.flink.runtime.taskexecutor.TaskManagerRunner       - TaskManager initialization failed.
org.apache.flink.configuration.IllegalConfigurationException: Failed to create TaskExecutorResourceSpec
	at org.apache.flink.runtime.taskexecutor.TaskExecutorResourceUtils.resourceSpecFromConfig(TaskExecutorResourceUtils.java:72)
	at org.apache.flink.runtime.taskexecutor.TaskManagerRunner.startTaskManager(TaskManagerRunner.java:356)
	at org.apache.flink.runtime.taskexecutor.TaskManagerRunner.<init>(TaskManagerRunner.java:152)
	at org.apache.flink.runtime.taskexecutor.TaskManagerRunner.runTaskManager(TaskManagerRunner.java:308)
	at org.apache.flink.runtime.taskexecutor.TaskManagerRunner.lambda$runTaskManagerSecurely$2(TaskManagerRunner.java:322)
	at org.apache.flink.runtime.security.NoOpSecurityContext.runSecured(NoOpSecurityContext.java:30)
	at org.apache.flink.runtime.taskexecutor.TaskManagerRunner.runTaskManagerSecurely(TaskManagerRunner.java:321)
	at org.apache.flink.runtime.taskexecutor.TaskManagerRunner.main(TaskManagerRunner.java:287)
Caused by: org.apache.flink.configuration.IllegalConfigurationException: The required configuration option Key: 'taskmanager.cpu.cores' , default: null (fallback keys: []) is not set
	at org.apache.flink.runtime.taskexecutor.TaskExecutorResourceUtils.checkConfigOptionIsSet(TaskExecutorResourceUtils.java:90)
	at org.apache.flink.runtime.taskexecutor.TaskExecutorResourceUtils.lambda$checkTaskExecutorResourceConfigSet$0(TaskExecutorResourceUtils.java:84)
	at java.util.Arrays$ArrayList.forEach(Unknown Source)
	at org.apache.flink.runtime.taskexecutor.TaskExecutorResourceUtils.checkTaskExecutorResourceConfigSet(TaskExecutorResourceUtils.java:84)
	at org.apache.flink.runtime.taskexecutor.TaskExecutorResourceUtils.resourceSpecFromConfig(TaskExecutorResourceUtils.java:70)
	... 7 more
2020-03-14 18:28:37,892 INFO  org.apache.flink.runtime.blob.TransientBlobCache              - Shutting down BLOB cache
2020-03-14 18:28:37,892 INFO  org.apache.flink.runtime.blob.PermanentBlobCache              - Shutting down BLOB cache

以为是没有权限创建文件夹,用管理员权限试了也报一样的错

搞了半天也没个结果,最后果断放弃了window环境。

在vmware上弄了个centos7,在centos7上搭建了jdk和flink环境,推荐centos7最少给2G内存,不然后面启动flink时会被内存不够的问题折腾死。

flink最少需要192M的内存才能启动,jdk1.8之后堆内存采用的是元空间,元空间是根据内存大小自动分配的。如果元空间给的太小,flink将会启动不起来。

The derived from fraction jvm overhead memory (19.200mb (20132659 bytes)) is less than its min value 192.000mb (201326592 bytes), min value will be used instead

在启动flink时可以通过jvm参数指定元空间大小

  • -XX:MetaspaceSize,初始空间大小,达到该值就会触发垃圾收集进行类型卸载,同时GC会对该值进行调整:如果释放了大量的空间,就适当降低该值;如果释放了很少的空间,那么在不超过MaxMetaspaceSize时,适当提高该值。

flink的cofig/flink-conf.yaml配置的jobmanager.heap.size和taskmanager.memory.process.size也不能超过元空间大小。它们默认大小分别是:1024m和1568m

配置方法:env.java.opts: -XX:MetaspaceSize=512m,指定元空间为512m

如果电脑内存够大的话,推荐还是直接给centos弄个2G吧,能省很多事。

运行./bin/start-cluster.sh,在linux上顺利启动,在win10上通过浏览器访问

image.png

win10上启动不了不知道是不是因为权限太严格导致的,希望有路过的大佬能在评论区指点一二

# Jenkins # RabbitMQ # Zookeeper # IDEA # Logstash # Kibana # ELK # NIO # Netty # Spring Cloud # Golang # DataX # Elasticsearch # React Native # Mysql # H2 # Socket # Spring Boot # Kafka # Mybatis # Sqlmap # Vue # Postgresql # Docker # Vert.x # Flutter # Flink # Redis
登录验证码开发记录
Flutter环境搭建
  • 文章目录
  • 站点概览
会Coding的猴子

会Coding的猴子

57 日志
19 分类
28 标签
RSS
Github
Creative Commons
© 2021 会Coding的猴子
由 Halo 强力驱动
|
主题 - NexT.Gemini v5.1.4

湘ICP备18011740号