Flink安装排坑指南

Flink安装排坑指南

Flink依赖与java环境,请先安装jdk,最好1.8以上,配置好JAVA_HOME环境变量

首先我是在window上安装flink的,下载flink

下载之后解压到任意目录,进入bin目录运行start-cluster.bat脚本启动flink服务,但是Task Slots却为0,查看log发现是TaskManager没有初始化成功

image.png

2020-03-14 18:28:37,888 ERROR org.apache.flink.runtime.taskexecutor.TaskManagerRunner       - TaskManager initialization failed.
org.apache.flink.configuration.IllegalConfigurationException: Failed to create TaskExecutorResourceSpec
	at org.apache.flink.runtime.taskexecutor.TaskExecutorResourceUtils.resourceSpecFromConfig(TaskExecutorResourceUtils.java:72)
	at org.apache.flink.runtime.taskexecutor.TaskManagerRunner.startTaskManager(TaskManagerRunner.java:356)
	at org.apache.flink.runtime.taskexecutor.TaskManagerRunner.<init>(TaskManagerRunner.java:152)
	at org.apache.flink.runtime.taskexecutor.TaskManagerRunner.runTaskManager(TaskManagerRunner.java:308)
	at org.apache.flink.runtime.taskexecutor.TaskManagerRunner.lambda$runTaskManagerSecurely$2(TaskManagerRunner.java:322)
	at org.apache.flink.runtime.security.NoOpSecurityContext.runSecured(NoOpSecurityContext.java:30)
	at org.apache.flink.runtime.taskexecutor.TaskManagerRunner.runTaskManagerSecurely(TaskManagerRunner.java:321)
	at org.apache.flink.runtime.taskexecutor.TaskManagerRunner.main(TaskManagerRunner.java:287)
Caused by: org.apache.flink.configuration.IllegalConfigurationException: The required configuration option Key: 'taskmanager.cpu.cores' , default: null (fallback keys: []) is not set
	at org.apache.flink.runtime.taskexecutor.TaskExecutorResourceUtils.checkConfigOptionIsSet(TaskExecutorResourceUtils.java:90)
	at org.apache.flink.runtime.taskexecutor.TaskExecutorResourceUtils.lambda$checkTaskExecutorResourceConfigSet$0(TaskExecutorResourceUtils.java:84)
	at java.util.Arrays$ArrayList.forEach(Unknown Source)
	at org.apache.flink.runtime.taskexecutor.TaskExecutorResourceUtils.checkTaskExecutorResourceConfigSet(TaskExecutorResourceUtils.java:84)
	at org.apache.flink.runtime.taskexecutor.TaskExecutorResourceUtils.resourceSpecFromConfig(TaskExecutorResourceUtils.java:70)
	... 7 more
2020-03-14 18:28:37,892 INFO  org.apache.flink.runtime.blob.TransientBlobCache              - Shutting down BLOB cache
2020-03-14 18:28:37,892 INFO  org.apache.flink.runtime.blob.PermanentBlobCache              - Shutting down BLOB cache

以为是没有权限创建文件夹,用管理员权限试了也报一样的错

搞了半天也没个结果,最后果断放弃了window环境。

在vmware上弄了个centos7,在centos7上搭建了jdk和flink环境,推荐centos7最少给2G内存,不然后面启动flink时会被内存不够的问题折腾死。

flink最少需要192M的内存才能启动,jdk1.8之后堆内存采用的是元空间,元空间是根据内存大小自动分配的。如果元空间给的太小,flink将会启动不起来。

The derived from fraction jvm overhead memory (19.200mb (20132659 bytes)) is less than its min value 192.000mb (201326592 bytes), min value will be used instead

在启动flink时可以通过jvm参数指定元空间大小

  • -XX:MetaspaceSize,初始空间大小,达到该值就会触发垃圾收集进行类型卸载,同时GC会对该值进行调整:如果释放了大量的空间,就适当降低该值;如果释放了很少的空间,那么在不超过MaxMetaspaceSize时,适当提高该值。

flink的cofig/flink-conf.yaml配置的jobmanager.heap.sizetaskmanager.memory.process.size也不能超过元空间大小。它们默认大小分别是:1024m和1568m

配置方法:env.java.opts: -XX:MetaspaceSize=512m,指定元空间为512m

如果电脑内存够大的话,推荐还是直接给centos弄个2G吧,能省很多事。

运行./bin/start-cluster.sh,在linux上顺利启动,在win10上通过浏览器访问

image.png

win10上启动不了不知道是不是因为权限太严格导致的,希望有路过的大佬能在评论区指点一二

# Flink 

评论

Your browser is out-of-date!

Update your browser to view this website correctly. Update my browser now

×