配置flink环境_flink collectionenvironment

下载

https://archive.apache.org/dist/flink/flink-1.9.0/

我们选择下载flink-1.9.0-bin-scala_2.11.tgz 。从名字上可以看出，需要scala_2.11的环境。
如果没有，需要在下面地址下载
https://www.scala-lang.org/download/
参考下面文档进行安装
https://www.jianshu.com/p/d7c94372020c

安装

本地环境

flink支持如下环境

• LocalEnvironment 本地模式执行
• RemoteEnvironment 提交到远程集群执行
• CollectionEnvironment 集合数据集模式执行
• OptimizerPlanEnvironment 不执行作业，仅创建优化的计划
• PreviewPlanEnvironment 提取预先优化的执行计划
• ContextEnvironment 用于在客户端上远程执行.
• DetachedEnvironment 用于在客户端上以分离模式进行远程执行

LocalEnvironment

这个本质上也不需要安装flink，只要使用flink依赖就好

    public static void main(String[] args) throws Exception {
        //定义socket的端口号
        int port;
        try{
            ParameterTool parameterTool = ParameterTool.fromArgs(args);
            port = parameterTool.getInt("port");
        }catch (Exception e){
            System.err.println("没有指定port参数，使用默认值9000");
            port = 9000;
        }

        //获取运行环境
        StreamExecutionEnvironment env = StreamExecutionEnvironment.getExecutionEnvironment();

        //连接socket获取输入的数据
        DataStreamSource<String> text = env.socketTextStream("127.0.0.1", port, "\n");

        //计算数据
        DataStream<WordWithCount> windowCount = text.flatMap(new FlatMapFunction<String, WordWithCount>() {
            public void flatMap(String value, Collector<WordWithCount> out) throws Exception {
                String[] splits = value.split("\\s");
                for (String word:splits) {
                    out.collect(new WordWithCount(word,1L));
                }
            }
        })//打平操作，把每行的单词转为<word,count>类型的数据
                .keyBy("word")//针对相同的word数据进行分组
                .timeWindow(Time.seconds(2),Time.seconds(1))//指定计算数据的窗口大小和滑动窗口大小
                .sum("count");

        //把数据打印到控制台
        windowCount.print()
                .setParallelism(1);//使用一个并行度
        //注意：因为flink是懒加载的，所以必须调用execute方法，上面的代码才会执行
        env.execute("streaming word count");

    }
    /**
     * 主要为了存储单词以及单词出现的次数
     */
    public static class WordWithCount{
        public String word;
        public long count;
        public WordWithCount(){}
        public WordWithCount(String word, long count) {
            this.word = word;
            this.count = count;
        }

        @Override
        public String toString() {
            return "WordWithCount{" +
                    "word='" + word + '\'' +
                    ", count=" + count +
                    '}';
        }
    }

}
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59

CollectionEnvironment

这个其实不需要安装flink，只要在maven中引入flink的依赖就可以

public class LocalTestJob {

    public static void main(String[] args) throws Exception{
// initialize a new Collection-based execution environment
        final ExecutionEnvironment env =new CollectionEnvironment();
        DataSet<String> users = env.fromCollection(Arrays.asList("AAA","BBB","CCC"));
        /* Data Set transformations ... */
        // retrieve the resulting Tuple2 elements into a ArrayList.
        List<String> result =new ArrayList();
        users.output(new LocalCollectionOutputFormat<String>(result));
        // kick off execution.
        env.execute();
        // Do some work with the resulting ArrayList (=Collection).
        for(String t : result){
            System.err.println("Result = "+t);
        }
    }
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17