赞
踩
Nginx:一个高性能的HTTP和反向代理web服务器,实际生产中一般用作负载均衡;
Kafka:一种高吞吐量的分布式发布订阅消息系统,实际生产中一般用作流式处理中间件;
Nginx和Kafka都有非常优秀的性能,实际生产中,往往都是通过Nginx记录日志生成日志文件,然后再通过Flume将日志文件的内容读取到Kafka,中间会有一个写文件和读文件操作,性能势必会受到一些影响;
下面介绍一下把Nginx的日志直接写入到Kafka的实现过程:
git主要用于拉取源码文件
yum install -y git
切换到/usr/local/src目录,然后将kafka的c客户端源码clone到本地
- cd /usr/local/src
- git clone https://github.com/edenhill/librdkafka
- cd librdkafka
- # 安装必要的包
- yum install -y gcc gcc-c++ pcre-devel zlib-devel
- ./configure
- make && make install
安装nginx整合kafka的插件,进入到/usr/local/src,clone nginx整合kafka的源码
- cd /usr/local/src
- git clone https://github.com/brg-liuwei/ngx_kafka_module
进入到nginx的源码包目录下(编译nginx,然后将将插件同时编译)
- cd /usr/local/src/nginx-1.20.2
- ./configure --add-module=/usr/local/src/ngx_kafka_module/
- make
- make install
- [root@hadoop100 conf]# vim nginx.conf
-
- # '$status $body_bytes_sent "$http_referer" '
- # '"$http_user_agent" "$http_x_forwarded_for"';
-
- #access_log logs/access.log main;
-
- sendfile on;
- #tcp_nopush on;
-
- #keepalive_timeout 0;
- keepalive_timeout 65;
-
- #gzip on;
- #
- # ############### kafka config ###########################
- kafka;
- kafka_broker_list hadoop101:9092 hadoop102:9092 hadoop103:9092;
- # ############### kafka config ###########################
-
- server {
- listen 80;
- server_name localhost;
-
- #charset koi8-r;
-
- #access_log logs/host.access.log main;
-
- #location / {
- # root html;
- # index index.html index.htm;
- #}
- # ############### kafka config ###########################
- location /kafka/test {
- kafka_topic test;
- }
- # ############### kafka config ###########################
- #error_page 404 /404.html;
-
- # redirect server error pages to the static page /50x.html
- #
- error_page 500 502 503 504 /50x.html;
- location = /50x.html {
- root html;
- }
-
bin/kafka-topics.sh --zookeeper hadoop101:2181 --create --topic test --partitions 3 --replication-factor 2
error while loading shared libraries: librdkafka.so.1: cannot open shared object file: No such file or directory
- echo "/usr/local/lib" >> /etc/ld.so.conf
- ldconfig
curl localhost/kafka/track -d "message send to kafka topic"
Copyright © 2003-2013 www.wpsshop.cn 版权所有,并保留所有权利。