kafka-nginx module模块添加与简单使用

Nginx Kafka Module环境搭建

参考官网：https://github.com/brg-liuwei/ngx_kafka_module/blob/master/README.md

进入目录 cd /usr/local/src/

1.上传下载的安装文件

2.进入libkafka目录 cd librdkafka

安装依赖 yum install -y gcc gcc-c++ pcre-devel zlib-level

编译安装安装完成的默认路径/usr/local/include

./configure

make && make install

3.进入nginx目录 cd nginx-1.8.1/ （编译nginx，然后将将插件同时编译）

./configure --add-module=/usr/local/src/ngx_kafka_module/

make

make install

4.修改配置文件，需要提前启动zk并创建要配置的主题信息

5.开启消费者接收消息

6.启动nginx 进入nginx sbin目录./nginx ，

1）报错，找不到kafka.so.1的文件 error while loading shared libraries: librdkafka.so.1: cannot open shared object file: No such file or directory

2）加载so库 echo "/usr/local/lib" >> /etc/ld.so.conf ldconfig

3）重新启动成功访问：

curl xxxxx/brokers/topics -d "hello ngx_kafka_module" -v

7.页面埋点采集配置

8.采集服务器js代码

(function () {

var params = {};

//Document对象数据

if(document) {

params.domain = document.domain || '';

params.url = document.URL || '';

params.title = document.title || '';

params.referrer = document.referrer || '';

}

//Window对象数据

if(window && window.screen) {

params.sh = window.screen.height || 0;

params.sw = window.screen.width || 0;

params.cd = window.screen.colorDepth || 0;

}

//navigator对象数据

if(navigator) {

params.lang = navigator.language || '';

}

//解析_maq配置

if(_maq) {

for(var i in _maq) {

var paramsName = _maq[i][0];

var paramsValue = _maq[i][1];

params[paramsName] = paramsValue;

}

//拼接参数串

var args = '';

for(var i in params) {

if(args != '') {

args += '&';

}

args += i + '=' + encodeURIComponent(params[i]);

}

//通过Image对象请求后端脚本

var img = new Image(1, 1);

img.src = 'http://xxxxx/log.gif?' + args;

})();

html页面的js脚本，访问采集服务器的js

var _maq = _maq || [];

$(function (){

$("button").click(function () {

var listenClick = $(this).attr("data-id");

var userName = $("#userId").val();

var date = getFormatDate();

_maq.push(['_setAccount', userName]);

_maq.push(['_setClick', listenClick]);

_maq.push(['_setClickTime', date]);

//解析_maq配置

var ma = document.createElement('script');

ma.type = 'text/javascript';

ma.async = true;

ma.src = 'http://xxxxx/ma/ma.js';

var s = document.getElementsByTagName('script')[0];

s.parentNode.insertBefore(ma, s);

})

function getFormatDate(){

var nowDate = new Date();

var year = nowDate.getFullYear();

var month = nowDate.getMonth() + 1 < 10 ? "0" + (nowDate.getMonth() + 1) : nowDate.getMonth() + 1;

var date = nowDate.getDate() < 10 ? "0" + nowDate.getDate() : nowDate.getDate();

var hour = nowDate.getHours()< 10 ? "0" + nowDate.getHours() : nowDate.getHours();

var minute = nowDate.getMinutes()< 10 ? "0" + nowDate.getMinutes() : nowDate.getMinutes();

var second = nowDate.getSeconds()< 10 ? "0" + nowDate.getSeconds() : nowDate.getSeconds();

return year + "-" + month + "-" + date+" "+hour+":"+minute+":"+second;

}

</script>

配置nginx.conf

配置kafka信息

kafka;

kafka_broker_list xxxxx:9092;

server {

listen 80;

server_name xxxxx;

location = /mykafka/brokers/topics {

kafka_topic nginx_topic;

}

在此接收js的log.gif请求，模拟post发送到kafka

location /log.gif {

#伪装成gif文件

default_type image/gif;

#本身关闭access_log，通过subrequest记录log

access_log off;

access_by_lua "

local uid = ngx.var.cookie___utrace

if not uid then

uid = ngx.md5(ngx.now() .. ngx.var.remote_addr .. ngx.var.http_user_agent)

end

ngx.header['Set-Cookie'] = {'__utrace=' .. uid .. '; path=/'}

if ngx.var.arg_domain then

ngx.location.capture('/i-log?' .. ngx.var.args .. '&utrace=' .. uid)

ngx.location.capture('/mykafka/brokers/topics',{method=ngx.HTTP_POST,body=ngx.var.args})

end

9.写入日志成功，kafka接收数据成功接入如下

最后编辑于：2020.11.14 10:17:33