NGINX common variables not passed to kafka topic

[qbadx]

Hi,

Would it be possible to pass some extra NGINX common variables to the data sent to the Kafka topic?

Currently, only the data of theNGINX variable $request_body is passed but , for instance, data from variables like $remote_addr would be great .

The goal is to have the IP of the client that is sending the data.

[brg-liuwei]

I have add this issue into TODO LIST.

[lifehacking]

nginx common var is very usefull

[brg-liuwei]

@lifehacking yep, I cannot agree more. PR is welcomed :)

[dkinon]

If anyone here is looking for a workaround in the interim, I am successfully injecting nginx variables into the $request_body using the lua nginx module.

[brg-liuwei]

@dkinon Yes, using lua_nginx_module injecting ngx variables into $request_body is easy to work, and I am an openresty user. But I still want to do some works to pass variables to kafka topic without lua module, cause not everyone use lua_nginx_module. :)

[adridi]

what about http headers - would it be possible to add them to the message body?

[dkinon]

@adridi http headers are actually the ngx variables (ngx.req.get_headers()) I'm injecting into the $request_body. Since they are just another ngx variable, I don't see why they couldn't technically be included. The bigger issue with including ngx variables into a kafka payload in some standard way is that kafka messages have no standard structure. It's just a text payload.

I prefer JSON so I make sure the incoming $request_body is JSON, I inject the ngx variables into the appropriate JSON key in $request_body, pass the resulting text payload of $request_body to the appropriate kafka topic using this module, and then parse that JSON structure on the other end with my kafka consumer. This works great for me but what if someone doesn't want to, or can't, use JSON... now the complexity of implementing this support in this module explodes.

[adridi]

@dkinon Totally Agree with your statement! I was referring to the existing module, I believe it should have set kafka message as = request header + request body. Both are equally important.

[dkinon]

@adridi I was also referring to the existing module, I merely used my lua_ngx module pseudocode example to illustrate one of the many formatted ways this could be done within the confines of the module. The larger point of my previous statement was discussing what format the headers in $request_header + $request_body would look like. Whatever the format, we wouldn't want this to be the default as it would definitely break downstream consumers as they exist today. I'm curious if anyone has suggestions on what those headers would look like?

[1349236893]

When I was testing,crul localhost/ kafka-d "this is a test data",kafka consumers could not accept the data

[1349236893]

当我测试时，crul localhost / kafka-d“这是一个测试数据”，kafka消费者无法接受数据
kafka and nginx No errors were reported

[brg-liuwei]

Could you please give me a whole test case, including the nginx configuration.

…

在 2019年6月25日，下午4:31，1349236893 ***@***.***> 写道： When I was testing,crul localhost/ kafka-d "this is a test data",kafka consumers could not accept the data — You are receiving this because you were assigned. Reply to this email directly, view it on GitHub <#13?email_source=notifications&email_token=ABNEOHZGLSVPBRT5X5WLWF3P4HJXZA5CNFSM4C4WKI5KYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGODYPPCYI#issuecomment-505344353>, or mute the thread <https://github.com/notifications/unsubscribe-auth/ABNEOHZ6ILDPI6JC5BA4CT3P4HJXZANCNFSM4C4WKI5A>.

[brg-liuwei]

当我测试时，crul localhost / kafka-d“这是一个测试数据”，kafka消费者无法接受数据
kafka and nginx No errors were reported

哦，麻烦贴一下你的nginx配置文件

[1349236893]

当我测试时，crul localhost / kafka-d“这是一个测试数据”，kafka消费者无法接受数据
kafka和nginx没有报错

哦，麻烦贴一下你的nginx的配置文件

稍等

[1349236893]

当我测试时，crul localhost / kafka-d“这是一个测试数据”，kafka消费者无法接受数据
kafka和nginx没有报错

哦，麻烦贴一下你的nginx的配置文件
#user nobody;
worker_processes 1;

#error_log logs/error.log;
#error_log logs/error.log notice;
#error_log logs/error.log info;

#pid logs/nginx.pid;

events {
worker_connections 1024;
}

http {
include mime.types;
default_type application/octet-stream;

#log_format  main  '$remote_addr - $remote_user [$time_local] "$request" '
#                  '$status $body_bytes_sent "$http_referer" '
#                  '"$http_user_agent" "$http_x_forwarded_for"';
#access_log  logs/access.log  main;
sendfile        on;
#tcp_nopush     on;
#keepalive_timeout  0;
keepalive_timeout  65;
#gzip  on;

kafka;
kafka_broker_list jaydon-hadoop2:9092jaydon-hadoop3:9092jaydon-hadoop4:9092; 	

server {
    listen       80;
    server_name  node-6.xiaoniu.com;
    #charset koi8-r;
    #access_log  logs/host.access.log  main;

	location = /kafka/bike {
            kafka_topic bike;
    }

	location = /kafka/user {
            kafka_topic user;
    }

    #error_page  404              /404.html;

    # redirect server error pages to the static page /50x.html
    #
    error_page   500 502 503 504  /50x.html;
    location = /50x.html {
        root   html;
    }

}

}

[brg-liuwei]

broker_list每个节点之间要以逗号分隔:

kafka_broker_list jaydon-hadoop2:9092,jaydon-hadoop3:9092,jaydon-hadoop4:9092; 

请确保主机名jaydon-hadoopX在你的nginx上可以被访问，如果你不确定的话，可以换成IPv4地址试试。

另外以后有问题请新建一个issue进行提问，本issue的主题和你提的问题无关，在这个issue下沟通会打扰到别人，谢谢。

[1349236893]

------------------ 原始邮件 ------------------ 发件人: "Wei Liu"<[email protected]>; 发送时间: 2019年6月25日(星期二) 下午4:36 收件人: "brg-liuwei/ngx_kafka_module"<[email protected]>; 抄送: "..."<[email protected]>;"Comment"<[email protected]>; 主题: Re: [brg-liuwei/ngx_kafka_module] NGINX common variables not passedto kafka topic (#13) 当我测试时，crul localhost / kafka-d“这是一个测试数据”，kafka消费者无法接受数据 kafka and nginx No errors were reported 哦，麻烦贴一下你的nginx配置文件 — You are receiving this because you commented. Reply to this email directly, view it on GitHub, or mute the thread.

[1349236893]

broker_list每个节点之间要以逗号分隔：

kafka_broker_list jaydon-hadoop2:9092,jaydon-hadoop3:9092,jaydon-hadoop4:9092; 

请中主机确保名jaydon-hadoopX在你的nginx的上可以被访问，如果你不确定的话，可以换成IPv4的地址试试。

另外以后有问题请新建一个问题进行提问，本课题的主题和你提的问题无关，在这个问题下沟通会打扰到别人，谢谢。

Okay, I'm sorry