How to Pass Clients/Visitors IP Through Varnish to Nginx

Varnish is a powerful open-source software known as an HTTP reverse proxy typically run in front of web servers such as Apache or Nginx.

Its primary function is to store the Varnish cache and remember the web server’s response to the user at the time of the first content access. It then returns the cached copy for subsequent requests from end users without asking the Nginx web server again.

This process can lead to a situation where the Nginx access logs display the local IP proxy (usually 127.0.0.1 if installed on the same server) instead of the user’s IP.

For example:

127.0.0.1 - - [16/Feb/2015:01:03:09 +0800] "GET /red-hat-details-next-linux-and-storage-platforms-for-cloud-big-data-era/?share=google-plus-1 HTTP/1.1" 302 5 "-" "Mozilla/5.0 (compatible; Yahoo! Slurp; http://help.yahoo.com/help/us/ysearch/slurp)"
127.0.0.1 - - [16/Feb/2015:01:03:15 +0800] "GET /how-to-install-and-configure-epel-repository-on-centos-5-8/ HTTP/1.1" 200 15212 "-" "Mozilla/5.0 (Windows NT 6.1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/40.0.2214.111 Safari/537.36"
127.0.0.1 - - [16/Feb/2015:01:03:22 +0800] "POST /ngx_pagespeed_beacon?url=http%3A%2F%2Fwww.ehowstuff.com%2Fhow-to-install-telnet-client-on-centos-6-3%2F HTTP/1.1" 404 564 "https://webhostinggeeks.com/howto/how-to-install-telnet-client-on-centos-6-3/" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/40.0.2214.111 Safari/537.36"
127.0.0.1 - - [16/Feb/2015:01:03:23 +0800] "GET /how-to-setup-squid-proxy-server-on-linux-centos-6-3/ HTTP/1.1" 200 16246 "-" "Mozilla/5.0 (compatible; Baiduspider/2.0; +http://www.baidu.com/search/spider.html)"

This can be problematic for software like awstats or log analysis software due to incomplete information about the visitors.

In this tutorial, we will guide you on how to relay your blog visitor IP address through Varnish, to Nginx logs. This process has been tested on CentOS 6.6 and CentOS 7. Before we start, please ensure that the http_realip_module has been enabled. This module allows changing the client’s IP address to value from request header (e.g., X-Real-IP or X-Forwarded-For). This module isn’t built by default, enable it with the configure option –with-http_realip_module.

See also  How to Uninstall Nginx on Ubuntu

Step 1: Include “X-Forwareded-For” in the sub vcl_recv default.vcl

[root@centos66 ~]# vi /etc/varnish/default.vcl
sub vcl_recv {
 # IP forwarding
 if (req.restarts == 0) {
 if (req.http.x-forwarded-for) {
 set req.http.X-Forwarded-For =
 req.http.X-Forwarded-For + ", " + client.ip;
 } else {
 set req.http.X-Forwarded-For = client.ip;
 }
 }
..
..

Step 2: Add the following in nginx.conf

[root@centos66 ~]# vi /etc/nginx/nginx.conf
http {
..
..
 set_real_ip_from 127.0.0.1;
 real_ip_header X-Forwarded-For;
..
..
}

Step 3: Restart Nginx web server and Varnish

[root@centos66 ~]# service nginx restart
[root@centos66 ~]# service varnish restart

Step 4: Check and monitor the nginx access log again

After performing the above steps, monitor the Nginx access log again. It should now return the actual visitor IP.

For example:

157.55.39.102 - - [16/Feb/2015:01:06:04 +0800] "GET /how-to-download-centos-6-2-iso/ HTTP/1.1" 200 14622 "-" "Mozilla/5.0 (compatible; bingbot/2.0; +http://www.bing.com/bingbot.htm)"
68.180.228.247 - - [16/Feb/2015:01:06:16 +0800] "GET /tag/centos-6-2/page/4/ HTTP/1.1" 200 14474 "-" "Mozilla/5.0 (compatible; Yahoo! Slurp; http://help.yahoo.com/help/us/ysearch/slurp)"
220.181.108.178 - - [16/Feb/2015:01:06:25 +0800] "GET /howto-guides/howto-centos/ HTTP/1.1" 200 13863 "-" "Mozilla/5.0 (compatible; Baiduspider/2.0; +http://www.baidu.com/search/spider.html)"
66.249.79.116 - - [16/Feb/2015:01:06:33 +0800] "GET /how-to-enable-root-login-on-ubuntu-14-04/ HTTP/1.1" 200 15547 "-" "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"

Commands Mentioned

  • vi /etc/varnish/default.vcl – Opens the Varnish configuration file in a text editor.
  • vi /etc/nginx/nginx.conf – Opens the Nginx configuration file in a text editor.
  • service nginx restart – Restarts the Nginx service.
  • service varnish restart – Restarts the Varnish service.
See also  How to Install and Configure Nginx on Ubuntu 18.04

FAQ

  1. What is Varnish in web hosting?

    Varnish is an open-source HTTP reverse proxy software that is typically run in front of web servers like Apache or Nginx. It is designed to store the Varnish cache and remember the web server’s response to the user at the time of the first content access. It then returns the cached copy for subsequent requests from end users without asking the web server again, thereby improving the performance of the website.

  2. What is the http_realip_module in Nginx?

    The http_realip_module is a module in Nginx that allows changing the client’s IP address to value from request header (e.g., X-Real-IP or X-Forwarded-For). This is particularly useful when you want to pass the original client’s IP address through proxies or load balancers.

  3. Why is it important to pass the client’s IP address through Varnish to Nginx?

    Passing the client’s IP address through Varnish to Nginx is important for accurate logging and analysis. Without this, the Nginx access logs would display the local IP proxy instead of the user’s IP, leading to incomplete information about the visitors. This can be problematic for software like awstats or log analysis software.

  4. What are the steps to relay the visitor’s IP address through Varnish to Nginx logs?

    The steps involve modifying the Varnish and Nginx configuration files to include the “X-Forwarded-For” header, setting the real IP from 127.0.0.1, and then restarting both the Nginx and Varnish services. After these steps, the Nginx access log should return the actual visitor IP.

  5. What does the command ‘service nginx restart’ do?

    The command ‘service nginx restart’ is used to restart the Nginx service. This is often necessary after making changes to the Nginx configuration file to ensure that the new settings take effect.

See also  How to Fix "upstream timed out (110: Connection timed out) while reading response header from upstream" in NGINX

Conclusion

Relaying your blog visitor’s IP address through Varnish to Nginx logs is a crucial step in ensuring accurate logging and analysis of your website’s traffic. By following the steps outlined in this tutorial, you can easily configure your server to pass the client’s IP address through Varnish to Nginx. This process has been tested on CentOS 6.6 and CentOS 7, and it involves modifying the Varnish and Nginx configuration files, setting the real IP from 127.0.0.1, and restarting both the Nginx and Varnish services.

Remember, before you start, ensure that the http_realip_module has been enabled. This module allows changing the client’s IP address to value from request header (e.g., X-Real-IP or X-Forwarded-For). This module isn’t built by default, enable it with the configure option –with-http_realip_module.

I hope this tutorial has been helpful in guiding you through the process.

Comments

Leave a Reply

Your email address will not be published. Required fields are marked *