Installing Varnish for your Magento Store on CentOS 6

You’ve compressed and minified your JavaScript and CSS, optimized your database, and tweaked your caching config, and yet you still can’t seem to get that time-to-first-byte to drop below one second. Tweaking dynamic applications like Magento can be frustrating. Sometimes there’s so much behind-the-scenes processing that goes into generating that catalog page that even your developers don’t understand where the problem lies. This is where code profiling becomes crucial. But this post isn’t about profiling or tweaking algorithms. This is a post for those of you who have done all that. This is a post about making your dynamic site fly with a great piece of software called Varnish.

What is Varnish

Varnish calls itself a “web application accelerator”. It’s essentially a really fast caching HTTP reverse proxy that sits in front of your web or application server. It proxies web requests to the application and caches the result, essentially making your dynamic site as fast as one composed of static HTML, images and scripts.

Varnish performs really well. Our tests show up to 1000x improvements in transactions per second. It’s also very flexible with it’s domain-specific configuration language, VCL, that’s based on C and Perl. It can act as a load balancer for multiple back-end application servers and supports both IP and Name-based virtual hosts. And best of all, Varnish is free software, licensed under the FreeBSD license.

Varnish and Magento

Varnish works really well out-of-the-box for sites with minimal session-dependent content. However, with eCommerce sites like Magento, there needs to be tight integration with the application so that users can add products to their cart or wishlist and check out. Fortunately Hostdedi has developed a free and open-source Magento-Varnish extension called Turpentine that has solved this problem. Not only does this extension have great features like allowing custom exceptions for different routes and allowing users to purge the cache from the Magento Admin, it also comes with pre-configured VCL configs for Varnish 2.1 and 3.0, making implementation easy. There are other great Varnish integration extensions for Magento out there, but the rest of this post will focus on installing and configuring Hostdedi Turpentine.

Caveats

Before we begin, I should mention a few caveats to consider before adding Varnish to your application stack. First, if you use any server-side analytic tools that sit behind Varnish, you need to understand that 90% of your traffic will not reach the application servers. Varnish will be serving the cached content for all non-exempt visitors (non-exempt meaning users who do not have an active cart or session). This means that if your traffic statistics depend on hits to the application or back-end web servers, then they will no longer reflect reality. Third-party tools that utilize JavaScript embedded in the page, like Google Analytics, should continue to work normally.

Second, the traffic that does pass to the backend servers will contain the IP address of the Varnish server instead of the client. This is because the client request is being proxied through Varnish. There are two ways to work around this. First, Varnish, like any proxy, can be configured to populate the X-Forwarded-For header with the “real” IP address of the client. If your back-end server is Apache, you can install mod_rpaf, which will rewrite the remote address with the IP contained in this header. The LiteSpeed web server can do this natively by toggling a configuration option, as I’m sure can other web servers.

With the above in mind, let’s move on to the installation process.

Step 1: Install Varnish

On CentOS 6 servers, varnish-2.1.5 is available in the EPEL repo. The Hostdedi Turpentine extension can generate a VCL for Varnish versions 2.1 and 3.0. Varnish 3.0 has some great features like ESI support, but you need to rebuild the 3.0 RPMs from the Fedora package if you want to use it. For simplicity, we’ll stick with the 2.1 package so that a simple “yum install” will work:

[bash]$ yum install varnish[/bash]

The varnish configuration files will be found under /etc/varnish, but we’ll skip that for now since we’ll be using the auto-generated configuration file that comes with the Turpentine extension.

Step 2: Install Hostdedi Turpentine

You can use the Magento Admin to install this extension, which can be found on Magento Connect. I prefer to use the “mage” command line tool. Note: detailed installation docs can be found on our github site.

[bash]$ ./mage install connect20.magentocommerce.com/community Nexcessnet_Turpentine[/bash]

The exension will be installed under “app/code/community/Nexcessnet/Turpentine/”. As mentioned, the configuration VCL is automatically generated by the Turpentine extension, so we don’t need to worry about editing it manually.

Step 3: Configure Varnish

Now we need to tell Varnish to sit in front of the web server and proxy requests to the web server. In this example, we’re using the Apache web server as the back-end, but the same concept applies to any web server you might be using. We’re also going to assume that all of this is being done on a single server, but keep in mind that this basic set-up can scale to any N-tier cluster.

First, open up the /etc/sysconfig/varnish file and and change the default port for Varnish to port 80.

[bash]VARNISH_LISTEN_PORT=80[/bash]

Second, to take advantage of Varnish’s memory-backed storage, change the VARNISH_STORAGE variable to the following:

[bash]VARNISH_STORAGE=”malloc,${VARNISH_STORAGE_SIZE}”[/bash]

As you can see, we need to reconfigure the web server to listen on port 8080, since port 80 is the HTTP port and Varnish needs to listen there. We do this next.

Open up the httpd.conf (locations vary per distro, CentOS and RHEL is /etc/httpd/conf/httpd.conf). You want to find the line that says “Listen 80” and change it to “Listen 8080”.

[bash]#

# Listen: Allows you to bind Apache to specific IP addresses and/or

# ports, instead of the default. See also the

# directive.

# Change this to Listen on specific IP addresses as shown below to

# prevent Apache from glomming onto all bound IP addresses (0.0.0.0)

#Listen 12.34.56.78:80[/bash]

Listen 8080

If you have VHOSTs or NameVirtualHosts, you need to update the ports in those directives as well.

[bash]NameVirtualHost 192.168.8.8:8080

…

…

etc.[/bash]

Once you’ve updated your Apache configs, it’s always good to check your syntax.

[bash]$ httpd -t

Syntax OK[/bash]

Assuming all is good, you can now restart Apache and start up Varnish.

[bash]$ service httpd restart && service varnish start[/bash]

You can verify that the port changes are correct using netstat.

[bash]$ netstat -ntlp | grep -w 80

tcp 0 0 0.0.0.0:80 0.0.0.0:* LISTEN 29770/varnishd[/bash]

Try loading your site in your browser. It should load without any problems. If the connection times out, then you’ve missed a step somewhere. Try connecting to your site on port 8080 to bypass varnish and make sure the backend web server is working (http://www.example.com:8080/). Also check to make sure you don’t have a firewall block in place.

Step 4: Enable VarnishCache in Magento

Change the “Backend Host” to your server’s IP address, and change the “Backend Port” to 80. You will also need to provide the IP and port of the Varnish admin service. By default, this is listening on localhost, port 6082. Finally, you need to add the “Varnish authentication key” for accessing the admin port. This can be found on CentOS in the /etc/varnish/secret file. Just copy/paste the line in that file into the form.

To finish up, just click “Save Config”, then navigate to System -> Varnish Management. Click on “Apply Configuration” to save the auto-generated VCL to the default location.

That’s it! Load some pages on the front end a few times and watch Varnish in action. You should see considerable improvements in response time once Varnish has cached the page. Try adding some products to your cart to confirm that Varnish is not caching them. You can even try running a benchmark against a few URLs using Apache Bench, Siege, or magespeedtest.com.

Conclusion

Varnish is a terrific tool for improving the performance and scalability of dynamic websites like Magento. Its flexibility and options can’t all be covered in a short blog post, but with later versions, you can take advantage of more advanced features, like “hole punching” with Edge Side Includes or load balancing. As always, it’s important to make sure you’re not falling prey to what I call “Performance Tuning Sickness”. Your goal should be to have an accessible, dynamic, functional application first, and then try to optimize it for maximal performance. Varnish should not be used as a band-aid over bad code.

Until next time, cheers.

Source link