While browsing my servers Munin reports, I recently noticed how used disk space was constantly increasing on a filesystem which should not have been seeing such growth. After a bit of digging, it appeared the
/usr/local/var/varnish/(host) directory was filled with dozens of sparse files all named
varnish.??????. What could have been happening ?
These files are actually the file backing for the memory-mapped cache used by Varnish by default to hold cache data. Turns out the
varnishd was being run from
/etc/init.d/varnish with a minimal set of options, which did not include the
-s storage option.
In such cases, Varnish defaults to using file-backed memory mapping, and creates a partly-randomly-named file to back the memory map, and preallocates them to the size limit. And, without a storage specification, this means the maximum addressable limit: 2GB on a 32-bit system. Luckily, these are sparse files, not occupying the full 2 GB allocated space, but over time, with periodis server restarts, this can become a problem.
The solutionThe solution is simply to tell Varnish to use a file with a constant name and defined size: that way, whenever Varnish restarts, it will simply reuse the same file name and not waste space creating a new file and forgetting about the earlier one. A start line like this in your will typically fix the problem:
varnishd -a :(port1) -b (host):(port2) -T localhost:(port3) -s file,/usr/local/var/varnish/cache,2G
And, of course, in many cases you will want to add a
-f (/some/config/file.vcl) for your custom rules.
Thanks DocWilco on #varnish for his explanations.
Good post. I recently experimented with varnish on my ioio.ca domain and found that it would stop working by morning. The exact command to execute was:
Then I discovered some bootstrap settings in /sites/all/default/settings.php that dealt specifically with drupal. I have not had a chance to experiment with those settings so time will tell if I resume varnish or not on my server. Also to note that after switching servers from my triple dute Centos 5 Lustre MGS/MDT Server/Client to a dedicated lustre client mounting the http document root on /mnt/ioio/html I had to make a new machine from the ground up and forgot to install php-xml and the SMTP mail wouldn't work for the longest time and so I uninstalled varnish in an attempt to debug but after two to three days of searhing and hacking discovered that php-xml if absent will not allow SMTP in drupal to work. THere are a great many posts out there in drupal land that might apply to php-xml and "my drupal not send email'. Hope this all fits.
Probably not sites/all/default/settings.php
sites/all/default/settings.phpas you wrote but rather
sites/default/settings.php, as the former will likely never be used during a page cycle.
Your ioio.ca site is down, BTW. dunno if that's related.