What to do when your Varnish directory fills up

Submitted by Frederic Marand on Fri, 2011-04-08 23:02

Add new comment

The symptoms

While browsing my servers Munin reports, I recently noticed how used disk space was constantly increasing on a filesystem which should not have been seeing such growth. After a bit of digging, it appeared the /usr/local/var/varnish/(host) directory was filled with dozens of sparse files all named varnish.??????. What could have been happening ?

The facts

These files are actually the file backing for the memory-mapped cache used by Varnish by default to hold cache data. Turns out the varnishd was being run from /etc/init.d/varnish with a minimal set of options, which did not include the -s storage option.

In such cases, Varnish defaults to using file-backed memory mapping, and creates a partly-randomly-named file to back the memory map, and preallocates them to the size limit. And, without a storage specification, this means the maximum addressable limit: 2GB on a 32-bit system. Luckily, these are sparse files, not occupying the full 2 GB allocated space, but over time, with periodis server restarts, this can become a problem.

The solution

The solution is simply to tell Varnish to use a file with a constant name and defined size: that way, whenever Varnish restarts, it will simply reuse the same file name and not waste space creating a new file and forgetting about the earlier one. A start line like this in your will typically fix the problem:

varnishd -a :(port1) -b (host):(port2) -T localhost:(port3) -s file,/usr/local/var/varnish/cache,2G

And, of course, in many cases you will want to add a -f (/some/config/file.vcl) for your custom rules.

Thanks DocWilco on #varnish for his explanations.

varnish restart

Good post. I recently experimented with varnish on my ioio.ca domain and found that it would stop working by morning. The exact command to execute was:

varnishd -f /etc/varnish/default.vcl -s malloc,1G -T 127.0.0.1:1000

[root@three init.d]# cat /etc/varnish/default.vcl
# This is a basic VCL configuration file for varnish.  See the vcl(7)
# man page for details on VCL syntax and semantics.                  
#                                                                    
# Default backend definition.  Set this to point to your content     
# server.                                                            
#                                                                    
backend default {                                                    
  .host = "127.0.0.1";                                               
  .port = "8080";                                                    
}                                                                    

sub vcl_recv {
              
    if (      
      req.http.user-agent ~ "^$"
      || req.http.user-agent ~ "^Java"
      || req.http.user-agent ~ "^Jakarta"
      || req.http.user-agent ~ "IDBot"   
      || req.http.user-agent ~ "id-search"
      || req.http.user-agent ~ "User-Agent"
      || req.http.user-agent ~ "compatible ;"
      || req.http.user-agent ~ "ConveraCrawler"
      || req.http.user-agent ~ "^Mozilla$"     
      || req.http.user-agent ~ "libwww"        
      || req.http.user-agent ~ "lwp-trivial"   
      || req.http.user-agent ~ "curl"          
      || req.http.user-agent ~ "PHP/"          
      || req.http.user-agent ~ "urllib"        
      || req.http.user-agent ~ "GT:WWW"        
      || req.http.user-agent ~ "Snoopy"        
      || req.http.user-agent ~ "MFC_Tear_Sample"
      || req.http.user-agent ~ "HTTP::Lite"     
      || req.http.user-agent ~ "PHPCrawl"       
      || req.http.user-agent ~ "URI::Fetch"     
      || req.http.user-agent ~ "Zend_Http_Client"
      || req.http.user-agent ~ "http client"     
      || req.http.user-agent ~ "PECL::HTTP"      
      || req.http.user-agent ~ "panscient.com"   
      || req.http.user-agent ~ "IBM EVV"         
      || req.http.user-agent ~ "Bork-edition"    
      || req.http.user-agent ~ "Fetch API Request"
      || req.http.user-agent ~ "PleaseCrawl"      
      || req.http.user-agent ~ "[A-Z][a-z]{3,} [a-z]{4,} [a-z]{4,}"
      || req.http.user-agent ~ "layeredtech.com"                   
      || req.http.user-agent ~ "WEP Search"                        
      || req.http.user-agent ~ "Wells Search II"                   
      || req.http.user-agent ~ "Missigua Locator"                  
      || req.http.user-agent ~ "ISC Systems iRc Search 2.1"        
      || req.http.user-agent ~ "Microsoft URL Control"             
      || req.http.user-agent ~ "Indy Library"                      
      || req.http.user-agent == "8484 Boston Project v 1.0"        
      || req.http.user-agent == "Atomic_Email_Hunter/4.0"          
      || req.http.user-agent == "atSpider/1.0"                     
      || req.http.user-agent == "autoemailspider"                  
      || req.http.user-agent == "China Local Browse 2.6"           
      || req.http.user-agent == "ContactBot/0.2"                   
      || req.http.user-agent == "ContentSmartz"                    
      || req.http.user-agent == "DataCha0s/2.0"                    
      || req.http.user-agent == "DataCha0s/2.0"                    
      || req.http.user-agent == "DBrowse 1.4b"                     
      || req.http.user-agent == "DBrowse 1.4d"                     
      || req.http.user-agent == "Demo Bot DOT 16b"                 
      || req.http.user-agent == "Demo Bot Z 16b"                   
      || req.http.user-agent == "DSurf15a 01"                      
      || req.http.user-agent == "DSurf15a 71"                      
      || req.http.user-agent == "DSurf15a 81"                      
      || req.http.user-agent == "DSurf15a VA"                      
      || req.http.user-agent == "EBrowse 1.4b"                     
      || req.http.user-agent == "Educate Search VxB"               
      || req.http.user-agent == "EmailSiphon"                      
      || req.http.user-agent == "EmailWolf 1.00"                   
      || req.http.user-agent == "ESurf15a 15"                      
      || req.http.user-agent == "ExtractorPro"                     
      || req.http.user-agent == "Franklin Locator 1.8"             
      || req.http.user-agent == "FSurf15a 01"                      
      || req.http.user-agent == "Full Web Bot 0416B"               
      || req.http.user-agent == "Full Web Bot 0516B"               
      || req.http.user-agent == "Full Web Bot 2816B"               
      || req.http.user-agent == "Guestbook Auto Submitter"         
      || req.http.user-agent == "Industry Program 1.0.x"           
      || req.http.user-agent == "ISC Systems iRc Search 2.1"       
      || req.http.user-agent == "IUPUI Research Bot v 1.9a"        
      || req.http.user-agent == "LARBIN-EXPERIMENTAL (efp@gmx.net)"
      || req.http.user-agent == "LetsCrawl.com/1.0 +http://letscrawl.com/"
      || req.http.user-agent == "Lincoln State Web Browser"               
      || req.http.user-agent == "LMQueueBot/0.2"                          
      || req.http.user-agent == "LWP::Simple/5.803"                       
      || req.http.user-agent == "Mac Finder 1.0.xx"                       
      || req.http.user-agent == "MFC Foundation Class Library 4.0"        
      || req.http.user-agent == "Microsoft URL Control - 6.00.8xxx"       
      || req.http.user-agent == "Missauga Locate 1.0.0"                   
      || req.http.user-agent == "Missigua Locator 1.9"                    
      || req.http.user-agent == "Missouri College Browse"                 
      || req.http.user-agent == "Mizzu Labs 2.2"                          
      || req.http.user-agent == "Mo College 1.9"                          
      || req.http.user-agent == "Mozilla/2.0 (compatible; NEWT ActiveX; Win32)"
      || req.http.user-agent == "Mozilla/3.0 (compatible; Indy Library)"       
      || req.http.user-agent == "Mozilla/4.0 (compatible; Advanced Email Extractor v2.xx)"
      || req.http.user-agent == "Mozilla/4.0 (compatible; Iplexx Spider/1.0 http://www.iplexx.at)"
      || req.http.user-agent == "Mozilla/4.0 (compatible; MSIE 5.0; Windows NT; DigExt; DTS Agent"
      || req.http.user-agent == "Mozilla/4.0 efp@gmx.net"                                         
      || req.http.user-agent == "Mozilla/5.0 (Version: xxxx Type:xx)"                             
      || req.http.user-agent == "Mozilla/5.0 (compatible; SISTRIX Crawler; http://crawler.sistrix.net/)"                                                                                                
      || req.http.user-agent == "Mozilla/5.0 (Windows NT 6.0) AppleWebKit/535.1 (KHTML, like Gecko) Chrome/13.0.782.112 Safari/535.1"                                                                   
      || req.http.user-agent == "MVAClient"                                                         
      || req.http.user-agent == "NameOfAgent (CMS Spider)"                                          
      || req.http.user-agent == "NASA Search 1.0"                                                   
      || req.http.user-agent == "Nsauditor/1.x"                                                     
      || req.http.user-agent == ""                                                                  
      || req.http.user-agent == "PBrowse 1.4b"                                                      
      || req.http.user-agent == "PEval 1.4b"                                                        
      || req.http.user-agent == "Poirot"                                                            
      || req.http.user-agent == "Port Huron Labs"                                                   
      || req.http.user-agent == "Production Bot 0116B"                                              
      || req.http.user-agent == "Production Bot 2016B"                                              
      || req.http.user-agent == "Production Bot DOT 3016B"                                          
      || req.http.user-agent == "Program Shareware 1.0.2"                                           
      || req.http.user-agent == "PSurf15a 11"                                                       
      || req.http.user-agent == "PSurf15a 51"                                                       
      || req.http.user-agent == "PSurf15a VA"                                                       
      || req.http.user-agent == "psycheclone"                                                       
      || req.http.user-agent == "RSurf15a 41"                                                       
      || req.http.user-agent == "RSurf15a 51"                                                       
      || req.http.user-agent == "RSurf15a 81"                                                       
      || req.http.user-agent == "searchbot admin@google.com"                                        
      || req.http.user-agent == "ShablastBot 1.0"                                                   
      || req.http.user-agent == "snap.com beta crawler v0"                                          
      || req.http.user-agent == "Snapbot/1.0"                                                       
      || req.http.user-agent == "sogou develop spider"                                              
      || req.http.user-agent == "Sogou Orion spider/3.0(+http://www.sogou.com/docs/help/webmasters.htm#07)"                                                                                             
      || req.http.user-agent == "sogou spider"                                                      
      || req.http.user-agent == "Sogou web spider/3.0(+http://www.sogou.com/docs/help/webmasters.htm#07)"                                                                                               
      || req.http.user-agent == "sohu agent"                                                        
      || req.http.user-agent == "SSurf15a 11"                                                       
      || req.http.user-agent == "TSurf15a 11"                                                       
      || req.http.user-agent == "Under the Rainbow 2.2"                                             
      || req.http.user-agent == "User-Agent: Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1)"    
      || req.http.user-agent == "VadixBot"                                                          
      || req.http.user-agent == "Mozilla/5.0 (Windows; U; Windows NT 6.1; ru; rv:1.9.2.16) Gecko/20110319 Firefox/3.6.16 WebMoney Advisor"                                                              
      || req.http.user-agent == "WebVulnCrawl.blogspot.com/1.0 libwww-perl/5.803"                   
      || req.http.user-agent == "Wells Search II"                                                   
      || req.http.user-agent == "WEP Search 00"                                                     
      || req.http.user-agent == "Opera/9.80 (Windows NT 6.1; U; MRA 6.0 (build 5680); ru) Presto/2.10.289 Version/12.00"                                                                                
      || req.http.user-agent == "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/535.11 (KHTML, like Gecko) Chrome/17.0.963.46 Safari/535.11 MRCHROME"                                                  
      || req.http.user-agent == "Opera/9.80 (Windows NT 5.1; U; MRA 5.10 (build 5288); ru) Presto/2.2.15 Version/10.10"                                                                                 
      || req.http.user-agent == "Mozilla/5.0 (compatible; YandexBot/3.0; +http://yandex.com/bots)"  
      || req.http.user-agent == "Mozilla/5.0 (compatible; Ezooms/1.0; ezooms.bot@gmail.com)"        
      || req.http.user-agent == "Opera/9.80 (Windows NT 6.1; WOW64; U; Edition Ukraine Local; ru) Presto/2.10.289 Version/12.00"                                                                        
      || req.http.user-agent == "Mozilla/5.0 (compatible; Baiduspider/2.0; +http://www.baidu.com/search/spider.html)"                                                                                   
      || req.http.user-agent == "Opera/9.80 (Windows NT 6.1; Win64; x64; U; ru) Presto/2.10.289 Version/12.00"                                                                                          
      || req.http.user-agent == "Mozilla/5.0 (Windows NT 6.1; rv:13.0) Gecko/20100101 Firefox/13.0.1"                                                                                                   
      || req.http.user-agent == "Mozilla/5.0 (Windows NT 5.1) AppleWebKit/536.5 (KHTML, like Gecko) Chrome/19.0.1084.52 Safari/536.5"                                                                   
      || req.http.user-agent == "Opera/9.80 (Windows NT 6.1; WOW64; U; en) Presto/2.10.289 Version/12.00"                                                                                               
      || req.http.user-agent == "Mozilla/5.0 (compatible; MJ12bot/v1.4.3; http://www.majestic12.co.uk/bot.php?+)"                                                                                       
      || req.http.user-agent == "Mozilla/5.0 (compatible; discoverybot/2.0; +http://discoveryengine.com/discoverybot.html)"                                                                             
      || req.http.user-agent == ""                                                                  
      || req.http.user-agent == ""                                                                  
    ) {                                                                                             
      error 403 "You are banned from this site.  Please contact via a different client configuration if you believe that this is a mistake.";                                                           
    }                                                                                               


    #if (req.http.User-Agent ~ "(yandexbot|baiduspider|sosospider)") { error 403 "forbidden"; }
#############################################
### This restores normal apache client ip ###
#############################################
#LogFormat "%{X-Forwarded-For}i %l %u %t "%r" %>s %b "%{Referer}i" "%{User-Agent}i"" combined

    remove req.http.X-Forwarded-For;                                                           
    set req.http.X-Forwarded-For = req.http.rlnclientipaddr;                                   
}                                                                                              
#                                                                                              
# Below is a commented-out copy of the default VCL logic.  If you                              
# redefine any of these subroutines, the built-in logic will be                                
# appended to your code.                                                                       
#                                                                                              
# sub vcl_recv {                                                                               
#     if (req.restarts == 0) {                                                                 
#       if (req.http.x-forwarded-for) {                                                        
#           set req.http.X-Forwarded-For =                                                     
#               req.http.X-Forwarded-For ", "  client.ip;                                      
#       } else {                                                                               
#           set req.http.X-Forwarded-For = client.ip;                                          
#       }                                                                                      
#     }                                                                                        
#     if (req.request != "GET" &&                                                              
#       req.request != "HEAD" &&                                                               
#       req.request != "PUT" &&                                                                
#       req.request != "POST" &&                                                               
#       req.request != "TRACE" &&                                                              
#       req.request != "OPTIONS" &&                                                            
#       req.request != "DELETE") {                                                             
#         /* Non-RFC2616 or CONNECT which is weird. */                                         
#         return (pipe);                                                                       
#     }                                                                                        
#     if (req.request != "GET" && req.request != "HEAD") {                                     
#         /* We only deal with GET and HEAD by default */                                      
#         return (pass);                                                                       
#     }                                                                                        
#     if (req.http.Authorization || req.http.Cookie) {                                         
#         /* Not cacheable by default */                                                       
#         return (pass);                                                                       
#     }                                                                                        
#     return (lookup);                                                                         
# }                                                                                            
                                                                                               
# sub vcl_pipe {                                                                               
#     # Note that only the first request to the backend will have                              
#     # X-Forwarded-For set.  If you use X-Forwarded-For and want to                           
#     # have it set for all requests, make sure to have:                                       
#    #  set bereq.http.connection = "close";                                                   
#     # here.  It is not set by default as it might break some broken web                      
#     # applications, like IIS with NTLM authentication.                                       
#     return (pipe);                                                                           
# }                                                                                            
#                                                                                              
# sub vcl_pass {                                                                               
#     return (pass);                                                                           
# }                                                                                            
#                                                                                              
# sub vcl_hash {                                                                               
#     set req.hash += req.url;                                                                 
#     if (req.http.host) {                                                                     
#         set req.hash += req.http.host;                                                       
#     } else {                                                                                 
#         set req.hash += server.ip;                                                           
#     }                                                                                        
#     return (hash);                                                                           
# }                                                                                            
#                                                                                              
# sub vcl_hit {                                                                                
#     if (!obj.cacheable) {                                                                    
#         return (pass);                                                                       
#     }                                                                                        
#     return (deliver);                                                                        
# }                                                                                            
#                                                                                              
# sub vcl_miss {                                                                               
#     return (fetch);                                                                          
# }                                                                                            
#                                                                                              
# sub vcl_fetch {                                                                              
#     if (!beresp.cacheable) {                                                                 
#         return (pass);                                                                       
#     }                                                                                        
#     if (beresp.http.Set-Cookie) {                                                            
#         return (pass);                                                                       
#     }                                                                                        
#     return (deliver);                                                                        
# }                                                                                            
#                                                                                              
# sub vcl_deliver {                                                                            
#     return (deliver);                                                                        
# }
#
# sub vcl_error {
#     set obj.http.Content-Type = "text/html; charset=utf-8";
#     synthetic {"
# <?xml version="1.0" encoding="utf-8"?>
# <!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Strict//EN"
#  "http://www.w3.org/TR/xhtml1/DTD/xhtml1-strict.dtd">
# <html>
#   <head>
#     <title>"} obj.status " " obj.response {"</title>
#   </head>
#   <body>
#     <h1>Error "} obj.status " " obj.response {"</h1>
#     <p>"} obj.response {"</p>
#     <h3>Guru Meditation:</h3>
#     <p>XID: "} req.xid {"</p>
#     <hr>
#     <p>Varnish cache server</p>
#   </body>
# </html>
# "};
#     return (deliver);
# }
[root@three init.d]#

Then I discovered some bootstrap settings in /sites/all/default/settings.php that dealt specifically with drupal. I have not had a chance to experiment with those settings so time will tell if I resume varnish or not on my server. Also to note that after switching servers from my triple dute Centos 5 Lustre MGS/MDT Server/Client to a dedicated lustre client mounting the http document root on /mnt/ioio/html I had to make a new machine from the ground up and forgot to install php-xml and the SMTP mail wouldn't work for the longest time and so I uninstalled varnish in an attempt to debug but after two to three days of searhing and hacking discovered that php-xml if absent will not allow SMTP in drupal to work. THere are a great many posts out there in drupal land that might apply to php-xml and "my drupal not send email'. Hope this all fits.

Probably not sites/all/default/settings.php

Probably not sites/all/default/settings.php as you wrote but rather sites/default/settings.php, as the former will likely never be used during a page cycle.

Your ioio.ca site is down, BTW. dunno if that's related.