If you run a website, there’s a good chance you regularly check Google Analytics to see how many visitors you’re getting, where they’re coming from and which pages they’re looking at. You might also use Google Search Console to monitor your search engine rankings and identify any problems Google has encountered while crawling your website.
However, there’s another source of information that I think is frequently overlooked, even by people who work in digital marketing, and that’s your website’s server log files.
Unlike Google Analytics, which relies on tracking code to collect information about visitors, server logs record requests made directly to your web server. They can tell you which pages are being requested, when those requests occurred, which IP addresses they came from and whether your server was able to respond successfully.
It might not sound particularly exciting, but you’d be surprised how much useful information is hiding in those files.
What Exactly Are Server Log Files?
Every time somebody visits a page on your website, their browser sends a request to the server hosting it. The server then responds by sending back the requested content, whether that’s an HTML page, an image, a stylesheet or something else.
Most web servers, including Apache and Nginx, can record these requests in files commonly referred to as access logs.
Each entry typically includes the visitor’s IP address, the date and time, the URL requested, the HTTP response code and information about the browser or software making the request.
For example, if somebody visits your homepage, you might see a request for / with a response code of 200, indicating that the server successfully delivered the page.
If somebody requests a page that no longer exists, you’ll probably see a 404 response instead.
And if your server encounters a problem, you might see a 500 error, indicating that something has gone wrong.
Most website owners will never need to look at this information, but if you’re trying to understand why something isn’t working properly, it can be extremely useful.
What Can Log Files Tell You About SEO?
From an SEO perspective, one of the most interesting things about server logs is that they allow you to see how search engine crawlers interact with your website.
Google uses automated software called Googlebot to discover and retrieve pages for its search engine. While Google Search Console provides information about crawling and indexing, your server logs can offer a much more detailed picture of which URLs are actually being requested.
For example, you might discover that Googlebot is repeatedly visiting hundreds of old URLs that no longer exist, or spending considerable time crawling pages that aren’t particularly important.
Alternatively, you might find that some of your most valuable pages haven’t been requested by Googlebot for weeks.
This doesn’t necessarily mean there’s a problem, as Google doesn’t crawl every page at the same frequency, but it can help identify areas worth investigating.
It’s particularly useful for larger ecommerce websites, where thousands of product pages, category pages and filtered URLs can create an enormous number of possible combinations for search engines to crawl.
Google only has a finite amount of resources available for crawling websites, something commonly referred to as a ‘crawl budget’. If those resources are being spent repeatedly requesting unimportant URLs, it can potentially delay the discovery of more valuable content.
Incidentally, it’s worth remembering that user-agent information can be faked, so a request claiming to come from Googlebot isn’t necessarily genuine. Google provides guidance on verifying its crawlers using IP addresses and DNS lookups.
Finding Problems You Didn’t Know Existed
Another useful application is identifying technical problems that might otherwise go unnoticed.
Let’s say your website has a page that occasionally returns a 500 error. Unless somebody reports it, or the problem occurs while you’re testing the website, you might never know there’s anything wrong.
However, those failed requests should appear in your server logs, making it possible to identify when the problem occurred and which URLs were affected.
You might also discover broken links, redirect loops or requests for files that no longer exist.
One particularly useful exercise is to look for URLs generating large numbers of 404 errors. Sometimes these will be requests for old pages that have been removed, but they can also reveal internal links that need correcting or external websites linking to incorrect addresses.
Of course, not every 404 is a problem that needs fixing. Websites constantly receive requests for URLs that have never existed, particularly from automated bots, so there’s little point trying to eliminate every error.
The important thing is identifying patterns that suggest something genuinely needs attention.
What About Security?
This is another area where server logs can be particularly revealing.
If you’ve ever examined the access logs of a publicly accessible website, you’ll probably have noticed some rather strange requests.
It’s not unusual to see automated systems trying to access WordPress administration pages, looking for vulnerable plugins or requesting files that have absolutely no business existing on your website.
Most of these attempts are simply automated scanning activity rather than somebody specifically targeting your business.
However, server logs can help identify suspicious behaviour, particularly when large numbers of requests originate from the same IP address or repeatedly target sensitive areas of your website.
They can also help explain sudden spikes in server load, which might otherwise be mistaken for an increase in genuine visitors.
Obviously, reviewing logs isn’t a substitute for proper website security, but it can provide useful evidence when something unusual happens.
How Do You Access Your Log Files?
That depends on where your website is hosted.
If you’re using a hosting control panel such as cPanel or Plesk, there’s a good chance you’ll find an option to download or view your access logs.
If you manage your own server, the files are commonly stored in directories such as /var/log/apache2/ or /var/log/nginx/, although their exact location will depend on your configuration.
For smaller websites, you can often get away with opening the files in a text editor, but larger sites can generate hundreds of thousands of entries, making manual analysis rather impractical.
Fortunately, there are tools available to help. Screaming Frog’s Log File Analyser, for example, can process server logs and provide useful information about search engine crawling behaviour without requiring you to manually inspect every request.
If you’re comfortable with a little programming, you can also analyse the files using Python, which makes it relatively straightforward to filter requests, identify patterns and produce reports.
Should You Be Checking Them Regularly?
Personally, I wouldn’t suggest that every small business owner needs to start downloading their server logs every morning.
For most websites, Google Analytics and Search Console will provide enough information to understand how the site is performing, and log file analysis probably isn’t going to transform your SEO overnight.
However, if you’re responsible for a large website, investigating technical SEO problems or trying to understand unexpected server behaviour, I’d strongly recommend taking a look.
There’s a considerable difference between knowing what Google Analytics says is happening on your website and seeing what requests your server is actually receiving.
And sometimes, the most useful information is the stuff your analytics software never recorded in the first place.