Web Analytics – Tools

I spent some time last week evaluating the web usage reports that we generate for the ISU Extension web site. I took a little different approach this time, in that I was looking at the entire site, rather than individual counties, departments, projects, etc. that are hosted as folders on the site.

To do the bulk of the analysis, I used reports generated by WebTrends, an extremely old version of it. Someday we’ll either upgrade (expensive) or change to another tool. For now, we still get a lot of good from the old version we’re running.

First thing I did was adjust the report for the entire server to eliminate as many robots/spiders/crawlers as I could. I did this by filtering out browsers that contained certain words, which took out many of them. Next, I filtered out all visits where the entry page was “/robots.txt”, which eliminate many more. This should catch the well behaved robots.

I then created a second report, which did some additional filtering. For the second report, I filtered out all visits where the client computer was from iastate.edu. I was mostly interested in how the public use our site. Unfortunately, this report does contain visits from staff members when they are not on the ISU network, for example if I access the site from home, it is included. I don’t know a good way to find and exclude these hits.

When I saw something in the report that seemed odd, I would use two additional tools to manipulate the actual IIS log file to see what was going on. First one was a UNIX style grep. I use this tool a lot to find words/phases in text files. I typically pipe the output into another text file, that I then load into Excel to analyze.

The second tool I use is Log Parser. Log Parser is a free download from Microsoft, and lets you do SQL queries against your log file. For some queries, like seeing the top ten pages requested from your server, this is an awesome tool.

I learned a lot about the way our web server is used. More on that in upcoming posts.

%d bloggers like this: