fixed 404s

[Mon Nov 3 11:18:58 2008] [error] [client 169.229.192.166] File does not exist: /usr/local/apache/htdocs/newsletter/01-4/2001-4.pdf
FIXED: -> 01-4.pdf

[Mon Nov 3 11:21:37 2008] [error] [client 169.229.192.166] File does not exist: /usr/local/apache/htdocs/aviation/2003/download.php
FIXED: -> /aviation2003downloads/
in /aviation/2003/index.php

also:
/aviation2000downloads/
FIXED: -> /aviation/2000/
in /aviation/2004/index.php, /aviation/2005/index.php


[Sun Nov 2 01:57:47 2008] [error] [client 169.229.192.166] File does not exist: /usr/local/apache/htdocs/sitemap/2004techtopics.pdf
[Sun Nov 2 01:57:47 2008] [error] [client 169.229.192.166] File does not exist: /usr/local/apache/htdocs/sitemap/2002techtopics.pdf
[Sun Nov 2 01:57:50 2008] [error] [client 169.229.192.166] File does not exist: /usr/local/apache/htdocs/sitemap/2006techtopics.pdf
[Sun Nov 2 01:58:17 2008] [error] [client 169.229.192.166] File does not exist: /usr/local/apache/htdocs/sitemap/1999techtopics.pdf
[Sun Nov 2 01:59:22 2008] [error] [client 169.229.192.166] File does not exist: /usr/local/apache/htdocs/sitemap/1998techtopics.pdf
[Sun Nov 2 01:59:22 2008] [error] [client 169.229.192.166] File does not exist: /usr/local/apache/htdocs/sitemap/2005techtopics.pdf
[Sun Nov 2 01:59:23 2008] [error] [client 169.229.192.166] File does not exist: /usr/local/apache/htdocs/sitemap/2001techtopics.pdf

fixed in: /sitemap/index.php

Newsletter

I also fixed a LOT of broken links in older newsletter folders (.html instead of .php, old paths that were replaced 3 or 4 years ago...)

Newsletters from late 2006 and all of 2007 didn't have index.php pages, so I copied HTML from the newsletter/archive page into new index pages for each issue in 2007. Still need to do 2006.

Finding 404s

Today I once again tail'ed the error log, and fixed a bunch of broken links to /freestuff/store.php

Then there are a lot of SSL errors...

Ooh! and I fixed some broken image links in the 2003 Summer newsletter...

analog / report magic on www

I want to:

• stop using analog in my home directory (use analog in /usr/local /usr/bin)
  
• move my webstats script to /user/web
  
• specify dnscache location, etc., on the command line in the script (using variable names) rather than in analog.cfg (that way if I upgrade from dnstran 1.5.2 to 1.5.3 I only need to rename the path once: in the variable assignment in the script.)
• use dnstran in /usr/local

also maybe

• archive apache access logs in /var/log instead of /export/www/htdocs
  

then the webstats script should be run by user web. today it failed because user steve can no longer create analog reports in /export/www/htdocs/ (and maybe never should have been able to)

sed and grep: "Stories" of web visitors

I've been thinking for a long time about crunching the web logs to produce "stories" showing how visitors navigate our site. We don't get so many visitors that it would be overwhelming to look at a day or two.

So I've started writing shell scripts to do the crunching.

They are located at /home/steve/webstats/trunk

The plan is, starting from a raw log file:
1. find just requests for pages (.php, .html, .pdf) with result code 200.

(Hmmm...
right now I am using grep -v to get rid of certain things that aren't (1.) above.
is it simpler to use something like:

grep '] "GET [^ ]*[php|pdf|html][^ ]* HTTP/1.." 200 '

or

sed '/] "GET [^ ]*[php|pdf|html][^ ]* HTTP\/1.." 200 /!d'

to get "successful requests for pages?"

sed doesn't do aaa OR bbb OR ccc very well; it looks like

sed -e '/AAA/b' -e '/BBB/b' -e '/CCC/b' -e d    # most seds
gsed '/AAA\|BBB\|CCC/!d'                        # GNU sed only

sed is very fast, though.

This works at about 10,000 raw log lines/sec:

sed '/" 200 /!d' $logfilename | \
sed   -e '/"GET [^ ]*\.php/b' \
     -e '/"GET [^ ]*\.pdf/b' \
     -e '/"GET [^ ]*\.ppt/b' \
     -e '/"GET [^ ]*\.avi/b' \
     -e '/"GET [^ ]*\.doc/b' \
     -e '/"GET [^ ]*\.mpg/b' \
     -e '/"GET [^ ]*\.flv/b' \
     -e '/"GET [^ ]*\.html/b' \
     -e '/"GET [^ ]*\/ HTTP/b' \
     -e d    > $temporary_story_file

More importantly, if I specify good files, then I have to modify the script every time we add a new file type. And if I forget, it's not easy to notice the absence of the overlooked file type.

grep, grep, grep takes a while to process one log file, about 15 minutes so far...

Tue May 20 20:44:19 PDT 2008

Lines in apache access log:
636818 /export/www/htdocs/weblogs/2008/Q2/access_log_200804

Dropping requests for images:

Tue May 20 20:54:32 PDT 2008
Lines left in temporary work file:
214027 /tmp/stories_14566

Dropping requests for javascripts:

Tue May 20 20:55:01 PDT 2008
Lines left in temporary work file:
203919 /tmp/stories_2_14566

Dropping requests for css:

Tue May 20 20:55:39 PDT 2008
Lines left in temporary work file:
192682 /tmp/stories_14566

Dropping partial or unsuccessful requests:

Tue May 20 20:59:28 PDT 2008
Lines left in temporary work file:
140429 /tmp/stories_2_14566

anyway, next is:
(1.5. weed out search / h@x0r bots)
2. sort by IP address and time
3. clean up the lines to make a tab-delimited table with these fields:
IP address datestamp URL requested referring URL Browser type

Of course you can buy tools that claim to do this...

Web Server Stats: What to do with hax0r traffic

In my ongoing quest to produce a measure of real human use of the site, I now come to malicious traffic.

First thing: how to identify malicious traffic?

In an earlier post I describe creating Fake hacker targets like /_vti_bin/owssvr.dll (on our LAMP server). I could grep access logs for known malicious URLs like that. Problem: high maintenance. The bad guys are always discovering and trying new vulnerabilities, and I would have to somehow keep up.

I could look for large numbers of 404s per host, since probes for vulnerabilities will fail. (Well, they would if I removed the fake targets.)

Fake hacker targets?

Today I created the following empty files:

• /display.php
• /errors.php
• /popup_window.php

these are often requested by script kiddies (display.php thousands of times a month); now we are serving empty docs instead of returning 404 errors.

Is this an improvement?

For us, it means fewer lines in the error logs, and more useful Analog reports.

Does it cause hassle for script kiddies? Presumably their script shows them all non-404 results; will they spend a few seconds trying to figure out our empty doc? And is it dangerous to attract this kind of attention?

Update January 7, 2008:

My colleague Jon Felder says "do it," so I'm adding:

• /_vti_bin/owssvr.dll
• /_vti_bin/shtml.exe/_vti_rpc
• /_vti_inf.html
• /account.php
• /calendar.php
• /cart_content.php
• /confirmUnsubscription.php
• /errors.php
• /fax_form.php
• /freestuff/account.php
• /freestuff/cart_content.php
• /freestuff/popup_window.php
• /freestuff/squirrelcart/js/cart_content.php
• /freestuff/squirrelcart/js/popup_window.php
• /freestuff/squirrelcart/js/squirrelcart/cart_content.php
• /freestuff/squirrelcart/popup_window.php
• /freestuff/squirrelcart/squirrelcart/cart_content.php
• /MSOffice/cltreq.asp
• /newsletter/05-3/errors.php
• /newsletter/confirmUnsubscription.php
• /newsletter/errors.php
• /newsletter/newsletter.php
• /newsletter/newsletter/newsletter.php
• /popup_window.php
• /squirrelcart/cart_content.php
• /squirrelcart/popup_window.php
• /store.php
• /test.php

HAH! Take that, h4x0r!

Fixed 404s

Besides the usual hax0r crud, I found and fixed these 404 errors:
/distancelearning/: symlink -> web-based-training
/helpdesk/: symlink -> services home page
(Hmmm... should I put in redirects instead?)