fixed 404s

[Mon Nov 3 11:18:58 2008] [error] [client 169.229.192.166] File does not exist: /usr/local/apache/htdocs/newsletter/01-4/2001-4.pdf
FIXED: -> 01-4.pdf

[Mon Nov 3 11:21:37 2008] [error] [client 169.229.192.166] File does not exist: /usr/local/apache/htdocs/aviation/2003/download.php
FIXED: -> /aviation2003downloads/
in /aviation/2003/index.php

also:
/aviation2000downloads/
FIXED: -> /aviation/2000/
in /aviation/2004/index.php, /aviation/2005/index.php


[Sun Nov 2 01:57:47 2008] [error] [client 169.229.192.166] File does not exist: /usr/local/apache/htdocs/sitemap/2004techtopics.pdf
[Sun Nov 2 01:57:47 2008] [error] [client 169.229.192.166] File does not exist: /usr/local/apache/htdocs/sitemap/2002techtopics.pdf
[Sun Nov 2 01:57:50 2008] [error] [client 169.229.192.166] File does not exist: /usr/local/apache/htdocs/sitemap/2006techtopics.pdf
[Sun Nov 2 01:58:17 2008] [error] [client 169.229.192.166] File does not exist: /usr/local/apache/htdocs/sitemap/1999techtopics.pdf
[Sun Nov 2 01:59:22 2008] [error] [client 169.229.192.166] File does not exist: /usr/local/apache/htdocs/sitemap/1998techtopics.pdf
[Sun Nov 2 01:59:22 2008] [error] [client 169.229.192.166] File does not exist: /usr/local/apache/htdocs/sitemap/2005techtopics.pdf
[Sun Nov 2 01:59:23 2008] [error] [client 169.229.192.166] File does not exist: /usr/local/apache/htdocs/sitemap/2001techtopics.pdf

fixed in: /sitemap/index.php

Newsletter

I also fixed a LOT of broken links in older newsletter folders (.html instead of .php, old paths that were replaced 3 or 4 years ago...)

Newsletters from late 2006 and all of 2007 didn't have index.php pages, so I copied HTML from the newsletter/archive page into new index pages for each issue in 2007. Still need to do 2006.

Finding 404s

Today I once again tail'ed the error log, and fixed a bunch of broken links to /freestuff/store.php

Then there are a lot of SSL errors...

Ooh! and I fixed some broken image links in the 2003 Summer newsletter...

Checking my web server for errors

My Apache error logs don't tell me enough about 404s. They don't tell me:

User-agent:
If the user-agent is a web crawler, I don't care about bad links on our site; they're probably already fixed and the crawler still has the bad link in their index.

Referrer:
If the 404 is from a bad link on our site, I want to know the originating page.

Query:
Most 404s come from cross-site scripting (XSS) attacks. Without the query part of the URL, it's impossible to distinguish these.

Today I am using tail and grep to check the last few days' worth of 404s:

$ tail -20000 access_log_www | grep -F '" 404'

As it happens, we get about 10,000 requests/day, so to see n days' log entries I just look at the last n0,000 lines in the access logs (tail -20000).

Only 0.2% of requests are 404s, so grep has to winnow through a lot of chaff to find them. To speed it up, I use grep -F, which turns off the regular expression engine.

The string 404 can appear in other places (the file size field, the user-agent field) so I grab the double quote in the last position of the URL field. Naturally that means I have to single-quote the search pattern.

Statistical aside:
How much time does grep -F save?

Here are the results of 10 greps on 20,000 lines, 5 with and 5 without the -F arg:

grep      real  0.486    user  0.460    sys  0.070
grep      real  0.493    user  0.460    sys  0.020
grep      real  0.507    user  0.430    sys  0.080
grep      real  0.510    user  0.490    sys  0.010
grep      real  0.541    user  0.500    sys  0.130
grep -F   real  0.455    user  0.480    sys  0.040
grep -F   real  0.458    user  0.420    sys  0.050
grep -F   real  0.458    user  0.430    sys  0.050
grep -F   real  0.458    user  0.470    sys  0.040
grep -F   real  0.523    user  0.600    sys  0.050

grep (no -F) costs:
7.87% in real time
-2.50% in user time (saves time?)
34.78% in system time (big percentage of a small number)
more than grep -F

put another way, grep -F saves:
7.29% in real time
-2.56% in user time (costs time?)
25.81% in system time (big percentage of a small number)
versus grep

Fake hacker targets?

Today I created the following empty files:

• /display.php
• /errors.php
• /popup_window.php

these are often requested by script kiddies (display.php thousands of times a month); now we are serving empty docs instead of returning 404 errors.

Is this an improvement?

For us, it means fewer lines in the error logs, and more useful Analog reports.

Does it cause hassle for script kiddies? Presumably their script shows them all non-404 results; will they spend a few seconds trying to figure out our empty doc? And is it dangerous to attract this kind of attention?

Update January 7, 2008:

My colleague Jon Felder says "do it," so I'm adding:

• /_vti_bin/owssvr.dll
• /_vti_bin/shtml.exe/_vti_rpc
• /_vti_inf.html
• /account.php
• /calendar.php
• /cart_content.php
• /confirmUnsubscription.php
• /errors.php
• /fax_form.php
• /freestuff/account.php
• /freestuff/cart_content.php
• /freestuff/popup_window.php
• /freestuff/squirrelcart/js/cart_content.php
• /freestuff/squirrelcart/js/popup_window.php
• /freestuff/squirrelcart/js/squirrelcart/cart_content.php
• /freestuff/squirrelcart/popup_window.php
• /freestuff/squirrelcart/squirrelcart/cart_content.php
• /MSOffice/cltreq.asp
• /newsletter/05-3/errors.php
• /newsletter/confirmUnsubscription.php
• /newsletter/errors.php
• /newsletter/newsletter.php
• /newsletter/newsletter/newsletter.php
• /popup_window.php
• /squirrelcart/cart_content.php
• /squirrelcart/popup_window.php
• /store.php
• /test.php

HAH! Take that, h4x0r!