robots.txt

Added a line:

Disallow: /freestuff/ # use /g3/ instead

editing robots.txt

# see http://www.robotstxt.org/

Today I looked through a list of our top-level directories and added lines to /robots.txt.

symbolic links

If we have several names for the same directory (e.g., /ped-safety/ and /pedsafety/) I added a Disallow: directive for the symbolic link.

Disallow: /ped-safety/  # use /pedsafety/ instead

This way, Google et al. will cache things with the preferred URL. Especially desirable for holdovers from an era when our site was organized completely differently:

Disallow: /resources/media/multimedia/  # use /videos/ instead

"honeypots"

These are common vulnerabilities that are randomly requested by hackers and spambots. I added zero-length empty files for the most-requested to eliminate 404s for common vulnerabilities that we don't have, so that our 404 reports would show actual broken links.

Disallow: /_vti_bin/    # honeypot

private directories

set up for various special projects; the URLs are not published. I had a couple Disallow: directives for these, but that's essentially publishing the secret URL, so I removed those directives.

how to tell the version of Redhat Linux installed on a machine from a command line

From Wijaya's Blog

Here is a sample:

$ uname -a
Linux myserver.mydomain.com 2.4.21-32.ELsmp #1 SMP Fri Apr 15 21:17:59 EDT 2005 i686 i686 i386 GNU/Linux

It return the server kernel's version: 2.4.21-32.ELsmp, but not something my mother could understand.
To get a layman term, we can query file /etc/redhat-release instead.

$ cat /etc/redhat-release
Red Hat Enterprise Linux AS release 3 (Taroon Update 5)

POSTED BY WIJAYA KUSUMO AT 2/06/2006 11:43:00 AM

Subversion trials and errors

Late yesterday I tried

$ cd htdocs
$ svn import -m "New import" file:///usr/local/svn/newrepos

which then started processing (what exactly was done?) many large files in the htdocs tree which, on reflection, I don't want in the repository (archived Apache access logs(!), movies which are never revised, etc.). The import then choked on a file for which permissions were wrong.

Two questions now:

1. How to undo all the processing that was done? (Maybe nothing was done since it choked?)
   
2. How to do it better next time?
   

I don't think that the huge files were actually copied to another location on the filesystem, since this shows that the space is unchanged (if they'd all really been copied, Used should have almost doubled):

$ df -h /
Filesystem            Size  Used Avail Use% Mounted on
/dev/sda3             131G   27G   98G  22% /

And it looks like the best way to undo is to use a system (not svn) rmdir. stspiffy says,

you can svn delete directories, but if you want to get rid of the entire repo forever-ever, ... the correct way to delete a repository is actually to delete the directory.

As far as How to do it better next time,

1. I think www, not htdocs, should be the 'root' of the project (so that libraries and other (admin, etc) servers are included in the same repository)
   [ah but maybe it's better to have separate repositories? probably not, if it makes updates and commits harder. (But it doesn't, right, since these commands don't require the repository url?) So actually, at an extreme, I could have separate repositories (with granular revision numbers) for each directory in htdocs!] Is it reasonable to have one repository for libraries (which hardly ever change) and another for htdocs (which change 1-4 times/year) and another for the clearinghouseAdmin server, etc? 
   
2. I could move archived access logs out of this tree. But what to do with other bulky files?
   
3. Probably the best thing to do is vi a folder-by-folder svn import script (vi, :r !ls, :%s/^.*$/svn import -m "New Import" & file:///usr/local/svn/newrepos, zap unwanted lines from the file and then run as a script)
   

Subversion notes

We have an existing project (web site).
We want to start using Subversion (svn) to document (and be able to undo) changes.
We have installed svn version 1.4.4 (r25188).
We need network access to the repository, at least for Daniel. (I may just continue editing on the server.)
Q: Which Subversion server process are we running? We have two choices. Quoting the Subversion FAQ:

either svnserve, which is small standalone program similar to cvs pserver, or Apache httpd-2.0 using a special mod_dav_svn module. svnserve speaks a custom protocol, while mod_dav_svn uses WebDAV as its network protocol.

Integration with Dreamweaver: it looks like DW CS4 will support svn.

Mac Maintenance - hidden desktop files

So I'm trying to get more life out of my PowerBook G4, and I only have 6.33G of 74.41G disk space free (8.5%).

Where's all the stuff?

12:03 PM 3016 > du -k -d 1 | sort -n -r
60046424        .
53814684        ./Music
4304660 ./Desktop
1553812 ./Library
165748  ./Documents
116480  ./Pictures
54708   ./Projects
34856   ./FontExplorer X
940     ./.Trash
72      ./.razorsql
20      ./.subversion
16      ./Movies
12      ./bin
12      ./.transmission
8       ./Shared
8       ./Public
8       ./NIVdlj0508-0623.fpbf
4       ./.ssh
4       ./.config
0       ./TheVolumeSettingsFolder
0       ./Network Trash Folder

The huge amount of storage in Music I'm not surprised by, but what's that 4.3G in the Desktop folder?

12:03 PM 3017 > ls -l Desktop/
total 93536
drwxr-xr-x   11 steve  steve       374 Apr  3 09:57 08-03
-rw-r--r--    1 steve  steve    779925 Mar 28 10:28 4_findings.pdf
-rw-r--r--    1 steve  steve  18585738 Mar 20 11:04 MPlayerOSX_1.0rc1.dmg
-rw-r--r--    1 steve  steve     60164 Mar 21 15:53 Picture 1.png
-rw-r--r--    1 steve  steve     68683 Mar 26 17:55 Picture 2.png
-rw-r--r--    1 steve  steve    206265 Mar 21 13:35 T2 Brief March 2008.pdf
-rw-r--r--    1 steve  steve   3469366 Mar 21 13:49 access31.pdf
-rw-r--r--    1 steve  steve  24706623 Mar 20 11:06 vlc-0.8.6e.dmg

OK, so I have 40M of disk image files. That still leaves 4G unaccounted for!

12:14 PM 3019 > ls -al Desktop/
total 93672
drwx------   22 steve  steve       748 Apr  2 08:15 .
drwxr-xr-x    3 steve  steve       102 Jul  3  2005 .$$ StuffIt Temp 1120423723
drwxr-xr-x    3 steve  steve       102 Nov 16  2006 .$$ StuffIt Temp 1163745308
drwxr-xr-x    3 steve  steve       102 Dec 27 19:14 .$$ StuffIt Temp 1198811673
drwxr-xr-x    3 steve  steve       102 Dec 27 19:19 .$$ StuffIt Temp 1198811989
drwxr-xr-x    3 steve  steve       102 Dec 28 19:56 .$$ StuffIt Temp 1198900615
drwxr-xr-x    3 steve  steve       102 Dec 28 19:57 .$$ StuffIt Temp 1198900637
drwxr-xr-x    3 steve  steve       102 Dec 28 19:57 .$$ StuffIt Temp 1198900669
drwxr-xr-x    3 steve  steve       102 Dec 28 19:58 .$$ StuffIt Temp 1198900722
drwxr-xr-x   41 steve  steve      1394 Apr  3 11:30 ..
drwxr-xr-x    3 steve  steve       102 Jun 26  2005 .BAHtemp-1363-1804289383
drwxr-xr-x    3 steve  steve       102 Apr 28  2005 .BAHtemp-366-1804289383
-rw-------    1 steve  steve     67588 Apr  3 11:44 .DS_Store
-rw-r--r--    1 steve  steve         0 Mar 17  2005 .localized
drwxr-xr-x   11 steve  steve       374 Apr  3 09:57 08-03
-rw-r--r--    1 steve  steve    779925 Mar 28 10:28 4_findings.pdf
-rw-r--r--    1 steve  steve  18585738 Mar 20 11:04 MPlayerOSX_1.0rc1.dmg
-rw-r--r--    1 steve  steve     60164 Mar 21 15:53 Picture 1.png
-rw-r--r--    1 steve  steve     68683 Mar 26 17:55 Picture 2.png
-rw-r--r--    1 steve  steve    206265 Mar 21 13:35 T2 Brief March 2008.pdf
-rw-r--r--    1 steve  steve   3469366 Mar 21 13:49 access31.pdf
-rw-r--r--    1 steve  steve  24706623 Mar 20 11:06 vlc-0.8.6e.dmg

Errrk! What are these hidden StuffIt Temp directories??

12:19 PM 3022 > du -k -d 1 Desktop/ | sort -nr
4304660 Desktop/
3985408 Desktop//.$$ StuffIt Temp 1198811989
113664  Desktop//.$$ StuffIt Temp 1198811673
83616   Desktop//.BAHtemp-1363-1804289383
57680   Desktop//.BAHtemp-366-1804289383
4236    Desktop//.$$ StuffIt Temp 1198900722
4236    Desktop//.$$ StuffIt Temp 1198900669
4236    Desktop//.$$ StuffIt Temp 1198900637
2448    Desktop//08-03
1536    Desktop//.$$ StuffIt Temp 1163745308
592     Desktop//.$$ StuffIt Temp 1198900615
172     Desktop//.$$ StuffIt Temp 1120423723

12:21 PM 3024 > du -k -a Desktop/.\$\$\ StuffIt\ Temp\ 1198811989/
3985408 Desktop/.$$ StuffIt Temp 1198811989//mencoder
3985408 Desktop/.$$ StuffIt Temp 1198811989/

12:21 PM 3025 > ls -al !$*
ls -al Desktop/.\$\$\ StuffIt\ Temp\ 1198811989/*
-rw-r--r--   1 steve  steve  4081057792 Dec 27 22:04 Desktop/.$$ StuffIt Temp 1198811989/mencoder

Wow! Removing those bumps my free space up to 10.39G! (14%) Nice.

Web Server Stats: What to do with hax0r traffic

In my ongoing quest to produce a measure of real human use of the site, I now come to malicious traffic.

First thing: how to identify malicious traffic?

In an earlier post I describe creating Fake hacker targets like /_vti_bin/owssvr.dll (on our LAMP server). I could grep access logs for known malicious URLs like that. Problem: high maintenance. The bad guys are always discovering and trying new vulnerabilities, and I would have to somehow keep up.

I could look for large numbers of 404s per host, since probes for vulnerabilities will fail. (Well, they would if I removed the fake targets.)

My Hard Drive

Maintenance:

Here's advice from
>Mac DevCenter, 4 years ago:

Once a month:

1. Repair the privileges.


2. Repair the drive.


3. Force the periodic maintenance.

After an application crash, a power failure or a force reboot

Immediately:

1. Repair the drive first.


2. Repair the privileges.

Then, if the computer slows down:

1. Force the periodic maintenance.


2. Update the prebinding.

Around the installation of a big new application or a system upgrade

Before, do this:

1. Repair the privileges.


2. Repair the disk.

After, do this:

If this is more than a simple OS incremental update, repair the privileges
and the file system again with the new, updated Disk Utility.

7/01/2007

System Profiler:
Volumes:
Macintosh HD:
Capacity: 74.41 GB
Available: 9.38 GB
File System: Journaled HFS+

NAV scan:
452714 files
123872 files in archives

Ran sw update (81M download)

Macintosh HD:
Capacity: 74.41 GB
Available: 9.42 GB
File System: Journaled HFS+

Huh! 40 M more available?


http://safari.peachpit.com/
Real World Mac Maintenance and Backups


11/01/2007

Disk Utility information:

Capacity : 74.4 GB (79,892,106,240 Bytes)
Free Space : 12.9 GB (13,877,231,616 Bytes)
Used : 61.5 GB (66,014,871,552 Bytes)
Number of Files : 347,112
Number of Folders : 85,270