ssh tunnel

I always have to look this up:

ssh  steve@www.techtransfer.berkeley.edu -L 8888/localhost/8800

The parts:

ssh

the secure shell command

user@host

your account on the host to which you want to tunnel

-L

ssh forwarding (tunneling) flag

8888

local port (could be 8800 in this case, just needs to be unused on your machine)

localhost

your machine

8800

port on remote host

then in (for example) your browser, you can enter the address

127.0.0.1:8888

and get the page at

www.techtransfer.berkeley.edu:8800

robots.txt

Added a line:

Disallow: /freestuff/ # use /g3/ instead

editing robots.txt

# see http://www.robotstxt.org/

Today I looked through a list of our top-level directories and added lines to /robots.txt.

symbolic links

If we have several names for the same directory (e.g., /ped-safety/ and /pedsafety/) I added a Disallow: directive for the symbolic link.

Disallow: /ped-safety/  # use /pedsafety/ instead

This way, Google et al. will cache things with the preferred URL. Especially desirable for holdovers from an era when our site was organized completely differently:

Disallow: /resources/media/multimedia/  # use /videos/ instead

"honeypots"

These are common vulnerabilities that are randomly requested by hackers and spambots. I added zero-length empty files for the most-requested to eliminate 404s for common vulnerabilities that we don't have, so that our 404 reports would show actual broken links.

Disallow: /_vti_bin/    # honeypot

private directories

set up for various special projects; the URLs are not published. I had a couple Disallow: directives for these, but that's essentially publishing the secret URL, so I removed those directives.

analog / report magic on www

I want to:

• stop using analog in my home directory (use analog in /usr/local /usr/bin)
  
• move my webstats script to /user/web
  
• specify dnscache location, etc., on the command line in the script (using variable names) rather than in analog.cfg (that way if I upgrade from dnstran 1.5.2 to 1.5.3 I only need to rename the path once: in the variable assignment in the script.)
• use dnstran in /usr/local

also maybe

• archive apache access logs in /var/log instead of /export/www/htdocs
  

then the webstats script should be run by user web. today it failed because user steve can no longer create analog reports in /export/www/htdocs/ (and maybe never should have been able to)

how to tell the version of Redhat Linux installed on a machine from a command line

From Wijaya's Blog

Here is a sample:

$ uname -a
Linux myserver.mydomain.com 2.4.21-32.ELsmp #1 SMP Fri Apr 15 21:17:59 EDT 2005 i686 i686 i386 GNU/Linux

It return the server kernel's version: 2.4.21-32.ELsmp, but not something my mother could understand.
To get a layman term, we can query file /etc/redhat-release instead.

$ cat /etc/redhat-release
Red Hat Enterprise Linux AS release 3 (Taroon Update 5)

POSTED BY WIJAYA KUSUMO AT 2/06/2006 11:43:00 AM

Subversion notes

We have an existing project (web site).
We want to start using Subversion (svn) to document (and be able to undo) changes.
We have installed svn version 1.4.4 (r25188).
We need network access to the repository, at least for Daniel. (I may just continue editing on the server.)
Q: Which Subversion server process are we running? We have two choices. Quoting the Subversion FAQ:

either svnserve, which is small standalone program similar to cvs pserver, or Apache httpd-2.0 using a special mod_dav_svn module. svnserve speaks a custom protocol, while mod_dav_svn uses WebDAV as its network protocol.

Integration with Dreamweaver: it looks like DW CS4 will support svn.

Web Survey Tools

It might be a good idea to have polls (quick one-question surveys) on our site.

How to find tools?

a look at SourceForge finds:

(search (+poll +php +mysql))

PHPOLL php - mysql poll system

looks active. email confirmation of votes; max one vote per e-mail

(search: php survey)

nabopoll (http://www.nabocorp.com/nabopoll/)

looks orphaned.

Unit Command Climate Assessment and Survey System (UCCASS) (pronounced yoo-kas)

nothing new since 2005.

(search (+survey +php +mysql))

Multi-Platform Survey Architect

nothing new since 2004

sed and grep: "Stories" of web visitors

I've been thinking for a long time about crunching the web logs to produce "stories" showing how visitors navigate our site. We don't get so many visitors that it would be overwhelming to look at a day or two.

So I've started writing shell scripts to do the crunching.

They are located at /home/steve/webstats/trunk

The plan is, starting from a raw log file:
1. find just requests for pages (.php, .html, .pdf) with result code 200.

(Hmmm...
right now I am using grep -v to get rid of certain things that aren't (1.) above.
is it simpler to use something like:

grep '] "GET [^ ]*[php|pdf|html][^ ]* HTTP/1.." 200 '

or

sed '/] "GET [^ ]*[php|pdf|html][^ ]* HTTP\/1.." 200 /!d'

to get "successful requests for pages?"

sed doesn't do aaa OR bbb OR ccc very well; it looks like

sed -e '/AAA/b' -e '/BBB/b' -e '/CCC/b' -e d    # most seds
gsed '/AAA\|BBB\|CCC/!d'                        # GNU sed only

sed is very fast, though.

This works at about 10,000 raw log lines/sec:

sed '/" 200 /!d' $logfilename | \
sed   -e '/"GET [^ ]*\.php/b' \
     -e '/"GET [^ ]*\.pdf/b' \
     -e '/"GET [^ ]*\.ppt/b' \
     -e '/"GET [^ ]*\.avi/b' \
     -e '/"GET [^ ]*\.doc/b' \
     -e '/"GET [^ ]*\.mpg/b' \
     -e '/"GET [^ ]*\.flv/b' \
     -e '/"GET [^ ]*\.html/b' \
     -e '/"GET [^ ]*\/ HTTP/b' \
     -e d    > $temporary_story_file

More importantly, if I specify good files, then I have to modify the script every time we add a new file type. And if I forget, it's not easy to notice the absence of the overlooked file type.

grep, grep, grep takes a while to process one log file, about 15 minutes so far...

Tue May 20 20:44:19 PDT 2008

Lines in apache access log:
636818 /export/www/htdocs/weblogs/2008/Q2/access_log_200804

Dropping requests for images:

Tue May 20 20:54:32 PDT 2008
Lines left in temporary work file:
214027 /tmp/stories_14566

Dropping requests for javascripts:

Tue May 20 20:55:01 PDT 2008
Lines left in temporary work file:
203919 /tmp/stories_2_14566

Dropping requests for css:

Tue May 20 20:55:39 PDT 2008
Lines left in temporary work file:
192682 /tmp/stories_14566

Dropping partial or unsuccessful requests:

Tue May 20 20:59:28 PDT 2008
Lines left in temporary work file:
140429 /tmp/stories_2_14566

anyway, next is:
(1.5. weed out search / h@x0r bots)
2. sort by IP address and time
3. clean up the lines to make a tab-delimited table with these fields:
IP address datestamp URL requested referring URL Browser type

Of course you can buy tools that claim to do this...

Tuesday 12 February

a.m.:

• Read about IP filtering with netfilter and iptables
• Read Blocking international network traffic to your server
• Read Traffic Safety Center headlines
• Edited 4 or 5 records in clearinghouse that had unqualified URLs

p.m.:

• Read about DNS and DNS appliances
• Read about specifying laptops
• 'tail' errors.log on www and fix broken links etc.

Google Public Service Search is dead -- Long Live Google Custom Search Engine!

The nav code that I uploaded ages ago to Google Public Service Search is broken. Although Public Service Search will still work, you can't make any changes; Google has discontinued support for administering Public Service Search, directing us to Google Custom Search Engine (CSE) instead.

These CSEs are cool; I've bookmarked several having to do with Transportation Agencies:

• Public Transit Agencies Search Engine
• State DOT Search Engine
• UTC Search Engine
• MPO Search Engine
• LTAP Center Search Engine

Apparently, free embedded search on a site is also managed through CSE.

I used our department account (username systems@techtransfer.berkeley.edu, password the usual) to configure a CSE.

Then on our web server I edited the search code for our pages with the new code (new values for hidden inputs in the search form). If you have your search form laid out in tables, as we do, you need to peer closely at Google's new code and make the necessary modifications by hand, rather than just cut'n'paste.

A big change is that the search results are in an iframe within a page hosted on our server.

I used the CSE control panel to change the appearance of the results, though I've had no luck making them narrower to fit inside our nested tables.

The page for our CSE can be found by logging in at:

http://www.google.com/coop/manage/cse

Draft Subversion email

Decisions:

o Keep the repository (repo) on www or on another server?

++ I vote for "on www." What Randall recommended: make the live htdocs/* folders a checked-out instance of the repo, so that we can make and check-in tiny changes right in those folders

o Repo storage:
Berkeley DB-based
FSFS-based

++ I vote "FSFS." (This is now the default.) Berkeley DB is a legacy holdover, and has corruption vulnerabilities.

o Remote access:
Since we have already set up ssh accounts, it's simplest/safest to use svn+ssh: the svn client opens a ssh session, which then spawns an svn process *as the ssh user* (This means that 1. repository must be owned by a group the user is a member of, 2. the repository must be group-writable, 3. the user's PATH must include the path_to_svn_libraries (/usr/bin/ on www).) I don't really understand this: there may also be umask issues where svnadmin? commands run by a user with the wrong umask set may not be group-writable.
via svnserve daemon (problem: passwords are stored as cleartext.)
via ssh tunnelling
via Apache/webdav
* regular system users using a Subversion client (as themselves) to access the repository directly via file:// URLs;
* regular system users connecting to SSH-spawned private svnserve processes (running as themselves) which access the repository;
* an svnserve process - either a daemon or one launched by inetd - running as a particular fixed user;
* an Apache httpd process, running as a particular fixed user.

++ I vote Apache/webdav: (right? I think that Windows users can just open a webdav resource in a Windows Explorer window.)


o what user/group will own the repository? web:web? do we want to create user:group svn:svn?

What data do you expect to live in your repository (or repositories), and how will that data be organized?
text files, binaries (pdf, jpeg, gif, wmv)
/export/www/* (one project root or two?)
o single repository for multiple projects, or to give each project its own repository?
/export/www/
clearinghouseAdmin/
htdocs/
htdocs-cp2info/
mediaLibrary/
phplib/
railsDevelopment/
ttplib/

? ? Maybe:
repository : Path
www : /export/www/htdocs (pages on www)
www-lib : /export/www/ (ttplib, phplib, admin tools etc)
++ actually, I vote for all in one: they are all related, and we should be able to to ask about (or modify, or migrate elsewhere) the entire history of a single project

Where will your repository live
www server
and how will it be accessed?
directly (command line), network server (WebDav?)
repository browsing interfaces

e-mail commit notification

data backup strategy
backed up with www backups? Right?
What types of access control and repository event reporting do you need?

Which of the available types of data store do you want to use?
FSFS