My WWWOFFLE tools. They at least work with WWWOFFLE Version 2.8.
Needn't be root to use:
Chop a WWWOFFLE index into smaller chunks -- else no fun each time I hit the BACK key. Also if one doesn't read their e.g. Last Time indexes before they get pushed off the back of the accessible list, this will keep it around until you get around to reading it.
Shows the changing state of the lasttime directory. To see only what WWWOFFLE is currently busy fetching, try wwwoffle-swat.
Helps find out why some files seem to get refetched unnecessarily.
Order the monitor directory by times.
Like wwwoffle-ls but with stat(1) output.
Must be root to use:
Remove WWWOFFLE outgoing requests via regular expression -- as often one wishes there was a quick way to get rid of a mistaken request.
Try to move outgoing URLs into higher priority to be fetched.
Remove things via regular expression from the WWWOFFLE cache -- an easy way to precision target something to blast from the cache.
Find the URLs that messed up last time and are candidates for perhaps requesting again.
Lists actively downloading files and asks if you want to stop them. During a WWWOFFLE fetch run, if one notices one is fetching some giant unwanted file, eating up precious bandwidth, this will stop it.
Debian etc. users should export
wwwoffle_cache=/var/cache/wwwoffle in
.bash_profile.
Which of these are more debugged? The ones with higher
Update Counts.
In wwwoffle.conf I can make the status of links show up even on a monochrome PDA with:
enable-modify-html=yes anchor-requested-end=$R anchor-cached-end=$
In the script I use to call up my ISP, I have the following real life saver, as currently WWWOFFLE offers no built in protection for when your carefully typed in POST encounters a down server, etc. One merely has to move the failed O* and U* pair of files back into outgoing, and hope the server will be up next time you call. Repeat the process until one day they are back up.
set -eu #WWWOFFLE failed POSTs recourse insurance:
d=~jidanni/tmp/wwwoffle-outgoing-backup
test -d $d
find $d -type f -mtime +22 -exec mv {} /var/tmp \; #like rm
cd ${wwwoffle_cache-/var/spool/wwwoffle}/outgoing
set -- `wwwoffle-ls outgoing|
awk '/POST/{m=$1;sub(/^O/,"U",$1);print m, $1}'`
test "$*"
cp -p $@ $d
We can also dig text out of a failed www-form-urlencoded POST with e.g.,
$ perl -MURI::Escape -pwe 'tr/+/ /; $_=uri_unescape($_); tr/\r//d'
Last modified: 2009-01-20 23:00:56 +0800