Podcasts

Google popped up with a link to some high-bandwidth Mars photos today, ensuring a sluggish day for internet users everywhere. Of course, this is a mere drop in the bandwidth bucket compared to NASA's self-aggrandizing NASAcast. Those arrogant [expletives]!

For a seemingly positive viewpoint on this latest fiscal atrocity see: Mars Orbiter Makes Port. This article is about podcasting.

Basically, podcasts are audio/video feeds using essentially the same technology as regular feeds, but with huge files. Thus illiterates can hog bandwidth, too. I don't have time for this now, but a cursory examination reveals a whole community of DJ wannabes streaming crap nobody listens to out to gee-whiz technolgy buffs for them to delete from their iPods. Just for good measure, we learned that MickeySoft has their own non-standard

On a related note RSS Specifications has some interesting thoughts on the RSS/atom dichotomy.

If you really must delve into this evil technology, Podcating News seems like a good place to start. Juice seems to be the most popular podcast client, although more clients are available. Of course you can check the article that "launched" this tirade -- NASA Podcast Help

Pod casting isn't going away just because I'm convinced it's a stupid waste of bandwidth. It will probably thrive. Maybe there is a way to use it for good instead of evil. I need some coffee.

RSS & atom

As I mentioned in a previous post, spiders (e.g. googlebot) crawl blogs differently than "regular" webpages. I didn't go into much detail, because I didn't actually know much about the process. Later I began fumbling around with Google Sitemaps which resulted in temporarily being blacklisted as a splog. (Incidentally, Blogspot corrected the problem promptly, much to my surprise.)

After using some online sitemap generators, which weren't entirely satisfactory, I stumbled onto Johannes Mueller's excellent GSiteCrawler. Whenever that program uploads a new sitemap, it pings Google Sitemaps, alerting them that new content is available. Aha! This two-way communication is the essence of web feed technology.

Suppose you have your own spider, dutifully traversing the web day in and day out. Obviously, you could cut down on the bandwidth it would require if it didn't have to crawl every page, but just the pages which had changed. Googlebot is perfectly capable of traversing a site, but employs its own algorithm to decide whether or not to re-index a particular page, thus driving webmasters insane.

Sitemaps don't really change this, but properly used, they give googlebot a "heads up" as to what contnet is new. New content is preferred over old content by some secret amount, so frequently updating sitemaps is the key to their successful use. If you don't have enough room for sitemaps, similar benefits can be had simply by including (properly maintained) "date" metatags for each page.

RSS and atom (remember them?) are two competing file specifications for this type of two-way communication, commonly referred to as web syndication. Presently the "standards" are under nearly constant revision, similar to the "browser wars" of a few years ago that made it practically impossible to use javascript. This site's web-feed is: http://wholeed.blogspot.com/atom.xml

Blogs Blogs Blogs

Googlebot can't seem to figure out what blog is about, so it's loading the pages up with "How to do a blog" AdSense ads. Swell.

Very well then, herewith is a short list of free blogspace providers:

• Bloglines
• MySpace
• Blog City
• Blog Easy
• Weblogs.us (WordPress)
• Skaffe
• Xanga
• we::blog
• Pitas.com
• News Is Free
• Live Journal
• Free Open Diary
• Easy Journal
• Diary Land
• Blogo Monster
• Blog Thing (Word Press 1.5)
• Smeego

The Robots Exclusion Protocol

Well-behaved 'bots comply with Robots Exclusion Protocol, which allows webmasters to specify pages or whole directories (Windows users read folders) that should not be crawled or indexed.

Most webmasters are moving heaven and earth to be included in various search-engines, so the default behavior -- "index, follow" -- is correct. Search-engines, on the other hand usually have very restrictive robots.txt files.

In part, this is to reduce the incredible traffic that would be generated by all the myriad search-engines crawling each other's sites. Another, probably more compelling reason is that a search-engine's database is its stock in trade, and most are very proprietary about sharing them with others

A sampling of robots.txt files from several prominent sites is instructive for those wishing to "tweak" their own robots.txt files:

• MSN Search
• Google
• Alexa
• A9
• ODP
• Altavista
• Free Find
• AOL Search
• Scrub The Web
• Blogger
• Bloglines (Ask.com)

In the course of compiling this list we learned that Teoma was recently acquired by Ask (Formerly "Ask Jeeves"). Now if somebody would only buy Yahoo! ...