isnoop.net blog

If you have a web tool that takes any sort text input for a search, you might find value in intelligently offering alternate spellings.

This technique is best used with an indexed list of proper words where it may be common to misspell based on phonetics. It is not meant to be applied to serial numbers.

First, create a new column in your index table for your alternate spelling key. Then, create a small batch script that fetches every entry from your index and runs the PHP metaphone() function on it. Store the resultant value in the new column you created. Also, make sure you update the code that adds new data to the table to populate your new column.

Now that you have your alternate spelling column populated, simply query the database with a metaphone() of the search term when a search turns up no results. You now have very rudimentary “did you mean” functionality, but it doesn’t stop there.

PHP’s metaphone() is an okay means of getting words similar to one another. In short, it boils the word down to a very basic pronunciation form where “buyh” and “bowwow” are alike. As you can see, metaphone can be far too general. What we need to do next is pick only the most relevant results from the similar words query.

The next step in finding more specific data is to find the words most similar to the one the user originally provided. I use the PHP levenshtein() function which calculates the difference between two strings. Simply ranking the results of the metaphone query by their levenshtein difference will provide you with a reasonable means of emulating the spelling correction/suggestion code used by popular search engines and spell checkers. This final step whittles words similar to “buyh” down to a manageable few, several of which may indeed be what you really meant.

For a demonstration of this type of code in action, check out my Compact Online Dictionary which I built for testing such ideas. The entire application spans only 100 lines of code and includes user-based result relevance ranking.

MacSaber 1.1 has been released, now with expanded 17″ PowerBook support and keyboard backlight effects. I invite you to take your very expensive laptop into a dark room and swing it around* for improved dramatic effect.

Download MacSaber 1.1.

*Don’t break your laptop, okay? If you do, don’t blame me.

UPDATE: Version 1.1 is now available. It includes keyboard backlight effects.

MacSaber 1.0 Final has been released. This new version includes support for iBooks and PowerBooks equipped with SMS. It also includes a “Check for updates” menu item and debug output if you are having problems (the report button is not yet available).

Compatibility notes:
iBook: SMS only exists on iBooks built after July 26, 2005. These iBooks are either 1.33GHz or 1.42GHz. All 1.42GHz iBooks have SMS. Only 1.33 GHz iBooks with a scrolling trackpad, 512MB RAM (stock), and 40GB HDD (stock) have SMS.

PowerBook: SMS only exists on PowerBooks built after January 31, 2005. These PowerBooks are either 1.5GHz or 1.67GHz. All 1.67GHz PowerBooks have SMS. All 1.5GHz PowerBooks without backlit keyboard have SMS. 1.5GHz PowerBooks with both backlit keyboard and the 8X superdrive have SMS.

Download MacSaber 1.0 here

Thank you to those who have provided their feedback. I have released MacSaber 1.0 with support for iBook and PowerBook machines. Please read here for more information.

I sure wish I had a PowerBook and an iBook right now. In lieu of that, I am calling on you to test for me.

Thank you for your help!

UPDATE: Version 1.1 is now available.

Now that you’ve spent entirely too much money on your fancy sudden motion sensor equipped Mac laptop, I predict you’ll soon be swinging it around like a loon.

Introducing MacSaber. Using your Mac’s sudden motion sensor, this software turns your computer into a Jedi weapon almost worthy of taking on the real thing by making authentic lightsaber sound effects. It senses speed for the lightsaber movement sounds and acceleration for different levels of striking sounds.

This beta is not yet compatible with the motion sensors in older Mac laptops, but I understand you can move the window arond to get the idea.

Update: Thanks go to Sak from Lot23 for the fancy icon!

Download the latest version of MacSaber

A co-worker and I attended PHP|Tek in Orlando last month. As part of the conference, we were able to test and become Zend Certified Engineers.

I’d like to thank all of the little people that helped me get this far. Thanks go to my wife for believing in me when nobody else did. Much love to my family; without them I wouldn’t be where I am today.

One love.

After last month’s Google AdWords Policy Reminder, I’ve decided to make it perfectly clear that you are by no means authorized to click my ads. You should avoid them at all costs, even to your own peril.

In case this isn’t clear, I have repurposed my magnetic words script to serve as entertainment on a new site dedicated to reminding you that you should not click my ads. Check it out:

http://dontclickmyads.com

For those with concerns about limited bandwidth and images that others might frequently link to, you will benefit from my anti-leech script generator.

This easy to use page will quickly generate a mod_rewrite script that you store in an .htaccess file in the root of your website or the directory you wish to protect. I’ve added images.google.com to the allowed domains by default so you don’t block innocent searchers.

This guide is a simple step-by-step guide to help you convert your currently live website to a handy-dandy revision-control system called Subversion. It also contains information on how to create a discrete development environment where you can modify your site and test your changes without affecting live traffic.

If you:
…want a place to beta test your code
…want to keep a revision history of your code
…have every modified or deleted a file on your website and wished you hadn’t done that
…realize that CVS sucks cat ass
You should try subversion!

Assumptions:
•You are hosted on a *nix machine*.
•You have SVN installed or can install it.
•You have SSH access to your site.
•Your web host allows for the creation of subdomains (if you want a dev environment).
•Your HTTP document root is not the root of your home directory. For example, when you log in to SSH, you see a list of system directories but one of them contains all of your web content.

Notes:
•The command syntaxes that I provide are those used by FreeBSD. If you’re savvy on another system and you notice a syntax difference, please speak up.
•Throughout this text, I will refer to your web documents directory’s absolute path as docroot, the repository as repos, and your home path (the full path from the drive root to your home directory) as homepath.
•The paths described above do not contain trailing slashes. Do not add a trailing slash to paths unless specifically indicated.
•Always always always back up your data, especially before attempting something like this!
•Subversion official documentation

Preparation:

You backed up, right?

1) First, start by running svn –version to verify you have Subversion installed. If not, you or your sysadmin should install it for you!

2) In your web root (~), create a directory named svn_import.

3) Within svn_import, create the following directories: trunk, branches, and tags. **

4) As a security precaution, add the following line to a file called .htaccess in your document root: RewriteRule (^|/)\.svn - [F]

5a) If you wish to add your entire site to revision control, use the command cp -R -p -v docroot/* ~/svn_import/trunk/ to copy your entire site into the import directory.
Note: The -p tag on the copy command is important, as some files on your site may have nonstandard permissions that you wish to preserve.

5b) If you only wish to add parts of your site, copy only those files and directories into ~/svn_import/trunk/

Creating and importing:

6) Create the repository: svnadmin create ~/.svnrepos

7) Import your site: svn import ~/svn_import file:///homepath/.svnrepos -m “Initial import”
Note: Yes, file:/// is correct. Make sure you’re using the full system path to your home directory for homepath

Creating your development area:
(optional, but highly recommended)

8) Using whatever means you use to manage your web site, create a subdomain for your web site called dev.yourdomain.com.

9) HTTP password protect your new subdomain, so people don’t go poking around your work in progress. After all, you’re creating this dev area for a little privacy while you’re working.

10) Check out the repository into the beta site document root: svn checkout file:///homepath/.svnrepos/trunk dev_docroot
When this process completes, you should see the message At revision 1.

11) Look over your new dev subdomain. All of your pages *should* work on this new site, but if you hard-coded any full system paths, these ought to be changed to relative or dynamic.

Converting your live site to SVN

12) Create a temporary checkout directory: mkdir ~/svn_stage

13) Check out the repository to the temp dir: svn checkout file:///homepath/.svnrepos/trunk ~/svn_stage

14) If you got this far without backing up your site content, you should do so right now and then kick yourself in the ass for not paying attention to important instructions.

15) Move your repository onto your live content: mv ~/svn_stage/* docroot/
This previous step may seem odd, but the SVN repository you just moved onto the live site contains lots of .svn directories that are required to make your site act like a repository. It also helps ensure the shift doesn’t interrupt your site traffic, and if you opted for #5b, it doesn’t harm the files you didn’t want to add into SVN but still want to keep.

16) Test your fancy new revision control-enhanced web site!

Post-Conversion

17) Remove ~/svn_import

18) Remove ~/svn_stage

19) Edit the file ~/.subversion/config and uncomment the global-ignores line. You may wish to add space-delimited files that you want SVN to ignore. If you chose to follow step 5b, you should add the files and directories you didn’t want to import here.

Usage
•Now that your site is under this wonderful new revision control system, you can FTP all of your file changes into the dev_docroot path and test them on your dev subdomain. Whenever you make a change you want to store to SVN, you should commit your changes with the following command:
svn commit dev_docroot
•If you don’t provide during a commit using the -m flag, you will be prompted for notes about the commit. Add a brief description of the changes you made so you can revisit that version later on.
•Once you have checked in your fancy changes, you can check them out in your changes by running the following: svn update docroot
•Both your regular and beta document root are now repositories. This way, you can technically make and commit changes from either location. Whether or not you choose to do that is up to you, but the entire point of a dev area is that you don’t ever program in the live path again. It is my suggestion that you don’t directly modify SVN-controlled files in your live path.
•Now might be a good time to remind you to RTFM. This manual is well written and contains gobs of good information on how to use SVN.

If you’re used to CVS
First, congrats. You’re on the road to saner and happier revision control.
Next, you should read this.
Perhaps the biggest shift that you should know about is that revisions in SVN are commit-based, not file based. It is not accurate to say “file X is version 56.” You should say “file X was modified in version 43.” The key difference here is that any number of files and directories can be changed in any revision.

As an interesting aside to all of this, I like to keep a repository of every binary I download and install (applications and drivers). So I don’t have to keep the entire collection on my laptop or desktop, I have a media server that is also running SVN for Windows. My laptop and desktop use the SVN server to store all of the binaries I’ve downloaded. I can delete all but the most recent versions of the apps so they don’t appear on my other computers, but I’ll always have a backup copy of old versions within SVN.

* This document was written based on FreeBSD commands. If you are running RedHat or some other linux, the command flags may differ slightly. Refer to your help documentation to verify command flag differences.

**Click here for an explanation of the trunk, branches, and tags directories. If you don’t care about branching or tagging, you really don’t need to worry your little head about this.

Nine months after closing down the Gmail Invite Spooler, the page remains one of the most popular landing pages on my site. Over the past several months, this page has averaged around 2,500 unique visitors a day. I’ll explain the arc of this wonderful service, but first I’d like to make one thing very clear:

Sorry, I do not have any Gmail invites. Please don’t ask me for Gmail invites. I am truly sorry that I cannot provide you with any. Please go to Google Mail for more information on how you may get your own account.

You may obtain an account without an invite these days. All you need is a cell phone.

Background
In 2004, Gmail was a very hot commodity. Since April 1st of that year, people were clamoring to get in on the exclusive beta of pre-IPO Google’s hottest new offering. In late summer, by the time I decided to write the spooler, Gmail invites were no longer selling for $100 or more on eBay, but there was a large amount of clutter on the internet with people asking for or offering invites.

The forums and blogs that I visited were littered with chatter about trading invites, but the givers and seekers didn’t seem to be coming together efficiently. It wasn’t uncommon to see multiple posts back to back asking for and offering invites.

My wife may not always like it, but when I see a problem, my mind immediately gets to work on a solution. This was one problem that I knew I could make a simple fix for in a matter of hours.

The First Incarnation
Throughout its lifetime, the basic workings of the page remained the same: People with gmail invites would send them to a specific email address. The spooler would then read those emails and store the invites in a database. Site visitors could come and claim available invites on a first-come, first-served basis. There was no backordering of invite requests. When demand exceeded supply, one had to wait until someone else donated some.

Originally the spooler was made solely for the use of the people on the forums I noticed suffered most from inefficient offers and requests. It was a very simple system that was only workable on a small scale, but I assumed it would only ever see a few hundred hits.

On the first day that I had the spooler open, I received 2,592 Gmail invites. The second day saw 4,574 more coming in. By the end of the second day, I had over 3,000 unclaimed invites.

It didn’t take long for word of the “magical free Gmail site” to leak out to the general internet. Within a few days, demand exceeded supply and I had to implement controls on the page to prevent people from refreshing constantly while waiting for a new invite to come in. I also got the first of many lessons in writing code with scaling in mind as I divorced the mailbox checking from page loading.

Ups and Downs
After a month of running the service, the average inbound invites per day dropped below 1,000 for the first time. It seemed that most of the people who had extra invites on hand had heard about the service and donated all they were willing; Google was not giving out new invites on a regular basis at that time. The inbound invites continued to decline through most of December 2004 until they hit a low around 50.

All this time, demand for invites remained strong. I recorded as many as 100,000 visitors and over a million hits per day. On December 20th, the drought was over as Google started to give Gmail users about five fresh invites each day. The average day saw around 2,500 new invites, but they were still being snapped up as soon as they came in. I implemented more restraints to prevent abuse and further streamlined my code in order to keep my server load at a reasonable level. During this time, my web statistics began to break down because Webalizer couldn’t process all of the data without choking.

Way, Way Up
On February 2nd, 2005, Google decided to open the flood gates. They began giving out around 100 new invites per day to Gmail users. My service experienced demand increases like I’d never seen before. For the first time, I was forced to benchmark my code and decide which methods to use based on how many milliseconds they took.

For only the second time, supply was greater than demand. Anyone wanting a Gmail invite could get one through my service without any delay. Unique visitors increased, but hits dropped way down since users had no need to refresh frequently to see if new invites had arrived.

Way, Way Down
Monday June 6th, 2005 was the day I received an email from Stephanie Hannon, Gmail’s Product Manager. Later that day, I had a conference call with Stephanie and her superior regarding my service. They felt that services like mine had become a threat to the quality of Gmail. Their reasons for making the service invite-only were many:

• Limit new subscribers
• Heighten demand and curiosity
• Limit accessibility of accounts to potential abusers

The last reason was the one that made them care about my site. Spammers and abusers have a higher threshold of entry without the spooler. Despite the fact that I think Google should do more on their part to prevent automated account creation and duplication, they do have more random people gaining access to invites through a service like mine.

In short, Google felt as if too many spammers and abusers were getting invites that they obtained from me and saw this as a threat.

Why I Pulled the Plug
I’ve received a few thousand emails asking for invites, complaining about how “unfair” this is, or asking for source code. In the early days after pulling the plug, I would respond to every request with an individually written response explaining the situation. This generated many replies suggesting I just re-open the system in defiance.

Aside from the fact that I really don’t wish to burn any bridges with Google (heck, maybe they’d forget all this and hire me if I ever applied), I have good technical reasons for not re-opening the spooler: My service relied on people with Gmail accounts constantly inviting the now blocked email address gmail@isnoop.net.

Google is no dummy. They know full well that they must track the email addresses that the invites are sent to. They can (and did) automatically invalidate every invite sent to my site. All 1,240,162 invites I had left over the day I shut the service down instantly became duds. To continue the service, I would have to change the method of catching new invites to one substantially more inconvenient for the donor.

In the end, insistence on keeping the spooler open would have certainly summoned the massive lawyering machine deep within the “don’t be evil” company and I don’t think reasonable person wants that fight.

Fast Forward to Today
The former Gmail Invite Spooler page is now a brief testament to what was once the most popular Gmail invite spooler on the internet.

The bulk of the current 2,500 visitors per day come from non-English speaking blog sites that haven’t yet gotten the message that the page is closed. While the rest of isnoop.net has a 66% US visitor rate, the spooler is only 17% US traffic; it holds the #1 slot by less than one percent.

Almost all of the dozens of emails and stray blog posts requesting invites ask the same thing (in broken English). I saw the need for folks who didn’t speak my native language to get the full story, so I wrote a simple script to help them out. This has helped reduce the confused request flow, but it has also crimped the last of my dwindling AdSense revenue. Oh well. I ran this site before it ever brought me a penny and I’ll continue to do so for as long as I have the energy.

I turn down all requests for the source code for the spooler. If Google doesn’t want me starting fires in their back yard, I’m certainly not going to give away my matches to all of the other neighborhood kids.

I have considered revamping the spooler for use with other invite-only services, but I’ve yet to see one of great enough popularity and of proper nature to justify the effort. I refuse to open up such a thing for a community-based website on the principal that it breaks the “six-degreesâ€? network they’re trying to build up by bringing in random people with no association to the inviter.

Media Coverage
Aside from a number of blogs and forums that mentioned the service, these are the print media references I am aware of:

Book: Google Search & Rescue for Dummies – 2005
Text

Book: Google Hacks – 2005
Text

Popular Science Magazine – June 2005
Close-up
Full page

The Mercury News (San Jose) – May 23, 2005
Online version

PC World – April 13, 2005
Online version

Sydney Morning Herald – April 9, 2005
Close-up
Full page
Online version

Are you seriously still reading this?
This post covers almost all of the points I regularly discuss with folks who have questions about the service. I hope this overly long post has satisfied your curiosity.