It’s relatively easy to roll own your own URL shortening service once you know the magic behind it. I’m going to explain the basic idea so you can start rolling out your own.

UPDATE: check out django-shorturls, a Django app recently created by Simon Willison and Jacob Kaplan-Moss, it uses a similar base 62 approach.

How it works

The basic idea of how these services work is by storing URLs in a database, each with its own unique ID, which is normally an integer. The ID is then magically converted into a series of characters which can be used to make up a complete URL that people can pass around to people.

When someone opens the shortened URL with the code, the server will convert the code back into the unique ID so that it can retrieve the URL in the database. Once the URL is retrieved then the server will send a HTTP redirect response to the browser.

The magic

It’s not magic, really.

The conversion function from an integer to the short code must produce as short code as possible while still being practical: the characters in the code must be valid for URLs, no “funny” characters (“?”, “/”, “&”, “%”, “+” are probably a bad idea).

One of the easiest functions to use for this purpose is to encode the integer (decimal, base 10 number) into base 36.

Normally we use base 10 to count. In base 10 numbers we only have 10 possibilities (0 to 9) for each digit before we need add another digit in front of it and roll over to the starting symbol (“10″ is two digits, because we have no more possibile symbol to use after “9″).

In base 16 (hexadecimal) we add 6 more symbols (“ABCDEF”) so that we can count to “F” (or 15) before we need to add a digit in front and roll over to the starting symbol (“10″, or 16 in hexadecimal). If you’re having trouble following, imagine having 16 fingers and 16 toes to count with.

Same idea for base 36, but we add the whole 26 Roman alphabets, “A” to “Z”. So “Z” is 35 in base 36, and “10″ is 36.

How short is short?

With a 3-digit base 10 numbers, we can only have 1000 possible combinations (0 to 999). If I remember my high school math correctly, the formula is N^D where N is the number of possibilities for each digit (or the base number, 10 in this case), and D is the number of digits (3 in this case), so 10³ = 1000.

The goal is to have as many combinations as possible with as little number of digits as possible. Since the number of digits will grow over time (as we store more URLs) then the only way we can increase the number of possible combinations is by increasing the base number.

That’s why converting from base 10 to base 36 works.

So why stop at 36, you ask. As a matter of fact, we can make use of both uppercase and lowercase letters of the alphabets so that we can use base 62. If you’re willing to have non-alphanumeric characters in your short codes, you can even push it further with a few more characters. In fact, this is what essentially base 64 encoding does. Be careful though, the commonly found base 64 encoding normally works on strings instead of numbers, converting 6 bits of data at a time, and it can add paddings, the output is 33.3% longer than the input. Quite different from what we want.

I would probably use base 62 because it’s slightly more pragmatic and it has less chance of screwing up, but if you insist on using base 64 (for numbers, not strings), then I suggest you use “-” and “=” as the additional symbols. I’d avoid underscores (“_”) because they can be hard to see in links, which are normally shown underlined.

The following table shows the number of possibilities for some of the bases:

Say, you add 1 million new URLs every month on average. With base 62, it would take a little over a year to use up all the possible 4 digits, and after that it would take another 75 years to use up all the possible 5 digits, and after that it would take another 4,658 years to use up all the possible 6 digits, and so on.

It would probably take even longer to use up the digits if you don’t add URLs that are already in the database. Conversely, you might use up the digits faster if you want to do per-user tracking, for example.

Sample code

Here’s a simple Python code that you can use to convert an integer to a base N “number” (actually string), and vice versa.

# Code adapted from: http://en.wikipedia.org/wiki/Base_36#Python_Conversion_Code
# This is base 62:
ALPHABETS = "0123456789ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz"

def base_n_decode(s, alphabets=ALPHABETS):
    n = len(alphabets)
    rv = pos = 0
    charlist = list(s)
    charlist.reverse()
    for char in charlist:
        rv += alphabets.find(char) * n**pos
        pos += 1
    return rv

def base_n_encode(num, alphabets=ALPHABETS):
    n = len(alphabets)
    rv = ""
    while num != 0:
        rv = alphabets[num % n] + rv
        num /= n
    return rv

Can’t we just use MD5?

No. We can’t use hash functions like MD5, because hash functions are one-way, irreversible, meaning we can’t get the original input just from knowing the hash code. Also, hash functions can have collisions, meaning from one hash code there is a remote possibility that it “maps” back to two or more URLs. Plus, the output of hash functions are normally long-ish fixed-length strings.

So, what are you waiting for?

Now that the magic is revealed, you can create your own URL shortening service. Be creative and innovative so you can compete with all the other URL shortening services out there.

As a bonus, you can also use this technique to shorten the permalinks or absolute URLs for your objects in your Django (or Rails, etc.) websites. So instead of http://example.com/post/12345 it would be http://example.com/post/3D7. Note that this has nothing to do with security whatsoever. The coded ID can still be reversed with some effort. And you shouldn’t rely on obfuscation to protect anything anyway.

Compiling psycopg2 using MacPorts as well as from the tarball would result in something like this when loaded from a web application that runs on Apache:

ImproperlyConfigured: Error loading psycopg2 module: dlopen(/Library/Python/2.5/site-packages/psycopg2/_psycopg.so, 2): no suitable image found.  Did find: /Library/Python/2.5/site-packages/psycopg2/_psycopg.so: no matching architecture in universal wrapper

So all I needed to do was force psycopg2 extension to be built for 64-bit as well as 32-bit. After extracting the latest psycopg2 source, do this instead:

LDFLAGS="-arch ppc -arch i386 -arch x86_64" CFLAGS="-arch ppc -arch i386 -arch x86_64" python setup.py build

Then sudo python setup.py install. After that, check the .so file, it should look something like this:

$ file /Library/Python/2.5/site-packages/psycopg2/_psycopg.so 
/Library/Python/2.5/site-packages/psycopg2/_psycopg.so: Mach-O universal binary with 3 architectures
/Library/Python/2.5/site-packages/psycopg2/_psycopg.so (for architecture i386): Mach-O bundle i386
/Library/Python/2.5/site-packages/psycopg2/_psycopg.so (for architecture ppc7400):      Mach-O bundle ppc
/Library/Python/2.5/site-packages/psycopg2/_psycopg.so (for architecture x86_64):       Mach-O 64-bit bundle x86_64

Don’t forget to restart Apache, and that should be it.

Update: It happened to PIL too, it was giving this error: The _imaging C module is not installed, so I had to do the same thing as above: LDFLAGS=... CFLAGS=... python setup.py build followed by sudo python setup.py install.

Joe Wreschnig, Quod Libet’s author, suggested that this could be implemented as a plugin. After an email conversation with him, and seeing a patch posted to the bug in launchpad, I started writing the plugin last night. Now the multimedia keys work again in Quod Libet. The plugin can be downloaded from launchpad or from here (rename to dbusmmkey.py). Simply put this file in ~/.quodlibet/plugins/events/ and enable it in Quod Libet.

PS. Python is so much fun!

The clickable smileys was not part of the original plugin, so I modified the plugin to provide that functionality. I notified Priyadi about my modifications, he quickly incorporated my changes and adjusted them to his code. He released version 4.0 of the plugin right away. :D

You can get the plugin from Priyadi’s im_smiley plugin, or from my local copy.

• muttrc
• keybindings
• colors
• gpg
• hooks
• lists
• mailcap
• smime
• vimrc

I put everything in $HOME/.mutt/ except the .muttrc file. You also need your own $HOME/.mutt/aliases containing your personal address book.

I use the same .vimrc when I’m in Linux, Windows, Solaris and Mac OS X. The file is mostly commented so you can change or customize it to your own liking. Some parts are taken from various sources but maybe I forgot to write credits in the comments, if you spot any please inform me.

Add Indonesian Wikipedia Search Plugin. Based on the original English Wikipedia plugin from mycroft.

Note: see my previous post if you’re using Firefox on a non-Windows platform.