It’s relatively easy to roll own your own URL shortening service once you know the magic behind it. I’m going to explain the basic idea so you can start rolling out your own.

UPDATE: check out django-shorturls, a Django app recently created by Simon Willison and Jacob Kaplan-Moss, it uses a similar base 62 approach.

How it works

The basic idea of how these services work is by storing URLs in a database, each with its own unique ID, which is normally an integer. The ID is then magically converted into a series of characters which can be used to make up a complete URL that people can pass around to people.

When someone opens the shortened URL with the code, the server will convert the code back into the unique ID so that it can retrieve the URL in the database. Once the URL is retrieved then the server will send a HTTP redirect response to the browser.

The magic

It’s not magic, really.

The conversion function from an integer to the short code must produce as short code as possible while still being practical: the characters in the code must be valid for URLs, no “funny” characters (“?”, “/”, “&”, “%”, “+” are probably a bad idea).

One of the easiest functions to use for this purpose is to encode the integer (decimal, base 10 number) into base 36.

Normally we use base 10 to count. In base 10 numbers we only have 10 possibilities (0 to 9) for each digit before we need add another digit in front of it and roll over to the starting symbol (“10″ is two digits, because we have no more possibile symbol to use after “9″).

In base 16 (hexadecimal) we add 6 more symbols (“ABCDEF”) so that we can count to “F” (or 15) before we need to add a digit in front and roll over to the starting symbol (“10″, or 16 in hexadecimal). If you’re having trouble following, imagine having 16 fingers and 16 toes to count with.

Same idea for base 36, but we add the whole 26 Roman alphabets, “A” to “Z”. So “Z” is 35 in base 36, and “10″ is 36.

How short is short?

With a 3-digit base 10 numbers, we can only have 1000 possible combinations (0 to 999). If I remember my high school math correctly, the formula is N^D where N is the number of possibilities for each digit (or the base number, 10 in this case), and D is the number of digits (3 in this case), so 10³ = 1000.

The goal is to have as many combinations as possible with as little number of digits as possible. Since the number of digits will grow over time (as we store more URLs) then the only way we can increase the number of possible combinations is by increasing the base number.

That’s why converting from base 10 to base 36 works.

So why stop at 36, you ask. As a matter of fact, we can make use of both uppercase and lowercase letters of the alphabets so that we can use base 62. If you’re willing to have non-alphanumeric characters in your short codes, you can even push it further with a few more characters. In fact, this is what essentially base 64 encoding does. Be careful though, the commonly found base 64 encoding normally works on strings instead of numbers, converting 6 bits of data at a time, and it can add paddings, the output is 33.3% longer than the input. Quite different from what we want.

I would probably use base 62 because it’s slightly more pragmatic and it has less chance of screwing up, but if you insist on using base 64 (for numbers, not strings), then I suggest you use “-” and “=” as the additional symbols. I’d avoid underscores (“_”) because they can be hard to see in links, which are normally shown underlined.

The following table shows the number of possibilities for some of the bases:

Say, you add 1 million new URLs every month on average. With base 62, it would take a little over a year to use up all the possible 4 digits, and after that it would take another 75 years to use up all the possible 5 digits, and after that it would take another 4,658 years to use up all the possible 6 digits, and so on.

It would probably take even longer to use up the digits if you don’t add URLs that are already in the database. Conversely, you might use up the digits faster if you want to do per-user tracking, for example.

Sample code

Here’s a simple Python code that you can use to convert an integer to a base N “number” (actually string), and vice versa.

# Code adapted from: http://en.wikipedia.org/wiki/Base_36#Python_Conversion_Code
# This is base 62:
ALPHABETS = "0123456789ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz"

def base_n_decode(s, alphabets=ALPHABETS):
    n = len(alphabets)
    rv = pos = 0
    charlist = list(s)
    charlist.reverse()
    for char in charlist:
        rv += alphabets.find(char) * n**pos
        pos += 1
    return rv

def base_n_encode(num, alphabets=ALPHABETS):
    n = len(alphabets)
    rv = ""
    while num != 0:
        rv = alphabets[num % n] + rv
        num /= n
    return rv

Can’t we just use MD5?

No. We can’t use hash functions like MD5, because hash functions are one-way, irreversible, meaning we can’t get the original input just from knowing the hash code. Also, hash functions can have collisions, meaning from one hash code there is a remote possibility that it “maps” back to two or more URLs. Plus, the output of hash functions are normally long-ish fixed-length strings.

So, what are you waiting for?

Now that the magic is revealed, you can create your own URL shortening service. Be creative and innovative so you can compete with all the other URL shortening services out there.

As a bonus, you can also use this technique to shorten the permalinks or absolute URLs for your objects in your Django (or Rails, etc.) websites. So instead of http://example.com/post/12345 it would be http://example.com/post/3D7. Note that this has nothing to do with security whatsoever. The coded ID can still be reversed with some effort. And you shouldn’t rely on obfuscation to protect anything anyway.

Bad usability, especially obvious ones, is one of my biggest pet peeves in web development. So when a website that I — and probably many others — frequently visit has a terrible usability, I have a mixed feeling of anger and frustration that I can only hold for so long. In the spirit of positive thinking, I’d like to channel out that anger and frustration through this (hopefully) constructive criticism.

A Closer Look at the Problem

Let’s take a look at IndosatM2‘s member site as an example.

First, I’d like to ask a question. What’s the main reason people go to their own ISP’s website? I bet checking their usage is one of them. It’s probably and arguably the number one reason, if not the only reason. It is for me at least, since my ISP doesn’t provide a Dashboard widget and no one has yet to write such widget to work with IndosatM2.

The most commonly used function should be as easy to reach and as quickly displayed as possible. It wouldn’t make sense to force the user to jump through many hoops just to get to the most frequently wanted information. So let’s count how many steps are required to check the usage for an IM2 subscriber.

Step 1: Logging In

The login form is not on the main indosatm2.com website, requiring users to click a link that goes to a separate login page. (Of course, one can bookmark the page for quicker future access.)

Things to note:

• CAPTCHA. On a login form? What’s the point?
• The size and the location of the login form compared to the text on the side.
• The order of the points in the text. The “Usage History” point is last.

How to improve:

• Remove the CAPTCHA. To prevent or deter brute force login attacks, use delayed response on failed login attempts within a short period of time. Block the IP address (either temporarily or permanently) after 50 repeated failed logins within 5 minutes, or whatever is reasonable.
• Make the login form bigger and place it in the center of the page. It is a login page after all, so the login form should be the center of attention. Move the text below the login form. Show the Usage History as the first item.

Step 2: Closing Useless Popup

Things to note:

• Needs to click on OK to make it go away.
• It serves no useful purpose.

How to improve:

• Remove it altogether. We would know if the login is successful or not. Only tell us when something is not working, not when something is working. It’s like a little child yelling proudly everytime they managed to accomplish the smallest things: it’s cute at first but it gets annoying after 3 times. Yes, Mr(s). Programmer, I know you have mad Javascript skills, but who are you trying to impress? Let me guess, you got the inspiration from Windows’ popup balloons?

Step 3: The Home Page

Things to note:

• The size of the image.
• How the image serves no useful purpose.
• Needs to scroll down to see the actual contents.
• How “Last Invoice” and “All Invoice” (sic) functions are highlighted in the home page.

How to improve:

• There’s already a menu on the right side. Make it hierarchical so that the user can get straight to the function they want rather than requiring them to see an extra page that only contain the submenus.
• The “Last Invoice” and “All Invoice” (sic) buttons can go away. They are no longer necessary. Instead, put the contents of step 4 and 5 directly into this page. Read more below.
• Remove the obnoxious image. At least, don’t make it so big, or move it somewhere else, don’t make it the first thing I see when I log in.
• This step can be eliminated.

Step 4: The Usage Page

Things to note:

• It already knows what month and year it is now (they are shown as the default).
• Needs to click “OK” to get the information for that month.
• Hmm. What does “Cancel” do?
• The obnoxious image that serves no useful purpose.
• Needs to scroll down to see the actual contents.

How to improve:

• Show the month selection box straight in the home page, instead of in a separate page.
• Make last month as the default value. See “How to improve” in step 5 for the reasoning.
• This step can be eliminated.

Step 5: Finally

Things to note:

• Again, the obnoxious image and the scrolling down.
• The inconsistent use of English and Indonesian.
• There’s no easy way to see data for other months, must go back to step 4.

How to improve:

• Show the results for the current month straight away in the home page. This is what most people want to see. And since you already know what month it is now, why don’t you just show me right away?
• The system can detect what day of the month it is now. If it’s still early in the month the system can also show the total usage for last month in this page.
• The month selection box in step 4 is always shown at the top (and optionally at the bottom) of the page with the default value set to last month.

Conclusion

The steps can be reduced down from 5 to only 2 without sacrificing any functionalities. The most useful and frequently used features are highlighted and shown as reasonable and sensible defaults. Logging in should be easy (username and password, that’s it). Then the home page, containing the current month’s usage (probably the most commonly requested information), is shown right away, without any annoying popup. If the user needs to see data for other months, then the month selection box should already be accessible from the same page, showing the next commonly wanted information (last month’s usage).

Other Examples?

There is another public website that I use quite often and I find that it has plenty of rooms for improvement in terms of usability: KlikBCA. But one can write a whole book on that website alone discussing what’s wrong and how to improve it.

Have you seen other public websites that can be improved in its usability?

Compiling psycopg2 using MacPorts as well as from the tarball would result in something like this when loaded from a web application that runs on Apache:

ImproperlyConfigured: Error loading psycopg2 module: dlopen(/Library/Python/2.5/site-packages/psycopg2/_psycopg.so, 2): no suitable image found.  Did find: /Library/Python/2.5/site-packages/psycopg2/_psycopg.so: no matching architecture in universal wrapper

So all I needed to do was force psycopg2 extension to be built for 64-bit as well as 32-bit. After extracting the latest psycopg2 source, do this instead:

LDFLAGS="-arch ppc -arch i386 -arch x86_64" CFLAGS="-arch ppc -arch i386 -arch x86_64" python setup.py build

Then sudo python setup.py install. After that, check the .so file, it should look something like this:

$ file /Library/Python/2.5/site-packages/psycopg2/_psycopg.so 
/Library/Python/2.5/site-packages/psycopg2/_psycopg.so: Mach-O universal binary with 3 architectures
/Library/Python/2.5/site-packages/psycopg2/_psycopg.so (for architecture i386): Mach-O bundle i386
/Library/Python/2.5/site-packages/psycopg2/_psycopg.so (for architecture ppc7400):      Mach-O bundle ppc
/Library/Python/2.5/site-packages/psycopg2/_psycopg.so (for architecture x86_64):       Mach-O 64-bit bundle x86_64

Don’t forget to restart Apache, and that should be it.

Update: It happened to PIL too, it was giving this error: The _imaging C module is not installed, so I had to do the same thing as above: LDFLAGS=... CFLAGS=... python setup.py build followed by sudo python setup.py install.

xkcd always cracks me up.

As usual, link for the uninitiated.

Why?

Having a revision control is really useful when we want to keep a history of contents in our web applications. This history data can be used to revert changes to a certain revision, find out by whom and when certain changes are made, and visualise the differences between revisions. Accidental overwriting of content is no longer a problem with revision control in place. Aside from history, revision control can also help in a multi-user environment where many people may potentially work on the same content. In my opinion, these are the main benefits of having revision control.

Cons

The only main drawback of having revision control is that extra storage space is needed to keep the revision data. The actual requirements vary, depending on the implementation. If done inefficiently (e.g. storing the whole revision content instead of just the differences), then the space requirement could be significant. However, storage is cheap nowadays, and in many cases the benefits far outweigh the drawbacks.

Traditional vs. web app.

Traditional revision control systems such as Subversion or CVS operate on files. Web application contents are normally stored in a database instead of files. This is one major difference that should be kept in mind.

Requirements

I’m going to give an example of two simple implementation approaches of revision control for web application content. But first I’m going to list the requirements for this simple implementation. Let say I already have a simple web application that is used by my family to manage our notes (think of a very simple family wiki). This is what I want to get from adding a revision control to the system:

• content history, ability to store, retrieve and rollback to certain revision, as well as to show differences between revisions
• audit, I want to know who did what and when
• all database, everything is stored in the database itself, no external files
• no cheating, I can’t use subversion or cvs behind the scene
• simple, minimal and straightforward modifications in the database structure and code
• generic, the design should be database- and language-agnostic
• storage, in this case I’m willing to pay the price of space for having a history of my notes, so storage space size is not a concern

Existing system

Let say this is the (simplified) existing database structure:

users:
    + id
    - name

notes:
    + id
    - status
    - title
    - content
    - created_by
    - created_at
    - last_modified_by
    - last_modified_at

First approach: the simple method

I think one of the simplest tricks that can be used to implement revision control is to add a revision_id column, and make it part of the primary key. And that is exactly what I’m going to do first. Here’s the modified database structure:

users:
    + id
    - name

notes:
    + id
    + rev_id
    - status
    - title
    - content
    - rev_user_id
    - rev_time

The users table is unchanged because I don’t need to put anything there under revision control. The primary key of the notes table now becomes a composite key (id, rev_id). The audit information now becomes the revision metadata, rev_user_id and rev_time, because now we can track who did what and when for each revision, not just the first and the last ones.

This method is the simplest that I can think of. There is a limitation, however, that every column in that table becomes under revision control. That’s probably fine for most needs, but there could be situations where only some of the columns should be under revision control, for example when storage space is a concern.

Now comes the more interesting part. How can we actually achieve common revision control tasks? I’m going to provide some example MySQL statements to illustrate how to do certain things:

• To list all published notes:

  SELECT DISTINCT id
  FROM notes
  WHERE status='published'

• To retrieve the latest revision of note 5:

  SELECT *
  FROM notes
  WHERE id=5
    AND rev_id=(SELECT MAX(rev_id)
        FROM notes
        WHERE id=5)

• To create a new revision of note 5 by user 3:

  BEGIN;
      SELECT @newid := MAX(rev_id) + 1 FROM notes WHERE id=5;
      INSERT INTO notes
      (id, rev_id, status, title, content, rev_user_id, rev_time)
          VALUES (5, @newid, 'published', '...', '...', 3, NOW());
  COMMIT;

• To edit a revision: simply create a new revision. Editing or deleting a revision is beating the purpose of having revision control in the first place. By never editing or deleting any revision we can rollback to a particular revision if we need to.

• To delete a revision: simply create a new revision with status set to ‘deleted’ and exclude the deleted notes from your view. Again, we never edit or delete any revisions, this way we can “resurrect” deleted notes by simply rollback to a previous revision.

• To list all revision metadata of note 5:

  SELECT rev_id, rev_user_id, rev_time
  FROM notes
  WHERE id=5

• To rollback note 5 to revision 2: create a new revision for note 5 by copying the content of revision 2 (excluding the revision metadata rev_user_id and rev_time) into the new revision.
• To show differences between revision 2 and 3 for note 5: retrieve both revisions and pass them into a diff program or its equivalent

Second approach: an alternative method

There is an alternative approach using a separate table to store revisions. This method practically addresses the aforementioned limitation of the simple method. Using the same example, the modified database structure now becomes:

users:
    + id
    - name

notes:
    + id
    - title

notes_revisions:
    + id
    + notes_id
    - status
    - content
    - rev_user_id
    - rev_time

In the above structure the title column is not under revision control. Although title may not be the best example to illustrate the storage space restriction requirement, I used it as an example to show that we can put certain columns outside revision control. The id column of notes_revisions is the revision id. If you’re using MySQL, then you can’t make notes_revisions(id) column as auto_increment in this case. The notes_id column is a foreign key referencing notes(id) and is part of the composite primary key.

And here’s how you do common revision control tasks using this alternative method:

• To list all published notes:

  SELECT n.*
  FROM notes n
  LEFT OUTER JOIN notes_revisions nr
      ON nr.notes_id = n.id
  WHERE nr.status='published'
    AND nr.id=( SELECT MAX(id)
          FROM notes_revisions
          WHERE notes_id=nr.notes_id)

• To retrieve the latest revision of note 5:

  SELECT n.*, nr.*
  FROM notes n
  LEFT OUTER JOIN notes_revisions nr ON nr.notes_id = n.id
  WHERE n.id=5
    AND nr.id=(SELECT MAX(id)
        FROM notes_revisions
        WHERE notes_id=5)

• To create a new revision of note 5 by user 3:

  BEGIN;
  SELECT @newrid := MAX(id) + 1 FROM notes_revisions WHERE notes_id=5;
  INSERT INTO notes_revisions
      (id, notes_id, status, content, rev_user_id, rev_time)
          VALUES (@newrid, 5, 'published', '...', 3, NOW());
  COMMIT;

• To list all revision metadata of note 5:

  SELECT id, rev_user_id, rev_time
  FROM notes_revisions
  WHERE notes_id=5

Other tasks are done in a similar manner to the simple method.

Addressing simultaneous edits

Many times it is useful to allow simultaneous editing of the same content without locking. The basic idea is to allow users to edit anyway and let the system merge the differences between simultaneous editing sessions when creating new revisions. To do this properly, we need the help of a diff program and a patch program (or their equivalents) to merge differences. I think this could make my application too complicated for my purposes. So I’ll settle for a less complicated workaround: during save, if a simultaneous edit is detected, then the application will deny the save request. Although I have to admit that this workaround is not better than locking. Maybe I will write another article in the future that discusses this specifically in more details.

How to detect simultaneous edits? When presenting a note for editing, the revision number must be attached to the form (e.g. as a hidden input field). After submitting the edited note and before creating a new revision, the system must first check that the revision number in the database is not higher than the current revision. If it is higher, then that means somebody else has edited the same note.

Conclusions

There are more to revision control than what is written in this article, and it could be a very complicated discussion topic. But basic revision control is good to have, and easy enough to implement in most web applications.

Related information

• Open Source Development with CVS, 3rd Edition
• Version Control with Subversion

Acknowledgements

The alternative approach was inspired by MediaWiki. Thank God for open source!

Karena saya orang teknis, saya cenderung mempunyai ide yang berawal dari design aplikasinya dahulu ketimbang memikirkan bagaimana saya akan mendapatkan contentnya. Saya sadar bahwa pada akhirnya aplikasi yang secanggih dan seuser-friendly apa pun jika tidak mempunyai isi yang bernilai kepada penggunanya sama saja bohong. Saya rasa ini adalah masalah klasik yang dihadapi web developers yang cenderung technical.

Membuat kamus dari scratch (nol) itu bukan pekerjaan mudah, dikerjakan full-time oleh seorang professional saja bisa memakan waktu bertahun-tahun barangkali. Alternatif yang ada adalah menggunakan sumber yang sudah ada dengan meminta ijin kepada pemiliknya (lebih kecil kemungkinannya tapi tidak ada salahnya dicoba), dan/atau membuat sendiri rame-rame a la Wiki. (Catatan teknis: Ini berarti ada dua lagi feature dari kamus online ini, yaitu bisa diedit oleh siapa saja (setelah login, dan mungkin menggunakan captcha), dan ada versioning control untuk jaga-jaga kalau kena vandalisme.) Sumber yang sudah ada yang bisa diadaptasi idealnya adalah yang berkualitas tinggi, seperti kamus edisi cetak yang dijual di toko-toko buku (walaupun tidak berarti semua kamus edisi cetak yang dijual di toko buku berkualitas tinggi), bukan sekedar kamus online buatan si Panjul atau si Unyil. Sejujurnya, dan cukup disayangkan, sampai saat ini saya belum menemukan satu pun kamus Inggris<->Indonesia online yang isinya berkualitas mendekati kamus edisi cetak.

Saya sudah punya gambaran garis besarnya untuk design aplikasinya untuk saat ini. Yang saya rasa lebih penting adalah memikirkan bagaimana mendapatkan isi kamus yang berkualitas. Ada ide atau saran lain barangkali?

Update: SEAsite dari Pusat Studi Asia Tenggara, Northern Illinois University menyediakan kamus Inggris-Indonesia yang mendekati apa yang saya cari. Mudah-mudahan jika waktu dan pemilik kamusnya mengijinkan saya akan konversi ke database dan interface yang seperti saya ceritakan di atas.

Ruby on Rails is what J2EE should have been.

Oh, man, was I ever excited! After finishing that tutorial, I immediately searched for information about the existance of an Oracle adapter for ActiveRecord. It turned out that there is one, it’s included in recent versions of ActiveRecord and is based on the ruby-oci8 driver. I started my development Oracle server and immediately generated a scaffold for a test model. There were two minor problems. The first problem was that my table and column names did not follow the proper naming conventions expected by Rails, so I had to override it in the model class with set_table_name (doc) and set_primary_key (doc). No worries. The second problem was that the web server seemed to freeze after about 5-10 seconds of inactivity. It must have been something to do with the oci8 driver or ActiveRecord’s oci8 implementation, because it worked flawlessly with mysql. I didn’t have time to investigate further. It was good enough for me for now. I’m sure if it really was a problem with the driver then someone more knowledegable will catch it real soon and we’ll have a fix.

So, in conclusion, I wish Ruby on Rails had existed many years ago therefore, it could save me many weeks worth of tedious and error-prone coding. It beats PHP and J2EE any day. I will definitely use Ruby on Rails for my next web development project. And to summarize for those of you that are lazy or don’t have the time to follow the tutorial, install Ruby on Rails, then create a table in your database called products with these columns: id, name, description. Then create the rails structure with rails productlist, go inside the newly-created productlist directory, edit config/database.yml accordingly then run ruby script/generate scaffold Product. Then run the built-in WEBrick web server with ruby script/server then open your browser and go to http://127.0.0.1:3000/products. Voilà! That was ‘hard’. Now you can concentrate on what’s important, your application and its design, not tedious, repetitive and error-prone coding.