Yesterday I was complaining the patent system meant that people were not encouraged to share good ideas when they had one, which means the world does not improve as fast as it could. I said that I was going to put my money where my mouth was, and start sharing my ideas. Well, here's the first idea, but it's not an idea for a piece of software, its an idea for a business.
I've had to deal with a number of IT recruitment agencies, both as someone looking to hire and as someone looking for a job. The worst ones have been bad, though one, ECM was good, from both perspectives. However, they all have the same core approach - text searching on a candidates CV. This might work ok if you are looking for a Java developer with 3 years experience. However, as Joel Spolsky has commented, you should be looking to hire the superstars. And the superstars might not know Java, but they could probably learn it in a week, and be a better programmer than one of the unwashed masses with 3 years experience. CV text search is obviously the wrong solution.
The right solution involves trying to find people who are smart and get things done. If you had an agency which measured how smart people were, and how good at getting things done, then it would be much more useful. If it had features so the superstars really stood out, you would want to use that agency if you were looking to hire a superstar. And you would want to use that agency if you were a superstar, because you would know you were going to work for a company where they appreciated you. You probably wouldn't want to use the agency if you were a poor programmer because it would be obvious very quickly, thus improving the quality of the hirees.
So how would you do this? Well people looking to hire clearly have money, or they couldn't afford to hire a superstar, but not much time, whereas people looking for a job probably have more time, especially those in college, but less money. So the site should be cheap but possibly time consuming for the hirees, and quick and time-saving but with the hirers paying for it. Monster already have this model.
My idea is have a site with lots of puzzles, intelligence tests, essay questions etc., ideally most of them automatically marked. In my experience, most superstars like solving puzzles, and like being told they are smarter than other people, and being able to see their rank. IQ tests may only show how well you do IQ tests, but they are probably well correlated with intelligence. Every test you allow people to take multiple times and improve their score. That way, people who are willing to work hard do better, and you get people who get things done nearer the top of the list.
There are a million and one different competitions and tests you could have. You could get people to submit code samples and run them through an automated, lint-like code quality detector. It wouldn't be perfect, but it would be fun to use, and competitive. You could ask people which software books they have read. You could even make them write a review of them. You could get other people to rate their reviews, and through an emergent process of how their reviews were rated, and how their ratings correlated with other people, you could probably get quite a good measure. You'd also end up with people knowing that if they want to do better they should read book X, or write code in the style Y, and so people could actually be inspired to improve through using the site. I'm sure you can think of more ideas for clever tests.
If you had enough different sensible tests, and gave people free choices of which ones they did, then I suspect the ranking you gained might be well-correlated with how smart people were and how good they are at getting things done. And it would help software companies hire superstars, and superstars be better recognized and find better jobs. Which should improve software for everyone. And the world would be a better place.