Don't create bad filtering systems

You’ll find that many systems get overloaded with too many low quality inputs. Often a solution to this is to add some kind of filter to catch those bad quality inputs. A common side effect is that the same filter unintentionally removes the best inputs at the same time.

Confused? Let’s try an example.

You open a job role, and you get 1000 applicants. Most of them are not qualified, but it’s hard to find the best applicants because of all the noise.

So instead of letting just anyone apply, you put a little puzzle in front of the application form. The non-qualified applicants generally don’t want to put in effort and they’re unable to solve the puzzle anyway. The number of applications you get drops by 90% and you don’t have any more unqualified people any more. Looks good!

But maybe the best applicants aren’t willing to jump through the extra hoop. They know they can find something good somewhere else. So while it appears like your filter is great and removing the bottom X% of applicants, it’s actually also removing the top X% too, in a way that’s hard for you to know (you just assume that the average quality has improved).

This works in all kinds of scenarios, not only hiring. Maybe you make your leads fill out a questionnaire to prove that they’re a good fit for what you’re selling? Maybe you have an onerous performance review process to weed out bad team members?

When you add a filter to an input, think about unintended consequences, and especially those that will be hard to measure.