One tiny change made this Python code over 2,000× faster. Same logic, same result. The only difference is a list versus a set.

Here's the video version (2 minutes):

The slow code

A common task: 20,000 users, a list of 20,000 banned emails, find which users are banned.

users = [...] # 20,000 emails banned = [...] # 20,000 banned emails (a list) flagged = [u for u in users if u in banned]

One line, looks perfectly fine. On my machine it took 1.31 seconds.

The fix: one function call 

banned = set([...]) # a set instead of a list flagged = [u for u in users if u in banned]

The comprehension doesn't change at all, and it returns the same 10,000 results. It now takes 0.61 milliseconds. That's over 2,000× faster, and building the set took less than a millisecond.

Why: how in works

  • x in list checks the items one by one until it finds a match → O(n)
  • x in set hashes x and jumps straight to the slot where it would be stored → O(1) on average
  • Dictionary keys work the same way: key in my_dict is O(1)

With lists, our example is up to 20,000 × 20,000 = 400 million comparisons. With a set, it's 20,000 hash lookups.

Measured: a single lookup at different sizes

Worst case (the item is at the end of the list):

n = 1,000 list 4.3 µs set 0.021 µs n = 100,000 list 455.0 µs set 0.027 µs n = 1,000,000 list 4560.7 µs set 0.027 µs

The list gets ~100× slower when the data gets 100× bigger. The set barely moves. At a million items, a set lookup was over 170,000× faster.

When to use a set (and when not to)

✅ Use a set when:

  • You check membership (in) many times against the same collection
  • You want to remove duplicates: set(items)
  • You need intersections or differences: a & b, a - b

❌ Keep a list when:

  • You need order, indexes, or duplicates
  • You only look something up once: converting to a set costs more than it saves

Try it on your code

Search your codebase for in followed by a big list inside a loop. You might find a free speedup.

I post one tested coding tip every day on YouTube at Refactorly. The 30-second version of this one is here.