Life is somewhat rare, but intelligent life that has managed to survive any local catastrophic events and build the means for space exploration is much rarer still.
I think we can safely assume that there is no time travel or faster than light space travel, since in that case we would be overrun by alien life forms and would not be wondering where they were. If that is true, then suppose that an intelligent alien civilization can travel for thousands of years to reach us, and that it would. The journey would need to be worthwhile for such a massive expenditure of resources. How would it decide to use its resources to visit Earth, specifically? It would have to have seen us with very powerful telescopes that could actually discern that our planet has life on it. It would need to observe that life doing something intelligent-looking, like building a civilization, etc., because it would be a waste of time to visit a planet of grazing herbivores, right? That is, unless it was just looking to colonize and/or take the resources.
If that all makes sense, then Earth creatures have only looked intelligent for a few thousand years, which makes the time in which we could have been discovered, an expedition to us mounted, and to have gotten here very short by interstellar travel standards. The alien civilization would have to be very close (within a few thousand light years, assuming near-light speeds are possible), would have to have been looking right at us at the dawn of "civilization," AND not be distracted by something more interesting or closer.
So, I think intelligent life just hasn't arrived yet, and it may be awhile. Hopefully, we'll still be here then.