Although this can be an unsettling thought, it's also a very powerful realization -- at least it has been for me.
Over the years, I've often found that the qualities that annoy me about others are either qualities I've possessed at some point in my past or qualities I would like to express but don't.
Both of these realizations have lead me to either an increased compassion for others or a sobering self-awareness, respectively. In this way, others can serve as mirrors for our own state of consciousness and help us to recognize the places where we still might need to do some healing and self work.
If you find yourself feeling irritated by someone, try looking inward first. I highly recommend thinking deeply about this quote from the highly influential Swiss psychoanalyst Carl Jung. Initially, it can be an unpleasant mental exercise but, well worth it in the long run.