Can I really hear the difference

YES! in a big way! Sorry, no references. Honestly, this falls into the "blue and red make purple" category. This means, you can just try it yourself. Go to a stereo store. Listen to one system. Listen to another. I guarantee they'll sound different.

We may not be the kings of the animal kingdom when it comes to senses but don't sell yourself short. Average humans can tell the subtlest differences between wine, cheese, faces, voices, music, etc. You can too. In fact music is the easiest of the previous examples. It requires absolutely NO training. You can already tell the difference to a very fine degree.