Why you can never directly compare
Here is the frustrating heart of it: there is no way to step inside someone else's mind and see their colors as they see them. You can both point at a ripe tomato, agree it is 'red,' pass every color-naming test identically, and still, in principle, be having different private experiences. Philosophers call the imagined case where one person's inner colors are systematically swapped the inverted spectrum, and it has been discussed since John Locke in the seventeenth century.
The Stanford Encyclopedia of Philosophy treats this as a serious puzzle in the study of consciousness. The reason it resists resolution is that language and behavior only track the relationships between colors, which stay the same even if the underlying feel were flipped. If your inner red and green were swapped from birth, you would still call tomatoes red and grass green, and no test of words or actions would reveal the difference.
Color perception really does vary
What is not just hypothetical is that color vision genuinely differs between people. Around one in twelve men has some form of color blindness, usually because one type of cone in the eye is missing or shifted, so their color world is measurably narrower. At the other end, a small number of people, almost always women, may be tetrachromats, carrying a fourth cone type that can in principle let them distinguish colors most of us cannot.
The most famous demonstration was 'the dress' in 2015, a photograph that some people saw as white and gold and others as blue and black. A study published in Current Biology confirmed the split was real and stable, driven by differing assumptions the brain makes about the lighting in the scene. It was a rare case where people's private color experiences visibly diverged over the very same image.
What this does and does not prove
These findings show that color is not simply out there in objects; it is constructed by eye and brain from wavelengths of light. Two people with different cones, or different assumptions about lighting, really can experience the same surface differently. That much is settled science.
What stays open is the harder question about people with ordinary, matched color vision. Even if your eyes and mine work identically, we still cannot verify that your inner experience of red feels the way mine does. Most philosophers suspect our experiences are broadly similar, since our biology and evolution are shared, but 'broadly similar' is a reasonable bet, not a proof. The private character of experience keeps the door open.