We been to many places and never been treated badly and this is a very first time we heard of racism abused. Just a bit worry. I am of indian origin and went with my wife non-indian to Riga recently.
You will get lots of staring out of curiosity, not many dark skinned people in Latvia. No-one will smile at you It may seem that people don't like you, but I think its just their culture.