Guy with his girlfirend started fight with me just for not giving him directions

I was walking by a trendy dressed type of couple around midnight on a saturday night. Usually I will walk on the other side of the road since nearly all drunks want to say something and mark their territory and prove themselves.

Since it was a couple I presumed they might be more harmless. How wrong I was.

Once I got side by side the guy immediately said in a condecending tone like he was demanding from me
'hey! where is x hotel!?' since i did not like his tone I tued very slowly and didn't answer him because answering him and tued back to walking.

Actually I don't even think he gave me chance to respond before following up with a racist remark 'i bet the bastard he doesn't even speak english' (since I am mixed race) which got a rise out of me and I said 'yes I really sound like a foreigner' and then under his breath he says 'prick'.

I am a bit ahead of him now so I just half tu and put my thumb up to him. He gets more angry with more cursing and so I tu fully around and beckon him to 'come and have a go then!' but rather than stand still I kept walking. Does that make me a 'wimp' who 'backed down from protecting my manhood'? I heard him taunting me saying that I am 'ruing away' but I didn't look back.

I trained in muay thai for two years and this guy was just a skiy trendy so I doubt he could fight but was probably coked up and spoiling for a fight. I feel like shit when things like this happen I always think 'why me?' or why are there so many such cretins in existence. Tough guys online like to say 'you were asking for it that shit would NEVER happen to me!' but I see plenty of big guys getting shit thrown at them on youtube videos so I don't think it is because of my 'weak demeanor'.

Should I have handled it differently?

In future I will avoid such situations altogether by always walking on the other side of the road where possible no matter if guys or girls. Only at night of course where such belligerence is rife. And if I have to walk by someone like that and they do say something I will try just ruing away at full speed. I don't mind ruing away I just don't like absorbing abuse and it is sort of a defiance because they expect you to just sit their and take their belittling.