I started with the following question:
Why is a man in women’s clothing called a transvestite,
while a woman can wear almost any man’s clothes she wants?
Where does this contrast come from?

It has to do with INDENTIFICATION!

IDENTIFICATION historically seen:
Man are strong, woman are weak.
A man descending into the weaker is a loser or worse

IDENTIFICATION socially:
In the western world you can dress however you want.
But a man dressed in women’s clothes is a weirdo, mostly described as a person who is extraordinarily strange or eccentric and will find himself socially excluded, can even expect violent reactions.

 IDENTIFICATION globally:
A man who dresses as a woman is tolerated almost nowhere in the world.
In many countries it is illegal, he can be arrested for it, thrown in jail, tortured, possibly sentenced to death.

.