I don't know if this is necessarily a chemistry question....
But the reason heat transverses to the shirt is because of a difference of temperature between the two objects, the shirt being cooler, and the heat will move from an area of high temperature to an area of low temperature to cause the environment as a whole to become equal. Much like how solvents move to areas of low solute concentration to areas of low solute concentration during diffusion.
