If cos and sin are both minus, you are in quadrant III where the cot(theta) is plus.
The reason for that is that cot(theta) = cos(theta)/sin(theta) which means that the two minuses will cancel out.
Now all you have to do is calculate sin(theta).
a^2 + b^2 = c^2
cos = adjacent / hypotenuse
a = 5 The adjacent side
b = ????
c = 13 The hypotenuse
5^2 + b^2 = 13^2
25 + b^2 = 169
b^2 = 144
b = 12
sin(theta) = -12/13
cot(theta) = -5/13//-12/13
cot(theta) = -5/-12 = 5/12
