Physics..

That's a no-no. Only reason why it doesn't matter this time is because they completely cancel each other out.

 Alright, 2nd attempt, or so help me jebus!

Code:

[|A|^2 + |B|^2 - 2|A||B|cos(θ)]n^2 = |A|^2 + |B|^2 - 2|A||B|cos(180-θ)
[|A|^2 + |B|^2 - 2|A||B|cos(θ)]n^2 = |A|^2 + |B|^2 + 2|A||B|cos(θ)
[2|A|^2 - 2|A|^2*cos(θ)]n^2 = 2|A|^2 + 2|A|^2cos(θ)
[1 - cos(θ)]n^2 = 1 + cos(θ)
n^2 = 1 + cos(θ) + cos(θ)*n^2
n^2 = 1 + (1 + n^2)*cos(θ)
(n^2 - 1) / (n^2 + 1) = cos(θ)

Well the way the problem is stated, it either doesn't matter, or the answer has to be in terms of |A| as well as n. Intuitively the magnitude of the vectors can change and the angles are still the same, so I just assumed they were unit vectors to make it cleaner. Probably not the most proper, but it gets the answer.

Hm, seems either you or I dropped a negative sign somewhere. I think it might've been me.

Edit: Oh, I see. I designated theta as the other angle.