I have the same problem, but I think it's the way our brains are conditioned.
Also, I don't know how useful it is to learn random intervals, around 80 % of the music around has a fixed key.
The reason in my opinion why min3/maj3 confusion happens is because we always think or want a reference ...