There's a lot of soy alarmists with biased motivations out there. I've also seen mixed reviews, but I was more concerned about their effect on adrenal function.
What I've read about soy isoflavones effects on estrogen is that they are cleared from the body within a few days, are between 10 and 1000 times weaker than your bodies' natural estrogen, and that they had a protective effect against chemical xenoestrogens - BPA, pesticides etc, by blocking the estrogen receptors, thereby reducing the number of sites the xenoestrogens can attach to and then indirectly encouraging them to be excreted from the body. But its really hard to find unbiased info. They were also good for gently normalising estrogen in females.
The estrogen in dairy is far stronger than soy.
I just try not to let it become a major part of my diet. I avoid eating anything with soy in it more than once a day (I do this with most foods anyway).
The other thing that concerns me though is that soy is poisonous raw. I'm not a raw foodie, but I have a fairly high proportion of raw, whole foods in my diet. But it still makes me think twice about things like that. But having said that, I still eat soy frequently.