Hi Mrss1981,
My situation is a bit different from yours, but thought i'd share my experience anyway so it can hopefully give you piece of mind
My DH was diagnosed with testicular cancer when he was 21, and as a result he had to have one of his testicles completely removed. After all his chemo and everything, his fertility was tested, and he was told by numerous doctors that if one testicle is no longer there or not functioning properly, then the other one will compensate for it. The tests showed that even with only one testicle, he actually still produces an above average amount of sperm, which I would never have thought possible, but it's true!
Having said this, we are about to enter month 6 of TTC (unless I get a BFP on Wednesday- FC!!), so I guess this hasn't been proved yet!
I know this is different from your situation, but I just wanted to let you know what my husband was told after having one of his removed. I imagine that even if one of your husbands isn't functioning properly, there should be no reason to assume that it will stop you TTC if the other one is fine
On another note, my DH's friend damaged one of his testicles when he was younger, and had to have it removed as a result, but him and his wife now have a beautiful 3 month old baby boy

and they only got married the September before last, so I don't think they were trying for very long.
Don't panic, it might be worth a test just to be sure, but you'll probably be fine
I hope this helps a bit xx