Will mouldy carpet mildew cause problem when growing veg?

I put my old carpet over my patch of soil over the winter to stop the weeds growing, as it was rife with them before. However now I have lifted it, there is mouldy stuff all over the carpet and white flecks of something al over the soil. I wanted to grow vegetables in the patch and make raised beds, but worried now that the toxins from the carpet have leached into the soil and will cause problems with the plants, either diseasing them, or making them poisonous to eat. Am I being paranoid or should I just grow raised beds for flowers here instead? Is there any way I can treat the soil to reverse the damage? When will I be able to plant veg here? Thanks for any advice you can give me :)