|Leucocoprinus birnbaumii (?)|
"What do I do?!?" you think, and "where did I go wrong?" Nothing* and nowhere.
If you have a small child or dog with compulsive hand to mouth syndrome (eats whatever it touches), you might remove the mushrooms, the species is not edible. But otherwise, they harm no one, certainly not your plant. Though some fungi are parasitic or pathogenic, in general fungi of the mushroom type are nothing more than an indication of the presence of decaying organic matter. In this case, potting soil. Many fungi are even beneficial. To learn more about that, read up on mycorrhizae.
This, however, is just a mushroom. I believe the species is Leucocoprinus birnbaumii. They get into the soil from other plants, spores in greenhouses and garden centers, and in potting soil (or potting soil components if you mix your own like I do) that is not fully sterilized. You can remove the mushrooms, but these are only fruiting bodies, the bulk of the organism is living quietly beneath the surface in a vast network of fine mycelia. I suggest you just enjoy their obnoxiously yellow presence for the few days they are present.
If anyone is interested, according to The Rainbow Beneath My Feet by Arleen and Alan Bessette, the species produces beige to yellowish brown dye depending on mordant. Bummer, I really hoped it would be sunshine yellow.