No, the cat poo will not affect your plants. Maybe you if you dig it up? There is something that I picked up on that you should know about. Lilies of all kinds are poisonous, toxic to cats. Worse, these plants actually attract cats. Very poisonous. Some cats might not be attracted right away sometimes never. I keep my ditties inside because cats out of doors become everyone's cats. Cats make this very regimented ROUTE in the neighborhood. Sort of a rectangular regular path they take everyday. There is no way you can keep cats away from lilies, mouse traps, mole traps, pesticide for mice and rats and in more rural areas, cougar, bobcat, badgers, raccoons...coyotes and dogs.
My cats were always out of door cats, too, until I learned about lilies being toxic to cats, I had my goats eaten by cougars, chickens eaten by raccoons...dog/badger to the death fights...ugh. Cats dragged away by coyotes and once an owl.
Day lilies, Canna Lilies, Siberian Iris...Calla Lilies might be the only one that is different with the lily name. Different genus and species all but still plants with the lily in their name are toxic to cats...I wish I hadn't found this site that lists all the toxic plants for cats.
list of plants toxic to cats