If you spray (or saturate) your plant with water mixed with a little salt and washing up liquid then the aphids should die within a week. You don't actually need the salt, but it probably helps a little bit.
If the aphid infestation is severe enough it can cause the leaves to come off the plant and the plant to get very sickly in various ways.
Another problem is that aphids can spread diseases. Like how mosquitoes are annoying for humans, but then can also spread malaria, aphids can just be pests or worse than pests. Therefore, especially if you have another similar plant, you should probably isolate the aphid infested one for some time to check it has not caught any diseases.
The more unhappy and stressed the plant is, the more susceptible it is to pests and diseases. So if it is dealing with undesirable conditions (such as the cold) as well as aphids, this will slow its recovery.