The composting process depends on a complex chain of organisms -- which can include ants -- to consume and thus break down the food in your compost pile. See this page from the University of Illinois extension service, which comments on ants in compost:
Ants - Ants feed on a variety of materials including fungi, seeds, sweets and other insects. They help the composting process by bringing fungi and other organisms into their nests. Ants can make compost richer in phosphorus and potassium by moving minerals around as they work.
Since the ants don't bother you, letting them work on your compost a bit would have some benefit -- you're enhancing the food web by allowing them to stay. The downside is that, by not turning the bin when the internal temperature drops, you may not achieve high enough temps for long enough to kill off weeds and harmful bacteria. But if your mix wasn't going to achieve this anyway, then it's not an issue.
See also the following diagram of the compost food web, taken from this chapter on composting:

(source: fao.org)