If you want the olive trees to thrive and survive, it certainly seems a choice must be made between moving the olives elsewhere, or removing the conifer. Without a close up photo of the needle arrangement on the conifer, it's difficult to say precisely what it is. It's therefore hard to know how big it is going to get, but from its growth habit, it does not look like a dwarf and is likely to get much wider than the bed it's planted in, which will prevent the olive trees from growing properly. Because the space is relatively small, a single olive tree with the Cordylines either side would be more appropriate anyway.
If you choose to remove the conifer, with most conifers, all you need to do is cut it right down to the ground, because the majority of them do not regenerate from old wood, rather than having to dig it out. Though not ideal,the roots and stump can be left in place to rot down over time on their own. Most conifers, though, do not usually have a very wide root spread, they tend to form a sort of clump of fibrous roots rather than spreading far and wide as deciduous trees do, so it may be possible to remove completely. If you decide to remove it, depending on how long the Olive trees have been in situ, you may be able to carefully lift those out, take the conifer plus roots out, then replant either one olive in the middle, or, if you must, one either side.
