It will really depend on the dog. A lot of people say strong smelling items such as Ammonia, vinegar, and citrus oils will deter a dog from coming near the area. Others say to use Chili Pepper. Potentially some plants with strong scents will deter just the same. However, my personal experience is that my dog is NOT deterred by any of that.
I had to resort to a physical barrier. I've got a larger dog, so a decorative fence from my local hardware was plenty to keep him out. For a small dog you could use a similar fence, but attach chicken wire to it. Even some of the small fences might be tight enough to keep a small dog out.
If you are worried about the look, you can get creative by first building a raised bed, oddly shaped rocks might be beneficial. Then for additional security, add some decently thick plants around the edge of the bed. For a small dog, there is a good chance this will look like a wall that they will be scared to jump on to.