The optimum propeller is the one that will drive the yacht at its designed maximum hull speed. An approximate formula for this is 1.3 times the square root of the waterline length in feet. For instance a yacht with a waterline length of 36 feet will have a theoretical maximum speed of 8 knots.
The engine needs to produce enough power to achieve this speed and also drive auxiliary items such as the alternator, fridge pump etc. The ideal combination of engine and propeller will produce maximum boat speed at 80% of maximum engine speed. This will enable the owner to cruise efficiently at speeds of up to 90% of maximum hull speed and still have some power in hand for an emergency.
An undersized prop will result in low boat speed, while an oversized prop will overload the engine before the optimum boat speed is reached, producing black smoke due to the burning of excess fuel.