The journey between Bucharest and Istanbul is made on-board the Bosphor Express. The service offers both First and Second Class sleeping cabins, as well as Second Class couchettes. The service is operated by Bulgarian Railways, using relatively old (but still perfectly comfortable) trains.
Part of the journey is shared with the Balkan Express train route between Belgrade and Istanbul and the Romania Express train between Bucharest and Thessaloniki.
Some people have reported frequent delays at both arrival and departure, so bear in mind that this might not be the most efficient rail journey. Nevertheless, the scenery you can expect to see should be worth it.
For further information, try this page at Turkey Travel Planner.
From March 2012 until late 2013, a rail-replacement bus is in operation between Istanbul and Kapikule as a result of engineering work.
If you are travelling towards Istanbul, the train will terminate at Kapikule and you will be transferred to a bus (accepts rail tickets and passes). The bus will arrive at the same time as the train is scheduled to. Travelling towards Belgrade, the bus departs from outside Istanbul Sirkeci Station at the same time as the train is meant to leave, and travels as far as Kapikule where passengers can board the train as usual.
Seeing as the overnight portion of the journey is made on a coach, buying a seated ticket rather than couchette may be adequate.