We think 5 days is the perfect amount of time to spend in New Orleans. You can't see everything yet, but you're leaving just before you get tired of partying, which will make you want to come back soon. This extra time provides a lot of flexibility, as you'll have time to spend the night in one of the enormous pre-war mansions located on the outskirts of the city. We stayed at Nottaway and it was really the highlight.
If you choose to do so, we suggest that you stay overnight on the last night, as it's easy to get to the airport the next morning from there. If you want to fit in more plantation mansions, stay in the middle of your 5 days by stopping at different mansions back and forth. We've included an alternate day itinerary to give you an idea of what we mean. We love having extra nights to mingle at the House of Blues or on a Natchez steamboat ride, instead of going to Bourbon Street alone each night.