Oh, p-u-u-u-u-l-l-l-l-l-e-e-e-e-e-z-e . . . . wear a jacket!
I've lived in California for all but 5 of my 60+ years, and travel to NOLA 1-2x per year since before Katrina . . . it's never been so cold that walking was out of the question. (That said, the rainfall can be drenching, and I'm not just talking hurricanes!)
When we're in New Orleans for a business conference, my wife and I typically stay in the FQ where the conference is being held -- usually the Royal Sonesta (just make sure to get a room that does NOT face Bourbon St.), though we often stay in the CBD at the Roosevelt, though before it re-opened, we stayed at the International House. Never had an issue with safety, as long as one uses common sense, and we do a LOT of walking while we're there. Indeed, between the two -- FQ and CBD -- I prefer the CBD: it's quieter at night and generally cleaner, though the FQ is kept cleaner than it has ever been!
That said, the Garden District can be beautiful, but it's also (relatively speaking) remote. You're in the GD, period. Depending upon where in the GD, you can walk down to Magazine St., but you are far more limited. We stay in the Garden District frequently -- often after the conference ends -- but we stay with friends in their beautiful antebellum mansion, so that's different than staying at, say, The Columns or the Ponchartrain.
it isn't necessarily that one is better than the other. While places are certainly more accessible by foot if you're in the Quarter or the CBD/Warehouse district, staying in the GD is simply a DIFFERENT way to experience New Orleans than if you stay, say, in the Quarter . . .
At night, we do take taxis to all the restaurants that aren't in the immediate vicinity. That is, if we're in the Quarter, we'll walk to the restaurant that's in the Quarter, but take a cab to and from the restaurant that's Uptown, Mid-town, in the Garden District, or even in the CBD. And -- yes -- every place will call a taxi for you, with the added benefit that they'll actually arrive! Taxis know they are dependent upon the restaurants for fares and generally respond more quickly to calls from them than they do from private parties. (Sometimes if you call the cab yourself, they may take far longer to get there or not get there at all; this is largely due to a safety issue, and depends to some extent on the current state of affairs vis-a-vis the number of taxicab robberies taking place at any given time. After all, with no glass partition between front and back seats, cab drivers can be viewed as an easy "mobile ATM" by some criminals.)