As far as I know, there isn't one dish that Atlanta is famous for. There are, however, several foods I hadn't had before I came to Atlanta. With any luck, these will reflect something of the city's food culture:
I had always thought of beef as the substance that fills tacos. Thanks to Taqueria del Sol, a popular taco chain with one instance in Decatur, I've been able to experience the fish taco. Now, if I have the choice, I'll always choose a fish taco over any other.
Bubble tea is a Taiwanese drink: small balls of tapioca immersed in sweet milky tea. There are all sorts of different flavors, and you can get an extra-large straw that's wide enough to suck up the chewy spheres. If you didn't know what was going on, it would feel a lot like accidentally sucking up a fly that was in your drink. I was lucky enough to have friends that explained everything to me beforehand.
I had frog legs at the home of one of the nicest and most "Southern" families I know. The legs were breaded and fried and much larger than I expected. Very nice dipped in tartar sauce.
Varsity hot dogs
The Varsity is a famous restaurant in Atlanta that serves hot dogs, sides for hot dogs, and a famous orange slushy. I'm putting this here because this place was recommended to me with great enthusiasm. Unfortunately, I couldn't tell what set their hot dogs apart from others; also, the high price of the orange slushy took away some of the joy of drinking it. A nice place, though.
From what I hear, cheese grits are classic breakfast material in these parts. I've certainly enjoyed them with bacon at many brunches. I'm not sure if I've ever had grits in Atlanta that weren't cheese grits.