In a medium saucepan, heat the cream, sugar, vanilla and a pinch of salt over medium-high heat until just below simmering. Remove the pan from the heat and set aside to steep for 30 minutes.
Bloom the gelatin by soaking it in ½ cup cold water for 5 minutes. Add the bloomed gelatin to the cream mixture and bring to a boil.
Carefully strain the hot cream mixture through a fine-mesh sieve placed over a blender. Slowly add the chocolate, blending on a low speed until incorporated. Place the buttermilk in a large bowl and strain the chocolate-cream mixture through into the milk, then stir until incorporated. Let the mixture come to room temperature, stirring occasionally, about 20 minutes.
Divide the mixture into 12 four-ounce ramekins; cover each with plastic wrap and refrigerate for 2 hours or up to overnight.
To serve, top the panna cottas with whipped cream, cacao nibs and a pinch of fleur de sel.