Make the lavender sugar: In a food processor or spice grinder, grind the lavender for about 2 seconds—yes, it goes by quickly! This will reduce the size of the petals just slightly and open them up a bit to bring out the flavor, without turning them into a powder.
In a small bowl, stir together the lavender and sugar. For optimal flavor, wait about 1 week before use. Store in an airtight container at room temperature for about 6 months.
Make the coffee: In a large bowl or pitcher, stir together the hot coffee and lavender sugar. Place the mix in the fridge for about 6 hours or overnight to chill.
When you are ready to make the cafés au lait, fill 4 glasses with ice. Pour 1 cup of the coffee and lavender mixture into each glass followed by ½ cup of whole milk. Stir until combined. Garnish each glass with a sprig of lavender.