Prepare the grill for direct cooking over medium heat (350° to 450°F).
Whisk the aioli ingredients and season with ¼ teaspoon salt and ¼ teaspoon pepper.
Scrub the potatoes under cold water and dry them with paper towels. Cut the potatoes lengthwise in half, and then cut each potato half lengthwise into ½-inch-thick slices. Place in a bowl and add the oil, 2 teaspoons salt, and 1 teaspoon pepper. Toss to coat.
Grill the potato slices over direct medium heat, with the lid closed, until tender and marked by the grill, 15 to 17 minutes, turning occasionally.
Serve the fries warm with rosemary-lemon aioli.