Heat oven to 350°F. Butter a 9×13-inch pan, line with parchment, and butter parchment.
Macerate peaches and raspberries with 1 tbsp sugar each. Whisk flour, baking powder, and salt in a bowl.
Beat butter and ½ cup sugar until fluffy. Mix in vanilla, whole egg, and egg yolk. Alternate adding dry ingredients and milk
Spread batter into a thin layer in the pan and bake for 10-12 mins until golden. Cool completely and cut into 3 rectangles.
Mix preserves, evaporated milk, salt, and peaches. Whip cream to stiff peaks, then fold into peach mixture.
Layer ice cream, cake, and raspberries in a plastic-wrapped loaf pan. Freeze 8 hrs. Top with macerated peaches and raspberries before serving.