Preheat oven to 325°F. Spray a round 2 quart baking dish with cooking spray.
Whisk eggs, half and half, vanilla, 1 cup of the brown sugar, cinnamon, nutmeg and salt in a large bowl until well blended.
Combine butter, remaining 1/3 cup brown sugar and pecans in a small bowl until combined.
Sprinkle 1/3 of the brown-sugar mixture on bottom of the prepared pan. Top with 1/2 of the toasted pound cake and banana; repeat layers.
Pour egg mixture evenly over the cake. Press down the cake cubes with the back of a spoon to cover; let stand 10 minutes. Sprinkle with remaining brown sugar mixture.
Cover the baking dish with foil. Bake 30 minutes. Remove cover and continue baking 30 minutes or until no liquid egg remains. Let cool for 10 minutes before serving. Garnish with whipped cream and a sprinkle of nutmeg, if desired.