Michael,
You will have to change the algorithm slightly.
I would continue to sum everything as linear units, and then divide through by the window bandwidth which can now be calculated with the following equation.
nWindowBW = RBW / BinSize;
Where RBW is what you asked for, and bin size is the frequency spacing between the sweep bins.
This should get rid of most of the variables used to calculate nWindowBW in your above example.
In the past we used a variable bandwidth flattop window, but now we use the standard flattop window with zero-padding to achieve arbitrary RBWs. This is a more traditional approach to arbitrary RBW selection.
Andrew
