ABorel,
Based on your comments, I think you want to be using the digital modulation mode rather than ARB. You are specifying symbols rather than I/Q samples. To clarify, it’s common to oversample the constellation symbols by upsampling and filtering with an ISI free filter like the Nyquist filters. If you had done all of the upsampling and filtering, then you would want to use ARB mode.
What I would try instead, is feeding your integer sequence into the sequence editor in digital mod mode. You can either adjust your sequence to match the constellation mapping, or create your own custom constellation mapping to match the one you have been using. Then the software will upsample and filter for you.
Without upsampling and filtering, you are creating sharp transitions, which will be filtered out by both the VSG software digital filters and hardware reconstruction filter which have ~40MHz of bandwidth. I suspect the smoothing effect is from these filters.
Let me know if you have follow up questions.
