Andrew, thanks for the quick and thorough reply.
I’m going to keep investigating this some more myself to try and understand what’s going on and verify the rest of the system. It just *feels* like a Queued sweep bug since any time I go above 1 (basically queueing at all) it starts to happen. But feelings aren’t fact, so we’ll do what we can to finish narrowing down the issue.
– How are you switching between your sources? Are you using the smSetSweepGPIO function?
— No. We have the SM200’s set to produce a waveform (go low when scanning, transition high when near the end of the sweep) and that transition to high then tells the FPGA to move to the next position.
– If yes to the question above, have you done any monitoring of those GPIO pins?
— We monitor counters in the FPGA for that rising edge, and they always seem to line up…but then again, when you’re doing something at multiple kHz, and there’s only ever an off-by-one bug it would be a needle in a haystack.
We don’t think that we are missing triggers because otherwise it wouldn’t go back! We know if we have a bad trigger line because then what was in slot 10 goes to 9 and then to 8 and then to 7, and so on. But this corrects itself! Which is just odd.
– Any way to verify they are cycling through all your states and not missing one periodically? Whether it’s hardware/software, that would be extremely helpful to know.
— Yea, that is one of our concerns. I don’t know why it would go away when we set the queue to one, unless we were for example missing a trigger and then catching up somehow? But I don’t know how the FPGA would do that since we’re just looking for an edge transition on a pin. One of the avenues we are investigating on our end. But again, it only happens when we queue sweeps…one thought would be that the waveform / time the GPIO is high before the next sweep happens is longer when we’re not queuing, so we “catch” it more regularly…but then again, why would it correct itself!?! It would process moving bins in that instance, I would think.
– Am I to understand that the “sweep offset” sticks around for many cycles? So if you are cycling through all 20 slots, are all 20 slots now off by 1 for some duration until it corrects itself? Or is it just slot 10 that shows up in slot 9 for several seconds until slot 9 reverts?
— Yes, we really have a couple hundred “slots” that we cycle through. And we kind of see the whole thing shift one left, and hold there for a bit and then shift one right, back to the original and desired location. Then it’ll go back and forth doing that. Really odd, honestly.
– Does using just 1 receiver make the problem go away? Even with 16 queued sweeps?
— Our hardware and software architecture is very very intertwined with having two SM200’s working in concert with each other. I think we might be able to hack some places in the code to reduce it down to one signal hound, but only for a little bit before buffers and other things start overflowing and getting too far out of sync elsewhere. Otherwise it’d be a huge effort to get it down to just one.
– Is it always a specific slot, e.g. 9/10? Or random?
— It appears to be all the slots kind of shifting by one. Almost like we started getting scans out of order for a bit or something. It seems more like the problem would be elsewhere. But it is 100% solved when we queue only 1 scan, and is apparent when we go for more than one scan. I honestly haven’t been able to come up with a theory as to why / how the queued/non-queued is at play here. But as far as we can tell, it is.
– Does the “sweep offset” happen on both receivers? Or just one?
— It does happen to both. We are looking into that more this rest of this week. We had an issue preventing us from looking at both and seeing if they do it at the same time, or if it’s a random correlation between the two.
– Let’s say you’re switching through slots 8,9,10,11. Does it now look like 8,10,10,11 for 15 seconds and then go back to 8,9,10,11?
It looks like 1,2,3,4,5,6,7,8,9 goes to 2,3,4,5,6,7,8,? for a while and then back to 1->9 in order. We’re investigating whether the ? is just old data in there, or if it’s 1 from the previous slot. It’s actually fairly hard for us to put energy *only* in one slot with our current setup, but figuring this out I think might provide a lot of insight, and that’s what we are attempting today.
It’s a tough nut to crack, because the entire system is orchestrating this at multiple kHz, and there’s a lot at play here, and we’ve tried a ton of things across all the other components, both hardware and software at play here, and it wasn’t until we only queued 1 sweep up did anything fix it.
With queued sweeps, is there maybe some extra allocation / deallocation (and thus garbage collection or other process) going on under the covers in the driver that’s maybe going on here?
