add support for vpixx pixel mode

[drammock]

No description provided.

[NeuroLaunch]

As written, this approach seems to preclude setting all the pixel bits to low. A simple fix would be to allow an empty bits array. In practice though, I-LABs will for now only be monitoring one of the digital output pins, so just setting one of the other bits to high is sufficient to get the bit of interest to be low.

[drammock]

fixed in cc69b92

[NeuroLaunch]

It might be nice if this and the other sync utility has VPIXX be a user-set option, defined at the top of the script. (Since our lab and others may not always have this hardware online during an AV test.) And consider defining the Vpixx color bit specification(s) at the top as well, for users to customize the test around their physical Vpixx connections.

[drammock]

fixed in cc69b92 and 25bb732

[larsoner]

So the general way of working in expyfun is to set everything that needs to be set, then let the sys config JSON make sure that things (triggering modes, response modes, etc.) get to and from the correct places / devices. As implemented, this code deviates from this pattern by protecting some setting from happening based on a conditional at the top of the file.

What I think would be better would be this set_vpixx_color command not to be protected by an if conditional. Then somewhere else, the vpixx-display-ness is controlled more globally based on whether or not a vpixx projector is connected.

The "expyfun-standard" way to do this I think would be

with ExperimentController(..., vpixx=None):

which means "decide whether or not to display vpixx pixel codes based on system expyfun.json". And end users can set ExperimentController(..., vpixx=True) on their systems to see this pixel appear even if they don't have vpixx.

Then at the end of the day the only thing that changes in this specific test is a single line addition of ec.set_vpixx_color("red"), and we make both ExperimentController and start_stimulus smarter in terms of how they handle vpixx-ness. So concretely the API would be:

1. Add SCREEN_VPIXX=False (or similar name) default to expyfun.json, which can be set to True on systems with vpixx
2. Add ExperimentController(..., vpixx=None), which uses SCREEN_VPIXX when None
3. Add start_stimulus(..., vpixx=None), which uses SCREEN_VPIXX when None

[larsoner]

Typically in expyfun we try to abstract complexity like this (I would have no idea how to switch to blue for example!). Ideally here the API would just allow for set_vpixx_color("r") or set_vpixx_color("red").

If vpixx supports full-spectrum (0-255, 0-255, 0-255) colors, then we should just use matplotlib to convert to int8, then convert this to vpixx internal coding.

If vpixx only supports a subset of colors, I'd prefer:

1. Add aliases for the most common ones (probably red, green, blue, black, white)?
2. Allow passing this list-of-int8-of-length-8 (?) to have more granular control

[NeuroLaunch]

The VPixx DataPixx video and Pixel Mode digital output use the RGB color scheme. 8-bits for each of the Red, Blue, Green. I agree with Dan (below) about abstracting the colors for Expyfun scripts, because in practice our lab would ever use only a subset of the digital output pins and we want exact control over those pins because they will be tied to digital inputs of the Megin or other acquisition system.

[larsoner]

So the general way of working in expyfun is to set everything that needs to be set, then let the sys config JSON make sure that things (triggering modes, response modes, etc.) get to and from the correct places / devices. As implemented, this code deviates from this pattern by protecting some setting from happening based on a conditional at the top of the file.

What I think would be better would be this set_vpixx_color command not to be protected by an if conditional. Then somewhere else, the vpixx-display-ness is controlled more globally based on whether or not a vpixx projector is connected.

The "expyfun-standard" way to do this I think would be

with ExperimentController(..., vpixx=None):

which means "decide whether or not to display vpixx pixel codes based on system expyfun.json". And end users can set ExperimentController(..., vpixx=True) on their systems to see this pixel appear even if they don't have vpixx.

Then at the end of the day the only thing that changes in this specific test is a single line addition of ec.set_vpixx_color("red"), and we make both ExperimentController and start_stimulus smarter in terms of how they handle vpixx-ness. So concretely the API would be:

1. Add SCREEN_VPIXX=False (or similar name) default to expyfun.json, which can be set to True on systems with vpixx
2. Add ExperimentController(..., vpixx=None), which uses SCREEN_VPIXX when None
3. Add start_stimulus(..., vpixx=None), which uses SCREEN_VPIXX when None

[larsoner]

... and if we do switch to this new API, I think the default behavior should be for len(bits) to be zero, i.e., don't draw anything unless the user has set_vpixx_color (and don't raise an error).

[drammock]

All good points @larsoner.

What I think would be better would be this set_vpixx_color command not to be protected by an if conditional.

Strictly speaking the conditional is unnecessary, it's just that setting the color has no effect if vpixx=False is used in the subsequent flip / start_stimulus calls.

I hesitated to do machine-config JSON for vpixx because I thought it would likely be experiment-specific (auditory paradigms might not want/care about super-precise flip times), which isn't usually the case for things like screen size, audio controller, etc. But I don't have a strong objection to it, so if you think it makes sense I can add a config.

Ideally here the API would just allow for set_vpixx_color("r")

Since the point of vpixx is (ab)using pixel color to send binary triggers, I'm assuming YAGNI on letting users specify colors as e.g. "r". My assumption is that users will know what bits they want to send, and will work backwards from there... so passing a sequence of bits is in a sense the least work (even though it's more typing) and passing a tuple of ints in [0,255] is slightly more work (and is a possible source of errors) but is less typing. Users going further and converting their sequence-of-bits into tuple-of-floats or strings just seems pointless 🤷🏻. So I lean YAGNI on specifying as a string or tuple of floats, but I can see the case for accepting an (R,G,B) tuple of ints in [0,255] instead of/in addition to a sequence of bit indices. Does that change your mind at all?

[larsoner]

I hesitated to do machine-config JSON for vpixx because I thought it would likely be experiment-specific (auditory paradigms might not want/care about super-precise flip times), which isn't usually the case for things like screen size, audio controller, etc. But I don't have a strong objection to it, so if you think it makes sense I can add a config.

In what I'm imagining, the experiment-specific part I think is whether set_vpixx_color is called at all (and when). If it's not called, like in existing / auditory experiments, then nothing will be drawn for those experiments even with a machine that's capable of vpixx-pixel-based "stamping" (because the default color should be nothing / empty / no-op).

My assumption is that users will know what bits they want to send, and will work backwards from there... so passing a sequence of bits is in a sense the least work

Oh okay I think I get it now... the experimenter shouldn't care about the color but rather the bits encoded. In your sync_test case, you are passing list(range(8)) because this happens to be red. But in a real experiment you might encode for example just [5] for trial type 5, which will map to some color but we don't care in practice what that color actually is. And the researcher should think about having 24 such binary values they could set, so if they have more than 24 different trial types, start thinking in terms of some binary code, etc.

In that case I wouldn't call it set_vpixx_color but rather set_vpixx_id or similar. The fact it maps to a color and a pixel on screen isn't the important part: that translation is done by expyfun and then undone by the projector automagically. The experimenter should be thinking and operating in terms of stamping a list of unique ints between 0 and 23 (inclusive).

... thinking about this use case actually I wonder if instead this should "just" be a part of identify_trial(..., vpixx_id=[...]), i.e., we only stamp a vpixx ID at the same time as we start the trial / do the initial flip with start_stimulus. And then if someone wants to stamp something with vpixx outside of that, we have a ec.stamp_vpixx(id_: list of int) that just immediately draws the ID in the pixel.

[larsoner]

... in other words, make the behavior more closely mirror ec.identify_trial and ec.stamp_triggers which is how we currently handle trial ID and TTL stamping

[drammock]

... in other words, make the behavior more closely mirror ec.identify_trial and ec.stamp_triggers which is how we currently handle trial ID and TTL stamping

I think I agree with this; at least with the idea that this should be exposed in the API as an ID and not a color. My hesitation is that the vpixx ID is inextricably tied to a flip event, and trial:flip_event is not 1-to-1:

• trials don't always start with flip (ec.start_stimulus(..., flip=False))
• there can be multiple flips per logical trial, and users might want multiple vpixx IDs for different flips within a single trial (think of RSVP paradigms, or visual stim on/off)
• ec.identify_trial(..., vpixx_id=[...]) sort of implies that there is / should be only one ID per trial

So I think the correct analogy is only to stamp_triggers (with the caveat that it's really more like a "call on next flip" function in terms of its timing)

In what I'm imagining, the experiment-specific part I think is whether set_vpixx_color is called at all (and when)

I'm actually leaning now toward eliminating the set_vpixx_color/id entirely, and passing the ID directly into the flip function. I.e., you can view what is experiment-specific as "whether a vpixx pixel gets drawn on any particular flip". So an API like flip(..., vpixx_id=()), and an edge case where start_stimulus(..., vpixx_id=(1,0,0), flip=False) could warn/raise to avoid a footgun (since an ID was passed but it won't show up until the next flip, possibly much later than stim start). With that API, it could still make sense to have a machine config; two possibilities are:

1. "should I ignore or honor all requests to draw a vpixx pixel", or
2. "how big should the vpixx pixel be" (0=disabled, 1=normal, >1=useful for debugging to make sure it shows up, without having to squint)

if we do that, then yeah, that should be settable as ExperimentController(..., use_vpixx) or ExperimentController(..., vpixx_size) or whatever

[larsoner]

ec.identify_trial(..., vpixx_id=[...]) sort of implies that there is / should be only one ID per trial
...
there can be multiple flips per logical trial, and users might want multiple vpixx IDs for different flips within a single trial (think of RSVP paradigms, or visual stim on/off)

So I think the correct analogy is only to stamp_triggers ...

I think this is the crux of how we see this differently. I'm not convinced about the first part -- you can want to stamp additional TTL triggers at arbitrary times with stamp_triggers, and yet we pass identify_trial(ttl_id=[...]) not start_stimulus(..., ttl_id=[...]). There are technical reasons for this separation, too, but so far trial identification been unified in the identify_trial function so far for three different modes of trial ID (ec_id, ttl_id, and el_id). (Side note: for ttl_id, this stamping/ID will also be deferred in the case of sound-card-based stamping, so there is precedence for this delayed "actual stamping" happening.)

I expect the most common anticipated use case of vpixx is that the user will want to use it to 1) identify a trial start of a paradigm that starts with a visual flip, and they might have a secondary goal of 2) "stamping" something later using vpixx. So I don't see the best analogy being with just stamp_triggers, but more closely aligned to (almost always) identify_trial + (less often) stamp_triggers, at least in a majority of expected use cases. But maybe I'm wrong about how people will use this!

trials don't always start with flip (ec.start_stimulus(..., flip=False))

Yeah this is true. While I think it's going to be pretty rare for visual paradigms that use vpixx, in my API proposal:

with ExperimentController(...):
    ...
    ec.identify_trial(ttl_id=[0, 1], vpixx_id=[5])
    ...
    ec.start_stimulus()

accidentally calling ec.start_stimulus(flip=False) could (and should) be coded to raise an error, just like in your new flip API proposal. So both proposals can safeguard against this mistake equally I think.

In the flip API proposal, this code would become:

with ExperimentController(...):
    ...
    ec.identify_trial(ttl_id=[0, 1])
    ...
    ec.start_stimulus(vpixx_id=[5])

Which is okay and workable, but (to me) slightly less clean conceptually consistent because now trial identification is done in two places instead of one. I can live with it if you're not convinced by the conceptual argument, though!

A hybrid option would be to allow both modes, essentially. Allow passing a vpixx_id in identify_trial, which will then be the value used when start_stimulus(..., vpixx_id=None) is called. As above, if vpixx_id is set and flip=False, it raises an error. If it was set in identify_trial and is set in start_stimulus, it also raises an error. Then either form of the code above works.

Either way, for non-stimulus-start ID vpixx-ids, I think flip(..., vpixx_id=[...]) works. While drawing any visual correctly is ultimately tied to doing it before a flip (which initially made me lean slightly toward something like .draw_vpixx(id_=[...]), the fact that it has to be done last and probably won't have any parameters needed other than the ID itself suggests adding it to flip is okay (and might prevent people from drawing it incorrectly / accidentally drawing something on top for example).

[drammock]

I can live with this:

with ExperimentController(..., use_vpixx=None) as ec:
    # None falls back to machine config  ↑↑↑↑
    ...
    ec.identify_trial(..., vpixx_id=(R,G,B))
    # ↑ silently does nothing if use_vpixx resolves to False
    ...
    ec.start_stimulus()
    # ↑ raises error if flip=False and vpixx_id isn't None
    #   (unless use_vpixx resolves to False? or even then?)
    ...
    ec.flip(vpixx_id=(R,G,B))
    # ↑ silently does nothing if use_vpixx resolves to False
    ec.trial_ok()

WDYT about the use_vpixx vs vpixx_size as the machine-level config? in my experience developing this, I set the pixel size to 10 or 100 to make it much easier to verify it was coming on / shutting off when expected... now that it works maybe YAGNI? I doubt users will ever need it at experiment time... do you think it's worth having an easy-to-twiddle knob for debugging purposes?

[larsoner]

In this case:

    ec.start_stimulus()
    # ↑ raises error if flip=False and vpixx_id isn't None
    #   (unless use_vpixx resolves to False? or even then?)

we want development on non-vpixx machines and vpixx machines to mirror one another as much as possible. If it raises an error on one it should on the other if possible, so it should raise an if you identify_trial(..., vpixx_id=[...]) but then start_stimulus(flip=False) regardless of whether or not that vpixx pixel is actually drawn to the buffer or not.

One could argue that the vpixx pixel should always be drawn if you set vpixx_id... no real harm when developing on a non-vpixx machine and arguably more accurately represents what will be done... so maybe we don't even need a use_vpixx and machine config at all? (When developing eyelink-based experiments, the calibration blocks still run and display etc. so I guess we could think of vpixx as being similar to that.)

In any case, if we do keep machine config stuff, I think SCREEN_VPIX=True | int makes sense, which can then be passed as ExperimentController(..., use_vpixx=bool | int) (using whatever names make sense for both). I agree passing int is nice for debugging and trivial to implement because bool(True) == 1.

[NeuroLaunch]

[drammock] I'm actually leaning now toward eliminating the set_vpixx_color/id entirely, and passing the ID directly into the flip function

This occurred to me as I was programming the dMEG protocol to use the Vpixx feature. Apologies for being lab-specific here in the pull request section. But in practice, I don't see a use case for our lab using this feature outside of wanting to strictly know the exact timing of a particular image flip (or really in most cases every image flip). If there are a lot of different types of images, then the VPixx has the additional advantage of inherently "stamping" them up to 24 bits. But for Expyfun scripts, I think we'd still be using our own stamping via the TDT or parallel port outputs, regardless of what the VPixx is doing.

In the dMEG experiment, for example, the stamping is still done using STI001 - 003. To capture the visual timing, I define a VON = [0] and VOFF = [1] state for the vpixx pixels (with [0] the single bit we are physically connected to that gets sent to STI004). Before the start_stimulus() command that generates the image of interest I have to give a set_vpixx_color(VON) command; and the subsequent flip() that turns if off is preceded by set_vpixx_color(VOFF). This dual line coding doesn't seem efficient, so I agree with Dan to set the color inside the flip call directly.

there can be multiple flips per logical trial, and users might want multiple vpixx IDs for different flips within a single trial (think of RSVP paradigms, or visual stim on/off)

Absolutely true. In fact, if you don't pull the monitored pixel back to some background state, the pixels could just remain at their set color for the entire experiment after the first flip. Even worse, if something else overdraws that pixel, like a new background color, the pixel will be at the mercy of the new final image. Thus the on/off states I defined for dMEG. So, basically if the VPixx is being used, there should be an expectation that every flip() will be setting the monitored pixel.

So that's not to say that every flip() command in an Expyfun script needs to have the pixel id specified. If the vpixx_id bit code isn't set for a particular flip, I think there are two reasonable options for handling this: 1) The pixel is nevertheless set to the last one used (continue to store it as an attribute), which makes sure the pixel is controlled; or 2) The pixel isn't set, and the pixel's value after the flip is at the mercy of whatever was just drawn to the buffer. This last one is what I did in dMEG for some of the small central cross images, where I knew the background wouldn't change. #1 is obviously the safer route; with #2, user beware.

[larsoner] One could argue that the vpixx pixel should always be drawn if you set vpixx_id... no real harm when developing on a non-vpixx machine and arguably more accurately represents what will be done... so maybe we don't even need a use_vpixx and machine config at all?

That sounds right to me. If you're not physically pulling out the digital info from the DataPixx port anyway, nothing is lost or gained by continuing to manipulate the upper-left pixel.