instruments in separate repo?

[guenp]

Not sure if this has been discussed before - but perhaps it's an idea to have a separate repo for instrument drivers?
This will keep the drivers nice and separate from the core code. Also, this will enable folks who use their own measurement code to use drivers & contribute. I like the idea of having universal instrument driver code which is completely usable on its own (without Qcodes dependencies).

See also: https://github.com/Galvant/InstrumentKit

[alexcjohnson]

Right, we did talk about this a while ago. It may make sense down the line. Right now I think it would be a pain, as it's nice to be able to simultaneously refactor the core and drivers, but we shouldn't wait too long if we're going to do this, lest interdependencies creep in that we're not even aware of.

I'm not sure about the goal of using driver code without qcodes dependencies though - there's a good deal of helper functionality that both drivers and core make use of. But folks who want to use qcodes drivers with their own measurements can still import them, right? qcodes is pretty light, and doesn't do anything just because you imported it.

[guenp]

If qcodes isn't supposed to do anything if you import a subclass, this should be incorporated in the manifesto, as well as in the guidelines and tests for driver creation.

[guenp]

@AdriaanRol regarding the python meeting at APS next week, I think this may become relevant if we want to collaborate with these folks :) I'll try to join on skype or hangouts, @alexcjohnson perhaps you'd also like to join if it's at a reasonable hour?

[alexcjohnson]

Even at an unreasonable hour I might join, but I get the impression it may be more cocktail hour than sitdown meeting, in which case skyping in wouldn't accomplish much. I'll let @AdriaanRol and @hsuominen represent qcodes and figure out if it makes sense to add me/us in remotely.

[alexcjohnson]

@guenp

If qcodes isn't supposed to do anything if you import a subclass, this should be incorporated in the manifesto, as well as in the guidelines and tests for driver creation.

I guess so... though packages that execute some active code just by being imported seem to me more the exception than the rule, and I'd rather if the onus were on THEM to say what they're doing - though per explicit is better than implicit I guess it wouldn't hurt to spell this out. You want to take a stab at incorporating it in the right places?

The poster child for this may be matplotib, that forced the ugly in_notebook hack so we don't import it in subprocesses. Though I guess based on this discussion we shouldn't even use that hack (except for importing notebook widgets, of course), we should just import matplotlib or pyqtgraph locally within the MatPlot and QtPlot constructors, and defer the error on missing packages until that point. I'll try that out once I get this instrument process fix #53 finished (almost there...)

[guenp]

@alexcjohnson So the meeting turned out to be a sit-down thing, I joined on hangouts. Most seemed in favor of a shared instrument repo. @AdriaanRol was not in favor, he said everyone should eventually converge to the same framework anyway (I'm guessing you meant this should be QCodes...? :)). The other person who wasn't in favor was worried we might end up with several versions of the same driver - however, as the end goal is to have it all in 1 place (this repo), all collaborating frameworks would depend on this repo, using their own adapter/api or something rather as interface to the instrument drivers. @hsuominen also mentioned such a repo would be useful for people who need to write drivers in different platforms/frameworks, as means of a 'copy-paste' manual.

Sharing a common convention for instrument drivers will only improve the quality for all repos out there. There's a huge amount of work to be done converting all drivers to the same standard (Qcodes or a shared one), so we might as well share that responsibility with the community. In other words, if we take this opportunity now, we'll essentially have a whole group of talented physicists/programmers out there who can help write drivers that will be compatible with Qcodes. The goal would be to make this a package with documentation, instrument testing and so on. Moreover, the more people are affiliated with the community, the more the code will spread into the physics community, such that people will have an easier time finding drivers and frameworks for their experiment and won't have to write their own (which obviously a lot of people are doing right now..). If we truly want to achieve a common standard for data acq software in python, instrument drivers would be the place to start.

@hsuominen @AdriaanRol did you have any other comments?

[AdriaanRol]

@guenp
Hi Guen, thanks for taking the time to write this down. I am a bit concerned that your report does not accurately reflect my position on this so I will attempt to clarify it.

I am very much in favor of collaborating on instrument drivers, I also agree that we need to involve as much of the community in this as we can. I do not however think that the proposed shared instrument repository with "adapters" is a good idea. Let me explain why

1. DRY,
   Having multiple copies of the drivers circulating in different repositories is a bad idea (DRY principle), an alternative that was proposed during the meeting (I think by @hsuominen ) was to have a central repository that contains references to the most up to date repositories where discussion can be concentrated and examples for the different frameworks can be found. I think having something like that can be a good start for collaboration, and I am not against that. I am also all in favor of giving anyone who wants access to our instrument drivers.
2. We already have a good standard
   @guenp what do you envision such a shared instrument repository to grow out into? It is unclear to me what you envision this repo to be if it would be succesful. However I can explain you how I would envision it. As you propose we should converge on some minimal standard of what an instrument driver is and add test suites, our framework specific instrument drivers should then talk to these "minimal" drivers. Suppose we converge on such a standard, at that point we will probably start abstracting these core features in some instrument class (in fact most frameworks have a very similar abstraction), the "framework specific" drivers should then be minimal wrappers around this. The drivers themselves are already supposed to be minimal wrappers around the VISA commands, I think adding in this other layer is not an idea that will pick up.
   The way I interpret this plan is to have a shared instrument driver where we converge on a minimal standard on how to do it. I think that QCodes instruments already satisfy this criterium. In fact I am basically using QCodes as a minimal provider of instrument drivers and do not see the need for "yet another standard" or a major modification of that.
3. We should concentrate efforts and not fragment them.
   About a year ago we met with the goal of seeing how we could collaborate and what a new standard in measurement software would look like. I think we should focus on bringing people in on this effort and focus on making the best possible framework there is, possibly improving our current classes to be a good standard is a much better approach than trying to come up with yet another standard.

Sorry for this long rant but I think QCodes and the way Instrument drivers work is completely fine, I just don't understand what you are trying to achieve Guen.

[guenp]

@AdriaanRol thanks for the comments! 1. this will only be the case in the beginning, in the end all frameworks will have no instrument drivers in the source anymore 2. I agree, I propose to start off with suggesting the Qcodes standard and see how the community responds, if it truly is the best possible we're as good as there 3. this is an effort to concentrate them - unfortunately, you can't just expect people to switch to your framework, they have put in work of their own & are used to it. by starting to merge a small part (instruments) you get them involved & working together. my end goal is to have the community work on up-to-date drivers together so we don't have to reinvent the wheel (also, Qcodes currently doesn't have a very large instrument database, so if we get a common standard that is based on the Qcodes vision we essentially have a bunch of folks writing drivers for us!).
Hope this is clear, I'm happy to talk about this on skype/hangouts of you still strongly disagree.

[AdriaanRol]

@guenp ,
Hi Guen,
I think we agree on far more than we both care to admit. I am all pro sharing and involving people, I just have some practical concerns and do not fully see the great advantages of refactoring and splitting out the drivers from the core of QCodes as I think it is already possible to use only the instrument drivers (which is basically how I am using QCodes at this moment).

unfortunately, you can't just expect people to switch to your framework,

Don't worry, I am not expecting that.