Repo structure with the new core

[timoML]

The next qudi release will allow to easily write and install custom packages (containing eg. guis, logic, hardware) on top of the bare qudi core. An example is the iqo-modules repo that bundles the toolchains Ulm IQO is using for our solid state color centers research.
The question open for dicussion here is: How to structure the main qudi repos for good maintainability while making it easy for external people to contribute?

As of now, all the hardware interfaces live in the iqo-modules repo. As @alrik-durand pointed out correctly, such a structure might cause fragmentation in the future if people outside the color center community start to create their own packages.
While they might not need iqo logic or toolchains, they inconveniently need to check out (and keep up-to date) the whole iqo-modules stuff with the current repo structure, if they want to use the hardware interface. At the same time there is a huge insenstive - namely standarization - to keep hardware interfaces aligned.

Currently, the IQO maintenance team sees three options to address the issue:

1. Move only the interfaces to a new intermediate "qudi-hw-basics" repo. This could be kept very stable, as their (hopefully good) design makes changes infrequent.
2. Move basic toolchains in seperate "qudi-basics" repo. This would involve interfaces plus the most common guis, logic and dummy hardware.
3. Keep it as it is (interfaces and all the iqo toolchains in "iqo-modules").

Any completly new thoughts or opinions on the above options are highly appreciated!

[alrik-durand]

Hi @timoML, thank you for this thread.

I indeed raised the issue because I think standardization of modules (ie. interfaces) is very important as it is the keystone that enable sharing complex logic and GUI modules.

Regarding the three options you list, I would like to express my personal opinion.

The problems of option 1

The problem with option 1 is that interfaces generally come with other stuff like dummy modules (so we can add them too but it's a slippery slope).
Also there is no point taking just the interfaces and rewriting the logic modules (unless someone really wants to do that).
Finally some generic logic/GUI (ex: PID, Spectrometer, switch) would be left out, opening the question of a way to properly share these modules between labs.

The problems of option 3

In option 3, anyone who wants the basic features needs to keep up with "iqo-modules", which might be confusing for people from other fields who just want a couple of feature like PID and switch GUI. Also it force the developers of "iqo-modules" to be extra careful in the review of the code changes as these might affect people in other labs. Sometimes it's nice to be able to do quick and dirty code and not having everybody pulling this changes !

The benefits of option 2

Option 2, where some generic modules (used across many community) form a "base" repository, offers some nice advantages.

• It allow to get started on new modules while relying of some pre-existing, well structured modules.
• It allows a clear separation between the code following the qudi guidelines, well tested and documented - and the code which we use everyday but does not have the same level of development.

To be exhaustive I should discuss the problem of this option :

The problems of option 2

It adds a layer of qudi which might be confusing, and needs to be well integrated with the rest of the code.
It raises the question of who review the code and validates the PR. I guess this is not a deal-beaker as most open source project manage it.

In the end I would vote in favor of option 2.
Regarding the actual implementation, I'm not sure how it should be done. I'm not very familiar with creating pip packages what it can do.
The end solution also needs to be usable with git to test new branches without too much trouble.

What is a generic module ?

It's obviously hard to define what is generic and what is specific. Maybe giving a single definition is not necessary.
Instead, I've looked at the alphabetic list of current interfaces and ticked the one I think should be included in the "base" repo.

• camera_interface : not very good (my fault it's my code...), will be obselete with the new ScienceCameraInterface (PR for spectrometer toolchain)
• confocal_scanner_interface : currently too much tied to NI hardware to be a generic interface
• data_instream_interface : generic
• data_logger_interface : as the docstring says : "TODO: This interface has no documentation nor dummy module or logic :'("
• fast_counter_interface : generic
• magnet_interface : specific to a hardware (example: ask method) and looks a lot like motor_interface
• odmr_interface : specific to the NV community. I guess an omni-scan like interface would be more generic
• pid_controller_interface : generic and does not need much change
• process_control_interface : generic
• process_interface : generic
• pulser_interface : generic, but needs a dedicated documentation (pulsed/pulser_interface.md is very short right now)
• simple_data_interface : redundant with process_interface and less used.
• simple_laser_interface : generic
• [?] slow_counter_interface : generic but a bit redundant with data_instream_interface on the first impression, I have no opinion on this
• spectrometer_interface : very short and will be obselete with the new ScienceCameraInterface / GratingSpectrometerInterface (PR for spectrometer toolchain)
• switch_interface : generic
• vacuum_pump_interface : A bit too specific ? Anyway there is also that : "This interface, while functional, is not used by any hardware nor logic at the moment."
• [?] wavemeter_interface : As the docstring says : "This interface is very similar in feature with slow counter" maybe too specific while not adding real features compared to other interface. I don't know.

Ongoing interfaces that would end up in the qudi-base :

• ScienceCameraInterface
• GratingSpectrometerInterface

What changes would be needed ?

Instead copy/pasting all chosen interface/modules from the old qudi to the new, it could be interesting to gradually add them if/when they strictly follow a set of guidelines agreed upon.
I suggested the use of an ADR a few years ago here to homogenize the interfaces, this new version can be an opportunity to do so.
This way modules from iqo-modules could be gradually transferred to the base repo when they are ready.

---

Sorry for the long post !
I'm curious to know what other people think about all this.

Alrik

[takuya-ulm]

I understand this issue as follows.

Aim:

• Increase maintainability
• Make easy for 'external' people

To think what 'external' people are, I review the roles of qudi.

Roles of qudi

1. manipulate electrons
2. Manipulate static devices (magnet, CW laser, vacuum_pum)
3. Scan
4. Manipulate EM fields (pulser)
5. Acquire data(fast_counter, camera, spectrometer
6. Process data(data_instream, data_logger, process, simple_data)

They can be roughly classified into the following.

Classifications of roles

• Electronics control
• Data handle
• Scan
• EM spectroscopy

Based on this, I classify the people as

Type of people (what they want to use)

1. IQO (core, logic, interface, hardware)
2. Diamond defects community(core, logic, interface, hardware)
3. Similar experiments with similar devices such as Laser and MW(core, interface, hardware)
4. Different experiments with similar devices such as Laser and MW(core +basics + modified interface)
5. Experiments without similar devices(core + basics)

My idea

As a physicist, I also hope that qudi is as general as possible, but in reality it is very hard to make qudi for all the physicists.
I prioritize the people in the order shown above.
In particular, it is difficult to take care of the modifications of the interfaces (group 4).
So, I would assume that the interfaces cannot be changed so easily.
Then, I prefer the option 2.
Of course, its genericity would be a problem, but personally I like to classify electronics and data process as basics.
In this case, people in group 3 can add what they want in their own repository.
I am not into the idea of separating the interface from the hardware.
There is still a plenty of room for discussion.

[NicolasStaudenmaier]

After the replies of Alrik and Takuya I don't have any real new thought to add. I agree with their opinion that option 2 is probably the best way. When thinking about the content of a "qudi-basics" repository, I would suggest that modules fulfilling one or better all of the following points should be included (at least partially mentioned already above):

• being abstract and versatile, i.e. not tailored to a special task and hardware only,
• being actively used since longer time and running stable,
• being properly documented to allow easy access to everyone,
• not being expected to undergo larger and especially breaking changes in a mid-term time.

I am not able to classify all the modules under these points, but I assume they give space for discussion.

I would furthermore support the idea of creating a centralized documentation (probably somewhere in the "qudi-core" repo) where detailed information to all qudi-module-repositories is listed. This would include all available repositories, not only from Ulm IQO but anyone else contributing to qudi.

[timoML]

Personally, I prefer option 1. I think of the interfaces as some sort of 'hardware abstraction layer'. As such, there is good reasoning to split them from the implementing logic:

• They live on a different time scale since they should change nearly never.
• The abstraction level is completely different.
• For me, they're also not meant to be instructive to newcomers. So we don't have to keep dummy modules or real implementations (which where I would point a new qudi developer to) close to them.

Also, I'm sceptical of splitting the interfaces into "generic" and "non-generic" ones. I think this depends too much on the viewpoint of certain users. At least for now, it's also few enough interfaces to maintain them and keep an overview over all of them in the same place (this might change over time).

[LukePiewalker]

I agree with @timoML on his last thoughts on option 1, especially when it comes to the longevity (hopefully) of the interfaces. However, that would then also mean, there is a third git repo to keep updated and track, right? Or would that be reside in some future "measurement module addon - blueprint" repo then?
I think, keeping the number of repos to track as well as installation steps to a minimum is something worth striving for, considering the average qudi user who is most likely interested in getting started quickly, even if they are not too familiar with the whole "software development" procedure.
Didn't @Neverhorst mention, that one can "bundle" repos together? So that basically each "measurement module addon" is bundeld together with the "interface repo"? So its kind of the "keep it how it is" option fused together with option 1?

One last thought regarding the interfaces in a seperate repo would be, that one/we would need to kick it up a notch regarding their documentation also with respect to expected behavior of the hw methods (block until done, etc ... ). When I was playing around with some of the modules recently, I think I would have been completely lost without the dummy implementation.

[Neverhorst]

In principle managing the dependencies ("interface repository" is automatically installed with qudi-iqo-modules) is easy using pip and PyPI. However it becomes a problem if you want to manage the installation location on your own and have the interfaces editable (pip editable mode -e).
It may be worth to have a look at git submodule but I have never used it and did not dive into the details yet.

[Geegee]

I will address the main points that come to my mind when looking at the different options.

Option 1:
From my point of view it improves the current situation as it further decouples fast changing parts of qudi
from long term stable parts of qudi as pointed out by @timoML . This is in line with the changes already done
by @Neverhorst to extract the qudi core functionality from measurement tool chains.

Option 2:
What I like about this option is that people can just clone the repository and start playing around with qudi.
It also would maybe help with further disussion as suggested by @NicolasStaudenmaier to prune the iqo-modules code base and with this get rid of a lot of unused code that is just noise.

Option 3:
The main drawback I see in the current situation is that the only major repository for measurement modules
is the iqo-modules repository, which contains a lot of noise that may lead new people into the wrong direction which leads them to give up on QuDi out of frustration.
The advantage being that it is the least amount of work.

One more thing, is Option 1 and Option 2 mutually exclusive or could we have both?

[Neverhorst]

Thank you all for the valuable input.
What I can distill from this discussion and personal conversations with developers and users is the following:

1. Some sort of outsourcing the "long-term-stability-code" makes sense from a developer point of view to disentangle it from lab-/experiment-specific qudi addon repositories.
2. However, our target audience are "scientific developers" and so we need to keep the installation of "the qudi basics" (i.e. qudi-core and any additional "basics" repos) in a development environment as easy and sleek as possible. Ideally hiding most of the repo structure and as much "git shenanigans" as possible.

The only way I see to combine these two requirements is a proper deployment (OS dependent executables, e.g. windows installer) of qudi. This deployment would check out (or bundle) all required repositories, create a proper Python runtime environment and install data files like icons and qudi config files etc. in OS specific locations that are known to qudi.
Using solely pip as we currently do limits our options quite drastically since it can only be used properly for distributing Python packages... not for deploying applications.
This is the logical next step for qudi anyways, independent of the isssue discussed here.
However this is no simple task and great care needs to be taken in order to not create more problems than it solves.
A possible (and quite crucial) first step would be to get the data file handling of qudi "in order" (see issue #17)