Reorganize Spyder repository

[spyder-bot]

From pierre.raybaut on 2013-03-17T03:48:24Z

If I had to rewrite Spyder from scratch, I would probably start with the following outline:

spyder/__init__.py
spyder/conf.py
spyder/app/spyder.py (and other modules)
spyder/lib/baseconfig.py
spyder/lib/utils/...
spyder/lib/widgets/...

And of course it's not too late to do well...

See also those comments (motivation): https://code.google.com/p/spyderlib/source/detail?r=82bbf24ae1546470a09170c04743c28e585945ee

Original issue: http://code.google.com/p/spyderlib/issues/detail?id=1320

[spyder-bot]

From pierre.raybaut on 2013-03-17T01:50:55Z

Blocking: spyderlib:944

[spyder-bot]

From pierre.raybaut on 2013-05-11T05:56:37Z

@carlos, @-Jed:
What do you think about this?

The problem is that it may take some time and it could be a real mess to merge things from a repository based on the old structure. Is it worth it?

Cc: ccordoba12 [email protected]

[spyder-bot]

From ccordoba12 on 2013-05-11T06:32:32Z

I like it, I think it's a cleaner and better organization to what we currently have.

Most probably things would need to be merged manually from one repo to another because of this. But what if you did the reorganization after 2.2.1? Let's say we release it by end of June and after that we'd merge only critical bugs in 2.2 (because 2.3 would be a couple of months away). That would make things more feasible and easier for all too.

[spyder-bot]

From [email protected] on 2013-05-11T09:16:09Z

I'd be in favor of some small changes to the package structure, but I'm struggling to see the value of reorganizing everything.

I think the primary point of my review comments as linked in the original post was to reorganize just a few things to make the dependency graph feel a bit less clumsy. Maybe the simplest example is in spyderlib.init.py here: https://code.google.com/p/spyderlib/source/browse/spyderlib/__init__.py#53 It feels clumsy for get_version to do "import spyderlib" We're already inside the init file that would get called as a result of this statement. The main target application lives in spyder.py with is inside the spyderlib package, and it just doesn't feel like the roll of spyder.py versus spyderlib.init.py is very clear right now. It would seem like, say, the Qt API selection code really belongs in spyderlib.init.py, since basically everything in spyderlib would need to respect that API selection. I think this type of clarity can be achieved without having to resort to major package structure changes.

[spyder-bot]

From pierre.raybaut on 2013-05-12T07:09:58Z

Blocking: -spyderlib:944

[spyder-bot]

From [email protected] on 2013-07-05T04:56:27Z

spyder/init.py
...
spyder/lib/baseconfig.py

If you have init.py at the top, then lib/ is unnecessary complication.

spyder/conf.py

Storing default configuration in .py file is unnatural - you lose the ability to configure and inspect Spyder config with external/system/administrative tools.

spyder/app/spyder.py (and other modules)

spyder/main.py

[spyder-bot]

From [email protected] on 2013-07-05T05:01:24Z

jed> the Qt API selection code really belongs in spyderlib.init.py,
jed> since basically everything in spyderlib would need to respect
jed> that API selection

There are some Spyder IDE utils that are independent of Qt, and loosing ability to run them without Qt installed (clean PYTHONPATH) or without heavy Qt import is a (major) disadvantage.

[spyder-bot]

From ccordoba12 on 2013-07-12T18:16:11Z

Pierre, I don't think this is going to happen for 2.3. Maybe in a future 3.0?

[spyder-bot]

From pierre.raybaut on 2013-07-13T05:14:36Z

I agree, there's no rush.

[spyder-bot]

From ccordoba12 on 2013-07-13T07:20:44Z

Ok :)

Labels: -MS-v2.3 MS-v2.4

[spyder-bot]

From [email protected] on 2013-09-09T07:24:23Z

Cc: [email protected]

[spyder-bot]

From [email protected] on 2015-01-30T03:21:44Z

I know this is on hold since... well a year and a half, but I still see value in reorganizing the structure.

Also I guess Git would be more "flexible" where the transitions are done?

External

external-py2 and external-py3 could be also reorganized... maybe becoming a namespace package having

external/py2
external/py3
external/init

Some packages that work without change could be left at the external dir level. Right now a external package has to be copied to both external-py2 and external-py3 even if it is the exact same thing.

Plugins

With plugins (autodiscovery, management and installation) on the way the repo would get rid of all the things inside spyderplugins, so at least that would get better. They could (should) also become namespace packages, that would allow easy autodiscovery as any installed plugin would always be in the spyderplugins namespace.

Should plugins be able to ship with external dependencies as well?

Random remark

Not saying we need something similar in spyder, but how Colin organized his cobol ide in a (M)VC seemed very interesting (and clean) to me and helps keep files short and to the point. https://github.com/OpenCobolIDE/OpenCobolIDE/tree/master/open_cobol_ide Cheers

[spyder-bot]

From [email protected] on 2015-01-30T03:27:50Z

Also I forgot, I am +1 on moving conf.py to another format.

Maybe *.ini, *.yaml?

spyder/conf/base.ini
spyder/conf/linux.ini / specific to platform
spyder/conf/win.ini / specific to platform
spyder/conf/osx.ini / specific to platform

?

[spyder-bot]

From ccordoba12 on 2015-01-30T15:39:28Z

My comments:

External

They probably be replaced by git submodules

Plugins

spyderplugins is not going to go away. We are just going to make it a namespace package (which we need to to make other plugins pip-installable).

We'll have to think about external deps.

Random remark

I don't agree with rewriting code just for the sake of rewriting it. If it can help us to solve present and fundamental shortcomings, I'm fine with it, else I won't bother.

I think code is just a medium to create an application, not its main object. We have to be aware that only a handful of people reads Spyder source code, compared to the thousands that use it daily. So our first priority should be to provide a platform that works intuitively and as people expects, not constantly rewriting our internal APIs to please ourselves with the most beautiful design.

I like to see good code as much as any other developer, but I can live with "good enough" code, if that gives me time to solve bugs and give people a stable program.

Config

I don't see any benefit in using ini or yaml files, instead of our current setup, which is working just fine. With ini or yaml we'd loose the benefit of using Python to programmatically generate and update our config. Or we'd have to write a layer to read ini/yaml and load it in a dictionary, so we can process it with Python, and then what would be the benefit?

The only thing I can think of is that ini/yaml are more readable than a dict. But that's not enough reason to touch a critical part of our infrastructure, that both Pierre and me tried really hard to make it as stable as possible (and unfortunately, we still get fail reports from time to time :-)

In a few words, if it's not broken we must not "fix" it. We really need to focus in solving problems that people are expecting for us to do (e.g. debugging, integrating the IPython notebook, making the Variable Explorer to inspect more objects, etc) instead of wasting time discussing this kind of things.

That's what I think about it :-)

Owner: ccordoba12
Cc: -ccordoba12
Labels: Cat-Miscelleneous

[spyder-bot]

From [email protected] on 2015-01-31T02:27:10Z

External

ok

Plugins

I meant away from the main repo

Random remark

Not for the sake of it, but to make it easier to dive into, I understand the comfort of having a lot of stuff in a single place, but for instance the editor is just bloated with stuff and hacks (python like etc...) it would be better to split it in more manageable/maintainable chunks.

I also agree in having a good enough code, but if making some changes would make it easier to contribute then it also should be consider into it, I would like for spyder to have more than just a handful of code readers

Config

Ok

Cheers

[spyder-bot]

From ccordoba12 on 2015-02-01T06:47:33Z

Plugins

I don't see the need to move our current spyderplugins (like pylint and breakpoints) away from the main repo because nobody else is maintaining them.

Random remark

Ok, in that case I think you should explain better what's the problem you're trying to solve with your proposals, else it sounds like pure redesign :-)

It's a good idea to make our editor more general and able to understand and handle other languages besides Python. I won't say codeeditor is full of hacks, it's just focused on editing Python to the exclusion of other languages.

I don't have time to work on it, but if you want to tackle it, please go ahead and do it.

[spyder-bot]

From [email protected] on 2015-02-01T07:15:42Z

Plugins

There is no strict need to do it, but it would make a more explicit separation between what is core and what is not. some plugins can be maintained in the same github repo ( https://github.com/spyder-ide )

Random remark

I do not have time right now (eventually I will), but I do have a clear idea of how to split it to make it more general and manageable so I will do some testing.

Cheers

[spyder-bot]

From ccordoba12 on 2015-02-01T08:08:22Z

Plugins

It's just more work for no gain :-) Breakpoints, pylint and profiler are not third-partyn plugins, because they have been distributed with Spyedr since a long time ago. It's just that their authors preferred to develop them in spyderplugins.

I think the contents of spyderplugins are irrelevant. The only important thing is that future authors can install there with pip or conda.

Random remark

Ok

[goanpeca]

@ccordoba12 is there a layout for the new repo organization, so we can have a "talk" on that??

Also I would like to move the "third party plugins" to normal spyder plugins (since you say they are part of the core of spyder), which would allow us to remove the spyderplugins folder inside the repo (and avoid in install)

Now that we are reworking how plugins are discovered, this would be the moment to unify these "core plugins developed as 3rd party ones", and refactor code that does plugin discovery

[goanpeca]

@ccordoba12 ?

[ccordoba12]

The layout is shown in the first comment of this issue. That one is what I intend to follow :-)

spyderplugins can't be removed (we've been over this point several times already). We need it to install 3rd party plugins with pip (which is one of your demands).

Besides, the plugins that live there (pylint, etc) are examples for other people to understand how to build 3rd party plugins for Spyder.

[goanpeca]

Of course spyderplugins can be removed (and should :) ), and for people to see how to make a plugin we will make documentation precisely on that.

I will make a PR on this later on, I believe you still do not understand what I mean....

Regarding the layout... this thread is a bit old, maybe it would be wise to ask the others, for sure they might have suggestions on the structure,

@Nodd , @SylvainCorlay , @blink1073

spyder/__init__.py
spyder/conf.py
spyder/app/spyder.py (and other modules)
spyder/lib/baseconfig.py
spyder/lib/utils/...
spyder/lib/widgets/...

[blink1073]

@goanpeca, I'm fine with that layout.

[ccordoba12]

@goanpeca, the new layout just adds a bit more clarity to how Spyder is organized internally, that's all. I don't think we should worry about it too much.

Of course spyderplugins can be removed (and should :) )

I don't agree with this proposal on (very) pragmatic grounds:

1. Pylint and the other plugins are working perfectly fine. There are very few bugs open about them.
2. Those plugins are not tightly coupled with our other plugins, so I don't see the need to make them first class plugins.
3. IMHO, this is just a proposal for correctness and nothing else. It doesn't help to solve one single bug or improve functionality in one bit. Again, my motto is: "if it's not broken, please don't fix it". There are plenty of bugs you can select on to tackle and fix, instead of insisting on this correctness aspect that you seem to be quite fond of ;-)
4. The last time someone wrote new documentation for Spyder was in April 2012 when Jed Ludlow wrote a new page for the IPython console. So "we should write documentation about it" doesn't appeal to me.

[goanpeca]

1. Great!, still it would add clarity to Spyder, so it is on the same line of this issue
2. Is about organization and making sense... with standalone plugins it makes absolutely no sense to have third party (but shipped with spyder) plugins. This will simplify the handling of plugins as it is one aspect to ignore. Spyderplugins folder is not needed inside the spyder repo.
3. You can call it whatever you want but it is in the same spirit of this issue... spyder is not broken because of how files are organized... it will just make it nicer... same with spyderplugins... they are not broken (actually they system kinda is) but it will look nicer. Yes I am quite fond of "correctness" and of how the repo is organized, cause it will make easier for people to contribute.
4. Fair enough. I will do the documentation on that.

[Nodd]

I would keep the spyderplugins directory, because even if there is a very good documentation, nothing beats a working example.
However it'll have to change a little bit I think, there should be one sub-directory for each plugin, containing ui, widgets, icons... (exactly like future third-party plugins). This was already discussed somewhere for third party plugins and internal plugins will have to follow the rules.

Also keeping them as plugins and not core widget will make testing plugin functionality easier.

[ccordoba12]

We're not going to continue this discussion forever. As the current maintainer, I won't merge PRs to remove spyderplugins, that's all, ok? :-)

I think @Nodd understood my point: a working example is much better than any documentation (which can became outdated quite easily). Besides @goanpeca, you can leave the project at any moment (by getting busy with your PhD, family, etc), and I really don't have time to maintain new docs or learn how to work with new plugins, when I know the current ones and they're working almost fine.

However, I'm not opposed to move each plugin to its own dir, as @Nodd mentioned here and I already agreed with on Gitter.