How to treat versions of the stdlib and external packages

[PMunch]

This is from the discussion during the 4th online Nim meetup, where stdlib module versioning was discussed. The common example used was the json module, so I will use that here as well.

The scenario is that a new implementation of json comes along that we want to replace the old one in the standard library. This module has a different API and can't simply be replaced as a drop-in because it would break everyones packages. The worrying idea that seemed to have some traction was to simply name this new module json2 or potentially put all new modules in std2 so it would become std2/json (I assume all the other standard library modules that depend on json would then be upgraded and placed in std2). In my mind this is a horrible solution, and not only because of the extremely unsightly names json2 or std2. This is reminiscent of how C would do it, and in fact has, simply because it lacks the package manager and versioning that Nim has. The fact is that the Nim standard library is already versioned, by the Nim version. The fact that users have grown to expect that the standard library doesn't change much in-between versions is simply because it historically hasn't changed tremendously.

My proposed solution is to simply allow imports to be versioned with the Nim standard library version they expect, otherwise it will use what the currently selected Nim version is shipping. The syntax could extend from the existing pragma syntax:

import json {.v1.4.6.}`
import macros, strutils {.v1.2.0.}`

Or it could be expressed in a different way entirely. This has several benefits:

• New users/programs don't have to think about what the latest version of a stdlib package is, simply import json would get you the current JSON module.
• Versions can be applied to blocks of modules without a lot of repetition
• It's immediately obvious where to find the documentation for the version imported
• We already have this versioning, so we don't need to sub-version the stdlib
• Would allow us to define a range of acceptable versions, or similar extensions
• Allows all focus to be on the newer version, the older json module would not need to be maintained for new compiler versions (If someone wanted to maintain it they could create an external module, or they'd have to pin their module to use an older Nim version)

The json2/std2/json similarly has a lot of disadvantages:

• What happens if a module in stdlib is in std2 because it depends on std2/json but itself wants to move to a newer API? Now we need to support the original, the std2 version, and a new std3 version
• Modules can change without updating the version, as the stdlib does today, causing confusion.
• Need to keep the old json module working on the latest compiler, or delete it and now only have a json2 module
• Extremely unsightly names

One critique of the import json {.v1.4.6.} syntax is that an unqualified import would pull the newest version. So if I have an external module mymodule that depended on the current JSON implementation it would potentially break when imported into a project that uses the new JSON implementation. However all Nimble modules already specify their required lowest Nim version. So this is no harder to solve that to assume that a Nimble module with a dependency on Nim >1.4.6 should also be served the 1.4.6 JSON module. This obviously requires the Nim compiler to have support for importing two different JSON modules into the same project, with potential passing of compatible data objects between them, but as far as I know this is a feature that is already planned and in the works.

These are just my two cents, but I think it resonates with more people from what I could gather in the discussion. In my opinion the json2/std2/json syntax is extremely ugly and would put me off using Nim if I saw that as one of the first things when I encountered the language.

[kaushalmodi]

Option B: Phase out old stdlibs as nimble pkgs

Here's another alternative (let's call it Option B) that can prevent stdlibs named like <STDLIB><NUMBER> or std<NUMBER>/<STDLIB>.

A new comer who comes to nim should simply need do import json (taking json as example stdlib in this discussion). They shouldn't need to worry about a json17 being possible available.

I am pasting my comment from Nim Forum here so that this discussion stays contained in this thread.

About the versioning of stdlib modules, would this work (a different idea than what I brought up in the meeting):

• The stdlib will still have one json.nim
  • The newer library should have all the features from the older json.nim even if the API is changing. People should not need to have to choose between old and new json because of feature parity.
• The older stdlib modules get published as nimble packages when newer ones come by
  • These get tested in Nim CI to ensure that they still keep on working with newer Nim versions
  • Nimble packages that choose to not track the latest std/json can freeze the old json version in the .nimble file (now that the old stdlib modules are available as nimble).
• This will prevent the stdlib fragmentation where some part of stdlib would be using json while some other parts be using json2, some other json3, and so on.

---

To add to what @PMunch said, few more disadvantages of json / json2 / json3 naming ..

• Such naming will make it impossible to phase out older json modules. Let's say json and json2 get deleted.. then it will be confusing to see just json3 hanging around.
• The documentation will just get bloated over time, with the TheIndex page just collecting older cruft from past years. We would want the Nim stdlib to point to the latest and greatest libs, not a collection of old not-so-good and newer and better libraries.
• This issue will be a repeat of re and nre (as pietroppeter brings up in this comment on Nim Forum).

---

Summary:

• Suggestion to pull out older modules out of stdlib and move then to nimble so that users still wanting to use the older versions can.
• But have just one version of a module in the stdlib.

[Araq]

But then if I actually use the non-ugly import std / json I'm aiming for a moving target. And readers of my code three years from now are not aware that I actually used json2. So effectively the only thing I can really do is to write import json {.v2.} which can also be written as import json_v2 and then we don't require yet another language feature.

[kaushalmodi]

But then if I actually use the non-ugly import std / json .. And readers of my code three years from now are not aware that I actually used json2

You would have the minimum or exact nim version dep in .nimble.

[Araq]

Such naming will make it impossible to phase out older json modules. Let's say json and json2 get deleted.. then it will be confusing to see just json3 hanging around.

I don't agree, why would it be confusing? json3 means that two versions existed before this version, nothing confusing about it.

[kaushalmodi]

why would it be confusing?

A new user would need to know that they need to import json3.. few years later, the then new user would need to import json5. Something about that doesn't feel right.

[Araq]

New users/programs don't have to think about what the latest version of a stdlib package is, simply import json would get you the current JSON module.

Nobody really has to think about this either way, you cannot just guess a module name and get away with it, you need to read about what's available somewhere (the index, tutorials). And these resources tell you which names to import.

[Araq]

You would have the minimum or exact nim version dep in .nimble.

And why exactly is this preferable? Meta information that is not part of your program directly is worse than what is accessible from the source code directly.

[Araq]

In my opinion the json2/std2/json syntax is extremely ugly and would put me off using Nim if I saw that as one of the first things when I encountered the language.

This is what concerns me most -- honestly, it feels like all the arguments against json2 are brought up just because it's "ugly". But that's just syntax and indeed most of the brought up arguments apply equally to any solution.

So ... if I come up with a sugar rule like "import json(2) is transformed into import json_v2" would that make it acceptable?

[kaushalmodi]

Meta information that is not part of your program directly is worse than what is accessible from the source code directly.

If we are going down the route of allowing multiple version to coexist, we need to do the package manager do the version juggling; baking version numbers into package names doesn't make sense... it's not scalable. If this is allowed, we will have stdlib get populated with arbitrary packages with version numbers encoded into them.

it feels like all the arguments against json2 are brought up just because it's "ugly"

It's more than that.

The stdlib will gradually gather a lot of old baggage. It won't be a collection of "standard" libraries, but just a collection of all versions of modules. E.g. re and nre.

[Araq]

Well ... it's still a "standard library", yes, it accumulates cruft, that's simply the nature of the beast. No, this doesn't mean a random collection of modules with trailing digits in their name (oh the horror! numbers! we should be scared!)

[Araq]

It won't be a collection of "standard" libraries, but just a collection of all versions of modules. E.g. re and nre.

We can embrace regex instead and deprecate and remove both re and nre. This solution doesn't get better just because it happened to be called regex instead of re3. Yet, actually, that is exactly what makes it acceptable to you, because "no digits in its name".

[Araq]

Python had urllib2 and it survived, https://docs.python.org/2.7/library/urllib2.html

[kaushalmodi]

Yet, actually, that is exactly what makes it acceptable to you,

You misunderstood me. I gave the example of re and nre because that's what I fear will happen for json, json2, etc. We will just collect all the old modules, even the ones with inefficient algo in the stdlib.

I didn't mean that re and nre are acceptable; it was the opposite.

[a-mr]

The proposal does not say how the versions are selected inside imported module itself.

If we do import json {.v2.} then should json.nim contain clauses like

when version(v2):
  ...

?

[Araq]

You misunderstood me.

I don't think so and the 'you' in my reply was a general 'you', don't take it personally. Just imagine we would have done that: Added regex, deprecated and then later removed 're' and 'nre'. Nobody would have complained, nobody would have made the connection to numbers and versioning and package management, everybody would have accepted it, Nim version X simply ships with a better regex library and the import is now import regex. Yet, if we we do the very same thing but name it re3, it's ugly and we need something better. The only difference is the digit in the name...

[stefantalpalaru]

import json {.v1.4.6.}

Let's not complicate the module system with dependency resolution and other fun package management features. We already control module search paths during compilation, allowing us to select specific module versions without changing the language.

The fact is that the Nim standard library is already versioned, by the Nim version.

But you can't expect the user to have more than one version of the standard library installed. When you start increasing the amount of magic during compilation, to silently install more stdlib versions, you're setting the user up for some interesting debugging sessions.

the extremely unsightly names json2 or std2

Usability trumps aesthetics.

When I see urllib, urllib2 and urllib3 in Python, I know those are incompatible, but similar forks of the same module.
When I see requests, I don't expect any API similarity with urllib.*. The pattern is clear and recognised already by most users.

Such naming will make it impossible to phase out older json modules.

Good. Backwards compatibility is very important for end users.

Let's say json and json2 get deleted.. then it will be confusing to see just json3 hanging around.

Confusing how? What can the user do instead of import json3 at that point?

pull out older modules out of stdlib and move then to nimble so that users still wanting to use the older versions can

Now that is confusing. So the json module suddenly errors out with a new Nim version, and the user is supposed to guess that it was moved to a separate package? Maybe one named json3? So now import json becomes import json3 in order to keep using an older version?

Counterintuitive and therefore dangerous.

[arnetheduck]

We seem to keep coming back to this topic, that the standard library is a place where code goes to die without there being an process for cleaning up the debris, and therefore, the amount of bad code keeps accumulating, creating confusion and frustration.

At the same time, it takes months to get critical security fixes into modules that need them, because it's hard to produce a working release when there are so many changes spread across the language, compiler and all the libraries - this creates an artificial dependency between all modules and the compiler which otherwise wouldn't have to be there, and which will keep slowing the pace of Nim development down over time. This is completely unsustainable.

The really easy and simple solution is to not add more stuff to the standard library, and move the existing libraries into separately versioned packages.

To get there, the following needs to happen:

• the package manager must support lock files - this is critical because code should never change without the programmer knowing about it, and the programmer should be able to select versions for each package separately
  • this is how the developer can bring in a security update or a bugfix to a single module without destroying months of work on getting the rest of the application stable based on a specific set of libraries that are known to work together
• the package manager needs to support multiple versions of the same package in the same application - this is the json vs json2 discussion from above - it's very common that transitive dependencies end up conflicting, and the way to resolve that module A uses json and module B uses json2 is that both are allowed to coexist in the binary - A and B don't have to be aware or affected by their respective needs. Sometimes, both need to be used in the same module, so a renaming mechanism is in order, so that one module can use both json and json2 (often to implement bridge / backwards compat)
• the package manager needs to support monorepo / multiple packages per git repo or we'll all go nuts when trying to upgrade two tightly related packages

There are a few more nice-to-haves, such as a vendoring option that will create a full, stand-alone release of the code.

In using the standard library today, when we type import tables or import json and the nim version changes, there's almost always an accompanying breaking change - this is because nim is a difficult language to keep API-compatible - most people don't understand the impact of adding a symbol to the global symbol namespace or raising an extra exception or defect here and there as API-breaking, but yet, this significantly changes the behavior of library, as was vividly was seen in the releases between 1.2.6 and 1.2.12, each release between breaking previously working code in different ways (and we're talking about bugfix releases here, which supposedly are "safe" - now apply the same thing to any release that also changes language rules and introduces new features).

The other thing we're seeing is that the old, fundamentally broken versions of the modules cannot be removed in any sane way - again, this is solved with packages: the unused modules fade from use and that's it, the historic versions continuing to exist as long as there's someone with a vested interest / enough skin in the game to keep it alive. In one fell stroke we also avoid the discussions around "but you should maintain this package for me".

The flawed assumption here is that when typing "import json", it's the latest version I'm after - it's often not, in any application that works already, or in a library that is not central to the application - I want the "most working" version of the library in those cases and that means using whatever version was used during development and not changing it until there's an explicit decision to do so (and test that decision).

The healthy way to treat package or module versions is that they are different libraries with similarities which in some cases end up being compatible, and when tested, can be upgraded together.

[saem]

• json2 is about multiple modules that serve the same workload (deal with json) in different ways, otherwise they can carry deprecated items in one module
• a version pragma on import is versioning modules
• a package is a bunch of Nim code in a folder, and so it was ostensibly developed together
• the version pragma proposed above presently is informing the package version
• looks like conflating versioning modules and packages
• currently, for versioning modules inside a package, the package maintainer has names and defines already
• what's still missing is a canonical list of issues to resolve that are expressed in a regular way to extract the key dimensions of the problem: varying rates change over time (project vs libs, compiler version, ...), types of changes (newer version, competing version, transitional, ...), and more
• it's hard to agree when talking about overlapping but different things

[arnetheduck]

[09:32] Varriount: @arnetheduck I really liked your comment regarding standard library versioning. I think, when it comes down to it, package management is an inevitable need.
How would you envision distributing the "standard library"? Would they come bundled with the compiler, as a set of pre-installed packages?

[11:05] arnetheduck: thanks - yeah, I mean, what I'm aiming for is to unleash the language so that libraries don't have to be gated and dependent on the release schedule of the compiler and at the same time free up time for the language devs to be working on things that are relevant to the language instead of maintaining a massive amount of obsolete modules - it also avoids having the standard library be an experimentation ground for unproven code that someone with the right commit access at some point thought was a good idea - inevitably, better ways to do things come around as the language evolves - it's a win-win for everyone, if you believe in people's ability to sort things out, and I generally do - ie given the right tools, this is the way to scale development and cater to a wider range of needs
[11:09] arnetheduck: I mean, you don't need that much of a standard library when the package manager is doing its job properly - the whole point is that if I want to update my json module, I should be able to do so without also getting the unrelated xml module changes I didn't ask for - certain important / endorsed packages can still be maintained as part of the nim-lang community repo to maintain their quality and serve as focal points for community building and collaboration (this tends to be the argument to put things into stdlib right now, and is still possible, if not easier, when the pieces are smaller because you can now delegate responsibilities easier)
[11:11] arnetheduck: the other thing like about this idea is that the language and compiler itself is also treated as one more library / dependency - you download project X that has been created and tested with Nim 1.Y in mind, and the PM/build system ensures that this exactly is what is built - language version included - when collaborating with others, this kind of reproducibility is a godsend, ie checkout and build is something that should never fail, the zero-to-hero story
[11:16] arnetheduck: some people see stdlib as an "ease-of-use" thing and there's certainly room to include certain versions of modules in there by default (ie the latest compatible versions at release time), because again, with the right package manager, these are just "first suggestions" and you can override them with other versions easily and locally for that particular application you're building - I see it more as a collection of packages whose primary value is curation - someone has taken the time to provide a quality set of packages that are good starting points for further development, and in a "standard" project config, there would be a dependency on this collection of packages "by default"
[11:22] arnetheduck: finally, certain libraries and code become part of the culture and idioms of the language - you can tell which ones based on how they slow down in maintenance and development but still get widespread usage over time - this is the point in time to consider them for inclusion in a standard library, not before - ie only when they've proven themselves to be solving an isolated problem naturally such that there is little doubt that there's any better way to solve it
[11:30] haxscramper: I think you should also add this to your comment on the RFC - specifically the "consider them for inclusion in a standard library, not before - ie only when they've proven themselves ..." part.
[11:31] haxscramper: Because to me, it is one of the most precise arguments made I've seen in these discussions
[11:32] haxscramper: But now after you said this I think it is the main fundamental flaw in this discussion - "what if"-based decision making
[11:33] haxscramper: Multiplied by nimble's lack of certain features

on what-if decision-making, per request in chat

[Araq]

I continue to fail to see the benefit in "instead of a stdlib, here is a list of Nimble packages". Not reviewed, using different styles and of different code qualities with different attitudes towards backwards compatibility. You always act like a fragile web of dependencies is a desirable thing to have and then you "only" need really good tooling to compensate for this design, but I think it's a terrible design to begin with.

[arnetheduck]

Not reviewed, using different styles and of different code qualities with different attitudes towards backwards compatibility.

This is probably the point that's misunderstood the most in these discussions - there's absolutely no reason why the packages that are moved out of the "stdlib" would have lower quality or adhere to different standards - they would / could still be maintained by the community of nim-lang - the aim is to untie the release schedule of packages from that of the compiler and the language, and allow each to be upgraded independently. At the same time provide a mechanism for replacement of obsolete code that works across a broad range of use cases, that is lacking today.

json is often cited as an example, for good reasons - it's slow, uses ridiculous amounts of memory, has API issues and doesn't actually implement the json standard, to start with - to the point that it gives Nim a bad reputation for being in the standard library, but above all because it causes problems for those that come across it - this would perhaps be acceptable if this was the only existing json implementation and Nim was a young language where the module was a first cut, but there are now much more sane implementations out there.

A mechanism to replace it is direly needed at this stage - one that doesn't break existing code that is happy with the status quo - but one that at the same time frees the community to move on and improve the situation, even if these improvements require changes to user code - at that point users have a choice: they can upgrade json and their code, or stick with the current version and its issues - without losing out on the other fixes and improvements that go into the language, and above all, with the ability to stay updated with security fixes such as the SSL certificate issue, or with critical bugfixes like that of the GC randomly collecting memory being fixed in 1.2.12.

Old versions continue to work as long as changes to Nim don't break them, and when Nim gains features, it's easy to gate newer versions of the package to Nim versions that supports the prerequisite feature - like the introduction of lent (this replaces the clunky since annotations as well).

Crucially, since the module is maintained by the Nim community, when a new, better version comes around that the curators of nim-lang agree is a worthy replacement, manuals, guides and tutorials that nil-lang controls are updated to point to the new version and users of the language can follow along _at their own pace. The social curation that now is tied to being a part of the std lib still continues to function.

compensate

It's the other way around - the current low standard of tooling is preventing a more dynamic development environment where releases of packages don't have to be artificially tied together to maintain a semblance of quality - the existence of the std lib is a crutch that compensates for the lack of adequate / minimum standard tooling and creates a, for the future, unsustainable bottleneck where the only thing people can imagine is to add more and more stuff to it - ie "module X in std lib is broken and needs module Y to fix it - let's add module Y to the standard lib also!".

Imagine that in order to fix a bug in libxml or apache web server you'd have to make a gcc release - it's nuts, and yet this is the reality of Nim today - or worse, when you try to create the equivalent of lighthttpd, people scorn you for "not improving the std lib apache instead" - this is even more nuts.

[beef331]

We already have each version of the stdlib on github through their respective branches, because of that we could do import std/json which without configuration fetches the present json from the Nim version you're using. With configuration you could specify using --override:json=1.2.0 on the CLI or override "json", "1.2.0" in Nimble, allowing you to pin to a specific stdlib version of the package and allowing the stdlib to progress with no concern of the past. This also could extend to enabling user constructed stdlibs as the overrides could support Nimble packages and the Nim stdlib.

[Varriount]

As inconvenient as it may be, "really good tooling" is what is expected from a programming language. It doesn't matter if a language is the fastest or most intuitive out there, if developers encounter too much friction attempting to set it up, build it, or use code developed by others, they are going to give up.

[jamesaorson]

It seems like the main complaints here are really about, "The overall quality and maintainability of the stdlib has become a burden on Nim development" rather than how to version packages. The whole issue of versioning is simply a symptom.

It might be a good reason to consider a new major release for Nim, making it Nim 2.0 with the main change being the stdlib, somehow ensuring backwards compatibility with existing nimble packages (this is the tricky part). Give Nim a fresh start with a lean and mean stdlib, along with a stricter acceptance criteria for stdlib code addiitions. Say the new stdlib has no json module to start: that's fine because most agree the nimble packages are better, and in this plan they are still valid! If each stdlib module is diligently planned out, then users will move FROM nimble packages to stdlib modules. Having holes in the stdlib in the shape of JSON support, YAML support, native UI support, etc will allow the community to decide what is important to them. Nimble packages are made. The community essentially "votes" by using packages. Popular packages either inspire stdlib support efforts, or end up being directly rolled into the stdlib if they meet acceptance criteria and the author supports such a move.

It may seem like a drastic step to take, but maybe it is necessary? In the modern day, does any Python dev speak of Python 1? Python 2 simply introduced some new data types, additional syntax, some new modules, some removed modules. It seemed like the greatest driving factor for Python 2 may have even just been a change in how development for Python itself was changing. Source: https://docs.python.org/3/whatsnew/2.0.html

Not that we should believe that if Python did it, it must be good or correct. Simply pointing out that there may be a reasonable path forward by taking these learned lessons and wiping the slate clean.

Given the tricky part of supporting Nimble packages from the previous major compiler version, this could be done if, similar to Python 2, the old stdlib was accessible somehow. This is where tooling comes in. Since nimble files already track the Nim version associated with them, when Nim2-compatible Nimble would see a Nim1 package, it knows the stdlib imports have a different root path. We can introduce deprecation warnings so Nimble package developers can update their packages to Nim2, directly reference the old stdlib, and begin to update to the new stdlib.

This is fundamentally very similar to an option that was discussed at the last Nim dev meetup, when it was suggested that we make a "new stdlib that is only compatible with itself". This is basically going down that path, but asking for a specific effort made to not make the same mistakes with the new stdlib. Don't allow the maintainability of the new stdlib to degrade for the sake of having everything and the kitchen sink.

[Araq]

In reality moving code around makes the problem worse -- we tried that with nim-lang/zip etc and more recently with 'fusion'. You need to ensure that the CI covers the Nim+fusion combinations that you actually try to support (N * M combinations where previously it was 1 * 1), you need to ensure that these modules show up in the generated documentation and when you ship them your users must understand to use the appropriate issue tracker. Then when a new language feature becomes available that the library would benefit from this library must migrate to a newer Nim compiler, so then only an older version of the library works with an older Nim compiler (or it uses the since mechanism) -- what if the library also receives bugfixes? Should they then be backported manually like we do it with Nim itself? That's expensive and risky -- most likely it won't get done.

But this is all a bit besides the point. The original problem was "how do we get something like json2 into the stdlib?" Here are some possible options:

• We don't add anything to the stdlib -- we invite people to import a different package instead. And only when we have a "time proven" json that the "community" agreed on we add it to the stdlib. But then what's the point? The happy users already have a dependency like "requires super_json > 1.0" and it keeps working. Why ever add anything to the stdlib then?
• We remove 'json' from the stdlib because it's "bad". And we keep removing bad libraries from Nim. And then we hope that new std modules don't have design flaws in them?
• We "fix" the existing stdlib's json. (But then that's the real problem -- the "fix" is a change in behavior and so a breaking change and it's much worse than adding a feature... Semver's prioritization which implies that bugfixes are harmless and features additions something to be concerned about is fundamentally flawed.)

[Clonkk]

IMO, import json2 also has issues.

What happens if we need to break part of the API of json2 ? Do we make a json3that is 90% identical to json2 ?

What happens once everyone stopped using import json ? Do we remove it ? Do we keep shipping and maintaining useless code ? Do we change import json to point to import json2 ?

Overall, I generally agree with @arnetheduck : The stdlib should be treated as a dependency. As such, this becomes a dependency resolution problem (that we tries to solve using the module system, somehow).

To put it differently, would we recommend a new Nimble package to be published every major version release ?

[Araq]

Do we make a json3that is 90% identical to json2 ?

If only there was a way to re-use code via templates, Nim's import-export mechanism, .deprecating individual procs rather than full modules...

[haxscramper]

need to ensure that the CI covers the Nim+fusion combinations that you actually try to support (N * M combinations where previously it was 1 * 1)

Aren't "important packages" already tested as a part of CI on every release/commit (i.e. we don't have 1*1 even now)?

[Araq]

They are tested yes, but only a single specific commit/version of every important package (well, usually "git master").

[arnetheduck]

To put it differently, would we recommend a new Nimble package to be published every major version release ?

absolutely not - the whole point is to not have to release things together: the compiler can make a release whenever and the package can make a release whenever, independently - of course sometimes it makes sense to release together (nim compiler gets lent feature, lent support is added to the library)

what changes is that at some point, the nim manual and other community resources start recommending other libraries and these will get picked up the same way the std lib libraries are picked up today - but it's a softer transition in which old code keeps working and existing users can upgrade at their leisure when it's worth it for them, not based on the need to make a Nim release, and conversely, it becomes a lot easier to make a package and/or Nim release to address critical issues.

moving code around makes the problem worse

only when the tooling to do so is inadequate - this is what tooling is for, to create an environment where these operations are not made artificially expensive. If you switch from nails to screws without a screwdriver, of course you'll be disappointed. Conversely, when all you have is the "add-to-stdlib-hammer", of course that looks like the only viable option - and more sadly, the path of least resistance.

Should they then be backported manually like we do it with Nim itself? That's expensive and risky -- most likely it won't get done.

Yes, of course it should be done, if there is an interest to do so, and that interest is driven by the people with skin in the game: the users of that version of the library - they can either persuade the maintainers to backport or do the work themselves - that's the mechanism that makes this a scalable solution. When it's a single package that focuses on one thing, it's also not expensive at all: you do the backport and you don't have to consider fixes to 100 other libraries to make the release with the backport - it's a much more tractable problem to fix a bug in json and release / backport it, than to navigate the history of a giant monolith and try to cut a release.

appropriate issue tracker.

with the right tooling (monorepo support in PM), the packages can even live in the same git repo for all the package manager cares - this is convenient even, for certain groups of packages (the html parser and the web server).

Why ever add anything to the stdlib then?

that's the point a bit: you don't have to, most of the time, unless the rough consensus becomes clear. With decent tooling, the incentive to add things to the std lib is much smaller. matplotlib is a popular python3 low-level package, but because it's so easy to create a virtualenv and pip install it, there's no practical difference between it being in the std lib and standalone - it turns out users can actually be trusted to be smart enough to use it anyway.

We remove 'json' from the stdlib because it's "bad".

If json were a standalone package, this would not be a problem: it would forever stay the same and easily accessible without any need to remove it - new code would move on to better libraries (often based on community recommendation, in manuals etc) and users upgrading would either follow along when they feel it's worth it, upgrading their code at a moment in time of their choosing or contribute to make the json library less bad without breaking their software, and without having to wait 6 months, best case, for a new Nim release. The point of the PM argument is to not keep adding to this problem - it's bad enough that json is there in the std lib already, to be adding even more stuff in there, or create the same monolith all over with fusion instead of actually addressing the root cause (poor tooling, mismatched incentives, poor quality code in std lib that can't compatibly be fixed, no socially acceptable way to get rid of it, artificial release bottlenecks etc etc)

how do we get something like json2 into the stdlib

don't. spend the effort on upgrading the tooling, then make an incompatible separate package and start recommending it in manuals. The old json module now goes into dormant state and barely requires maintenance, except an addition in the manual pointing users to the saner alternative. All languages have dark corners from their early stages - at some point deprecation and removal might become the right thing for the poor json module as well - for example when Nim 2.0 happens with ARC/ORC becoming default and all code must be rewritten and retested anyway, but again, with a decent PM there's much less pressure to do so: people that run into its issues have an easy solution now - and those that are fine with json don't lose out either: if they're only parsing 5 line config files with strings in them, the json issues don't affect them.

[Clonkk]

To put it differently, would we recommend a new Nimble package to be published every major version release ?

absolutely not - the whole point is to not have to release things together

The question was rhetorical; of course it's a terrible idea to release new package every version. Yet, I feel like this is what we're doing with import json2. I agree that module inside the stdlib should be treated as separate entities (which doesn't mean there shouldn't be a stdlib distributed as a coherent unit)

[Araq]

with the right tooling (monorepo support in PM), the packages can even live in the same git repo for all the package manager cares - this is convenient even, for certain groups of packages (the html parser and the web server).

Hmm, "monorepo support in PM", will give this a serious thought.

[dom96]

with the right tooling (monorepo support in PM), the packages can even live in the same git repo for all the package manager cares - this is convenient even, for certain groups of packages (the html parser and the web server).

Hmm, "monorepo support in PM", will give this a serious thought.

Just on this point- Nimble supports this already. We even have a repo in nim-lang that utilises this, graveyard, it's done like this: https://github.com/nim-lang/packages/blob/master/packages.json#L12009.

[Araq]

Just on this point- Nimble supports this already.

Yeah, I know. So let's make this concrete, how do we use it to bundle nim-regex with the next major Nim release?

[dom96]

@Araq depends what your goals are. I'm guessing, but do you want the package to be bundled with Nim and also installable via Nimble? Why?

[Araq]

I want to bundle it in the tarball/zipfiles because "just nimble install x" is not allowed in some environments, no matter how much we like Nimble. Plus the downsides (no git history) don't apply to the zipfile as it has no .git subdirectory to begin with.

It shall be update-able via Nimble.

[arnetheduck]

I want to bundle it in the tarball/zipfiles because "just nimble install x" is not allowed in some environments, no matter how much we like Nimble.

the normal way is that the package manage has a vendor or dist command that just dumps all dependencies, transitive also, into the local folder and that's it - in fact, the current model of keeping a global directory with nimble stuff is quite broken and makes this more difficult than it has to be - a PM for a language should never affect the installation unless specifically asked, instead keeping everything in the project folder - it's trivial to understand, wipe, trouble-shoot etc - when you work on multiple projects there are no conflicts and the IDE is happy as well.

For such bundling, the metadata must of course follow along so that the packages can be identified. Now, with good tooling, also this feature becomes much better for users: instead of having a standard library with lots of stuff they're not interested in, and lacking the stuff they are interested in, a bundle command creates an offline version of all their dependencies - the same pattern repeats: the incentive for having things in a standard library goes down because whoever is constrained by download restrictions can nonetheless benefit from the ecosystem of libraries by creating a personal standard library.

Now, the really cool thing would be if the bundle command also included a dep on the language itself - when you bundle a project, you get the compiler with the package and it's treated like just one more library - this creates a fully reproducible zero-to-hero story rivalled by few out there.

Of course, later the PM can optimize things and cache and so on, but that doesn't affect the mental / abstract model under which it operates.

[dom96]

I want to bundle it in the tarball/zipfiles because "just nimble install x" is not allowed in some environments, no matter how much we like Nimble.

That's not the reason this should be done. The reason should be: we want this to be the officially supported way to do regex in Nim.

It shall be update-able via Nimble.

Unfortunately to support this properly non-trivial changes will be required. There may be simple "hacky" ways to accomplish this though and we might want to do so anyway.

[Araq]

The reason should be: we want this to be the officially supported way to do regex in Nim.

Yes, that too.

Unfortunately to support this properly non-trivial changes will be required.

Please elaborate.

[dom96]

Unfortunately to support this properly non-trivial changes will be required.

Please elaborate.

So we can easily bundle a package with Nim:

• Put the package source code in $nim/lib/pkgs (or similar)
• Add --path:$nim/lib/pkgs/regex to the nim.cfg file

The problem is that Nimble has no clue this exists, so if the user wants a newer regex package then how can Nimble tell Nim to use that newer package?

[Araq]

The problem is that Nimble has no clue this exists, so if the user wants a newer regex package then how can Nimble tell Nim to use that newer package?

Here is an idea: "nimble install regex" installs regex like it always does and there is a mechanism so that --nimblePath is prefered over --path:$nim/lib/pkgs/regex. Maybe a bit ugly.

[arnetheduck]

Here is an idea:

the other question when bundling is what to do with packages that don't declare their dependency on regex - ideally, they should not see the bundled package until they explicitly declare that they want it - this is the way to solve the "no clue it exists" issue and prepare existing code to correctly declare what it's using, such that in the locked versions it depends on, it doesn't get surprising updates.

[andreaferretti]

My 2c: it doens't really matter if people don't use, say, a sanctioned JSON library, but mix and match them according to their need, as long as they are interoperable. So in my opinion the stdlib should ship datatypes for JSON (and HTTP, HTML, regexes, and so on) and never change them, while library authors should be encouraged to do whatever they please, but ensure that their libraries work with the official datatypes, so that one can, say, parse and HTTP request with lib1, add some headers to it with lib2 and finally call the request with lib3

[mildred]

I see this issue has not been talked a lot lately, is it considered for Nim v2? I think this would be a great way to allow standard library improvement.

[PMunch]

@mildred do you mean my original proposal? Or any of the other ideas put forth in this thread? AFAIK there hasn't been any consensus on what to do about imports.