Link Lwt-related libraries in README.md

[smithjessk]

Addresses #366. As mentioned in the REAMDE, I generated the list of libraries with opam list --unavailable --depends-on=lwt --depopts --recursive --short | grep lwt. This might miss some libraries, but I believe it's a good start. I used wget to check that all the URLs work.

[Drup]

Do we really want a mostly auto-generated list in the Readme that will be outdated the instant it's commited?
If that list was curated with nice descriptions, maybe ... but right now, that's not an interesting list at all.

[aantron]

@smithjessk, thanks for the PR. These are packages, some of which correspond to libraries that are worth listing, but it requires a bit of curation to figure out what to include.

For example, angstrom-lwt is actually part of angstrom, so we would want to list it under that name, maybe with package name in parentheses, and give a brief description.

Also, I think some of the packages are deprecated. Not immediately sure. We want to look for the ones that are actively maintained.

The easiest way to find all this out is to type opam info blah for each package blah, and look at the development repo URL. This URL can also be found online by searching for each package at opam.ocaml.org. We'd want to visit the repo of the project each package comes from and have a look around.

[smithjessk]

@Drup @aantron Thanks for the quick comments.

Is there still interest in this PR being on this project? I ask because of @Drup's comment. If so, I'd be happy to:

• Change the list to libraries (with the corresponding OPAM package name)
• Remove deprecated and irrelevant libraries
• Add descriptions to said libraries

[aantron]

@smithjessk Yes, please. That plan sounds good.

I'm also wondering why e.g. Opium isn't in the list. I haven't tried running the command recently, so I don't know if it's not being picked up through the dependency graph.

[aantron]

Two more things I'd suggest:

1. For the projects part of Mirage, figure out which ones have an "independent" usage. Everything else can be listed as one entry "Mirage" or "Mirage unikernel," I think.
2. Some libraries are in opam mainly, I think, for easy installation, and don't really have many general users. These tend to be very application-specific, like very specific database software, etc. We should probably omit those, too. Basically, if something is narrow enough that its users already know they want to use it, and will search for it by name, we should omit it. On the other hand, if something gives a good idea of what's in the "Lwt ecosystem" for general users, we should include it. The ecosystem category would include things like web servers (Cohttp), etc.

[smithjessk]

Opium isn't included because, as a starting point, I only kept packages whose names contained "lwt". The listed opam command produced ~750 packages, and I wanted to limit the libraries to those that were deeply connected to Lwt.

I see now that this is a bad filter because of the important false negatives (e.g., Opium, the main Cohttp package), so I'll manually go through the results and see what comes out. It shouldn't be too bad. I plan to have an updated + improved list by the end of the week.

[aantron]

Ok, wow, I didn't realize it was that many. I usually run that command through grep, or start a script that downloads everything and then runs grep, so I never actually counted.

[Drup]

That might be because of --rec. It'll find all the packages that depends on lwt and all those that depends on them. Obviously, that's a lot. :)
Maybe it's better to keep only the direct dependencies ? At least at first. That's "only" 330 packages. :>

[smithjessk]

Sounds fair. I considered that before posting my last comment, but, since I'm new to the Ocaml world, I wasn't sure if that would miss important projects.

Could we drop --depopts too? That drops the count to 259.

[aantron]

Libraries depending through --depopts is the "old" way of depending on Lwt for libraries that can also be used without it. For example, it was used in the past by Cohttp, so a year or so ago we wouldn't have been able to find Cohttp without including --depopts. Many packages still depend on Lwt that way, so I think we should keep the flag.

Dropping --recursive as suggested by @Drup is probably more likely to not clip anything "really important," so I would start with that if you want to trim the count. But there may be packages not depending directly on Lwt, that still materially use it, and are worth looking at.

Two things regarding the number of packages:

• Even after filtering 750 (or 330) packages, we might not want to list all the "good" results. It could just be too many for the README. For that, I'd suggest listing a few in the README, and maybe opening some PR(s) to awesome-ocaml, and linking to a section in awesome-ocaml from the end of the list in the README.
• I realize you may not want to actually filter 750 packages. That's about half of the total number published in opam, so it would be a pretty good survey of the entire OCaml ecosystem :) Feel free to recruit help, save this for later, do something else, etc. At least we already learned how big a task solving this issue might be :)

[smithjessk]

Hey- sorry for the delay. I went through the ~330 by hand and tried to pick out a good starting set. Does anyone see any false negatives/positives? In the meantime, I'll start writing up small descriptions for these.

alcotest
angstrom
arakoon
bap
cohttp
conduit
cstruct
csv
datakit
ezjsonm
faraday
gdbprofiler
hardcaml
iocaml
irmin
js_of_ocaml
jupyter
logs
lwt-parallel
mirage
ocplib-resto
ocsigenserver
opium
session
stog

[aantron]

No problem, I can imagine how much work it was :) So thank you :)

Pretty much everything in that list is a very solid project, but I suggest taking from it only alcotest, angstrom, cohttp, cstruct, ezjsonm, faraday, logs, lwt-parallel, and opium.

This is because most of the other projects are mainly using Lwt. From Lwt's point of view, those are so high up the dependency chain, that they are practically applications for us. To make this concrete, take either Jupyter or IOCaml. If a developer stumbles on the Lwt repo and wants to know what else is going to make developing their other application easier with Lwt, they won't likely find Jupyter or IOCaml immediately useful for that (neat as Jupyter and IOCaml are in general, of course :)). By contrast, they may find Alcotest very useful for easily writing nice tests of their Lwt code, ezjsonm for parsing JSON in an Lwt-using codebase, etc.

So, the projects I'm suggesting to leave are the ones that somehow directly enhance the experience of coding with Lwt. They interact with Lwt on what Lwt does, which is concurrency and I/O, and are on a similar level in the dependency graph.

Of course, this is all open to interpretation :)

[smithjessk]

I added a new commit with the libraries you mentioned. Please let me know if there's anything else I can do.

Actually, the large number of libraries made this a perfect starter issue. I learned a lot about the important players in the Ocaml ecosystem. Thanks for being patient!

[aantron]

Thanks for being thorough!

I'm glad you found it useful. You probably now have a better survey of the OCaml ecosystem than almost everyone, me included :)