Higher-level Go API

[barakmich]

Tracking for this discussion that keeps happening -- now it's got a number.

Basically, it'd be great to write queries (or, more explicitly, build iterator trees) without having to be so mechanistic in building the iterator trees. Exposing such an API makes it easier for people using Cayley as a library, makes it easier to write query languages (that have repeated patterns in a lot of places) and, perhaps, make it easier to talk about external API connections -- essentially, the wire format for building iterator trees. Which would allow interesting bindings for other languages without catting together Javascript.

This may be directly related to the Gremlin API, in that one may influence the other and vice versa.

[sdboyer]

this sorta seems, maybe, like an appropriate juncture to discuss what i'm trying to do in https://github.com/sdboyer/gogl, as the chief thing of value (i think) i've produced so far is some essential patterns around graph interfaces.

then again, i've not gotten to property graphs/triple stores at all yet. but if this is what you mean by "higher-level Go API", then i'm happy to bend my work in that direction.

[barakmich]

@sdboyer -- Seems right, to a degree. Do you have a favorite example?

So iterators as ways of getting sets of things (in fact, exactly sets, if you drop the path data) seems like it plays a little into your story as well, though from my (albeit cursory) look, you're talking a little more about the extended library -- which does play in here, but is slightly orthogonal.

If I can think out loud a bit...

So, unless you know why and how building iterator trees work, it's not going to make much sense to the standard library writer. Query languages help, because they effectively do the same thing for you; they give you a way to start and work with your data.

So a similar thing might exist in Go-land, and it may (but doesn't have to) look a lot like Gremlin.

Thinking about it in terms of abstract data types....
Traversals are things that take Paths to Paths, Morphisms are just traversals as first-class objects, and the thing that people kind of need is a way to start with some Node or set or something and lift that into a Path. These are the basic building blocks of queries.

Do correct me if I'm wrong, but I sense you're interested in, basically, other data types. Things like taking Paths and coming back with a topological list of Paths (via some pattern), or given two sets of nodes, the set of Paths that connect them (going from Node pair to Path, DFS style) or what have you.

Which is why it is related, but different at the same time. And you could imagine using the same building blocks behind your more complicated traversals.

[barakmich]

So, copy and pasting from #58:
"In thinking about attacking #74, I think it may be useful to build a graph.Query object to wrap much of this. Then the gremlin and MQL parsers/output can share more code at least."

This makes sense. I think one of the things that may be worth doing is to write a RunQuery (or somesuch) file in graph/ -- perhaps with the aforementioned graph.Query-- that, for the purposes today, takes care of the for { _, ok := it.Next() } loop I have everywhere in one location, with some flags and options for how and what to query, whether to optimize, etc.

This also centralizes it for other such functions/new evaluations/etc, as per this thread. So this may spin off a new issue, if that sounds reasonable, but all the better.

[pbnjay]

Here's my brain-dump as well. Basically it'd be cool to have something like this:

type Query interface {
  TreeWalk(walkFunc func(path []Iterator))
  AddIterator(type string, options map, subIts ...Iterator) (Iterator, error)
  Optimize() bool
  Exec() error
  Results() map
}

With TreeWalk I can display stats estimates without executing the query, so that I can manually refine it. Or perform other operations on the tree without too much pain.

AddIterator is akin to the current buildIterator methods in various forms. Basically allows you to dynamically create iterators without special-case code (would require a more augmented iterator registration system). I dont think this method prototype has enough info, but you get the idea.

Optimize() is what you think, Exec() is the RunQuery() @barakmich mentions with the ability to return errors instead of panic-ing, Results() returns data ready to be remapped to your own format.

[barakmich]

Let's start with Optimize(), Exec() and Results(), which are pretty clearly going to need to happen anyway, and we'll come around to TreeWalk and AddIterator.

I might make Optimize a bool flag, that Exec() respects, along with the potential for tracing or limits or something, but that's the only thing in that part that I could think of. I take it then that the query languages would generate things with these interfaces and call an appropriate RunQuery(graph.Query) function?

[kortschak]

Can you explain this a bit more?

Results() returns data ready to be remapped to your own format.

[pbnjay]

@barakmich TreeWalk is fairly trivial to implement and can be potentially
used to simplify things like Optimize. Also, Exec = your RunQuery, which
you really want as part of Query to fully encapsulate the Go API. Somewhere
in here we need to pass a TripleStore as well. Or is RunQuery part of
TripleStore?

@kortschak basically some sort of generic way to represent both Gremlin and
MQL outputs such that all you have to do is a basic transform on it. I
don't know if its possible or reasonable but its an idea =)
Can you explain this a bit more?

Results() returns data ready to be remapped to your own format.

—
Reply to this email directly or view it on GitHub
#64 (comment).

[kortschak]

Have a look at database/sql to see how Brad did it. There Exec() returns a Result interface. This is probably the way to go - also naming to be consistent with that package would be nice where possible.

[pbnjay]

Might be closer to database/sql/driver since this is behind the query
language, but yeah that's a good idea.
On Jul 21, 2014 7:24 PM, "Dan Kortschak" [email protected] wrote:

Have a look at database/sql to see how Brad did it. There Exec() returns a
Result interface. This is probably the way to go - also naming to be
consistent with that package would be nice where possible.

—
Reply to this email directly or view it on GitHub
#64 (comment).

[pbnjay]

Naw you're right, TripleStore is more like the driver interface. Query would be similar to a prepared statement I think, and the Result interface would likely be a thin wrapper on the IteratorTree, so I guess this does front-load the optimization and possible other options into Exec(). I'll think about it some more.

[pbnjay]

Forgot to mention the other benefit of AddIterator's dynamic nature - it means that internal data can be made more accessible. Which will be especially important for allowing triplestores to optimize them more. Right now a triplestore backend cannot access the direction(without typecast) and value in a LinksTo iterator, and the (currently used) Comparison iterator cannot export Operator and Value access as well. If we use a data map for these features, then a Postgres Comparison iterator and the base Comparison iterator can both use the same keys and be used interchangeably. This allows greater extensibility for more iterator types, and less boilerplate code compared to exported struct fields or interfaces.

Hope all that makes sense =)

[villadora]

Is that go api make cayley in some way like embeded mode?

[sdboyer]

@villadora i think it's safe to say that an embedded mode would be one benefits to having a higher-level Go API. i think another would simply be ease in moving between doing work on in-memory graphs and easily moving them into a persistence layer.

at least, that's the kind of thing i have in mind :)

[flosse]

I hope that's the right place to ask:
Where can I find some introductions for using cayley as a library? Is there a hello world example? And do you know a real world example of a go application that's using cayley (no matter which API is used) ?

[iddan]

Documentation for using Cayley as a lib can be found here: https://github.com/cayleygraph/cayley/blob/master/docs/Quickstart-As-Lib.md.
As far as I understand this feature was resolved and can be closed. @robertmeta @dennwc

[dennwc]

@iddan Yes, I think we can close it. Most of it is implemented I believe. For everything else we have a set of more focused issues open.