Unable to use published datasets in a different client #2336

nikhilrajrs · 2018-11-05T12:57:40Z

I get the error "Inputs contain futures that were created by another client" when I try to join a published dataset within a client different from the one that originally published it. The whole flow can be summarised as follows:

Client 1
df = dd.read_csv(...)
client.persist(df)
client.publish_dataset(ds_name=df)

Client 2
df = client.get_dataset("ds_name")
df2 = dd.read_csv(...)
df2.join(df)
client.persist(df2)

Is this expected behavior?

For reference, this check was added in the commit c02ea63#diff-96a27223dc91b5c9ea3d03684d79ad3f%5D which is part of the pull request #2227

The text was updated successfully, but these errors were encountered:

VMois · 2018-11-11T12:22:17Z

Are your code for publishing dataset should looks like this:

df = dd.read_csv(...)  
df = client.persist(df) 
client.publish_dataset(my_dataset=df)

Because client.persist() returns futures and you need to reference those futures for publishing dataset, but in your flow example you just call client.persist() without assigning it to any new variable. Please, check docs examples (https://distributed.readthedocs.io/en/latest/api.html#distributed.Client.publish_dataset) and add your actual code.

nikhilrajrs · 2018-11-12T09:23:26Z

Sorry for being not precise enough.
The code I use is indeed:

names = dd.read_csv('/shared-store-path/names.csv')
names = names.set_index("ID")
names = client1.persist(names)
client1.publish_dataset(names=names)

I have attached a full example that can be used to reproduce the behavior. Shown below is the output from the notebook I used to test -

from dask.distributed import Client
from dask.distributed import wait
import dask.dataframe as dd

client1 = Client("scheduler-address:8786")
client2 = Client("scheduler-address:8786")

names = dd.read_csv('/shared-store-path/names.csv')
names = names.set_index("ID")
names = client1.persist(names)
wait(names)

DoneAndNotDoneFutures(done={<Future: status: finished, type: DataFrame, key: ('sort_index-6f9583110f3b77c24727c1e970735470', 0)>}, not_done=set())

client1.publish_dataset(names=names)

roles = dd.read_csv('/shared-store-path/roles.csv')
roles = roles.set_index("ID")

names_dataset = client2.get_dataset("names")

roles = roles.join(names_dataset)

roles.head()

	Role	Name
ID
1	Developer	John
2	Developer	Doe
3	Developer	Johnny
4	Developer	Doey
5	Tester	Hello

client2.persist(roles)

---------------------------------------------------------------------------

ValueError                                Traceback (most recent call last)

<ipython-input-20-7cb62cd7bd89> in <module>()
----> 1 client2.persist(roles)


/misc/anaconda3-amp/lib/python3.6/site-packages/distributed/client.py in persist(self, collections, optimize_graph, workers, allow_other_workers, resources, retries, priority, fifo_timeout, actors, **kwargs)
   2561                                          user_priority=priority,
   2562                                          fifo_timeout=fifo_timeout,
-> 2563                                          actors=actors)
   2564 
   2565         postpersists = [c.__dask_postpersist__() for c in collections]


/misc/anaconda3-amp/lib/python3.6/site-packages/distributed/client.py in _graph_to_futures(self, dsk, keys, restrictions, loose_restrictions, priority, user_priority, resources, retries, fifo_timeout, actors)
   2210                     msg = ("Inputs contain futures that were created by "
   2211                            "another client.")
-> 2212                     raise ValueError(msg)
   2213 
   2214             if restrictions:


ValueError: Inputs contain futures that were created by another client.

testfiles.zip

VMois · 2018-11-12T19:23:16Z

Okay. Thanks for full example. I will try to run it tomorrow and see what will happen.

VMois · 2018-11-13T17:15:16Z

Actually, I have reproduced your error, but not using your code. It enough to publish a dataset with one client, then using second client (different process) get a dataset, modify something and if you will try to make client.persist() it will throw above error. I think it's a general error. My small example:

client = Client('localhost:8786')
client.restart()

df = dd.read_parquet('test_data/data.parquet')
df = client.persist(df)
client.publish_dataset(test=df)

Now, you can open a new terminal:

client = Client(...)
df = client.get_dataset('test')
new_df = df['some'] > 0.5
new_df = client.persist(new_df)
# error here...

Do you have any ideas @mrocklin ?

Python: 3.6.6
Dask: 0.20.1
Dask distributed: 1.24.1

VMois · 2018-11-16T09:57:26Z

Any updates/suggestions about this problem @mrocklin @martindurant ?

martindurant · 2018-11-16T14:02:53Z

(I am at a conference and unlikely to be able to look into this right now)

itozapata · 2019-04-22T20:52:25Z

did anything ever happen with this ticket. i seem to have run into the same situation and dont see any information anywhere else about it.

My situation is i am persisting a dataset the first time i load it and someone else is picking it up and adding more to the graph something akin to

df = client.get_dataset('test')
new_df = df['some'] > 0.5
new_df = client.compute(new_df)

and it is giving me this error

martindurant · 2019-04-23T00:11:31Z

The original posting above used versions that are now somewhat out of date. Would you mind trying with the latest versions? I just ran a very similar test and didn't see any error.

itozapata · 2019-04-23T20:32:55Z

sounds good after continuing to look i think i am getting something just slightly different. i will try and package up an example. I was able to get it working by not creating my own client and instead using get_client on my computer to make sure that the same client was being used with the client.compute call, but the behavior that i was experiencing didnt seem to make sense to me.

I will attempt to get something small to illustrate the situation shortly

tshatrov · 2019-06-10T10:07:16Z

We're also having troubles working with this check. There's a bunch of datasets published by a loader script, and another application is launching tasks that use .loc[...].compute() to retrieve a part of published dataset as a pandas dataframe and then perform some calculations with that dataframe. The way I see it, if the published dataset contains futures, and dask-worker retrieves it, then future.client is always a different object than the client of the worker, so future.client is self check will always fail. So how are we supposed to use published datasets from within workers?

mrocklin · 2019-06-10T17:34:26Z

future.client should become the client on the local worker.

tshatrov · 2019-06-11T12:26:13Z

That's not the case, the futures from the retrieved dataframe have a client with a different id (which is different for every session)

In [1]: from distributed import Client

In [2]: cl = Client("*redacted*")

In [3]: cl.id
Out[3]: 'Client-99fd4b86-8c42-11e9-b54a-48df3757a8a0'

In [4]: df = cl.get_dataset('test')

In [5]: list(df.dask.values())[0].client.id
Out[5]: 'Client-aaa7eacc-8c42-11e9-b54a-48df3757a8a0'

In [7]: dask.__version__
Out[7]: '1.2.2'

In [9]: distributed.__version__
Out[9]: '1.28.1'

TomAugspurger · 2019-06-13T20:57:02Z

Can anyone provide a reproducible example? This doesn't do it, I suspect because we aren't getting references to the client in the task graph

from distributed import LocalCluster, Client
import dask.array as da


def main():
    cluster = LocalCluster(dashboard_address=None)
    data = da.ones(3)

    with Client(cluster) as a:
        a.publish_dataset(data=data)
        with Client(cluster, set_as_default=False) as b:
            print(b.get_dataset('data').compute())


if __name__ == '__main__':
    main()

TomAugspurger · 2019-06-14T11:37:36Z

Oh @tshatrov, can you repeat your test in #2336 (comment) after calling optimize on the graph? That's when things would be replaced:

distributed/distributed/client.py

Line 2519 in 587be8d

def _optimize_insert_futures(self, dsk, keys):

tshatrov · 2019-06-17T16:23:05Z

STR:

Launch dask-scheduler and dask-worker pointing at dask-scheduler.
Launch ipython and do the following

import dask
from dask.distributed import Client

uri = <dask-scheduler's uri>
cl = Client(uri)
ts = dask.datasets.timeseries()
ts = ts.persist()
cl.publish_dataset(timeseries=ts)

def get_ts():
    from dask.distributed import worker_client
    with worker_client() as client:
       return client.get_dataset('timeseries').compute()

cl.submit(get_ts).result()

As a result I get ValueError: Inputs contain futures that were created by another client.

I can't reproduce this on LocalCluster.

I have not figured out how to use optimize_insert_futures to make this not fail. As far as I see it doesn't change the dataframe.

TomAugspurger · 2019-06-17T20:06:55Z

Thanks @tshatrov. FWIW, I can't reproduce the ValueError locally with that. Perhaps others can.

pranav-kohli · 2019-06-21T13:03:58Z

Any updates on the issue, even I am facing the same problem as mentioned by @tshatrov

martindurant · 2019-06-21T13:14:28Z

Hm, the example code also works OK for me. I wonder what might be different on your system, @pranav-kohli - what exact versions are you using?

pranav-kohli · 2019-06-24T06:26:37Z

I am using the following setup
dask==1.1.0
distributed==1.25.2

mrocklin · 2019-06-24T06:30:09Z

You should maybe try updating Dask and distributed and see if the problem persists.

…

On Mon, Jun 24, 2019 at 8:26 AM pranav-kohli ***@***.***> wrote: I am using the following setup dask==1.1.0 distributed==1.25.2 — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#2336?email_source=notifications&email_token=AACKZTAV5HOAZ4SEQDXX2F3P4BSJ5A5CNFSM4GBZSIQ2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGODYL4TAQ#issuecomment-504875394>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AACKZTEQ5XMEWRX4JFZDVLTP4BSJ5ANCNFSM4GBZSIQQ> .

tshatrov · 2019-06-24T09:55:14Z

As I posted above the problem persists on the most recently released versions.

pranav-kohli · 2019-07-01T09:57:20Z

This is what i used to replicate the issue on the latest versions
dask = 2.0.0
distributed = 2.0.1

from distributed import Client
daskClient = Client('localhost:8786')
df = pd.DataFrame(data=[[1]], columns=['a'])
df = dd.from_pandas(df, npartitions=1)
df = df.persist()
print(daskClient.id)
print(list(df.dask.values())[0].client.id)
future = daskClient.submit(getCompute, df)

def getCompute(df):
    #print(df.id)
    print(list(df.dask.values())[0].client.id)
    df.compute()

The printed ids are different, hence the future.client is self check fails

TomAugspurger · 2019-07-01T12:20:34Z

Thanks @pranav-kohli. Unfortunately this is the output I get for (a lightly modified) version of your script :/

Client-7ef457a4-9bfa-11e9-abd5-186590cd1c87
Client-7ef457a4-9bfa-11e9-abd5-186590cd1c87

import pandas as pd
from distributed import Client
import dask.dataframe as dd


def getCompute(df):
    #print(df.id)
    print(list(df.dask.values())[0].client.id)
    df.compute()


if __name__ == '__main__':
    daskClient = Client()
    df = pd.DataFrame(data=[[1]], columns=['a'])
    df = dd.from_pandas(df, npartitions=1)
    df = df.persist()
    print(daskClient.id)
    print(list(df.dask.values())[0].client.id)
    future = daskClient.submit(getCompute, df)

pranav-kohli · 2019-07-01T12:22:03Z

@TomAugspurger So actually in the test case we are printing the same id twice

print(daskClient.id)
print(list(df.dask.values())[0].client.id)

Can you check your dask worker log which prints from getCompute function?
My dask worker log shows a different client Id

TomAugspurger · 2019-07-08T16:11:32Z

Is anybody who's able to reproduce this issue locally able to debug further?

mrocklin · 2019-08-02T18:40:22Z

Here is a reproducer:

from distributed.deploy.ssh2 import SSHCluster
from distributed import Client
import pandas as pd
import dask.dataframe as dd


def getCompute(df):
    return df.compute()


async def f():
    async with SSHCluster(
        hosts=["localhost", "localhost", "localhost"],
        worker_kwargs={"nthreads": 4},
        connect_kwargs={"known_hosts": None},
        asynchronous=True,
    ) as cluster:

        client = await Client("localhost:8786", asynchronous=True)
        df = pd.DataFrame(data=[[1]], columns=["a"])
        df = dd.from_pandas(df, npartitions=1)
        df = df.persist()
        future = client.submit(getCompute, df)
        await future


if __name__ == "__main__":
    import asyncio

    asyncio.get_event_loop().run_until_complete(f())

Traceback

Traceback (most recent call last):
  File "foo.py", line 28, in <module>
    asyncio.get_event_loop().run_until_complete(f())
  File "/Users/mrocklin/miniconda/envs/dev/lib/python3.7/asyncio/base_events.py", line 573, in run_until_complete
    return future.result()
  File "foo.py", line 23, in f
    await future
  File "/Users/mrocklin/workspace/distributed/distributed/client.py", line 232, in _result
    six.reraise(*exc)
  File "/Users/mrocklin/miniconda/envs/dev/lib/python3.7/site-packages/six.py", line 692, in reraise
    raise value.with_traceback(tb)
  File "foo.py", line 8, in getCompute
    return df.compute()
  File "/Users/mrocklin/workspace/dask/dask/base.py", line 175, in compute
    (result,) = compute(self, traverse=False, **kwargs)
  File "/Users/mrocklin/workspace/dask/dask/base.py", line 446, in compute
    results = schedule(dsk, keys, **kwargs)
  File "/Users/mrocklin/workspace/distributed/distributed/client.py", line 2500, in get
    actors=actors,
  File "/Users/mrocklin/workspace/distributed/distributed/client.py", line 2389, in _graph_to_futures
    raise ValueError(msg)
ValueError: Inputs contain futures that were created by another client.

jacobtomlinson · 2019-08-02T18:49:17Z

I am able to reproduce following @tshatrov's instructions. I've slightly modified it to use get_cluster.

import distributed
client = distributed.Client('tcp://localhost:8786')

import dask
ts = dask.datasets.timeseries()
ts = ts.persist()
client.publish_dataset(timeseries=ts)

def get_ts():
    from dask.distributed import get_client
    with get_client() as client:
        return client.get_dataset('timeseries').compute()

client.submit(get_ts).result()

ValueError: Inputs contain futures that were created by another client.

Dask and distributed 2.1.0.

This helps to normalize scheduler addresses before comparison Fixes dask#2336

jacobtomlinson · 2019-08-07T09:53:06Z

Here is a new minimum reproducible example. The key here is that the clients connects to the scheduler on a different IP to the worker.

# Run dash scheduler on a machine
$ dask-scheduler

# Connect to that scheduler with a worker on the same machine via localhost
$ dask-worker localhost:8786

# Connect to the cluster on a different IP
import distributed
client = distributed.Client('10.1.2.3:8786')  # Or whatever the LAN IP is

# Persist some data and publish it as a dataset
import dask
df = dask.datasets.timeseries().persist()
client.publish_dataset(df=df)

# Try and grab the dataset from within a delayed task
@dask.delayed
def remote_head():
    client = distributed.get_client()
    df = client.get_dataset('df')
    return df.head()

remote_head().compute()

# This results in an error
ValueError: Inputs contain futures that were created by another client.

crusaderky · 2019-11-12T11:34:33Z

I"m stumbling on the same problem on distributed 2.7.0.

Another minimal example:

client = distributed.Client("localhost:8786")

a, = client.persist([delayed(1, pure=True)])
print(a) # Delayed('int-62645d78d66e2508256b7ab60a38b944')
print(a.compute())  # 1
print(client.compute([a])[0].result())  # 1

client.publish_dataset(foo=a)
b = client.get_dataset("foo")

print(b)  # Delayed('int-62645d78d66e2508256b7ab60a38b944')
print(b.compute()) # 1
print(client.compute([b])[0].result())  # ValueError: Inputs contain futures that were created by another client.

The issue disappears if I connect to a LocalCluster instead.

Workaround:

print(client.gather([b.dask[b.key]]))  # 1

This issue is particularly troublesome when using the asynchronous client together with non-trivial collections (read: anything other than a delayed), since the compute() method does not work (dask/dask#5580) so one would have to reassemble the output by hand starting from output of the futures.

pranav-kohli · 2020-03-11T12:51:25Z

Still facing the same issue with Dask 2.6.0 and Distributed 2.6.0.

import pandas as pd
import dask.dataframe as dd
from distributed import Client

def getCompute(df):
    print("client id inside")
    print(list(df.dask.values())[0].client.id)
    df.compute()

daskClient = Client('localhost:8786')
df = pd.DataFrame(data=[[1]], columns=['a'])
df = dd.from_pandas(df, npartitions=1)
df = df.persist()
print(daskClient.id)
future = daskClient.submit(getCompute, df)
print(future.result())

Error:
ValueError: Inputs contain futures that were created by another client.

jrbourbeau · 2020-03-11T14:31:12Z

@pranav-kohli I'm not able to reproduce the error with the latest dask and distributed release (version 2.12.0):

In [1]: import pandas as pd
   ...: import dask.dataframe as dd
   ...: from distributed import Client
   ...:
   ...: def getCompute(df):
   ...:     print("client id inside")
   ...:     print(list(df.dask.values())[0].client.id)
   ...:     df.compute()
   ...:
   ...: daskClient = Client()
   ...: df = pd.DataFrame(data=[[1]], columns=['a'])
   ...: df = dd.from_pandas(df, npartitions=1)
   ...: df = df.persist()
   ...: print(daskClient.id)
   ...: future = daskClient.submit(getCompute, df)
   ...: print(future.result())
Client-cb589a84-63a4-11ea-9e5d-a0999b120aab
client id inside
Client-worker-ccab6e5c-63a4-11ea-9e6b-a0999b120aab
None

Can you update those packages and see if the problem persists?

tshatrov · 2020-03-11T15:58:51Z

Isn't this code using LocalCluster? We already know this bug does not reproduce on a LocalCluster.

martindurant · 2020-03-11T16:02:13Z

LocalCluster has separate processes by default, just like any other deployment

jrbourbeau · 2020-03-11T21:33:33Z

Isn't this code using LocalCluster? We already know this bug does not reproduce on a LocalCluster.

Ah, I missed that when reading through the previous comments. I am able to reproduce the ValueError when not using LocalCluster

martindurant · 2020-03-12T13:17:32Z

How weird! Now that should give us something to diagnose by, but I am pretty mystified.
I wonder, @jrbourbeau , do you get the problem with LocalCluster if you use spawn (which would end up with the client objects having different IDs in each worker) ?

jrbourbeau · 2020-03-12T15:02:05Z

The default multiprocessing method was updated to spawn in #3461. I checked that the problem does not occur with LocalCluster when using either spawn or forkserver multiprocessing methods

pborgen · 2020-05-09T02:13:25Z

This is what i used to replicate the issue on the latest versions
dask = 2.0.0
distributed = 2.0.1
from distributed import Client
daskClient = Client('localhost:8786')
df = pd.DataFrame(data=[[1]], columns=['a'])
df = dd.from_pandas(df, npartitions=1)
df = df.persist()
print(daskClient.id)
print(list(df.dask.values())[0].client.id)
future = daskClient.submit(getCompute, df)

def getCompute(df):
    #print(df.id)
    print(list(df.dask.values())[0].client.id)
    df.compute()
The printed ids are different, hence the future.client is self check fails

I have this same issue with the latest. Anyone one have any ideas?
dask==2.16.0
distributed==2.16.0

crusaderky · 2020-05-09T22:09:40Z

@pborgen my PR resolves the issue for get_dataset() specifically - which is what the opening post was about. What you're doing in that snippet is sending over an arbitrary python object which happens to contain Futures and expect them to be recreated correctly when the Worker deserializes them - which is a different problem, albeit related.

pborgen · 2020-05-10T00:02:26Z

@pborgen my PR resolves the issue for get_dataset() specifically - which is what the opening post was about. What you're doing in that snippet is sending over an arbitrary python object which happens to contain Futures and expect them to be recreated correctly when the Worker deserializes them - which is a different problem, albeit related.

Is there currently a issue to capture this?

crusaderky · 2020-05-11T10:15:47Z

@pborgen there is one now: #3790

Note that, as a workaround, you can use publish, which is currently the only sanctioned way to move collections across clients:

from distributed import Client, get_client

daskClient = Client('localhost:8786')
df = pd.DataFrame(data=[[1]], columns=['a'])
df = dd.from_pandas(df, npartitions=1)
df = df.persist()
daskClient.publish_dataset(foo=df)
try:
    future = daskClient.submit(getCompute, "foo")
    future.result()
finally:
    daskClient.unpublish_dataset("foo")

def getCompute(name):
    df = get_client().get_dataset(name)
    df.compute()

The downside is that, if for any reason the Client is SIGKILL'ed or loses network connectivity before the end of the computation, you'll end up with a memory leak on the cluster.

pborgen · 2020-05-11T15:22:33Z

Thanks for your help.....I upgraded my dev box and all my machines that are part of the cluster to 2.16.0. I am using dask cli with one machine dedicated to the scheduler and 3 other dask workers... When I run with one worker everything works fine. But if I run with 2 workers I get the below error:

Error:
File "c:\python38\lib\site-packages\dask\base.py", line 166, in compute
(result,) = compute(self, traverse=False, **kwargs)
File "c:\python38\lib\site-packages\dask\base.py", line 444, in compute
results = schedule(dsk, keys, **kwargs)
File "c:\python38\lib\site-packages\distributed\client.py", line 2646, in get
used as an optimization to avoid recomputation.
File "c:\python38\lib\site-packages\distributed\client.py", line 2543, in _graph_to_futures

ValueError: Inputs contain futures that were created by another client.
Versions:
dask==2.16.0
distributed==2.16.0

crusaderky · 2020-05-11T15:33:39Z

@pborgen run what? the POC that you posted, or my latest one with publish_dataset/get_dataset?

pborgen · 2020-05-11T15:52:43Z

I am running my code...It is pretty similar to what you posted though..... Your code seems to run fine on my cluster.....I am going to try to run your code with a very large dask dataframe to see if I can reproduce...

pborgen · 2020-05-11T15:54:21Z

I should say that this error I get only happens after things seem to be running for for a minute or so...I am also using a ddf that is created from a parquet file that is 140mb and about 40millions rows

pborgen · 2020-05-11T15:55:16Z

inside my task I am querying this very large parquet file

jcrist · 2020-05-11T15:57:06Z

@pborgen, if you can create a minimal reproducible example, please file a new issue and we can discuss there.

pborgen · 2020-05-11T15:59:11Z

    def task_big_data(id):
        from dask.distributed import get_client

        dataset = get_client().get_dataset('my_dataset')
        df = dataset[dataset.ID== id].compute()  <-----this is where I am getting my error(it does run for a bit before getting the error....Maybe it errors when the task is run on another worker?)

pborgen · 2020-05-11T15:59:58Z

@pborgen, if you can create a minimal reproducible example, please file a new issue and we can discuss there.

Thanks for you help...I will try to get this to you today or tomarrow.

pborgen · 2020-05-11T16:04:06Z

Do you know of a way to programmaticly create a very large dataframe? like at least 40 million rows and 8 columns.

jcrist · 2020-05-11T16:08:28Z

There are some functions for doing this in dask.datasets (e.g. dask.datasets.timeseries). You could also use dask.dataframe.from_dask_array and dask.array.random to make some large random numeric dataframes.

pborgen · 2020-05-11T17:28:03Z

Created #3791

mrocklin added a commit to mrocklin/distributed that referenced this issue Aug 2, 2019

Resolve scheduler address in get_client

742df33

This helps to normalize scheduler addresses before comparison Fixes dask#2336

mrocklin mentioned this issue Aug 2, 2019

Resolve scheduler address in get_client #2918

Open

crusaderky mentioned this issue Nov 12, 2019

Published Future retrieved from async client is synchronous #3227

Closed

crusaderky mentioned this issue Apr 20, 2020

Client.get_dataset to always create Futures attached to itself #3729

Merged

jcrist closed this as completed in #3729 May 7, 2020

crusaderky mentioned this issue May 11, 2020

Futures unpickled on Client/Worker are attached to the wrong Client #3790

Open

pborgen mentioned this issue May 11, 2020

Cannot call compute() on a dask dataframe when inside a task #3791

Closed

chinmaychandak mentioned this issue Jun 22, 2020

Publishing results as Dask datasets. python-streamz/streamz#343

Closed

Unable to use published datasets in a different client #2336

Unable to use published datasets in a different client #2336

Comments

nikhilrajrs commented Nov 5, 2018

VMois commented Nov 11, 2018

nikhilrajrs commented Nov 12, 2018 • edited Loading

VMois commented Nov 12, 2018

VMois commented Nov 13, 2018

VMois commented Nov 16, 2018 • edited Loading

martindurant commented Nov 16, 2018

itozapata commented Apr 22, 2019 • edited Loading

martindurant commented Apr 23, 2019

itozapata commented Apr 23, 2019

tshatrov commented Jun 10, 2019

mrocklin commented Jun 10, 2019

tshatrov commented Jun 11, 2019

TomAugspurger commented Jun 13, 2019

TomAugspurger commented Jun 14, 2019

tshatrov commented Jun 17, 2019

TomAugspurger commented Jun 17, 2019

pranav-kohli commented Jun 21, 2019

martindurant commented Jun 21, 2019

pranav-kohli commented Jun 24, 2019

mrocklin commented Jun 24, 2019 via email

tshatrov commented Jun 24, 2019 • edited Loading

pranav-kohli commented Jul 1, 2019

TomAugspurger commented Jul 1, 2019

pranav-kohli commented Jul 1, 2019 • edited Loading

TomAugspurger commented Jul 8, 2019

mrocklin commented Aug 2, 2019

Traceback

jacobtomlinson commented Aug 2, 2019

jacobtomlinson commented Aug 7, 2019 • edited Loading

crusaderky commented Nov 12, 2019

pranav-kohli commented Mar 11, 2020

jrbourbeau commented Mar 11, 2020

tshatrov commented Mar 11, 2020

martindurant commented Mar 11, 2020

jrbourbeau commented Mar 11, 2020

martindurant commented Mar 12, 2020

jrbourbeau commented Mar 12, 2020

pborgen commented May 9, 2020

crusaderky commented May 9, 2020

pborgen commented May 10, 2020

crusaderky commented May 11, 2020

pborgen commented May 11, 2020

crusaderky commented May 11, 2020

pborgen commented May 11, 2020

pborgen commented May 11, 2020

pborgen commented May 11, 2020

jcrist commented May 11, 2020

pborgen commented May 11, 2020

pborgen commented May 11, 2020

pborgen commented May 11, 2020

jcrist commented May 11, 2020 • edited Loading

pborgen commented May 11, 2020

nikhilrajrs commented Nov 12, 2018 •

edited

Loading

VMois commented Nov 16, 2018 •

edited

Loading

itozapata commented Apr 22, 2019 •

edited

Loading

tshatrov commented Jun 24, 2019 •

edited

Loading

pranav-kohli commented Jul 1, 2019 •

edited

Loading

jacobtomlinson commented Aug 7, 2019 •

edited

Loading

jcrist commented May 11, 2020 •

edited

Loading