Update to use fog-google v1 #180

seanmalloy · 2018-02-05T06:11:30Z

I am attempting to update vagrant-google to use v1.2.0 of the
fog-google library. I have made some progress, but there is still
a lot of work to do.

Any feedback on this PR is welcome.

seanmalloy · 2018-02-05T06:14:33Z

I'm currently running into the below error ...

$ bundle exec vagrant up --provider=google
bundler: failed to load command: vagrant (/home/sean/projects/vagrant-google/vendor/bundle/ruby/2.4.0/bin/vagrant)
NoMethodError: undefined method `get_object' for #<Fog::Compute::Google::Disk:0x00005636e28b6f90>
  /home/sean/projects/vagrant-google/vendor/bundle/ruby/2.4.0/gems/fog-google-1.2.0/lib/fog/compute/google/models/disk.rb:70:in `get_as_boot_disk'
  /home/sean/projects/vagrant-google/lib/vagrant-google/action/run_instance.rb:153:in `call'
  /home/sean/projects/vagrant-google/vendor/bundle/ruby/2.4.0/bundler/gems/vagrant-8ee7b01855e9/lib/vagrant/action/warden.rb:34:in `call'
  /home/sean/projects/vagrant-google/lib/vagrant-google/action/warn_ssh_keys.rb:28:in `call'
  /home/sean/projects/vagrant-google/vendor/bundle/ruby/2.4.0/bundler/gems/vagrant-8ee7b01855e9/lib/vagrant/action/warden.rb:34:in `call'
  /home/sean/projects/vagrant-google/lib/vagrant-google/action/warn_networks.rb:28:in `call'
  /home/sean/projects/vagrant-google/vendor/bundle/ruby/2.4.0/bundler/gems/vagrant-8ee7b01855e9/lib/vagrant/action/warden.rb:34:in `call'
  /home/sean/projects/vagrant-google/lib/vagrant-google/action/sync_folders.rb:33:in `call'
  /home/sean/projects/vagrant-google/vendor/bundle/ruby/2.4.0/bundler/gems/vagrant-8ee7b01855e9/lib/vagrant/action/warden.rb:34:in `call'
  /home/sean/projects/vagrant-google/vendor/bundle/ruby/2.4.0/bundler/gems/vagrant-8ee7b01855e9/lib/vagrant/action/builtin/provision.rb:80:in `call'
  /home/sean/projects/vagrant-google/vendor/bundle/ruby/2.4.0/bundler/gems/vagrant-8ee7b01855e9/lib/vagrant/action/warden.rb:34:in `call'
  /home/sean/projects/vagrant-google/vendor/bundle/ruby/2.4.0/bundler/gems/vagrant-8ee7b01855e9/lib/vagrant/action/warden.rb:95:in `block in finalize_action'
  /home/sean/projects/vagrant-google/vendor/bundle/ruby/2.4.0/bundler/gems/vagrant-8ee7b01855e9/lib/vagrant/action/warden.rb:34:in `call'
  /home/sean/projects/vagrant-google/vendor/bundle/ruby/2.4.0/bundler/gems/vagrant-8ee7b01855e9/lib/vagrant/action/builder.rb:116:in `call'
  /home/sean/projects/vagrant-google/vendor/bundle/ruby/2.4.0/bundler/gems/vagrant-8ee7b01855e9/lib/vagrant/action/runner.rb:66:in `block in run'
  /home/sean/projects/vagrant-google/vendor/bundle/ruby/2.4.0/bundler/gems/vagrant-8ee7b01855e9/lib/vagrant/util/busy.rb:19:in `busy'
  /home/sean/projects/vagrant-google/vendor/bundle/ruby/2.4.0/bundler/gems/vagrant-8ee7b01855e9/lib/vagrant/action/runner.rb:66:in `run'
  /home/sean/projects/vagrant-google/vendor/bundle/ruby/2.4.0/bundler/gems/vagrant-8ee7b01855e9/lib/vagrant/action/builtin/call.rb:53:in `call'
  /home/sean/projects/vagrant-google/vendor/bundle/ruby/2.4.0/bundler/gems/vagrant-8ee7b01855e9/lib/vagrant/action/warden.rb:34:in `call'
  /home/sean/projects/vagrant-google/lib/vagrant-google/action/connect_google.rb:47:in `call'
  /home/sean/projects/vagrant-google/vendor/bundle/ruby/2.4.0/bundler/gems/vagrant-8ee7b01855e9/lib/vagrant/action/warden.rb:34:in `call'
  /home/sean/projects/vagrant-google/vendor/bundle/ruby/2.4.0/bundler/gems/vagrant-8ee7b01855e9/lib/vagrant/action/builtin/box_check_outdated.rb:79:in `call'
  /home/sean/projects/vagrant-google/vendor/bundle/ruby/2.4.0/bundler/gems/vagrant-8ee7b01855e9/lib/vagrant/action/warden.rb:34:in `call'
  /home/sean/projects/vagrant-google/vendor/bundle/ruby/2.4.0/bundler/gems/vagrant-8ee7b01855e9/lib/vagrant/action/builtin/config_validate.rb:25:in `call'
  /home/sean/projects/vagrant-google/vendor/bundle/ruby/2.4.0/bundler/gems/vagrant-8ee7b01855e9/lib/vagrant/action/warden.rb:34:in `call'
  /home/sean/projects/vagrant-google/vendor/bundle/ruby/2.4.0/bundler/gems/vagrant-8ee7b01855e9/lib/vagrant/action/builtin/handle_box.rb:56:in `call'
  /home/sean/projects/vagrant-google/vendor/bundle/ruby/2.4.0/bundler/gems/vagrant-8ee7b01855e9/lib/vagrant/action/warden.rb:34:in `call'
  /home/sean/projects/vagrant-google/vendor/bundle/ruby/2.4.0/bundler/gems/vagrant-8ee7b01855e9/lib/vagrant/action/builder.rb:116:in `call'
  /home/sean/projects/vagrant-google/vendor/bundle/ruby/2.4.0/bundler/gems/vagrant-8ee7b01855e9/lib/vagrant/action/runner.rb:66:in `block in run'
  /home/sean/projects/vagrant-google/vendor/bundle/ruby/2.4.0/bundler/gems/vagrant-8ee7b01855e9/lib/vagrant/util/busy.rb:19:in `busy'
  /home/sean/projects/vagrant-google/vendor/bundle/ruby/2.4.0/bundler/gems/vagrant-8ee7b01855e9/lib/vagrant/action/runner.rb:66:in `run'
  /home/sean/projects/vagrant-google/vendor/bundle/ruby/2.4.0/bundler/gems/vagrant-8ee7b01855e9/lib/vagrant/machine.rb:227:in `action_raw'
  /home/sean/projects/vagrant-google/vendor/bundle/ruby/2.4.0/bundler/gems/vagrant-8ee7b01855e9/lib/vagrant/machine.rb:202:in `block in action'
  /home/sean/projects/vagrant-google/vendor/bundle/ruby/2.4.0/bundler/gems/vagrant-8ee7b01855e9/lib/vagrant/environment.rb:592:in `lock'
  /home/sean/projects/vagrant-google/vendor/bundle/ruby/2.4.0/bundler/gems/vagrant-8ee7b01855e9/lib/vagrant/machine.rb:188:in `call'
  /home/sean/projects/vagrant-google/vendor/bundle/ruby/2.4.0/bundler/gems/vagrant-8ee7b01855e9/lib/vagrant/machine.rb:188:in `action'
  /home/sean/projects/vagrant-google/vendor/bundle/ruby/2.4.0/bundler/gems/vagrant-8ee7b01855e9/lib/vagrant/batch_action.rb:82:in `block (2 levels) in run'

I opened fog/fog-google#300 as I believe this might be a bug in the fog-google library.

seanmalloy · 2018-02-06T06:57:18Z

Example Vagrantfile used for testing is below. There are going to be quite a few code changes required to get this working.

Vagrant.configure("2") do |config|
  config.vm.box = "google/gce"
  config.vm.synced_folder ".", "/vagrant", disabled: true

  config.vm.provider :google do |google, override|
    google.google_project_id = "smalloy-sbx1"
    google.google_client_email = "[email protected]"
    google.google_json_key_location = "./smalloy-sbx1-eee15af2a503.json"

    google.image = "projects/centos-cloud/global/images/centos-7-v20180129"
    google.preemptible = true
    google.auto_restart = false
    google.on_host_maintenance = "TERMINATE"

    # See resource.networkInterfaces[].accessConfigs[] below for details
    #
    # https://cloud.google.com/compute/docs/reference/rest/v1/instances/insert
    google.network = [ {
      :network        => "projects/smalloy-sbx1/global/networks/sandbox-1",
      :subnetwork     => "projects/smalloy-sbx1/regions/us-central1/subnetworks/us-central1-sandbox-1",
      :access_configs => [{ :name => "External NAT" }],  # <-- required so that we get an external IP
    } ]
    google.tags = { :items => ["ssh"] }

    override.ssh.username = "sean.malloy"
    override.ssh.private_key_path = "~/.ssh/id_rsa"
  end
end

seanmalloy · 2018-02-09T05:16:24Z

With the above example Vagrantfile I can successfully run vagrant up --provider=google, but vagrant cannot connect to the GCE instance. It seems like the host information is nil. I need to debug this further.

DEBUG ssh: == Net-SSH connection debug-level log END ==
 INFO retryable: Retryable exception raised: #<Errno::ECONNREFUSED: Connection refused - connect(2) for 127.0.0.1:22>
 INFO ssh: Attempting to connect to SSH...
 INFO ssh:   - Host: 
 INFO ssh:   - Port: 22
 INFO ssh:   - Username: sean.malloy
 INFO ssh:   - Password? false
 INFO ssh:   - Key Path: ["/home/sean/.ssh/id_rsa"]

Here are some notes for myself, so I can continue debugging this in the future.

Debugging command ...

VAGRANT_LOG=debug bundle exec vagrant up --provider=google

https://github.com/hashicorp/vagrant/blob/master/plugins/communicators/ssh/communicator.rb#L311
https://github.com/KohlsTechnology/vagrant-google/blob/update-fog-google/lib/vagrant-google/action/run_instance.rb#L189

seanmalloy · 2018-02-09T06:48:42Z

I fixed the SSH glitch with more Vagrantfile hacking. Now the "up", "ssh", and "destroy" commands can be executed successfully. Although the "destroy command does not remove the disk, not sure if this is the expected behaviour or not.

Next up ...

Write some Ruby code to get rid of the Vagrantfile hacks. :-)

seanmalloy · 2018-02-11T22:42:55Z

Updated Vagrantfile example used for testing ...

Vagrant.configure("2") do |config|
  config.vm.box = "google/gce"
  config.vm.synced_folder ".", "/vagrant", disabled: true

  config.vm.provider :google do |google, override|
    google.google_project_id = "smalloy-sbx1"
    google.google_client_email = "[email protected]"
    google.google_json_key_location = "./smalloy-sbx1-eee15af2a503.json"

    google.image = "projects/centos-cloud/global/images/centos-7-v20180129"
    google.preemptible = true
    google.auto_restart = false
    google.on_host_maintenance = "TERMINATE"

    # See resource.networkInterfaces[] below for details
    #
    # https://cloud.google.com/compute/docs/reference/rest/v1/instances/insert
    #
    # TODO: having to specify the full path to the subnetwork seems very verbose
    google.network     = "sandbox-1"
    google.subnetwork  = "us-central1-sandbox-1"

    google.tags = ["ssh"]

    override.ssh.username = "sean.malloy"
    override.ssh.private_key_path = "~/.ssh/id_rsa"
  end
end

seanmalloy · 2018-02-12T03:23:25Z

seanmalloy · 2018-02-14T05:59:50Z

Need PR fog/fog-google#304 to fix config options preemptible, on_host_maintenance, and auto_start.

seanmalloy · 2018-02-19T05:25:29Z

Need PR fog/fog-google#305 merged to fix config options scopes and service_accounts.

seanmalloy · 2018-02-28T06:12:46Z

The last feature I need to fix is the disk auto delete feature. This is not working. After running vagrant destroy only the GCE instance is deleted, but not the disk. I confirmed this is working as expected from the master branch, so it seems like a regression when updating to fog-google v1.

Notes for my future self when I work on this in the near future.

https://github.com/fog/fog-google/blob/master/lib/fog/compute/google/models/disk.rb#L69
https://cloud.google.com/compute/docs/reference/rest/v1/instances/setDiskAutoDelete

seanmalloy · 2018-02-28T06:18:50Z

@Temikus and @erjohnso what are your thoughts on dropping support for the google_key_location config option? It seems like the JSON key type is preferred over the P12 format, and P12 is considered legacy.

erjohnso · 2018-02-28T14:52:43Z

Hey @seanmalloy - first, thank you very much for taking on this work! It's been a long haul to get the fog client updated and I'm glad to see other projects starting to pick it up.

You're correct that P12 is legacy and will likely be dropped at some point. However, it's still an option for users to create that type of key, so it probably makes sense to keep that feature around. I would definitely suggest making JSON key the default and even add documentation that P12 is on the outs to discourage any new users from going that route.

Also, I'm very appreciative of this work. You have a standing invitation to join me and the team for lunch next time you make it to Seattle! Feel free to hit me up anytime at erjohnso[at]google[dot]com.

seanmalloy · 2018-02-28T15:51:29Z

@erjohnso sounds good I will also get the P12 key stuff tested to make sure it works. I think I tried it a few weeks back and it was not working. I'm hoping to have this PR ready for review in the near future. Most of the functionality seems to be working now. Just have a few loose ends to finish off.

seanmalloy · 2018-03-04T07:08:54Z

Waiting for the below pull request to be merged to fix the disk auto delete feature of "vagrant destroy".

fog/fog-google#307

TimRots · 2018-03-05T19:58:40Z

Awesome work Sean :) 👍

seanmalloy · 2018-03-06T06:26:01Z

@erjohnso and @Temikus it looks like v1 of the fog-google library deprecated the option for using the P12 key type. From what I can tell we now have to use JSON keys instead.

See this block of code for details ...

https://github.com/fog/fog-google/blob/master/lib/fog/google/shared.rb#L60-L64

Here is the stack trace I get when try to use the google.google_key_location option in my Vagrantfile.

ArgumentError: Deprecated argument no longer works: google_key_location
  /home/sean/projects/vagrant-google/vendor/bundle/ruby/2.4.0/bundler/gems/fog-google-c42060f42df5/lib/fog/google/shared.rb:63:in `initialize_google_client'
  /home/sean/projects/vagrant-google/vendor/bundle/ruby/2.4.0/bundler/gems/fog-google-c42060f42df5/lib/fog/compute/google/real.rb:14:in `initialize'
  /home/sean/projects/vagrant-google/vendor/bundle/ruby/2.4.0/gems/fog-core-2.0.0/lib/fog/core/service.rb:115:in `new'
  /home/sean/projects/vagrant-google/vendor/bundle/ruby/2.4.0/gems/fog-core-2.0.0/lib/fog/core/service.rb:115:in `new'
  /home/sean/projects/vagrant-google/vendor/bundle/ruby/2.4.0/gems/fog-core-2.0.0/lib/fog/core/services_mixin.rb:16:in `new'
  /home/sean/projects/vagrant-google/vendor/bundle/ruby/2.4.0/gems/fog-core-2.0.0/lib/fog/compute.rb:54:in `new'
  /home/sean/projects/vagrant-google/lib/vagrant-google/action/connect_google.rb:45:in `call'
  /home/sean/projects/vagrant-google/vendor/bundle/ruby/2.4.0/bundler/gems/vagrant-04fdab961e25/lib/vagrant/action/warden.rb:34:in `call'
  /home/sean/projects/vagrant-google/vendor/bundle/ruby/2.4.0/bundler/gems/vagrant-04fdab961e25/lib/vagrant/action/builtin/config_validate.rb:25:in `call'
  /home/sean/projects/vagrant-google/vendor/bundle/ruby/2.4.0/bundler/gems/vagrant-04fdab961e25/lib/vagrant/action/warden.rb:34:in `call'
  /home/sean/projects/vagrant-google/vendor/bundle/ruby/2.4.0/bundler/gems/vagrant-04fdab961e25/lib/vagrant/action/builder.rb:116:in `call'
  /home/sean/projects/vagrant-google/vendor/bundle/ruby/2.4.0/bundler/gems/vagrant-04fdab961e25/lib/vagrant/action/runner.rb:66:in `block in run'
  /home/sean/projects/vagrant-google/vendor/bundle/ruby/2.4.0/bundler/gems/vagrant-04fdab961e25/lib/vagrant/util/busy.rb:19:in `busy'
  /home/sean/projects/vagrant-google/vendor/bundle/ruby/2.4.0/bundler/gems/vagrant-04fdab961e25/lib/vagrant/action/runner.rb:66:in `run'
  /home/sean/projects/vagrant-google/vendor/bundle/ruby/2.4.0/bundler/gems/vagrant-04fdab961e25/lib/vagrant/machine.rb:227:in `action_raw'
  /home/sean/projects/vagrant-google/vendor/bundle/ruby/2.4.0/bundler/gems/vagrant-04fdab961e25/lib/vagrant/machine.rb:202:in `block in action'
  /home/sean/projects/vagrant-google/vendor/bundle/ruby/2.4.0/bundler/gems/vagrant-04fdab961e25/lib/vagrant/environment.rb:592:in `lock'
  /home/sean/projects/vagrant-google/vendor/bundle/ruby/2.4.0/bundler/gems/vagrant-04fdab961e25/lib/vagrant/machine.rb:188:in `call'
  /home/sean/projects/vagrant-google/vendor/bundle/ruby/2.4.0/bundler/gems/vagrant-04fdab961e25/lib/vagrant/machine.rb:188:in `action'
  /home/sean/projects/vagrant-google/lib/vagrant-google/provider.rb:45:in `state'
  /home/sean/projects/vagrant-google/vendor/bundle/ruby/2.4.0/bundler/gems/vagrant-04fdab961e25/lib/vagrant/machine.rb:514:in `state'
  /home/sean/projects/vagrant-google/vendor/bundle/ruby/2.4.0/bundler/gems/vagrant-04fdab961e25/lib/vagrant/machine.rb:145:in `initialize'
  /home/sean/projects/vagrant-google/vendor/bundle/ruby/2.4.0/bundler/gems/vagrant-04fdab961e25/lib/vagrant/vagrantfile.rb:79:in `new'
  /home/sean/projects/vagrant-google/vendor/bundle/ruby/2.4.0/bundler/gems/vagrant-04fdab961e25/lib/vagrant/vagrantfile.rb:79:in `machine'
  /home/sean/projects/vagrant-google/vendor/bundle/ruby/2.4.0/bundler/gems/vagrant-04fdab961e25/lib/vagrant/environment.rb:694:in `machine'
  /home/sean/projects/vagrant-google/vendor/bundle/ruby/2.4.0/bundler/gems/vagrant-04fdab961e25/lib/vagrant/plugin/v2/command.rb:180:in `block in with_target_vms'
  /home/sean/projects/vagrant-google/vendor/bundle/ruby/2.4.0/bundler/gems/vagrant-04fdab961e25/lib/vagrant/plugin/v2/command.rb:204:in `block in with_target_vms'
  /home/sean/projects/vagrant-google/vendor/bundle/ruby/2.4.0/bundler/gems/vagrant-04fdab961e25/lib/vagrant/plugin/v2/command.rb:186:in `each'
  /home/sean/projects/vagrant-google/vendor/bundle/ruby/2.4.0/bundler/gems/vagrant-04fdab961e25/lib/vagrant/plugin/v2/command.rb:186:in `with_target_vms'
  /home/sean/projects/vagrant-google/vendor/bundle/ruby/2.4.0/bundler/gems/vagrant-04fdab961e25/plugins/commands/up/command.rb:87:in `execute'
  /home/sean/projects/vagrant-google/vendor/bundle/ruby/2.4.0/bundler/gems/vagrant-04fdab961e25/lib/vagrant/cli.rb:46:in `execute'
  /home/sean/projects/vagrant-google/vendor/bundle/ruby/2.4.0/bundler/gems/vagrant-04fdab961e25/lib/vagrant/environment.rb:269:in `cli'
  /home/sean/projects/vagrant-google/vendor/bundle/ruby/2.4.0/bundler/gems/vagrant-04fdab961e25/bin/vagrant:151:in `<top (required)>'
  /home/sean/projects/vagrant-google/vendor/bundle/ruby/2.4.0/bin/vagrant:23:in `load'
  /home/sean/projects/vagrant-google/vendor/bundle/ruby/2.4.0/bin/vagrant:23:in `<top (required)>'

What are your thoughts on this? Can I remove the support for the P12 key?

I have tested all of the other options, and everything seems to be working. I just need to do a few code clean ups before this PR is ready for review.

seanmalloy · 2018-03-07T05:11:27Z

The final count down ...

remove google_key_location config option
fix instance group status wait loop
fix deprecation warning when adding to an instance group
update Gemfile and Gem Spec to use released version of fog-google

seanmalloy · 2018-03-08T06:40:58Z

I created fog/fog-google/issues/308 to get a better understanding of how to get rid of the deprecation warning when adding instance to an instance group. Not 100% sure if this is a bug in the fog-google library or not.

seanmalloy · 2018-03-08T07:04:26Z

Created fog/fog-google/issues/309 requesting a new release of the fog-google gem.

seanmalloy · 2018-03-09T04:18:20Z

I cleaned up my commit messages. Everything seems to be working. The only remaining task is to remove the google_key_location option. I should have time to do this last task in the next few days.

@erjohnso and @Temikus let me know if you have any feedback on this PR. It should be good enough for a review now.

The fog-google v1 library dropped support for the P12 key. Now only JSON keys are supported.

seanmalloy · 2018-03-10T04:26:05Z

I removed support for the google_key_location config option. Now only JSON key files can be used. the v1 version of the fog-google library dropped support for P12 keys. I believe this pull request is now code complete, and is ready for review.

@erjohnso and @Temikus I'm looking forward to feed back on this PR.

https://cloud.google.com/compute/docs/labeling-resources

seanmalloy · 2018-03-10T06:07:39Z

As a bonus I've just pushed up a commit to add support for GCE labels. I hope you like it!

https://cloud.google.com/compute/docs/labeling-resources

erjohnso · 2018-03-12T16:47:27Z

I like bonus features! Thank you @seanmalloy!

I don't know when @Temikus will have a chance to dig in. But I likely won't have a good chunk of time until this coming weekend due to travel and busy dayjob.

seanmalloy · 2018-03-12T22:11:38Z

@erjohnso I totally understand your time constraints. Thanks for the update!

erjohnso · 2018-03-19T10:28:52Z

Hiya @seanmalloy,

Ok, looked it over and it looks good to me. Probably fixed a few bugs along the way. 👍

Given that P12 will stop working, I think we'll want to bump this to 2.0.0 release. To get ready for that, could you also update the README and CHANGELOG?

seanmalloy · 2018-03-20T02:46:27Z

@erjohnso I updated the CHANGELOG. I believe I already made the required updates to the README. Let me know I missed anything.

erjohnso · 2018-03-20T14:13:56Z

I'll work on the 2.0 release - thank you very much for this fantastic work @seanmalloy!

Temikus · 2018-03-22T05:42:21Z

Sorry, I lapsed off of GitHub for the last 2 months, just started looking into my notifications. @erjohnso I can cut a release and run some tests while I'm at it tomorrow. Or you're already on top of it?

Temikus · 2018-03-22T05:42:39Z

And huge thanks to Sean, this is great :)

erjohnso · 2018-03-22T13:33:20Z

@Temikus - go for it! Thank you!

Temikus · 2018-03-27T08:44:58Z

@seanmalloy QQ since I want to make sure I'm not breaking something - why did you add a runtime dependency on ffi here and strict-locked the version?

This is causing issues in latest version of Vagrant (2.0.3) since vagrant's vendored install requires ffi 1.9.23.

EDIT: Let me open a bug for this. #183

Update to use fog-google v1

8e1f559

seanmalloy changed the title ~~WIP: upgrade to fog-google v1.x~~ Update to use fog-google v1 Mar 9, 2018

Remove google_key_location config option

debc782

The fog-google v1 library dropped support for the P12 key. Now only JSON keys are supported.

Add support for adding labels to GCE instance

8bff7f8

https://cloud.google.com/compute/docs/labeling-resources

Update CHANGELOG for 2.0.0 release

72c4579

erjohnso merged commit a86df95 into mitchellh:master Mar 20, 2018

This was referenced Mar 23, 2018

Release 2.0 #181

Merged

google-api-client warning messages when provisioning multiple instances #54

Closed

seanmalloy deleted the update-fog-google branch March 27, 2018 03:27

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Update to use fog-google v1 #180

Update to use fog-google v1 #180

seanmalloy commented Feb 5, 2018

seanmalloy commented Feb 5, 2018

seanmalloy commented Feb 6, 2018 •

edited

Loading

seanmalloy commented Feb 9, 2018

seanmalloy commented Feb 9, 2018

seanmalloy commented Feb 11, 2018 •

edited

Loading

seanmalloy commented Feb 12, 2018 •

edited

Loading

seanmalloy commented Feb 14, 2018

seanmalloy commented Feb 19, 2018

seanmalloy commented Feb 28, 2018 •

edited

Loading

seanmalloy commented Feb 28, 2018

erjohnso commented Feb 28, 2018

seanmalloy commented Feb 28, 2018 •

edited

Loading

seanmalloy commented Mar 4, 2018 •

edited

Loading

TimRots commented Mar 5, 2018

seanmalloy commented Mar 6, 2018

seanmalloy commented Mar 7, 2018 •

edited

Loading

seanmalloy commented Mar 8, 2018

seanmalloy commented Mar 8, 2018

seanmalloy commented Mar 9, 2018

seanmalloy commented Mar 10, 2018 •

edited

Loading

seanmalloy commented Mar 10, 2018

erjohnso commented Mar 12, 2018

seanmalloy commented Mar 12, 2018

erjohnso commented Mar 19, 2018

seanmalloy commented Mar 20, 2018

erjohnso commented Mar 20, 2018

Temikus commented Mar 22, 2018

Temikus commented Mar 22, 2018

erjohnso commented Mar 22, 2018

Temikus commented Mar 27, 2018 •

edited

Loading

Update to use fog-google v1 #180

Update to use fog-google v1 #180

Conversation

seanmalloy commented Feb 5, 2018

seanmalloy commented Feb 5, 2018

seanmalloy commented Feb 6, 2018 • edited Loading

seanmalloy commented Feb 9, 2018

seanmalloy commented Feb 9, 2018

seanmalloy commented Feb 11, 2018 • edited Loading

seanmalloy commented Feb 12, 2018 • edited Loading

seanmalloy commented Feb 14, 2018

seanmalloy commented Feb 19, 2018

seanmalloy commented Feb 28, 2018 • edited Loading

seanmalloy commented Feb 28, 2018

erjohnso commented Feb 28, 2018

seanmalloy commented Feb 28, 2018 • edited Loading

seanmalloy commented Mar 4, 2018 • edited Loading

TimRots commented Mar 5, 2018

seanmalloy commented Mar 6, 2018

seanmalloy commented Mar 7, 2018 • edited Loading

seanmalloy commented Mar 8, 2018

seanmalloy commented Mar 8, 2018

seanmalloy commented Mar 9, 2018

seanmalloy commented Mar 10, 2018 • edited Loading

seanmalloy commented Mar 10, 2018

erjohnso commented Mar 12, 2018

seanmalloy commented Mar 12, 2018

erjohnso commented Mar 19, 2018

seanmalloy commented Mar 20, 2018

erjohnso commented Mar 20, 2018

Temikus commented Mar 22, 2018

Temikus commented Mar 22, 2018

erjohnso commented Mar 22, 2018

Temikus commented Mar 27, 2018 • edited Loading

seanmalloy commented Feb 6, 2018 •

edited

Loading

seanmalloy commented Feb 11, 2018 •

edited

Loading

seanmalloy commented Feb 12, 2018 •

edited

Loading

seanmalloy commented Feb 28, 2018 •

edited

Loading

seanmalloy commented Feb 28, 2018 •

edited

Loading

seanmalloy commented Mar 4, 2018 •

edited

Loading

seanmalloy commented Mar 7, 2018 •

edited

Loading

seanmalloy commented Mar 10, 2018 •

edited

Loading

Temikus commented Mar 27, 2018 •

edited

Loading