Please allow configuration via python

[akvadrako]

Using mixcoatl as a library isn't very elegant, because we have to set environmental variables before all import statements. Also, we need to interact with more than one DCM account in the same app. So the logic is like:

• set fake environmental variables
• import mixcoatl
• before each method call, configure it with the set methods on the Config() object

It would be nice to have a more conventional API.

[bdwilliams]

When importing the entire library via import mixcoatl it triggers the Config() method to be called and throws exceptions if the required fields are not set. Here is a quick sample that shows a way around this:

#!/usr/bin/env python
import csv
from mixcoatl import resource_utils, utils
from mixcoatl.infrastructure.server import Server
from mixcoatl.settings.load_settings import settings as mixcoatl_settings
with open('creds.csv', 'rb') as f:
    reader = csv.reader(f)
    cred_list = list(reader)
    for line in cred_list:
        mixcoatl_settings.set_access_key(line[0])
        mixcoatl_settings.set_secret_key(line[1])
        mixcoatl_settings.set_endpoint(line[2])
        
        print "Results with ACCESS KEY #",line[0]
        print utils.print_format(Server.all(), 'json')

By only importing mixcoatl.settings.load_settings you can manually set these variables on the fly without the exceptions. I also added some quick (mostly untested) logic for reading through a CSV file of credentials and then making a list server call against each discovered endpoint.

Sample creds.csv format:

ACCESS_KEY,SECRET_KEY,ENDPOINT_URL

Let me know if this works or not and we can continue to troubleshoot.

[akvadrako]

Hi Brian, I'm not interested in work-arounds and hacks and this doesn't address the problem of multiple DCM accounts being used in one process.

[zomGreg]

Hi Devin,

Thanks for using mixcoatl! We're happy to have you in the cloud!

We hope you like working with clouds and DCM as much as we do. Sorry we couldn't answer your question with a quick solution. Mixcoatl is open-source and community driven, please fork the mixcoatl repository and make a PR if you have a solution you'd like to contribute. We'll keep this issue open and we appreciate your using DCM and mixcoatl!

zomgreg

[akvadrako]

Hi zomGreg,

that might be possible, but before spending time on it I would like to know that the general design is acceptable. Our goal is:

• to make it safe to import mixcoatl anywhere (like in tests) without any preparation
• allow multiple DCM accounts to be used simultaneously (in threads, for example)

To achieve this, I would add a config parameter to Server, Resource and other places where it is needed. And it would be passed around as necessary. For backwards compatibility, it could default to the global, automatically initialised config object.

Would that be accepted?

[zomGreg]

Hi Devin,

I not sure I'm able to comment intelligently on your design proposal, so let me take a step back for a moment to try to understand the bigger picture.

It sounds to me like you're trying to work with multiple DCM accounts. This is something that the DCM API, and by extension, mixcoatl, allows natively.

For example, with a single set of DCM user API credentials, I can do something like this, listing all the DCM cloud accounts to which my user has access.

➜  handsfree git:(feature/billing_codes) dcm-list-accounts
+------------+--------------------+----------+----------------+--------+
| Account ID | Account Name       | Customer | Default Budget | Status |
+------------+--------------------+----------+----------------+--------+
| 201        | AWS                | 200      | 200            | ACTIVE |
| 202        | CloudStack44       | 200      | 200            | ACTIVE |
| 203        | CloudStack45       | 200      | 200            | ACTIVE |
| 204        | OpenStack Icehouse | 200      | 200            | ACTIVE |
| 205        | vSphere            | 200      | 200            | ACTIVE |
+------------+--------------------+----------+----------------+--------+

From there, I can, for example, list all regions in account 201:

➜  handsfree git:(feature/billing_codes) dcm-list-regions
+-----------+----------------+-------+----------------+----------------+--------+
| Region ID | Provider ID    | Cloud | Region Name    | Description    | Status |
+-----------+----------------+-------+----------------+----------------+--------+
| 12        | ap-northeast-1 | 1     | ap-northeast-1 | ap-northeast-1 | ACTIVE |
| 18        | ap-southeast-1 | 1     | ap-southeast-1 | ap-southeast-1 | ACTIVE |
| 17        | ap-southeast-2 | 1     | ap-southeast-2 | ap-southeast-2 | ACTIVE |
| 10        | eu-central-1   | 1     | eu-central-1   | eu-central-1   | ACTIVE |
| 13        | eu-west-1      | 1     | eu-west-1      | eu-west-1      | ACTIVE |
| 11        | sa-east-1      | 1     | sa-east-1      | sa-east-1      | ACTIVE |
| 14        | us-east-1      | 1     | us-east-1      | us-east-1      | ACTIVE |
| 15        | us-west-1      | 1     | us-west-1      | us-west-1      | ACTIVE |
| 16        | us-west-2      | 1     | us-west-2      | us-west-2      | ACTIVE |
+-----------+----------------+-------+----------------+----------------+--------+

and then I can list all servers in region 16

➜  handsfree git:(feature/billing_codes) dcm-list-servers -r 16
+-----------+--------+-------------+-----------------------------+----------------+----------+--------+-----------+---------+------------------------------+
| Server ID | Region | Provider ID | Server Name                 | Public IP      | Platform | Budget | Product   | Status  | Start Date                   |
+-----------+--------+-------------+-----------------------------+----------------+----------+--------+-----------+---------+------------------------------+
| 73        | 16     | i-adsf      | asdf                        | 01.01.01.01    | UNKNOWN  | 200    | m1.medium | RUNNING | 2015-03-24T16:05:51.000+0000 |
| 71        | 16     | i-asdf      | asdf                        | 01.01.01.01    | UBUNTU   | 200    | m1.medium | RUNNING | 2015-03-24T19:23:14.000+0000 |
| 70        | 16     | i-asdf      | KSTESTOSPv3.7.3 Dont delete | 12.12.12.12    | RHEL     | 200    | m1.small  | RUNNING | 2014-12-09T12:49:00.000+0000 |
| 72        | 16     | i-afasdf    | kstestRHELPEv3.3.0          | None           | RHEL     | 200    | m1.small  | STOPPED | 2015-01-29T06:42:25.000+0000 |
+-----------+--------+-------------+-----------------------------+----------------+----------+--------+-----------+---------+------------------------------+

mixcoatl can also facilitate "generic" REST calls like:

➜  handsfree git:(feature/billing_codes) dcm-get geography/Region/16
{
  "regions": [
    {
      "cloud": {
        "cloudId": 1
      },
      "customer": {
        "customerId": 200
      },
      "description": "us-west-2",
      "jurisdiction": "US",
      "name": "us-west-2",
      "providerId": "us-west-2",
      "regionId": 16,
      "status": "ACTIVE"
    }
  ]
}

which is useful for passing arbitrary data.

Please see our most recent wiki entry
Please also see our:

testing hooks (GET)

testing hooks (POST)

Importing mixcoatl libraries for DCM server, account, and other objects is a feature we're constantly working to improve. For example, you can see account module information here And there are pages that are created for each mixcoatl module.

Making mixcoal have more of an SDK feel is something that's on our project roadmap, but it will take time.

Cheers,
Greg

[akvadrako]

Hi Greg, in your post I think you are mixing up DCM accounts and cloud accounts. We need to support multiple DCM accounts with different DCM API endpoints. And we don't want to use mixcoatl-specific ways of importing or mocking - it should just work like any other library. I'm offering to make a pull request that gives mixcoatl that API-feel, but it's only worth the effort if the design is acceptable to the owners.

Is that the case?

[jaronson]

The limiting of mixcoatl API queries to a single DCM account has been a known limitation with no great workaround, as you saw above. I for one would like to see the PR as you've described. It definitely sounds like a viable candidate for inclusion.

[buzztroll]

I would like to throw in with akvadrako. I need such a feature as well. The proposal sounds reasonable to me and I do not think it would be too hairy either.

The idea here is that you may want to have a single long running python program that interacts with DCM on behalf of many different DCM users at the same time and in overlapping requests. Keeping account information in global state will not work for this (even if you change the global state you quickly run into a case where 2 accounts are trying to set/use it at the same time). Associating a config object with each request has my +1.

[igable]

This was closed with release 1.0.1, take a look at the release notes for details on how to do this.

[igable]

I also want to draw @akvadrako attention to this now being fixed.