Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

DynamoDB support #297

Merged
merged 18 commits into from
Sep 21, 2017
Merged

DynamoDB support #297

merged 18 commits into from
Sep 21, 2017

Conversation

saratlingamarla
Copy link
Contributor

@saratlingamarla saratlingamarla commented Sep 11, 2017

Before submitting pull requests, please see the
Development documentation
and specifically the Pull Request Guidelines.

IMPORTANT: Please take note of the below checklist, especially the first two items.

Pull Request Checklist

  • All pull requests should be against the develop branch, not master.
  • All pull requests must include the Contributor License Agreement (see below).
  • Code should conform to the Development Guidelines:
    • pep8 compliant with some exceptions (see pytest.ini)
    • 100% test coverage with pytest (with valid tests). If you have difficulty
      writing tests for the code, feel free to ask for help or submit the PR without tests.
    • Complete, correctly-formatted documentation for all classes, functions and methods.
    • documentation has been rebuilt with tox -e docs
    • Connections to the AWS services should only be made by the class's
      connect() and connect_resource() methods, inherited from
      awslimitchecker.connectable.Connectable
    • All modules should have (and use) module-level loggers.
    • Commit messages should be meaningful, and reference the Issue number
      if you're working on a GitHub issue (i.e. "issue #x - "). Please
      refrain from using the "fixes #x" notation unless you are sure that the
      the issue is fixed in that commit.
    • Git history is fully intact; please do not squash or rewrite history.

Contributor License Agreement

By submitting this work for inclusion in awslimitchecker, I agree to the following terms:

  • The contribution included in this request (and any subsequent revisions or versions of it)
    is being made under the same license as the awslimitchecker project (the Affero GPL v3,
    or any subsequent version of that license if adopted by awslimitchecker).
  • My contribution may perpetually be included in and distributed with awslimitchecker; submitting
    this pull request grants a perpetual, global, unlimited license for it to be used and distributed
    under the terms of awslimitchecker's license.
  • I have the legal power and rights to agree to these terms.

@saratlingamarla
Copy link
Contributor Author

I'm getting an AttributeError, not sure why -

region_name = self.conn._client_config.region_name
AttributeError: '_DynamodbService' object has no attribute 'conn'

:returns: dict of limit names to :py:class:`~.AwsLimit` objects
:rtype: dict
"""
region_name = self.conn._client_config.region_name
Copy link
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You need a call to self.connect() before this line.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

that's exactly what I did

Copy link
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ok. Can you then try to post the full traceback, and debug-level output?

@jantman
Copy link
Owner

jantman commented Sep 11, 2017

Sorry I haven't had a chance to look at this yet, it might be another day or two before I can.

That AttributeError is also causing the unit tests to fail.

The AttributeError is because self.get_limits() is missing a call to self.connect().

@saratlingamarla
Copy link
Contributor Author

adding a self.connect() in get_limits() still gives me the same Error.

@jantman jantman added the ready label Sep 13, 2017
@jantman
Copy link
Owner

jantman commented Sep 13, 2017

@saratlingamarla I should have some time in the next few days to finally work on the next release. It's up to you if you'd like me to assist with this PR however I can, or just pick it up where it is right now and finish it up myself.

If you'd like some help debugging the AttributeError, please (1) make sure the code pushed up in your branch is updated with what you're running, and (2) paste the full debug-level (-vv) output and the full traceback.

@saratlingamarla
Copy link
Contributor Author

saratlingamarla commented Sep 14, 2017

@jantman please take a look at the full debug level output :

(awslimitchecker) sarat@sarat-Inspiron-3542:~$ awslimitchecker -vv
awslimitchecker 0.11.0 is AGPL-licensed free software; all users have a right to the full source code of this version. See <https://github.com/jantman/awslimitchecker>
2017-09-13 23:05:02,220 [DEBUG checker.py:200 - awslimitchecker.checker._boto_conn_kwargs() ] Connecting to region None
Traceback (most recent call last):
  File "/home/sarat/Desktop/awslimitchecker/bin/awslimitchecker", line 11, in <module>
    load_entry_point('awslimitchecker==0.11.0', 'console_scripts', 'awslimitchecker')()
  File "/home/sarat/Desktop/awslimitchecker/local/lib/python2.7/site-packages/awslimitchecker/runner.py", line 405, in console_entry_point
    r.console_entry_point()
  File "/home/sarat/Desktop/awslimitchecker/local/lib/python2.7/site-packages/awslimitchecker/runner.py", line 362, in console_entry_point
    ta_refresh_timeout=args.ta_refresh_timeout
  File "/home/sarat/Desktop/awslimitchecker/local/lib/python2.7/site-packages/awslimitchecker/checker.py", line 157, in __init__
    boto_conn_kwargs)
  File "/home/sarat/Desktop/awslimitchecker/local/lib/python2.7/site-packages/awslimitchecker/services/base.py", line 104, in __init__
    self.limits = self.get_limits()
  File "/home/sarat/Desktop/awslimitchecker/local/lib/python2.7/site-packages/awslimitchecker/services/dynamodb.py", line 149, in get_limits
    self.connect()
  File "/home/sarat/Desktop/awslimitchecker/local/lib/python2.7/site-packages/awslimitchecker/connectable.py", line 85, in connect
    if self.conn is not None:
AttributeError: '_DynamodbService' object has no attribute 'conn'

@jantman
Copy link
Owner

jantman commented Sep 15, 2017

Ok, it took me a while to figure out exactly what was going on here.

This is the only service that has per-region limits, so this is the only service that's calling self.connect() in get_limits(). It turns out there's a slight ordering issue that was causing this to break.

In awslimitchecker/services/base.py init() you'll need to switch lines 103-4 and 105-6 around so that self.conn and self.resource_conn are created before get_limits() is called. I.e. change this:

        self.limits = {}
        self.limits = self.get_limits()
        self.conn = None
        self.resource_conn = None

to:

        self.conn = None
        self.resource_conn = None
        self.limits = {}
        self.limits = self.get_limits()

That should at least get the code executing.

@saratlingamarla
Copy link
Contributor Author

debug output -

(awslimitchecker) sarat@sarat-Inspiron-3542:~/Desktop/awslimitchecker$ awslimitchecker -vv
awslimitchecker 0.11.0@bf4fa7dc is AGPL-licensed free software; all users have a right to the full source code of this version. See https://github.com/jantman/awslimitchecker
2017-09-15 22:46:02,011 [DEBUG checker.py:200 - awslimitchecker.checker._boto_conn_kwargs() ] Connecting to region None
Traceback (most recent call last):
File "/home/sarat/Desktop/awslimitchecker/bin/awslimitchecker", line 11, in
load_entry_point('awslimitchecker', 'console_scripts', 'awslimitchecker')()
File "/home/sarat/Desktop/awslimitchecker/src/awslimitchecker/awslimitchecker/runner.py", line 405, in console_entry_point
r.console_entry_point()
File "/home/sarat/Desktop/awslimitchecker/src/awslimitchecker/awslimitchecker/runner.py", line 362, in console_entry_point
ta_refresh_timeout=args.ta_refresh_timeout
File "/home/sarat/Desktop/awslimitchecker/src/awslimitchecker/awslimitchecker/checker.py", line 157, in init
boto_conn_kwargs)
File "/home/sarat/Desktop/awslimitchecker/src/awslimitchecker/awslimitchecker/services/base.py", line 106, in init
self.limits = self.get_limits()
File "/home/sarat/Desktop/awslimitchecker/src/awslimitchecker/awslimitchecker/services/dynamodb.py", line 149, in get_limits
self.connect()
File "/home/sarat/Desktop/awslimitchecker/src/awslimitchecker/awslimitchecker/connectable.py", line 88, in connect
self.conn = boto3.client(self.api_name, **kwargs)
File "/home/sarat/Desktop/awslimitchecker/local/lib/python2.7/site-packages/boto3/init.py", line 83, in client
return _get_default_session().client(*args, **kwargs)
File "/home/sarat/Desktop/awslimitchecker/local/lib/python2.7/site-packages/boto3/session.py", line 263, in client
aws_session_token=aws_session_token, config=config)
File "/home/sarat/Desktop/awslimitchecker/local/lib/python2.7/site-packages/botocore/session.py", line 836, in create_client
client_config=config, api_version=api_version)
File "/home/sarat/Desktop/awslimitchecker/local/lib/python2.7/site-packages/botocore/client.py", line 71, in create_client
verify, credentials, scoped_config, client_config, endpoint_bridge)
File "/home/sarat/Desktop/awslimitchecker/local/lib/python2.7/site-packages/botocore/client.py", line 283, in _get_client_args
verify, credentials, scoped_config, client_config, endpoint_bridge)
File "/home/sarat/Desktop/awslimitchecker/local/lib/python2.7/site-packages/botocore/args.py", line 45, in get_client_args
endpoint_url, is_secure, scoped_config)
File "/home/sarat/Desktop/awslimitchecker/local/lib/python2.7/site-packages/botocore/args.py", line 111, in compute_client_args
service_name, region_name, endpoint_url, is_secure)
File "/home/sarat/Desktop/awslimitchecker/local/lib/python2.7/site-packages/botocore/client.py", line 356, in resolve
service_name, region_name)
File "/home/sarat/Desktop/awslimitchecker/local/lib/python2.7/site-packages/botocore/regions.py", line 122, in construct_endpoint
partition, service_name, region_name)
File "/home/sarat/Desktop/awslimitchecker/local/lib/python2.7/site-packages/botocore/regions.py", line 135, in _endpoint_for_partition
raise NoRegionError()
botocore.exceptions.NoRegionError: You must specify a region.

@jantman
Copy link
Owner

jantman commented Sep 17, 2017

Ok, thanks. That looks better.

Do you think you could run it so the code actually executes, i.e. with your AWS credentials set in the environment? I assume if you're writing this for your own need, you'd want to know that the limit and usage information is accurate...

@saratlingamarla
Copy link
Contributor Author

saratlingamarla commented Sep 21, 2017

Yes, sorry about the delay, I verified the usages and they look right:

(awslimitchecker) sarat@sarat-Inspiron-3542:~/Desktop/awslimitchecker$ awslimitchecker -u
awslimitchecker 0.11.0@97d348c0 is AGPL-licensed free software; all users have a right to the full source code of this version. See https://github.com/jantman/awslimitchecker
WARNING:awslimitchecker.trustedadvisor:Cannot check TrustedAdvisor: AWS Premium Support Subscription is required to use this service.
AutoScaling/Auto Scaling groups 0
AutoScaling/Launch configurations 0
CloudFormation/Stacks 1
Dynamodb/Global Secondary Indexes max: arn:aws:dynamodb:us-east-1:339518693606:table/test2=2 (arn:aws:dynamodb:us-east-1:339518693606:table/test1=0, arn:aws:dynamodb:us-east-1:339518693606:table/test2=2)
Dynamodb/Local Secondary Indexes max: arn:aws:dynamodb:us-east-1:339518693606:table/test1=0 (arn:aws:dynamodb:us-east-1:339518693606:table/test1=0, arn:aws:dynamodb:us-east-1:339518693606:table/test2=0)
Dynamodb/Read Capacity (region) 20
Dynamodb/Read Capacity (table) max: arn:aws:dynamodb:us-east-1:339518693606:table/test2=15 (arn:aws:dynamodb:us-east-1:339518693606:table/test1=5, arn:aws:dynamodb:us-east-1:339518693606:table/test2=15)
Dynamodb/Table Count (region) 2
Dynamodb/Write Capacity (region) 20
Dynamodb/Write Capacity (table) max: arn:aws:dynamodb:us-east-1:339518693606:table/test2=15 (arn:aws:dynamodb:us-east-1:339518693606:table/test1=5, arn:aws:dynamodb:us-east-1:339518693606:table/test2=15)
EBS/Active snapshots 0
EBS/Active volumes 0
EBS/Cold (HDD) volume storage (GiB) 0
EBS/General Purpose (SSD) volume storage (GiB) 0
EBS/Magnetic volume storage (GiB) 0
EBS/Provisioned IOPS 0
EBS/Provisioned IOPS (SSD) storage (GiB) 0
EBS/Throughput Optimized (HDD) volume storage (GiB) 0
EC2/Elastic IP addresses (EIPs) 0
EC2/Max active spot fleets per region 0
EC2/Max launch specifications per spot fleet
EC2/Max spot instance requests per region 0
EC2/Max target capacity for all spot fleets in region 0
EC2/Max target capacity per spot fleet
EC2/Rules per VPC security group max: sg-bd1ec5cd=1 (sg-bd1ec5cd=1, sg-04823d74=1)
EC2/Running On-Demand EC2 instances 0
EC2/Running On-Demand c1.medium instances
EC2/Running On-Demand c1.xlarge instances
EC2/Running On-Demand c3.2xlarge instances

@jantman
Copy link
Owner

jantman commented Sep 21, 2017

Ok, thanks so much for this! I'm going to be working on finishing up the tests for this either today or tomorrow; when I do, I'll push the fixed tests to your branch and then merge this.

@jantman jantman added this to the 1.0.0 milestone Sep 21, 2017
@codecov-io
Copy link

codecov-io commented Sep 21, 2017

Codecov Report

Merging #297 into develop will not change coverage.
The diff coverage is 100%.

Impacted file tree graph

@@          Coverage Diff           @@
##           develop   #297   +/-   ##
======================================
  Coverage      100%   100%           
======================================
  Files           25     26    +1     
  Lines         1901   1970   +69     
  Branches       293    298    +5     
======================================
+ Hits          1901   1970   +69
Impacted Files Coverage Δ
awslimitchecker/services/dynamodb.py 100% <100%> (ø)
awslimitchecker/services/base.py 100% <100%> (ø) ⬆️
awslimitchecker/tests/services/__init__.py 100% <0%> (ø) ⬆️

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 1768e3b...11251b2. Read the comment docs.

@jantman jantman merged commit 25c7f58 into jantman:develop Sep 21, 2017
@jantman jantman removed the ready label Sep 21, 2017
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants