-
Notifications
You must be signed in to change notification settings - Fork 104
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
crc32 computation partly redundant? #281
Comments
Is really 7% of all cpu time spent in crc32 during a normal "attic create", that sounds very high? the crc32 is very useful for server side repository checks and repairs without needing access to any encryption keys. |
BTW I would hope that is really crc32c since that is supported in hardware On Thu, Apr 16, 2015 at 6:12 PM, Jonas Borgström [email protected]
|
The numbers are from a cProfile run and ofc may vary depending on the hw and attic version used. I used my code which spends quite some less cycles overall, so the (relative) percentage of time spent for crc32 might be higher than in release attic while the (absolute) amount of cycles is likely similar.
|
Related: Note: i tried "crc-32c" from crcmod (with the compiled C extension). I wasn't faster than zlib.crc32. Guess it would need a python lib with C extension that actually uses the hw acceleration (if available). |
Guess we can close this one, (server side) repo checks w/o having to decrypt/decompress is a good thing. |
While profiling attic, I found that crc32() calls make up 4.2s of a total of 60s "attic create" runtime.
It seems that attic computes a crc32 over some object metadata (id, length, tag) AND the payload data.
Isn't this redundant considering that the payload data already has a sha256 (unencrypted repo) or even a HMAC (encrypted repo) inside? So, could the payload data be excluded from the crc32 computation without losing anything?
The text was updated successfully, but these errors were encountered: