tmp/suds getting full of *.px files

[datacubed]

I'm not sure if this an issue or not, so I apology now if I'm in the wrong place.
Our server hit us with a "ran out of disk space", and when looking around found the tmp/suds folder being quite a few GBs inside.
Once the *.px files were deleted, everything returned to normal.

I know usually /tmp/ is cleared on reboot - but we try not to reboot our servers too often. Is it possible for me to avoid this via a function in the API, or do I just need to write a command that clears the tmp/suds director after we've used the Bing API.

Cheers

[khvn26]

Hi! Check out our fork where this issue is solved: #108

[qitia]

I hope this can help you.

[datacubed]

Hi! Check out our fork where this issue is solved: #108

Hey this post says "the branch has conflicts" - should I assume to wait for this to be resolved before using?

I hope this can help you.

@qitia from this post do you recommend adding the fix as mentioned:
#80 (comment)

In the post the last comment mentions moving away from suds ? Is that something happening with this library? Or will there be an update that handles this cache "bloating" (as ive seen it called) ?

[qitia]

hey @datacubed , what I recommend is try below method, with remove_suds_cache set to true. This will help you clear the cache folder on app's exist.

from os import path
from tempfile import gettempdir
import atexit

def remove_suds_cache(remove_suds_cache=False):
    if remove_suds_cache:
        shutil.rmtree(path.join(gettempdir(), 'suds'), ignore_errors=True)
    pass

# Set remove_suds_cache=True to remove the temp suds cache
atexit.register(remove_suds_cache, remove_suds_cache=False)

I hope this can help.

[khvn26]

There's an internal suds.Client instance in the library code which is used to construct SOAP object factories. It's instantiated with suds' default cache which pollutes the /tmp/suds directory because a bug in suds' default cache implementation.

Our fork makes this instance use a dummy in-memory cache (basically a dict), which will load Bing Ads' SOAP XSDs to RAM but won't touch the disk at all. It overwrites the instantiating code, hence the conflicts. You should be fine using it.

[d601]

I recently ran into this issue so I took a look at the solutions people have posted here. I found them problematic because of various issues, including 1.) deleting the /tmp/suds directory at exit, which I imagine would cause problems if you have lots of programs running simultaneously, and 2.) waiting for #108 to get merged, which might not happen soon.

The core issue is that suds-jurko 0.6 defines a mangle() function in its reader class that is broken in newer versions of python. The old one does this:
h = abs(hash(name))
I believe hash() was changed at some point to return different values for the same name in different processes. The development version of suds-jurko has this instead:
h = md5(name.encode()).hexdigest()
which behaves deterministically for the same name.

I wrote some code to monkeypatch suds 0.6 into behaving correctly:

import suds.reader

try:
    from hashlib import md5
except ImportError:
    # 'hashlib' package added in Python 2.5 so use the now deprecated/removed
    # 'md5' package in older Python versions.
    from md5 import md5



def mangle(self, name, x):
    """
    Mangle the name by hashing the I{name} and appending I{x}.

    @return: The mangled name.
    @rtype: str

    """
    h = md5(name.encode()).hexdigest()
    return '%s-%s' % (h, x)

print("Monkeypatching suds Reader classes")
suds.reader.Reader.mangle = mangle
suds.reader.DocumentReader.mangle = mangle
suds.reader.DefinitionsReader.mangle = mangle

print("Monkeypatching complete, importing bingads")
from bingads import *

(It's entirely possible the above code could be simplified - I just know that what I've written works.)
If you have some library code you can drop this in as bingads.py and then do from <yourlibrary>.bingads import <stuff>. This will make sure suds gets patched before bingads starts doing anything. I tested this in python 3.6 and it seems to work.

IMO a better solution than #108 or trying to fix the broken reader code would be to just use a version of suds that's actually maintained, like suds-community. Or switch to zeep. But others have already suggested both these options in other issues.

[qitia]

moved to suds-community since 13.0.11.1

[addisonjones-gc]

Hi all, just chiming in to say I'm still getting this issue just in different form. I'm using bingads in an Airflow DAG and if imported anywhere in a file with the @dag or @task decorator, it creates a new directory and .px file in /tmp/. This is causing our Pods to be evicted with the ever growing files in the /tmp folder which is a real drag.

Example:

/tmp
  /tmpatzruj5fsuds-default-cache
    suds-9dbf36d0033995011d074a4d7e7f1669-document.px
    version

No idea what's causing this, but it's clearly the suds.cache.py __init__() method which is seeing None for the location and setting the default location.

Using bingads 13.0.13, suds-community 1.1.1 .

Any amount of help would be excellent.

[addisonjones-gc]

As a follow up to the above, updating to the 13.0.14 version of the SDK fixed the default-cache folder creep issue. Our scheduler is still full of tmp* directories, but at least they're empty now.