save the repr of template variables in database

[wxtim]

These changes close #4862

Requirements check-list

• I have read CONTRIBUTING.md and added my name as a Code Contributor.
• Contains logically grouped changes (else tidy your branch by rebase).
• Does not contain off-topic changes (use other PRs for other changes).
• Applied any dependency changes to both setup.cfg and conda-environment.yml.
• Appropriate tests are included (unit and/or functional).
• Appropriate change log entry included.

[oliver-sanders]

@oliver-sanders - From comments at conference

I think we should be either storing the "raw" strings or the Python repr() of the parsed types.

Storing the type as a separate field won't work because types can be nested e.g. {'foo': [{1, 2}, True, 42, 'string']}.

#4862 (comment)

[oliver-sanders]

Need to test manually but looks good.

[hjoliver]

Looks good, tested as working 🎉

[wxtim]

I've badly messed up branches - fixing before undrafting.

[MetRonnie]

I think a changelog entry wouldn't go amiss. Also a short description of the fix in the PR body, for the benefit of us from the future?

[oliver-sanders]

FYI: For piece of mind I had a quick crack at checking that repr representations support everything that literal_eval can parse. Here's what I came up with, as expected it didn't find any examples upto 12 chars long.

Compute heavy, good to check but we don't need anything like this in the test battery.

from ast import literal_eval                                              
from itertools import combinations_with_replacement                       
                                                                          
                                                                          
def range2(*args):                                                        
    for number in range(*args):                                           
        print(f'# {number}')                                              
        yield number                                                      
                                                                          
                                                                          
def generate(length):                                                     
    """Generate random strings with "length" characters."""               
    for chars in combinations_with_replacement(                           
        '\'"-_+=()<>[]{},./a1',                                           
        length                                                            
    ):                                                                    
        yield ''.join(chars)                                              
                                                                          
                                                                          
def parse(string):                                                        
    """Return True if the string is a valid Python literal."""            
    try:                                                                  
        literal_eval(string)                                              
    except Exception:                                                     
        return False                                                      
    return True                                                           
                                                                          
                                                                          
def test(string):                                                         
    """Ensure repr/literal_eval can serialise/deserialise this value."""    
    value1 = literal_eval(string)    
    try:    
        value2 = literal_eval(repr(value1))    
    except Exception:    
        return False    
    return value1 == value2    
     
     
invalid_reprs = (    
    string    
    for length in range2(0, 20)    
    for string in generate(length)    
    if parse(string)    
    if not test(string)    
)    
     
for item in invalid_reprs:    
    print(item) 

Some of the things literal_eval accepts are interesting though e.g:

• {},.1 (implicit tuple)
• """""""" (convoluted string)
• ((((())))) (empty tuple)

Worth being aware that these are valid template variables!

[wxtim]

Can I get a final check @oliver-sanders and @MetRonnie

[hjoliver]

FYI: For piece of mind I had a quick crack at checking that repr representations support everything that literal_eval can parse.

Nice, but isn't that exactly the purpose of repr (so it had better work!)?

[oliver-sanders]

Nice, but isn't that exactly the purpose of repr

In theory, yes, in practice however, it is implemented on a type-by-type basis (e.g. TaskProxy(repr(TaskProxy())) will not work). So long as literal_eval supports the output of repr for all of the types supported by literal_eval we are good.