What is "args.use_templated_goals"?

[soyeonm]

Hello,

With the latest code, when I try to preprocess data, I get an error due to "args.use_templated_goals". The code runs fine when I manually set use_templated_goals = False in def preprocess_splits of preprocess.py. However, when I manually set it to use_templated_goals = True,
I get the following error:

$ python models/eval/eval_seq2seq.py --model_path models/pre-trained_models/best_seen.pth --eval_split valid_seen --data data/json_2.1.0 --model models.model.seq2seq_im_mask --gpu --num_threads 1 --preprocess
{'tests_seen': 1533,
'tests_unseen': 1529,
'train': 21023,
'valid_seen': 820,
'valid_unseen': 821}
Loading: models/pre-trained_models/best_seen.pth

Preprocessing dataset and saving to pp folders ... This is will take a while. Do this once as required:
Preprocessing tests_seen
N/A% (0 of 1533) | | Elapsed Time: 0:00:00 ETA: --:--:--Traceback (most recent call last):
File "models/eval/eval_seq2seq.py", line 54, in
eval = EvalTask(args, manager)
File "/home/symin95_gmail_com/alfred/models/eval/eval.py", line 45, in init
dataset.preprocess_splits(self.splits)
File "/home/symin95_gmail_com/alfred/data/preprocess.py", line 74, in preprocess_splits
self.process_language(ex, traj, r_idx, use_templated_goals=use_templated_goals)
File "/home/symin95_gmail_com/alfred/data/preprocess.py", line 102, in process_language
task_desc = sample_templated_task_desc_from_traj_data(traj)
File "/home/symin95_gmail_com/alfred/gen/utils/game_util.py", line 351, in sample_templated_task_desc_from_traj_data
pddl_params = traj_data['pddl_params']
KeyError: 'pddl_params'

1. What does use_templated_goals = True do? Can you copy and paste an example that is generated because of use_templated_goals = True?
2. What should I fix in the code with the above error to get it working with use_templated_goals = True?
3. I am using the "lite" version of the data (json_2.1.0 instead of json_feat_2.1.0). Could it be because I use "json_2.1.0" instead of "json_feat_2.1.0"?

Thank you!

[soyeonm]

Just to add, if I run the code as it is (without manually setting "args.use_templated_goals" to True or False in preprocess.py), I get the error that there is no args.use_templated_goals in the namespace. That is why I manually set this to something.

[MohitShridhar]

@soyeonm,

1. use_templated_goals trains the model with synthetic (non-human annotated) goal descriptions e.g. "put X on the Y", where X and Y are filled-in with object and receptacle categories. Whereas by default, the model is trained with human-annotated goals from the ALFRED dataset. I added use_templated_goals as an additional option since some of the human annotated goals in the dataset are incomplete e.g "prepare the apple". Note that none of the results in the paper used templated goals.
2. The templated goals are only available during training time as a form of data augmentation; you probably need to train a model with both templated and human-annotated goals to make it robust. But during test time, you only have access to human-annotated goals since that's part of the ALFRED challenge.
3. Probably not. It's related to 2.

Thanks for pointing out the issue with eval_seq2seq.py. You are doing the right thing by setting use_templated_goals=False. I'll fix this soon.

Hope this helps!

[MohitShridhar]

Fixed.