[PP] ValueError in handle_macro_file_links

[jtcohen6]

Not sure if this is already fixed in one of the open PRs. It looked similar to, but not exactly the same as, some of the other open issues.

The case where this cropped up for me is a pretty weird one, but I've been able to reliably reproduce it. In my work for #3510, I was adding a macro named snowflake__truncate_relation to plugins/snowflake/dbt/include/snowflake/macros/adapters.sql. If I remove that macro, do a full re-parse, then add it back and use partial parsing, I see the error again.

If the unique_id is not in source_file.macros (potentially because it was removed already), it raises the a ValueError when it tries to remove it from the list here:

https://github.com/dbt-labs/dbt/blob/9c58f3465bf9907a2b62942de548f80650cd6288/core/dbt/parser/partial.py#L391

2021-07-27 17:38:12.543828 (MainThread): Partial parsing enabled: 0 files deleted, 0 files added, 1 files changed.
2021-07-27 17:38:12.574882 (MainThread): Sending event: {'category': 'dbt', 'action': 'invocation', 'label': 'end', 'context': [<snowplow_tracker.self_describing_json.SelfDescribingJson object at 0x116d1fdf0>, <snowplow_tracker.self_describing_json.SelfDescribingJson object at 0x116d1fd00>, <snowplow_tracker.self_describing_json.SelfDescribingJson object at 0x116d1fd30>]}
2021-07-27 17:38:12.578412 (MainThread): Flushing usage events
2021-07-27 17:38:12.870885 (MainThread): Encountered an error:
2021-07-27 17:38:12.871082 (MainThread): list.remove(x): x not in list
Traceback (most recent call last):
  File "/Users/jerco/dev/product/dbt/core/dbt/main.py", line 127, in main
    results, succeeded = handle_and_check(args)
  File "/Users/jerco/dev/product/dbt/core/dbt/main.py", line 205, in handle_and_check
    task, res = run_from_args(parsed)
  File "/Users/jerco/dev/product/dbt/core/dbt/main.py", line 258, in run_from_args
    results = task.run()
  File "/Users/jerco/dev/product/dbt/core/dbt/task/runnable.py", line 408, in run
    self._runtime_initialize()
  File "/Users/jerco/dev/product/dbt/core/dbt/task/runnable.py", line 124, in _runtime_initialize
    super()._runtime_initialize()
  File "/Users/jerco/dev/product/dbt/core/dbt/task/runnable.py", line 81, in _runtime_initialize
    self.load_manifest()
  File "/Users/jerco/dev/product/dbt/core/dbt/task/runnable.py", line 68, in load_manifest
    self.manifest = ManifestLoader.get_full_manifest(self.config)
  File "/Users/jerco/dev/product/dbt/core/dbt/parser/manifest.py", line 170, in get_full_manifest
    manifest = loader.load()
  File "/Users/jerco/dev/product/dbt/core/dbt/parser/manifest.py", line 212, in load
    project_parser_files = partial_parsing.get_parsing_files()
  File "/Users/jerco/dev/product/dbt/core/dbt/parser/partial.py", line 131, in get_parsing_files
    self.update_in_saved(file_id)
  File "/Users/jerco/dev/product/dbt/core/dbt/parser/partial.py", line 213, in update_in_saved
    self.update_macro_in_saved(new_source_file, old_source_file)
  File "/Users/jerco/dev/product/dbt/core/dbt/parser/partial.py", line 271, in update_macro_in_saved
    self.handle_macro_file_links(old_source_file, follow_references=True)
  File "/Users/jerco/dev/product/dbt/core/dbt/parser/partial.py", line 391, in handle_macro_file_links
    source_file.macros.remove(unique_id)
ValueError: list.remove(x): x not in list

[gshank]

Jeremy: This code has changed in a couple of tickets. I don't see from the log how to reproduce it. Could you try with the latest in develop (or 0.20.latest) and see if it still happens?

[jtcohen6]

@gshank I retried this with the latest version of develop, and I was able to reproduce the issue. Again, this is a niche one, since it involves adding/removing macros from internal packages (in this case dbt_snowflake), so I don't know how likely it is to come up in practice. Users don't tend to edit internal packages, outside of upgrading to a new adapter plugin version. I'm interested in it insofar as it could represent an error that crops up elsewhere.

Steps to reproduce:

1. Install dbt from source (pip install -r editable-requirements.txt)
2. Parse a sample project to create target/partial_parse.msgpack
3. Add any macro to an internal project (e.g. plugins/postgres/dbt/include/postgres/macros/adapters.sql)
4. Re-parse the project with partial parsing enabled:

Partial parsing enabled: 0 files deleted, 0 files added, 1 files changed.
Encountered an error:
list.remove(x): x not in list

5. Trigger a full re-parse (dbt --no-partial-parse parse)
6. Remove the previously added macro
7. Re-parse the project with partial parsing enabled:

Partial parsing enabled: 0 files deleted, 0 files added, 1 files changed.
Encountered an error:
list.remove(x): x not in list