fix(gatewayapi): don't set listener ResolvedRefs based on routes ResolvedRefs

[michaelbeaumont]

Current behavior isn't part of the spec, conformance tests coming soon to enforce this

Checklist prior to review

• Link to relevant issue as well as docs and UI issues --
• This will not break child repos: it doesn't hardcode values (.e.g "kumahq" as a image registry) and it will work on Windows, system specific functions like syscall.Mkfifo have equivalent implementation on the other OS --
• Tests (Unit test, E2E tests, manual test on universal and k8s) --
• Do you need to update UPGRADE.md? --
• Does it need to be backported according to the backporting policy? (this GH action will add "backport" label based on these file globs, if you want to prevent it from adding the "backport" label use no-backport-autolabel label) --
• Do you need to explicitly set a > Changelog: entry here or add a ci/ label to run fewer/more tests?

[johnharris85]

LGTM