Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add alibi-detect version to metadata #236

Closed
arnaudvl opened this issue Apr 26, 2021 · 3 comments · Fixed by #339
Closed

Add alibi-detect version to metadata #236

arnaudvl opened this issue Apr 26, 2021 · 3 comments · Fixed by #339
Assignees

Comments

@arnaudvl
Copy link
Contributor

No description provided.

@jklaise
Copy link
Contributor

jklaise commented Sep 14, 2021

Check alibi for an example of setting it, the module also has code for validating and raising a warning if mismatched upon loading: https://github.com/SeldonIO/alibi/blob/90ff57ed7d81bbfcbc8c544fda1472418356f293/alibi/saving.py#L93

@jklaise
Copy link
Contributor

jklaise commented Sep 16, 2021

To do this properly we would also have to do #216. To clarify, if we fetch the __version__ string from alibi.version at the time of saving the detector (the object doesn't have an associated version attribute), what can happen is the following:

  • Detector gets saved and obtains a _version: 0.7.2 entry in the meta attribute
  • Detector gets loaded in a new version of the library, we check and raise a warning
  • Detector gets saved again and because of fetching the runtime __version__ it's version entry is updated _version: 0.7.3
    This is misleading and can lead to broken artefacts because we've overwritten the version with the new (wrong) runtime value.

The better way to do is is an in alibi (see above), where each detector gets a _version class attribute upon declaration of the class. Then this is fetched upon saving and added to metadata. When a detector is reloaded in a different version a warning is issued and when it is re-saved, the correct (old) _version is fetched from the detector class attribute (the whole class is serialized by dill so this is not over-written by the new runtime class definition) and saved, so we avoid overwriting the version in the detector metadata with the incorrect (runtime) version.

@jklaise
Copy link
Contributor

jklaise commented Sep 17, 2021

Tinkering with alibi I noticed that this doesn't actually work as intended and a previously-saved explainer loaded into a new alibi version is assigned the new (incorrect) version attribute.

One easy way to fix this is instead of raising a warning just raise an exception and completely disallow trying to use an artefact with a different version of the library.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging a pull request may close this issue.

2 participants