Show more details for Validation errors

[WolfgangFahl]

Operating system

uname -a
Linux 065ec9c0a5c8 5.10.104-linuxkit #1 SMP Thu Mar 17 17:08:06 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux

nbgrader --version

Python version 3.9.13 | packaged by conda-forge | (main, May 27 2022, 16:56:21)
[GCC 10.3.0]
nbgrader version 0.7.0

Expected behavior

I assume
https://python-jsonschema.readthedocs.io/en/latest/errors/
allows to show details of a validation error

Actual behavior

failed to validate, metadata is corrupted

Steps to reproduce the behavior

use nbgrader update on a broken .ipynb file

[perllaghu]

Do you have any specific examples?

My experience is that the errors are generally complex to identify, and often need a human to snake their way between options.

For example: if a student duplicates a cell, you get errors - so the challenge is to work out which is the "correct" cell [it's not always the first or last one... or may have been replaced all together]

[WolfgangFahl]

Currently i have some 50 broken notebooks out of some 1500 - i can't share the notebooks for privacy reasons and have exactly the problem to identify what happened to these notebooks. I intend to patch the code that the details are displayed. Pull requests are a problem for me since we are not using the most current version of nbgrader.

[WolfgangFahl]

I have the following ValidationError types:

• 8 ?
• 18 Markdown grade cell is not marked as a solution cell
• 4 Markdown solution cell not marked as a grade cell: ####
• 24 Duplicate grade id:
• 4 Markdown solution cell not marked as a grade cell:

which i analyzed with the following bash/awk script:

#!/bin/bash
# WF 2022-07-08
# analyze log
cat /tmp/update.log | awk '
BEGIN { 
  squote="\x27" 
  cell="cell-xxxxxxxxxxxxx"
  problem=""
}
# ignore lines which are ok
/Updating metadata for notebook:/  { next }
# [UpdateApp | ERROR] Notebook './434728/UB-8/UB-8.ipynb' failed to validate, metadata is corrupted
/Notebook/{ 
  path=$5
  gsub(squote,"",path)
  print path,cell, problem, detailproblem
  count[problem]+=1
  cell="cell-xxxxxxxxxxxxx"
  problem=""
  next
}
#  jsonschema.exceptions.ValidationError: Markdown grade cell 'cell-bfa4f1e80f1ee60a' is not marked as a solution cell
/jsonschema.exceptions.ValidationError/ { 
  detailproblem=$0
  problem=$2" "$3" "$4" "$6" "$7" "$8" "$9" "$10" "$11" "$12
  gsub("jsonschema.exceptions.ValidationError:","",problem)
  cell=$5
  gsub(squote,"",cell)
  next
}
#{ print }
END {
  for (problem in count ) {
    print count[problem],problem
  }
}
'

[jhamrick]

IIRC when I initially implemented this, I couldn't find a way to surface the error from jsonschema. That might have changed, though, so it's worth looking into again.

[WolfgangFahl]

The JSON schema error is actually shown - for me the problem was linking it back to the cell that caused it. As far as i understand it the idea is to have pairs of cells from the source notebook and the submission notebook to be matched. When this matching fails this calls for trouble. We have seen manyfold technical and user behavior reasons for such problems in our learning environment with some 150 student groups working on 10 different notebooks. This is one of the areas where the robustness of the nbgrader software might be improved to more clearly point to the underlying problems. The whole point of digitization is somewhat defeated if indididual problems need a lot of technical debugging.