Add response statistics

[kthui]

Related PRs:

• Add response statistics server#6869
• Add response statistics common#112
• Add response statistics square_backend#16

Add response statistics API.

Example statistics endpoint output to client:

{
  'model_stats': [{
    'name': 'square_int32',
    ...
    'inference_stats': {
      ...
      'response_stats': {
        '0': {
          'compute_infer': {'count': 3, 'ns': 1200514434},
          'compute_output': {'count': 3, 'ns': 600346215},
          'success': {'count': 2, 'ns': 1200431061},
          'fail': {'count': 1, 'ns': 600429588},
          'empty_response': {'count': 0, 'ns': 0}
        },
        '1': {
          'compute_infer': {'count': 3, 'ns': 1200250644},
          'compute_output': {'count': 3, 'ns': 600358040},
          'success': {'count': 1, 'ns': 600202056},
          'fail': {'count': 2, 'ns': 1200406628},
          'empty_response': {'count': 0, 'ns': 0}
        },
        '2': {
          'compute_infer': {'count': 3, 'ns': 1200252841},
          'compute_output': {'count': 2, 'ns': 400214433},
          'success': {'count': 1, 'ns': 600183751},
          'fail': {'count': 1, 'ns': 600188918},
          'empty_response': {'count': 1, 'ns': 400094605}
        },
        '3': {
          'compute_infer': {'count': 2, 'ns': 800160632},
          'compute_output': {'count': 1, 'ns': 200133237},
          'success': {'count': 0, 'ns': 0},
          'fail': {'count': 1, 'ns': 600214152},
          'empty_response': {'count': 1, 'ns': 400079717}
        },
        '4': {
          'compute_infer': {'count': 1, 'ns': 400086093},
          'compute_output': {'count': 1, 'ns': 200104114},
          'success': {'count': 0, 'ns': 0},
          'fail': {'count': 1, 'ns': 600190207},
          'empty_response': {'count': 0, 'ns': 0}
        },
        '5': {
          'compute_infer': {'count': 1, 'ns': 400089434},
          'compute_output': {'count': 0, 'ns': 0},
          'success': {'count': 0, 'ns': 0},
          'fail': {'count': 0, 'ns': 0},
          'empty_response': {'count': 1, 'ns': 400089434}}
        }
      }
    ...
    }
  }]
}

[oandreeva-nv]

TMLG, but I would also let @rmccorm4 take a look

[rmccorm4]

LGTM overall, main comment is on the use of "zero" compute time for "empty".

[rmccorm4]

Nice work! 🚀