ERROR: Exception in ASGI application: IndexError: index 24 is out of bounds for axis 0 with size 24, and meanwhile the search hangs.

[thuzhf]

🐛 Bug

aim up server log:

Search hangs in aim web UI:

To reproduce

I am not sure how to reproduce exactly. Maybe if you run lots of experiments/runs and do search (no matter whether you search while the experiments are being run or after the experiments are being run) and it will occur.

Expected behavior

The search does not hang. And the aim up server log does not have exceptions.

Environment

• Aim Version (e.g., 3.0.1): 3.16.2 & 3.17.1
• Python version: 3.9.16
• pip version: 23.0.1
• OS (e.g., Linux): Linux
• Any other relevant information

[SGevorg]

@thuzhf thanks for sharing this. this looks to be blocker on your end?

[thuzhf]

@SGevorg What is blocked on my end? The server shows this exception log, but is still running. And the aim web UI search will hang when it reaches 255 of 1798.

[SGevorg]

ah I see, Aim hangs across all explorers on the UI?

[thuzhf]

@SGevorg I just tested all explorers, and found only metrics explorer hangs. And runs explorer only shows the first 45 matched runs.

[SGevorg]

Thanks @thuzhf :-)
Strange. Have you tracked things other than metrics?
Runs explorer is virtualized and auto-loads the rest of the runs if you scroll down (if it doesn't then that's a bug, but hope not haha).

[thuzhf]

@SGevorg

1. Yes, I only tracked metrics.
2. Yes, runs explorer auto-loads the rest of the runs if I scroll down.

[gorarakelyan]

@thuzhf the search hangs as the mentioned exception is raised. Seems the issue appears when trying to sort metrics values based on steps. Do you specify steps when tracking metrics?

[thuzhf]

@gorarakelyan Do you mean steps like follows? I didn't specify these steps.

[thuzhf]

@SGevorg Today I encountered searching in all explorers hung, and there are not any error logs from aim server.
And searching in every explorer hung here:

[SGevorg]

this is definitely terrible news.
@mihran113 @gorarakelyan @roubkar fyi

[roubkar]

@thuzhf does the browser page freeze or can you still use the UI (navigate to other pages from the sidebar, hit the cancel button, etc..)?
would like to understand whether you're facing a UI performance issue or there's no response from the server

[thuzhf]

@roubkar It does not freeze. I can still use the UI like you gave as examples. It seems there's no response from the server.

[mihran113]

Hey @thuzhf! I've artificially recreated this case on my end. As the tracking call for aim.Run is not atomic, there can be a case when the step data is already written, and the process terminated before the value data was written, and this will result in an error mentioned above. I've opened a pull request that will prevent such cases from happening in the future.

Regarding on how to fix your logs:
I've pushed some changes to a branch named: fix/temporary-fix-for-index-error, you can install from the source of that branch using
pip install git+https://github.com/aimhubio/aim.git@fix/temporary-fix-for-index-error
this will print the corrupted metric's run hash, metric name and context in the terminal of aim up command, then using that information you'll need to override the corrupted step via this simple script:

from aim import Run
run = Run(repo={path_to_your_repo}, run_hash={run_hash}, read_only=False)
run.track(0, name={metric_name}, context={context}, step=24)

Let me know if you encounter any issues during the process, will be glad to help.

[mihran113]

Closing the issue as the fix was shipped with aim v3.17.4.