Fix kernel panic due to tsd_exit in ZFS_EXIT(zsb) #3247
Conversation
|
Nice analysis of this issue. You're right, we can't call As the code stands now we depend on Ideally we want Another way to go about fixing this would be to implement some form of basic garbage collection for the tsd hash. Periodically walking the hash to verify each pid still exists would be sufficient. The tricky part would be to handle the case of recycled pids. The advantage of an approach like this would be that we could then remove the |
|
@behlendorf |
|
Sure, if we can determine when they're no longer needed. At the site of the previous tsd_exit call should work. |
|
I've updated the patch. zfs_allow_log_key is especially interesting. Apparently it was never cleaned up unless the same process calls to zpl functions, and it cross syscalls so there's no guarantee that it will get clean up after my patch. |
The following panic would occur under certain heavy load: [ 4692.202686] Kernel panic - not syncing: thread ffff8800c4f5dd60 terminating with rrw lock ffff8800da1b9c40 held [ 4692.228053] CPU: 1 PID: 6250 Comm: mmap_deadlock Tainted: P OE 3.18.10 openzfs#7 The culprit is that ZFS_EXIT(zsb) would call tsd_exit() every time, which would purge all tsd data for the thread. However, ZFS_ENTER is designed to be reentrant, so we cannot allow ZFS_EXIT to blindly purge tsd data. Instead, we rely on the new behavior of tsd_set. When NULL is passed as the new value to tsd_set, it will automatically remove the tsd entry specified the the key for the current thread. rrw_tsd_key and zfs_allow_log_key already calls tsd_set(key, NULL) when they're done. The zfs_fsyncer_key relied on ZFS_EXIT(zsb) to call tsd_exit() to do clean up. Now we explicitly call tsd_set(key, NULL) on them. Signed-off-by: Chunwei Chen <[email protected]>
|
@behlendorf |
The following panic would occur under certain heavy load:
[ 4692.202686] Kernel panic - not syncing: thread ffff8800c4f5dd60 terminating with rrw lock ffff8800da1b9c40 held
[ 4692.228053] CPU: 1 PID: 6250 Comm: mmap_deadlock Tainted: P OE 3.18.10 #7
The culprit is that ZFS_EXIT(zsb) would call tsd_exit() every time, which
would purge all tsd data for the thread. However, ZFS_ENTER is designed to be
reentrant, so we cannot allow ZFS_EXIT to blindly purge tsd data.
Instead, when we are doing rrw_exit, if we are removing the last rrn entry,
then we calls tsd_exit_key(rrw_tsd_key), which would only remove the
rrw_tsd_key tsd entry and also the PID_KEY tsd entry if it is the only entry
left for this thread.
The zfs_fsyncer_key tsds rely on ZFS_EXIT(zsb) to call tsd_exit() to do clean
up. Now we need to explicit call tsd_exit_key() on them. We also clean up the
zfs_allow_log_key when it's not needed.
Signed-off-by: Chunwei Chen [email protected]