DAOS-17111 cart: Fix csm_alive_count

[liw]

In swim, csm_alive_count may underflow because some
cst->cst_state.sms_status changes in csm overlook the count. Moreover,
not counting SUSPECT members seems to be a mistake. Consider a
membership of three, {x, y, z}. If x enters a state where it can't
receive any SWIM messages, and it picks y in the next period, then it
will suspect y, causing csm_alive_count to drop from 3 to 2, which
prevents x from declaring an "outage". (In the subsequent period, x will
suspect z, causing csm_alive_count to drop from 2 to 1 quickly.) Since x
keeps pinging SUSPECT members, it seems reasonable to count them in and
expect them to send messages to x until they become DEAD.

This patch fixes the underflow, and counts SUSPECT members in addition
to ALIVE members in csm_alive_count (renamed to
csm_alive_or_suspect_count).

Before requesting gatekeeper:

• Two review approvals and any prior change requests have been resolved.
• Testing is complete and all tests passed or there is a reason documented in the PR why it should be force landed and forced-landing tag is set.
• Features: (or Test-tag*) commit pragma was used or there is a reason documented that there are no appropriate tags for this PR.
• Commit messages follows the guidelines outlined here.
• Any tests skipped by the ticket being addressed have been run and passed in the PR.

Gatekeeper:

• You are the appropriate gatekeeper to be landing the patch.
• The PR has 2 reviews by people familiar with the code, including appropriate owners.
• Githooks were used. If not, request that user install them and check copyright dates.
• Checkpatch issues are resolved. Pay particular attention to ones that will show up on future PRs.
• All builds have passed. Check non-required builds for any new compiler warnings.
• Sufficient testing is done. Check feature pragmas and test tags and that tests skipped for the ticket are run and now pass with the changes.
• If applicable, the PR has addressed any potential version compatibility issues.
• Check the target branch. If it is master branch, should the PR go to a feature branch? If it is a release branch, does it have merge approval in the JIRA ticket.
• Extra checks if forced landing is requested
  • Review comments are sufficiently resolved, particularly by prior reviewers that requested changes.
  • No new NLT or valgrind warnings. Check the classic view.
  • Quick-build or Quick-functional is not used.
• Fix the commit message upon landing. Check the standard here. Edit it to create a single commit. If necessary, ask submitter for a new summary.

[github-actions]

Ticket title is '[SWIM] Zombie Node Messes Up SWIM'
Status is 'Open'
https://daosio.atlassian.net/browse/DAOS-17111

[jxiong]

Have you figured out that why it's okay for us to not update/extend the suspecting timeout if the number of alive_or_suspect_count is less or equal than 2? I tend to think it's applicable no matter how many ranks there are.

[jxiong]

LGTM, just a single pending question