[core][distributed] support layer size undividable by pp size in pipeline parallel inference

[youkaichao]

fixes #6114

[andoorve]

There's a subtle point here:

vllm/vllm/executor/distributed_gpu_executor.py

Lines 38 to 46 in 966fe72

	num_blocks = self._run_workers("determine_num_available_blocks", )
	# Since we use a shared centralized controller, we take the minimum
	# number of blocks across all workers to make sure all the memory
	# operators can be applied to all workers.
	num_gpu_blocks = min(b[0] for b in num_blocks)
	num_cpu_blocks = min(b[1] for b in num_blocks)
	return num_gpu_blocks, num_cpu_blocks

We take min of blocks across all workers - as a result of this the GPU mem utilization of 0...n-2th will be slightly lower than expected.

I don't think it's a big deal, just something to be aware of.

[youkaichao]

We take min of blocks across all workers - as a result of this the GPU mem utilization of 0...n-2th will be slightly lower than expected.

Good point. I also noticed this. Some GPU memory waste, but it is better than not being supported.

[andoorve]

LGTM