[cmake] [c++] use .dylib extension unconditionally on macOS

[jameslamb]

Short Description

Proposes the following changes to CMake-based builds of LightGBM:

• remove APPLE_OUTPUT_DYLIB CMake option
• always produce lib_lightgbm.dylib on macOS

Benefits of these changes

• simplifies CMake configuration
• improves the probability of lib_lightgbm working correctly with tools that use string matching on file extensions to guess file format (instead of examining the magic bytes)
  • for example, I think this makes it more likely that CMake find_library() calls or custom find_*() CMake scripts in other projects would be able to find lib_lightgbm on macOS
• might help with some of the mac-specific packaging work I'm focusing on (e.g. Provide binary python package for macOS M1 #5328)

Notes for Reviewers

This is just changing the file name, not the actual format

The shared library LightGBM currently produces on macOS is actually in mach-o format, not ELF. The CMake customization removed in this PR literally just changes the file name.

cmake -B build -S .
cmake --build build --target _lightgbm -j4
file ./lib_lightgbm.so
# ./lib_lightgbm.so: Mach-O 64-bit dynamically linked shared library arm64

I couldn't find a strong reason that LightGBM needs to produce a lib_lightgbm.so on macOS

Looks like that was first added in 70c31c0 back in December 2016. @guolinke do you remember why?

The APPLE_OUTPUT_DYLIB option was then added in #204.

Why call this "breaking"?

Changing this filename could cause the following types of problems for applications depending on LightGBM on macOS:

• any scripts expecting to find lib_lightgbm.so will no longer find it
• setting a path to a lib_lightgbm.so in environment variable LD_PRELOAD will no longer work
• RPATHs or similar embedded in binaries will not find LightGBM

In my opinion, these are worthwhile costs in exchange for the benefits.

How were these particular changes identified?

I searched like this:

git grep -E '\.so'
git grep -E 'lightgbm.*so'

I believe I've caught all the places that need to be changed. Thankfully this project has pretty good CI coverage on macOS (including Python, R, and SWIG), so I think we can be confident that this is working whenever CI for all macOS jobs passes.

[borchero]

Seems reasonable 😄 👍

[jameslamb]

Thanks very much!

@guolinke could you also comment when you have time? As you might remember more context for why this was first added back in #6418 (review).

[guolinke]

I don't remember, but this change looks reasonable to me

[jameslamb]

Alright thanks for checking, and for the review @borchero !

Let's proceed with this then.