Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

dependency got using find_library is not deduplicated for static library builds #2150

Closed
bruce-richardson opened this issue Aug 2, 2017 · 7 comments · Fixed by #5073
Closed

dependency got using find_library is not deduplicated for static library builds #2150

bruce-richardson opened this issue Aug 2, 2017 · 7 comments · Fixed by #5073
Labels
bug compilers dependencies
Milestone
0.51.0

Comments

@bruce-richardson
Copy link
Contributor

bruce-richardson commented Aug 2, 2017 •
edited
Loading

Simple test case attached. [Original issue found when porting DPDK project to meson, where we got errors due to too many parameters.]

Setup: liba, which depends on libdl, and pthreads. Libb which depends on liba. App which depends on both liba and libb

Shared build looks ok, no duplication seen with latest HEAD:

$ ninja -v -j1
 <snip for brevity>
 [8/8] ccache cc  -o app/app 'app/app@exe/app.c.o' -Wl,--no-undefined -Wl,--as-needed -Wl,--start-group liba/liba.so libb/libb.so -pthread -ldl -Wl,--end-group -ldl '-Wl,-rpath,$ORIGIN/../liba:$ORIGIN/../libb' -Wl,-rpath-link,/home/bruce/temp/meson-test/shared_build/liba:/home/bruce/temp/meson-test/shared_build/libb

Static library shows a large amount of duplication for -ldl, but not pthreads, despite both being listed as dependencies in the same places:

$ ninja -v -j1
 <snip for brevity>
 [6/6] ccache cc  -o app/app 'app/app@exe/app.c.o' -Wl,--no-undefined -Wl,--as-needed -Wl,--start-group liba/liba.a libb/libb.a -pthread -ldl -Wl,--end-group -ldl -ldl -ldl -ldl -ldl

meson-test.zip
@bruce-richardson
Copy link
Contributor Author

I also see the same problem with DPDK code if I put in '-ldl' in the link_args of declare_dependency, rather than passing it as a dependency explicitly.

@bruce-richardson bruce-richardson mentioned this issue Apr 5, 2018
Closed
@Ralith
Copy link

Ralith commented Apr 12, 2018

This is affecting me as well; I have a project in which 5 dynamic libraries, obtained using find_library and depended upon transitively via static libraries built within the project, explode into 151(!) -l flags.

@textshell textshell mentioned this issue Apr 15, 2018
Open
nirbheek added a commit that referenced this issue Jun 5, 2018
@nirbheek
pkgconfig deps: Also resolve paths to shared libraries
5726a9e 
This allows us to more aggressively de-dup them, and also sets RPATHs
to all libraries that are not in the system linker paths so that
binaries can be run uninstalled without any special steps.

These RPATHs will be wiped on install, so they do not affect
reproducible builds.

De-duping:
Fixes #2150
Fixes #2118

RPATHs:
Fixes #314
Fixes #2881

Also fixes the uninstalled usage portions of:
#3038
#3077
nirbheek added a commit that referenced this issue Jun 5, 2018
@nirbheek
pkgconfig deps: Also resolve paths to shared libraries
434d702 
This allows us to more aggressively de-dup them, and also sets RPATHs
to all libraries that are not in the system linker paths so that
binaries can be run uninstalled without any special steps.

These RPATHs will be wiped on install, so they do not affect
reproducible builds.

De-duping:
Fixes #2150
Fixes #2118

RPATHs:
Fixes #314
Fixes #2881

Also fixes the uninstalled usage portions of:
#3038
#3077
nirbheek added a commit that referenced this issue Jun 7, 2018
@nirbheek
pkgconfig deps: Also resolve paths to shared libraries
0e5e2bf 
This allows us to more aggressively de-dup them, and also sets RPATHs
to all libraries that are not in the system linker paths so that
binaries can be run uninstalled without any special steps.

These RPATHs will be wiped on install, so they do not affect
reproducible builds.

De-duping:
Fixes #2150
Fixes #2118

RPATHs:
Fixes #314
Fixes #2881

Also fixes the uninstalled usage portions of:
#3038
#3077
nirbheek added a commit that referenced this issue Jun 7, 2018
@nirbheek
pkgconfig deps: Also resolve paths to shared libraries
8bd2d6e 
This allows us to more aggressively de-dup them, and also sets RPATHs
to all libraries that are not in the system linker paths so that
binaries can be run uninstalled without any special steps.

These RPATHs will be wiped on install, so they do not affect
reproducible builds.

De-duping:
Fixes #2150
Fixes #2118
Fixes #3071

RPATHs:
Fixes #314
Fixes #2881

Also fixes the uninstalled usage portions of:
#3038
#3077
nirbheek added a commit that referenced this issue Jun 8, 2018
@nirbheek
pkgconfig deps: Also resolve paths to shared libraries
783a1a3 
This allows us to more aggressively de-dup them, and also sets RPATHs
to all libraries that are not in the system linker paths so that
binaries can be run uninstalled without any special steps.

These RPATHs will be wiped on install, so they do not affect
reproducible builds.

De-duping:
Fixes #2150
Fixes #2118
Fixes #3071

RPATHs:
Fixes #314
Fixes #2881

Also fixes the uninstalled usage portions of:
#3038
#3077
nirbheek added a commit that referenced this issue Jun 10, 2018
@nirbheek
pkgconfig deps: Also resolve paths to shared libraries
5038bbf 
This allows us to more aggressively de-dup them, and also sets RPATHs
to all libraries that are not in the system linker paths so that
binaries can be run uninstalled without any special steps.

These RPATHs will be wiped on install, so they do not affect
reproducible builds.

De-duping:
Fixes #2150
Fixes #2118
Fixes #3071

RPATHs:
Fixes #314
Fixes #2881

Also fixes the uninstalled usage portions of:
#3038
#3077
nirbheek added a commit that referenced this issue Jun 10, 2018
@nirbheek
pkgconfig deps: Also resolve paths to shared libraries
8d42050 
This allows us to more aggressively de-dup them, and also sets RPATHs
to all libraries that are not in the system linker paths so that
binaries can be run uninstalled without any special steps.

These RPATHs will be wiped on install, so they do not affect
reproducible builds.

De-duping:
Fixes #2150
Fixes #2118
Fixes #3071

RPATHs:
Fixes #314
Fixes #2881

Also fixes the uninstalled usage portions of:
#3038
#3077
@bruce-richardson
Copy link
Contributor Author

This issue is still not fixed from my point of view. Even the basic test case discussed in the original bug report still shows 4 copies of "-ldl" listed in the build.ninja file. When building DPDK having converted our "-lm" and "-ldl" project linker flags to proper dependencies fails to build properly for static builds due to us exceeding the open files threshold.

ccache gcc -o examples/dpdk-vm_power_manager 'examples/c590b3c@@dpdk-vm_power_manager@exe/vm_power_manager_channel_manager.c.o' 'examples/c590b3c@@dpdk-vm_power_manager@exe/vm_power_manager_channel_monitor.c.o' 'examples/c590b3c@@dpdk-vm_power_manager@exe/vm_power_manager_main.c.o' 'examples/c590b3c@@dpdk-vm_power_manager@exe/vm_power_manager_parse.c.o' 'examples/c590b3c@@dpdk-vm_power_manager@exe/vm_power_manager_power_manager.c.o' 'examples/c590b3c@@dpdk-vm_power_manager@exe/vm_power_manager_vm_power_cli.c.o' 'examples/c590b3c@@dpdk-vm_power_manager@exe/vm_power_manager_oob_monitor_x86.c.o' -Wl,--no-undefined -Wl,--as-needed -Wl,-O1 -Wl,--whole-archive -Wl,--start-group drivers/librte_common_cpt.a drivers/librte_common_dpaax.a drivers/librte_common_octeontx.a drivers/librte_bus_dpaa.a drivers/librte_bus_fslmc.a drivers/librte_bus_ifpga.a drivers/librte_bus_pci.a drivers/librte_bus_vdev.a drivers/librte_bus_vmbus.a drivers/librte_mempool_bucket.a drivers/librte_mempool_dpaa.a drivers/librte_mempool_dpaa2.a drivers/librte_mempool_octeontx.a drivers/librte_mempool_ring.a drivers/librte_mempool_stack.a drivers/librte_pmd_af_packet.a drivers/librte_pmd_ark.a drivers/librte_pmd_atlantic.a drivers/librte_pmd_avp.a drivers/librte_pmd_axgbe.a drivers/librte_pmd_bond.a drivers/librte_pmd_bnx2x.a drivers/librte_pmd_bnxt.a drivers/librte_pmd_cxgbe.a drivers/librte_pmd_dpaa.a drivers/librte_pmd_dpaa2.a drivers/librte_pmd_e1000.a drivers/librte_pmd_ena.a drivers/librte_pmd_enetc.a drivers/librte_pmd_enic.a drivers/librte_pmd_failsafe.a drivers/librte_pmd_fm10k.a drivers/librte_pmd_i40e.a drivers/librte_pmd_iavf.a drivers/librte_pmd_ice.a drivers/librte_pmd_ifc.a drivers/librte_pmd_ixgbe.a drivers/librte_pmd_kni.a drivers/librte_pmd_liquidio.a drivers/librte_pmd_netvsc.a drivers/librte_pmd_nfp.a drivers/librte_pmd_null.a drivers/librte_pmd_octeontx.a drivers/librte_pmd_pcap.a drivers/librte_pmd_qede.a drivers/librte_pmd_ring.a drivers/librte_pmd_sfc.a drivers/librte_pmd_softnic.a drivers/librte_pmd_szedata2.a drivers/librte_pmd_tap.a drivers/librte_pmd_thunderx.a drivers/librte_pmd_vdev_netvsc.a drivers/librte_pmd_vhost.a drivers/librte_pmd_virtio.a drivers/librte_pmd_vmxnet3.a drivers/librte_pmd_aesni_gcm.a drivers/librte_pmd_aesni_mb.a drivers/librte_pmd_caam_jr.a drivers/librte_pmd_ccp.a drivers/librte_pmd_dpaa_sec.a drivers/librte_pmd_dpaa2_sec.a drivers/librte_pmd_null_crypto.a drivers/librte_pmd_octeontx_crypto.a drivers/librte_pmd_openssl.a drivers/librte_pmd_crypto_scheduler.a drivers/librte_pmd_virtio_crypto.a drivers/librte_pmd_octeontx_compress.a drivers/librte_pmd_qat.a drivers/librte_pmd_zlib.a drivers/librte_pmd_dpaa_event.a drivers/librte_pmd_dpaa2_event.a drivers/librte_pmd_octeontx_event.a drivers/librte_pmd_opdl_event.a drivers/librte_pmd_skeleton_event.a drivers/librte_pmd_sw_event.a drivers/librte_pmd_dsw_event.a drivers/librte_pmd_bbdev_null.a drivers/librte_pmd_skeleton_rawdev.a drivers/librte_pmd_dpaa2_cmdif.a drivers/librte_pmd_dpaa2_qdma.a drivers/librte_pmd_ifpga_rawdev.a -Wl,--no-whole-archive -Wl,--no-as-needed -lnuma lib/librte_eal.so.9.1 lib/librte_kvargs.so.1.1 lib/librte_mempool.so.5.1 lib/librte_ring.so.2.1 lib/librte_net.so.1.1 lib/librte_mbuf.so.5.1 lib/librte_ethdev.so.11.1 lib/librte_cmdline.so.2.1 lib/librte_meter.so.2.1 drivers/librte_pmd_bnxt.so.2.1 drivers/librte_bus_pci.so.2.1 lib/librte_pci.so.1.1 drivers/librte_bus_vdev.so.2.1 drivers/librte_pmd_i40e.so.2.1 lib/librte_hash.so.2.1 drivers/librte_pmd_ixgbe.so.2.1 lib/librte_security.so.2.1 lib/librte_cryptodev.so.6.1 lib/librte_power.so.1.1 lib/librte_timer.so.1.1 drivers/librte_common_dpaax.so.1.1 lib/librte_eventdev.so.6.1 lib/librte_rawdev.so.1.1 drivers/librte_bus_dpaa.so.2.1 drivers/librte_bus_fslmc.so.2.1 drivers/librte_common_octeontx.so.1.1 lib/librte_sched.so.2.1 lib/librte_ip_frag.so.1.1 drivers/librte_mempool_dpaa.so.1.1 drivers/librte_mempool_dpaa2.so.2.1 lib/librte_vhost.so.4.1 lib/librte_kni.so.2.1 drivers/librte_bus_vmbus.so.2.1 drivers/librte_mempool_octeontx.so.1.1 lib/librte_pipeline.so.3.1 lib/librte_port.so.3.1 lib/librte_table.so.3.1 lib/librte_lpm.so.2.1 lib/librte_acl.so.2.1 lib/librte_gso.so.1.1 drivers/librte_common_cpt.so.1.1 lib/librte_reorder.so.1.1 lib/librte_compressdev.so.1.1 drivers/librte_pmd_dpaa.so.1.1 drivers/librte_pmd_dpaa2.so.2.1 drivers/librte_pmd_dpaa2_sec.so.2.1 drivers/librte_pmd_octeontx.so.1.1 lib/librte_bbdev.so.1.1 drivers/librte_bus_ifpga.so.2.1 -Wl,--no-as-needed -lvirt /usr/lib64/libjansson.so -lm -ldl /usr/lib64/libbsd.so -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl /usr/lib64/libz.so -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lpcap -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lsze2 -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lIPSec_MB -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lIPSec_MB -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl /usr/lib64/libcrypto.so -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -lm -ldl -Wl,--end-group -pthread '-Wl,-rpath,$ORIGIN/../lib:$ORIGIN/../drivers' -Wl,-rpath-link,/home/bruce/prototypes/both_libraries/build-gcc-static/lib:/home/bruce/prototypes/both_libraries/build-gcc-static/drivers
/usr/bin/ld: cannot open linker script file /usr/lib/gcc/x86_64-redhat-linux/8/../../../../lib64/libm.so: Too many open files

@nirbheek nirbheek reopened this Mar 12, 2019
@nirbheek nirbheek added this to the 0.51.0 milestone Mar 12, 2019
@nirbheek
Copy link
Member

Sigh, this keeps cropping up over and over. -pthread is getting de-duplicated but not -ldl and -lm.

@bruce-richardson
Copy link
Contributor Author

bruce-richardson commented Mar 12, 2019 •
edited
Loading

the following change seems to help here, i.e. it fixes things for me in DPDK, but we still have some duplication in the artifical test case I started. The one thing I'm wondering about is if there is ever a valid reason to have the one dependency appear twice in the dependency list? Do we need to be able to have -la -lb -la for external deps?

diff --git a/mesonbuild/build.py b/mesonbuild/build.py
index d456ab86..20f0cdb2 100644
--- a/mesonbuild/build.py
+++ b/mesonbuild/build.py
@@ -1030,8 +1030,9 @@ This will become a hard error in a future Meson release.''')
                 # Deps of deps.
                 self.add_deps(dep.ext_deps)
             elif isinstance(dep, dependencies.Dependency):
-                self.external_deps.append(dep)
-                self.process_sourcelist(dep.get_sources())
+                if dep not in self.external_deps:
+                    self.external_deps.append(dep)
+                    self.process_sourcelist(dep.get_sources())
             elif isinstance(dep, BuildTarget):
                 raise InvalidArguments('''Tried to use a build target as a dependency.
 You probably should put it in link_with instead.''')

nirbheek added a commit that referenced this issue Mar 12, 2019
@nirbheek
compilers: Try harder to dedup builtin libs
8b06553 
-ldl, -lm, -lpthread should always be de-duplicated, no matter what.

Closes #2150
@nirbheek nirbheek mentioned this issue Mar 12, 2019
Merged
@nirbheek
Copy link
Member

Do we need to be able to have -la -lb -la for external deps?

You're right, we may not need this anymore now that we use --start/end-group. We can merge this for 0.51 if you'd like to open a PR for it?

nirbheek added a commit that referenced this issue Mar 13, 2019
@nirbheek
compilers: Try harder to dedup builtin libs
a6e5440 
-ldl, -lm, -lpthread should always be de-duplicated, no matter what.

Closes #2150
bruce-richardson added a commit to bruce-richardson/meson that referenced this issue Mar 13, 2019
@bruce-richardson
do not duplicate external dependencies in list
4793538 
Since the "-l<lib>" flags in the build.ninja file are passed in
"--start-group"/"--end-group" flags, there should be no need to have any
library listed twice, even if there are circular dependencies. Therefore we
can eliminate duplicates. For speed, rather than deduplicating at the end
of the process, it's faster to not add the duplicate flags in the first
place.

This should help fix mesonbuild#2150
@bruce-richardson bruce-richardson mentioned this issue Mar 13, 2019
Merged
nirbheek added a commit that referenced this issue Mar 13, 2019
@nirbheek
compilers: Try harder to dedup builtin libs
97dc00d 
Compiler internal libs should always be de-duplicated, no matter what.

Closes #2150
nirbheek pushed a commit that referenced this issue Mar 13, 2019
@bruce-richardson @nirbheek
do not duplicate external dependencies in list
cfe82db 
Since the "-l<lib>" flags in the build.ninja file are passed in
"--start-group"/"--end-group" flags, there should be no need to have any
library listed twice, even if there are circular dependencies. Therefore we
can eliminate duplicates. For speed, rather than deduplicating at the end
of the process, it's faster to not add the duplicate flags in the first
place.

This should help fix #2150
nirbheek added a commit that referenced this issue Mar 13, 2019
@nirbheek
compilers: Try harder to dedup builtin libs
18b0b0a 
Compiler internal libs should always be de-duplicated, no matter what.

Closes #2150

Test case is by Bruce Richardson in the issue.
nirbheek added a commit that referenced this issue Mar 14, 2019
@nirbheek
compilers: Try harder to dedup builtin libs
0a25bab 
Compiler internal libs should always be de-duplicated, no matter what.

Closes #2150

Test case is by Bruce Richardson in the issue.
nirbheek added a commit that referenced this issue Mar 14, 2019
@nirbheek
compilers: Try harder to dedup builtin libs
0ab833d 
Compiler internal libs should always be de-duplicated, no matter what.

Closes #2150

Test case is by Bruce Richardson in the issue.
vt-alt pushed a commit to altlinux/specs that referenced this issue Mar 14, 2019
@YuriSedunov
224994: meson-0.49.2-alt3
bc73c2c 
- backported fix for mesonbuild/meson#2150
nirbheek added a commit that referenced this issue Mar 14, 2019
@nirbheek
compilers: Try harder to dedup builtin libs
07818da 
Compiler internal libs should always be de-duplicated, no matter what.

Closes #2150

Test case is by Bruce Richardson in the issue.
nirbheek added a commit that referenced this issue Apr 16, 2019
@nirbheek
compilers: Try harder to dedup builtin libs
8a2c2c9 
Compiler internal libs should always be de-duplicated, no matter what.

Closes #2150

Test case is by Bruce Richardson in the issue.
@krader1961
Copy link
Contributor

FYI, Commit 07818da merged in response to this issue breaks building programs on OpenBSD that need -lexecinfo. The proximate cause is that the change causes this probe for libexecinfo to fail because the dirs argument is ignored:

cc.find_library('execinfo', required: false, dirs: ['/usr/local/lib'])

See #5432.

tbeloqui pushed a commit to pexip/meson that referenced this issue Aug 22, 2019
@bruce-richardson @tbeloqui
do not duplicate external dependencies in list
17db978 
Since the "-l<lib>" flags in the build.ninja file are passed in
"--start-group"/"--end-group" flags, there should be no need to have any
library listed twice, even if there are circular dependencies. Therefore we
can eliminate duplicates. For speed, rather than deduplicating at the end
of the process, it's faster to not add the duplicate flags in the first
place.

This should help fix mesonbuild#2150
tbeloqui pushed a commit to pexip/meson that referenced this issue Aug 22, 2019
@nirbheek @tbeloqui
compilers: Try harder to dedup builtin libs
a263ea4 
Compiler internal libs should always be de-duplicated, no matter what.

Closes mesonbuild#2150

Test case is by Bruce Richardson in the issue.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
bug compilers dependencies
Projects
None yet
Milestone
0.51.0
Development

Successfully merging a pull request may close this issue.

do not duplicate external dependencies in list bruce-richardson/meson
4 participants
@nirbheek @Ralith @krader1961 @bruce-richardson