Failed to build triton server on an aarch64 device with error "error: ‘size_t’ does not name a type"

[brightdark]

Description
Failed to build triton server on an aarch64 device with error:

/data/user/triton/server-2.15.0/build/server/../../src/core/constants.h:88:11: error: ‘size_t’ does not name a type
constexpr size_t STRING_CORRELATION_ID_MAX_LENGTH_BYTES = 128;
^~~~~~

[ 64%] Building CXX object src/core/CMakeFiles/server-library.dir/model_config_utils.cc.o
cd /data/user/triton/server-2.15.0/citritonbuild/tritonserver/build/server/src/core && /usr/bin/c++ -DTRITON_ENABLE_ENSEMBLE=1 -DTRITON_ENABLE_HTTP=1 -DTRITON_ENABLE_LOGGING=1 -DTRITON_ENABLE_STATS=1 -DTRITON_ENABLE_TRACING=1 -DTRITON_VERSION="2.15.0" -I/data/user/triton/server-2.15.0/build/server/../.. -I/data/user/triton/server-2.15.0/citritonbuild/tritonserver/build/server -I/data/user/triton/server-2.15.0/citritonbuild/tritonserver/build/third-party/libevent/include -I/data/user/triton/server-2.15.0/citritonbuild/tritonserver/build/third-party/protobuf/include -I/data/user/triton/server-2.15.0/citritonbuild/tritonserver/build/server/_deps/repo-common-build/protobuf -I/data/user/triton/server-2.15.0/citritonbuild/tritonserver/build/server/_deps/repo-core-src/include -I/data/user/triton/server-2.15.0/citritonbuild/tritonserver/build/server/_deps/repo-common-src/include -Wall -Wextra -Wno-unused-parameter -Werror -Wno-deprecated-declarations -g -rdynamic -fPIC -std=gnu++11 -MD -MT src/core/CMakeFiles/server-library.dir/model_config_utils.cc.o -MF CMakeFiles/server-library.dir/model_config_utils.cc.o.d -o CMakeFiles/server-library.dir/model_config_utils.cc.o -c /data/user/triton/server-2.15.0/src/core/model_config_utils.cc
In file included from /data/user/triton/server-2.15.0/build/server/../../src/core/memory.h:29:0,
from /data/user/triton/server-2.15.0/src/core/memory.cc:27:
/data/user/triton/server-2.15.0/build/server/../../src/core/constants.h:88:11: error: ‘size_t’ does not name a type
constexpr size_t STRING_CORRELATION_ID_MAX_LENGTH_BYTES = 128;
^~~~~~
[ 65%] Building CXX object src/core/CMakeFiles/server-library.dir/model_repository_manager.cc.o
cd /data/user/triton/server-2.15.0/citritonbuild/tritonserver/build/server/src/core && /usr/bin/c++ -DTRITON_ENABLE_ENSEMBLE=1 -DTRITON_ENABLE_HTTP=1 -DTRITON_ENABLE_LOGGING=1 -DTRITON_ENABLE_STATS=1 -DTRITON_ENABLE_TRACING=1 -DTRITON_VERSION="2.15.0" -I/data/user/triton/server-2.15.0/build/server/../.. -I/data/user/triton/server-2.15.0/citritonbuild/tritonserver/build/server -I/data/user/triton/server-2.15.0/citritonbuild/tritonserver/build/third-party/libevent/include -I/data/user/triton/server-2.15.0/citritonbuild/tritonserver/build/third-party/protobuf/include -I/data/user/triton/server-2.15.0/citritonbuild/tritonserver/build/server/_deps/repo-common-build/protobuf -I/data/user/triton/server-2.15.0/citritonbuild/tritonserver/build/server/_deps/repo-core-src/include -I/data/user/triton/server-2.15.0/citritonbuild/tritonserver/build/server/_deps/repo-common-src/include -Wall -Wextra -Wno-unused-parameter -Werror -Wno-deprecated-declarations -g -rdynamic -fPIC -std=gnu++11 -MD -MT src/core/CMakeFiles/server-library.dir/model_repository_manager.cc.o -MF CMakeFiles/server-library.dir/model_repository_manager.cc.o.d -o CMakeFiles/server-library.dir/model_repository_manager.cc.o -c /data/user/triton/server-2.15.0/src/core/model_repository_manager.cc
[ 67%] Building CXX object src/core/CMakeFiles/server-library.dir/numa_utils.cc.o
cd /data/user/triton/server-2.15.0/citritonbuild/tritonserver/build/server/src/core && /usr/bin/c++ -DTRITON_ENABLE_ENSEMBLE=1 -DTRITON_ENABLE_HTTP=1 -DTRITON_ENABLE_LOGGING=1 -DTRITON_ENABLE_STATS=1 -DTRITON_ENABLE_TRACING=1 -DTRITON_VERSION="2.15.0" -I/data/user/triton/server-2.15.0/build/server/../.. -I/data/user/triton/server-2.15.0/citritonbuild/tritonserver/build/server -I/data/user/triton/server-2.15.0/citritonbuild/tritonserver/build/third-party/libevent/include -I/data/user/triton/server-2.15.0/citritonbuild/tritonserver/build/third-party/protobuf/include -I/data/user/triton/server-2.15.0/citritonbuild/tritonserver/build/server/_deps/repo-common-build/protobuf -I/data/user/triton/server-2.15.0/citritonbuild/tritonserver/build/server/_deps/repo-core-src/include -I/data/user/triton/server-2.15.0/citritonbuild/tritonserver/build/server/_deps/repo-common-src/include -Wall -Wextra -Wno-unused-parameter -Werror -Wno-deprecated-declarations -g -rdynamic -fPIC -std=gnu++11 -MD -MT src/core/CMakeFiles/server-library.dir/numa_utils.cc.o -MF CMakeFiles/server-library.dir/numa_utils.cc.o.d -o CMakeFiles/server-library.dir/numa_utils.cc.o -c /data/user/triton/server-2.15.0/src/core/numa_utils.cc
[ 68%] Building CXX object src/core/CMakeFiles/server-library.dir/payload.cc.o
cd /data/user/triton/server-2.15.0/citritonbuild/tritonserver/build/server/src/core && /usr/bin/c++ -DTRITON_ENABLE_ENSEMBLE=1 -DTRITON_ENABLE_HTTP=1 -DTRITON_ENABLE_LOGGING=1 -DTRITON_ENABLE_STATS=1 -DTRITON_ENABLE_TRACING=1 -DTRITON_VERSION="2.15.0" -I/data/user/triton/server-2.15.0/build/server/../.. -I/data/user/triton/server-2.15.0/citritonbuild/tritonserver/build/server -I/data/user/triton/server-2.15.0/citritonbuild/tritonserver/build/third-party/libevent/include -I/data/user/triton/server-2.15.0/citritonbuild/tritonserver/build/third-party/protobuf/include -I/data/user/triton/server-2.15.0/citritonbuild/tritonserver/build/server/_deps/repo-common-build/protobuf -I/data/user/triton/server-2.15.0/citritonbuild/tritonserver/build/server/_deps/repo-core-src/include -I/data/user/triton/server-2.15.0/citritonbuild/tritonserver/build/server/_deps/repo-common-src/include -Wall -Wextra -Wno-unused-parameter -Werror -Wno-deprecated-declarations -g -rdynamic -fPIC -std=gnu++11 -MD -MT src/core/CMakeFiles/server-library.dir/payload.cc.o -MF CMakeFiles/server-library.dir/payload.cc.o.d -o CMakeFiles/server-library.dir/payload.cc.o -c /data/user/triton/server-2.15.0/src/core/payload.cc
c++: internal compiler error: Killed (program cc1plus)

[Kernel]
Linux linaro 4.9.38-v6.3.0-00428-gcce7bea #2 SMP Mon Sep 28 06:14:07 CST 2020 aarch64 GNU/Linux
[OS]
Distributor ID: Debian
Description: Debian GNU/Linux 9.9 (stretch)
Release: 9.9
Codename: stretch
[version]
gcc (Debian 6.3.0-18+deb9u1) 6.3.0 20170516
g++ (Debian 6.3.0-18+deb9u1) 6.3.0 20170516
Python 3.5.3 (default, Sep 27 2018, 17:25:39)
GNU Make 4.1 Built for aarch64-unknown-linux-gnu
cmake version 3.21.4

-- The C compiler identification is GNU 6.3.0
-- The CXX compiler identification is GNU 6.3.0
-- Detecting C compiler ABI info
-- Detecting C compiler ABI info - done
-- Check for working C compiler: /usr/bin/cc - skipped
-- Detecting C compile features
-- Detecting C compile features - done
-- Detecting CXX compiler ABI info
-- Detecting CXX compiler ABI info - done
-- Check for working CXX compiler: /usr/bin/c++ - skipped
-- Detecting CXX compile features
-- Detecting CXX compile features - done
-- RapidJSON found. Headers: /usr/local/include
-- RapidJSON found. Headers: /usr/local/include
-- Looking for pthread.h
-- Looking for pthread.h - found
-- Performing Test CMAKE_HAVE_LIBC_PTHREAD
-- Performing Test CMAKE_HAVE_LIBC_PTHREAD - Failed
-- Looking for pthread_create in pthreads
-- Looking for pthread_create in pthreads - not found
-- Looking for pthread_create in pthread
-- Looking for pthread_create in pthread - found
-- Found Threads: TRUE
-- Found Protobuf: /data/user/triton/server-2.15.0/citritonbuild/tritonserver/build/third-party/protobuf/bin/protoc-3.8.0.0 (found version "3.8.0.0")
-- Using protobuf 3.8.0.0
-- Found Boost: /usr/include (found version "1.62.0")
-- Using Boost 1.62.0
-- Using libevent 2.1.8
-- Using protobuf 3.8.0.0
-- Using libevhtp 1.2.18
-- Using tracing
-- Configuring done
-- Generating done
-- Build files have been written to: /data/user/triton/server-2.15.0/citritonbuild/tritonserver/build/server

[g++/c++环境]

Using built-in specs.
COLLECT_GCC=c++
Target: aarch64-linux-gnu
Configured with: ../src/configure -v --with-pkgversion='Debian 6.3.0-18+deb9u1' --with-bugurl=file:///usr/share/doc/gcc-6/README.Bugs --enable-languages=c,ada,c++,java,go,d,fortran,objc,obj-c++ --prefix=/usr --program-suffix=-6 --program-prefix=aarch64-linux-gnu- --enable-shared --enable-linker-build-id --libexecdir=/usr/lib --without-included-gettext --enable-threads=posix --libdir=/usr/lib --enable-nls --with-sysroot=/ --enable-clocale=gnu --enable-libstdcxx-debug --enable-libstdcxx-time=yes --with-default-libstdcxx-abi=new --enable-gnu-unique-object --disable-libquadmath --enable-plugin --enable-default-pie --with-system-zlib --disable-browser-plugin --enable-java-awt=gtk --enable-gtk-cairo --with-java-home=/usr/lib/jvm/java-1.5.0-gcj-6-arm64/jre --enable-java-home --with-jvm-root-dir=/usr/lib/jvm/java-1.5.0-gcj-6-arm64 --with-jvm-jar-dir=/usr/lib/jvm-exports/java-1.5.0-gcj-6-arm64 --with-arch-directory=aarch64 --with-ecj-jar=/usr/share/java/eclipse-ecj.jar --enable-multiarch --enable-fix-cortex-a53-843419 --enable-checking=release --build=aarch64-linux-gnu --host=aarch64-linux-gnu --target=aarch64-linux-gnu
Thread model: posix
gcc version 6.3.0 20170516 (Debian 6.3.0-18+deb9u1)
COLLECT_GCC_OPTIONS='-v' '-E' '-shared-libgcc' '-mlittle-endian' '-mabi=lp64'
/usr/lib/gcc/aarch64-linux-gnu/6/cc1 -E -quiet -v -imultiarch aarch64-linux-gnu - -mlittle-endian -mabi=lp64
ignoring nonexistent directory "/usr/local/include/aarch64-linux-gnu"
ignoring nonexistent directory "/usr/lib/gcc/aarch64-linux-gnu/6/../../../../aarch64-linux-gnu/include"
#include "..." search starts here:
#include <...> search starts here:
/usr/lib/gcc/aarch64-linux-gnu/6/include
/usr/local/include
/usr/lib/gcc/aarch64-linux-gnu/6/include-fixed
/usr/include/aarch64-linux-gnu
/usr/include
End of search list.
stdin
built-in
command-line
command-line
/usr/include/stdc-predef.h 1 3 4
command-line 2
stdin
COMPILER_PATH=/usr/lib/gcc/aarch64-linux-gnu/6/:/usr/lib/gcc/aarch64-linux-gnu/6/:/usr/lib/gcc/aarch64-linux-gnu/:/usr/lib/gcc/aarch64-linux-gnu/6/:/usr/lib/gcc/aarch64-linux-gnu/
LIBRARY_PATH=/usr/lib/gcc/aarch64-linux-gnu/6/:/usr/lib/gcc/aarch64-linux-gnu/6/../../../aarch64-linux-gnu/:/usr/lib/gcc/aarch64-linux-gnu/6/../../../../lib/:/lib/aarch64-linux-gnu/:/lib/../lib/:/usr/lib/aarch64-linux-gnu/:/usr/lib/../lib/:/usr/lib/gcc/aarch64-linux-gnu/6/../../../:/lib/:/usr/lib/
COLLECT_GCC_OPTIONS='-v' '-E' '-shared-libgcc' '-mlittle-endian' '-mabi=lp64'

Triton Information
2.15.0 & 2.14.0

Are you using the Triton container or did you build it yourself?
build from source with no container

To Reproduce

• download release source code .zip file
• unzip
• install dependencies under instruction of <Release Notes> in "Installation and Usage" Section
• Build RapidJson from source( version in apt repo is too old)
• build triton server by build.py
  ./build.py --build-type=Debug --install-dir=/data/user/triton/server-2.15.0/install --cmake-dir=/data/user/triton/server-2.15.0/build --build-dir=/data/user/triton/server-2.15.0/citritonbuild --no-container-build --enable-logging --enable-stats --enable-tracing --endpoint=http --repo-tag=common:r21.10 --repo-tag=core:r21.10 --repo-tag=backend:r21.10 --backend=ensemble --backend=identity

Expected behavior
Build success.

[brightdark]

I modified the file src/core/constants.h
add #include <stddef.h> at the beginning.

As a result, the compilation continues to run but failed again with the following error:

Any suggestion to what I should do? upgrade gcc/g++ version?

Thx!

[brightdark]

I modified the file src/core/constants.h add #include <stddef.h> at the beginning.

As a result, the compilation continues to run but failed again with the following error:

Any suggestion to what I should do? upgrade gcc/g++ version?

Thx!

The above problem is due to out of memory. Add swap file will solve the issue.

!!! ATTENTION !!!
boost/stacktrace.hpp is needed which is in boost >=1.65.
I need to build boost from source. I will tell after success.

[brightdark]

I check the boost version in NGC docker which is 1.70.
With built boost dev libs(version 1.70) from source (1.65+ should be OK), building of triton succeed at the end.

Summary:

1. With lower version of gcc/g++ (e.g. 6.3.0), add #include <stddef.h> at the beginning of src/core/constants.h.
2. Building on aarch platforms with limited memory, you should add swap space to 2GB.
3. Make sure cmake Version>=3.21, otherwise build it from source.
4. If the RapidJson Lib in apt sources is too old, build it from source or download and install aarch pre-built files.
5. Make sure version of boost libs >=1.65, otherwise build it from source.