Added snort3 & libdaq recipes #789

RNGesus-exe · 2024-02-13T15:34:07Z

No description provided.

meta-networking/recipes-connectivity/daq/libdaq_git.bb

kraj · 2024-02-13T16:37:39Z

meta-networking/recipes-connectivity/snort/snort3_git.bb

+LIC_FILES_CHKSUM = "file://COPYING;md5=78fa8ef966b48fbf9095e13cc92377c5"
+
+DEPENDS = "luajit libdaq libdnet flex hwloc libpcap libpcre zlib libtirpc"
+RDEPENDS:${PN} += "bash"


Can we remove this direct dependency ? perhaps convert the shell to not need bash extentions or move the concerning scripts to package of its own.

I tried to remove this dependency by creating a patch for the bash script but there are a lot of changes to be made in this script and I don't need it for my use case.

OK, can be move the script to a package of its own. ? something like below

PACKAGES += "${PN}-scripts"

FILES:${PN}-scripts = "/path/to/script/name"

RDEPENDS:${PN}-scripts = "bash"

sure, I'll try making this change, Also can I push all my changes in one single commit rather than making a commit for each recipe?

Usually one commit per recipe is good

meta-networking/recipes-connectivity/snort/snort3_git.bb

kraj · 2024-02-14T16:48:25Z

there is a build failure as well - https://errors.yoctoproject.org/Errors/Details/752967/

kraj · 2024-02-14T16:51:57Z

also make sure its only enabled for architectures supporting luajit otherwise we see errors like this

12:09:02 luajit was skipped: incompatible with host riscv64-yoe-linux (not in COMPATIBLE_HOST)
12:09:10 ERROR: Nothing RPROVIDES 'snort3-dev' (but /mnt/jenkins/workspace/yocto-world-glibc/sources/meta-openembedded/meta-networking/recipes-connectivity/snort/snort3_git.bb RDEPENDS on or otherwise requires it)
12:09:10 No eligible RPROVIDERs exist for 'snort3-dev'
12:09:10 NOTE: Runtime target 'snort3-dev' is unbuildable, removing...
12:09:10 Missing or unbuildable dependency chain was: ['snort3-dev']
12:09:15 ERROR: Nothing RPROVIDES 'snort3' (but /mnt/jenkins/workspace/yocto-world-glibc/sources/meta-openembedded/meta-networking/recipes-connectivity/snort/snort3_git.bb RDEPENDS on or otherwise requires it)
12:09:15 No eligible RPROVIDERs exist for 'snort3'
12:09:15 NOTE: Runtime target 'snort3' is unbuildable, removing...
12:09:15 Missing or unbuildable dependency chain was: ['snort3']

RNGesus-exe · 2024-02-15T22:50:03Z

@kraj Can you tell me how to reproduce the above errors on my end. I ended up isolating the bash dependent script into a seperate package but I want to get the errors on my end so I can fix luajit + yocto build error

kraj · 2024-02-17T03:13:15Z

@kraj Can you tell me how to reproduce the above errors on my end. I ended up isolating the bash dependent script into a seperate package but I want to get the errors on my end so I can fix luajit + yocto build error

I am building with clang/libc++, one way to reproduce it would be to use yoe distro https://github.com/yoedistro/yoe-distro

. ./envsetup.sh qemuriscv64
bitbake snort3

RNGesus-exe · 2024-02-19T21:17:20Z

I added the following code block to my snort3 recipe, is this what you want? I read the luajit recipe and apparently, it is not supported for the following architectures yet. I reproduced the error using the approach you gave above as well.

# mips64/ppc/ppc64/riscv64/riscv32 is not supported
COMPATIBLE_HOST:mipsarchn32 = "null"
COMPATIBLE_HOST:mipsarchn64 = "null"
COMPATIBLE_HOST:powerpc = "null"
COMPATIBLE_HOST:powerpc64 = "null"
COMPATIBLE_HOST:powerpc64le = "null"
COMPATIBLE_HOST:riscv64 = "null"
COMPATIBLE_HOST:riscv32 = "null"

kraj · 2024-02-19T21:48:04Z

Yes that would fix the compatiblity errror I am still worried about the build failures seen with clang and libc++

RNGesus-exe · 2024-02-20T12:06:46Z

Yup, that build error is still occurring for qemuarm, I am resolving it for now

RNGesus-exe · 2024-02-20T13:44:00Z

Update: I fixed the build error for qemuarm machine, by passing "-lm" flag to the libdaq recipe

LDFLAG += "-lm"

Now, libdaq is building but when building snort3, I get a new error something as below

/yoe/build/tmp/work/cortexa15t2hf-neon-yoe-linux-gnueabi/snort3/git/git/src/helpers/process.cc:191:56: error: use of undeclared identifier 'unw_strerror'; did you mean 'strerror'?
|   191 |         ssp.printf("unw_getcontext failed: %s (%d)\n", unw_strerror(ret), ret);
|       |                                                        ^~~~~~~~~~~~
|       |                                                        strerror

I added libunwind in DEPENDS and passed -lunwind as LDFLAG but the error still occurring so trying to resolve that

RNGesus-exe · 2024-02-20T17:35:15Z

@kraj I have created a patch (which, I'll push once the following issue gets resolved) for the llibdaq build error as show below

| arm-yoe-linux-gnueabi-ld.lld: error: undefined symbol: ceilf
| >>> referenced by math.h:632 (/usr/include/c++/v1/math.h:632)
| >>>               libdaq_static_fst_la-daq_fst.o:(std::__1::pair<std::__1::__hash_iterator<std::__1::__hash_node<std::__1::__hash_value_type<FstKey, FstNode*>, void*>*>, bool> std::__1::__hash_table<std::__1::__hash_value_type<FstKey, FstNode*>, std::__1::__unordered_map_hasher<FstKey, std::__1::__hash_value_type<FstKey, FstNode*>, FstKeyHash, std::__1::equal_to<FstKey>, true>, std::__1::__unordered_map_equal<FstKey, std::__1::__hash_value_type<FstKey, FstNode*>, std::__1::equal_to<FstKey>, FstKeyHash, true>, std::__1::allocator<std::__1::__hash_value_type<FstKey, FstNode*>>>::__emplace_unique_key_args<FstKey, std::__1::pair<FstKey const, FstNode*>>(FstKey const&, std::__1::pair<FstKey const, FstNode*>&&)) in archive ../modules/fst/.libs/libdaq_static_fst.a

libdaq is working now but snort3 produces a build error as follows

/yoe/build/tmp/work/cortexa15t2hf-neon-yoe-linux-gnueabi/snort3/git/git/src/helpers/process.cc:191:56: error: use of undeclared identifier 'unw_strerror'; did you mean 'strerror'?
|   191 |         ssp.printf("unw_getcontext failed: %s (%d)\n", unw_strerror(ret), ret);
|       |                                                        ^~~~~~~~~~~~
|       |                                                        strerror

This is the exact line number where the error is produced during snort3 compilation process.cc line 191 error
The library libunwind.h on github has the function unw_strerror but when I checked the meta-clang layer configuration, I noticed this meta_clang layer.conf

So, to sum it up meta_clang provides lib_cxx and lib_cxx provides libunwind (which does not have the unw_sterror). My goal is not to add clang support to snort3 so, Can you tell me how to add incompatibility for snort3 using clang.

kraj · 2024-02-20T17:38:22Z

This error means a dependency on libunwind is missing.

RNGesus-exe · 2024-02-20T17:40:16Z

This error means a dependency on libunwind is missing.

I added it already, but it doesn't work. As meta-clang provides it's own libunwind

kraj · 2024-02-20T17:56:14Z

Can you set TC_CXX_RUNTIME = "gnu" and see if that helps ?

RNGesus-exe · 2024-02-20T18:21:49Z

Can you set TC_CXX_RUNTIME = "gnu" and see if that helps ?

same error

kraj · 2024-02-20T18:25:11Z

Can you set TC_CXX_RUNTIME = "gnu" and see if that helps ?

same error

Add it in meta-clang/conf/nonclangable.conf like below

TC_CXX_RUNTIME:pn-snort3:toolchain-clang = "gnu"

RNGesus-exe · 2024-02-20T18:40:50Z

Can you set TC_CXX_RUNTIME = "gnu" and see if that helps ?

same error

Add it in meta-clang/conf/nonclangable.conf like below
TC_CXX_RUNTIME:pn-snort3:toolchain-clang = "gnu"

nope, still the same error Yocto Build Error

RNGesus-exe · 2024-02-20T18:48:37Z

I checked the environment bitbake -e libunwind, it is still being provided by libcxx

FILE="/home/rngesusexe/Programming/yocto/yoe/sources/meta-clang/recipes-devtools/clang/libcxx_git.bb"

kraj · 2024-02-20T18:51:58Z

OK, can you post your updated patches to the PR, I want to take a look.

kraj · 2024-02-20T19:00:00Z

Check for TC_CXX_RUNTIME what is it set to

I am adding this recipe as snort3 depends on it. snort3 recipe will be provided in next commit. See more: https://github.com/snort3/libdaq Signed-off-by: Khawaja Shaheryar <[email protected]>

RNGesus-exe · 2024-02-20T19:13:49Z

Check for TC_CXX_RUNTIME what is it set to

I have pushed the new changes, but here the variable is set as following
TC_CXX_RUNTIME="llvm"

kraj · 2024-02-20T19:38:13Z

Check for TC_CXX_RUNTIME what is it set to

I have pushed the new changes, but here the variable is set as following TC_CXX_RUNTIME="llvm"

Even when you set to gnu ? If so then that’s the issue

RNGesus-exe · 2024-02-20T19:42:18Z

Check for TC_CXX_RUNTIME what is it set to

I have pushed the new changes, but here the variable is set as following TC_CXX_RUNTIME="llvm"

Even when you set to gnu ? If so then that’s the issue

Yup I added in meta-clang/conf/nonclangable.conf
TC_CXX_RUNTIME:pn-snort3:toolchain-clang = "gnu"

RNGesus-exe · 2024-02-20T22:10:44Z

@kraj is there any way to make libunwind from poky/meta/recipes-support/libunwind/libunwind_1.6.2.bb be used instead of the meta-clang one.

kraj · 2024-02-20T23:42:23Z

Please test 8db172c

this should fix the problem.

kraj · 2024-02-20T23:55:10Z

Addition QA warnings

WARNING: snort3-3+git-r0 do_package_qa: QA Issue: File /usr/lib/pkgconfig/snort.pc in package snort3-dev contains reference to TMPDIR [buildpaths]
WARNING: snort3-3+git-r0 do_package_qa: QA Issue: File /usr/src/debug/snort3/3+git/src/js_norm/pdf_tokenizer.cc in package snort3-src contains reference to TMPDIR
File /usr/src/debug/snort3/3+git/src/js_norm/js_tokenizer.cc in package snort3-src contains reference to TMPDIR [buildpaths]

For once I have addressed them via

9736478

However, I expect submitters to be able to handle these in general.

RNGesus-exe · 2024-02-21T00:13:46Z

Please test 8db172c

this should fix the problem.

Tested, it is working

RNGesus-exe · 2024-02-21T00:14:58Z

Addition QA warnings

WARNING: snort3-3+git-r0 do_package_qa: QA Issue: File /usr/lib/pkgconfig/snort.pc in package snort3-dev contains reference to TMPDIR [buildpaths]
WARNING: snort3-3+git-r0 do_package_qa: QA Issue: File /usr/src/debug/snort3/3+git/src/js_norm/pdf_tokenizer.cc in package snort3-src contains reference to TMPDIR
File /usr/src/debug/snort3/3+git/src/js_norm/js_tokenizer.cc in package snort3-src contains reference to TMPDIR [buildpaths]

For once I have addressed them via

9736478

However, I expect submitters to be able to handle these in general.

I didn't have these QA warnings at my end as the build was failing or else I would have resolved them, nonetheless thankyou for resolving them from your side.

I am adding this recipe as snort2 is legacy now. See more: https://github.com/snort3/snort3 Signed-off-by: Khawaja Shaheryar <[email protected]>

kraj · 2024-02-22T21:00:36Z

applied to master branch as of 7132d7b

shr-project · 2024-02-24T07:43:24Z

In multilib builds I'm also seeing:

ERROR: lib32-snort3-3+git-r0 do_populate_sysroot: QA Issue: snort.pc failed sanity test (tmpdir) in path lib32-snort3/3+git/sysroot-destdir/usr/lib/pkgconfig [pkgconfig]

And the ${SRCPV} shouldn't be in PV, see:
https://git.openembedded.org/meta-openembedded/commit/?id=21f956598ddb1bb08855fbc9a3293acf957154d1

shr-project · 2024-02-24T08:41:39Z

I'll send fixes for both issues to ML shortly.

RNGesus-exe · 2024-04-16T11:31:59Z

@shr-project How can I reproduce this error

In multilib builds I'm also seeing:

ERROR: lib32-snort3-3+git-r0 do_populate_sysroot: QA Issue: snort.pc failed sanity test (tmpdir) in path lib32-snort3/3+git/sysroot-destdir/usr/lib/pkgconfig [pkgconfig]

moreover can you kindly give some info on how to resolve this issue.

shr-project · 2024-04-16T11:42:16Z

See the fixes I've sent to ML and were already merged (see https://git.openembedded.org/meta-openembedded/log/?qt=grep&q=snort3)

Changelog: https://github.com/c-ares/c-ares/releases/tag/v1.32.0 Features: Add support for DNS 0x20 to help prevent cache poisoning attacks, enabled by specifying ARES_FLAG_DNS0x20. Disabled by default. PR openembedded#800 Rework query timeout logic to automatically adjust timeouts based on network conditions. The timeout specified now is only used as a hint until there is enough history to calculate a more valid timeout. PR openembedded#794 Changes: DNS RR TXT strings should not be automatically concatenated as there are use cases outside of RFC 7208. In order to maintain ABI compliance, the ability to retrieve TXT strings concatenated is retained as well as a new API to retrieve the individual strings. This restores behavior from c-ares 1.20.0. PR openembedded#801 Clean up header inclusion logic to make hacking on code easier. PR openembedded#797 GCC/Clang: Enable even more strict warnings to catch more coding flaws. 253bdee MSVC: Enable /W4 warning level. PR openembedded#792 Bugfixes: Tests: Fix thread race condition in test cases for EventThread. PR openembedded#803 Windows: Fix building with UNICODE. PR openembedded#802 Thread Saftey: ares_timeout() was missing lock. 74a64e4 Fix building with DJGPP (32bit protected mode DOS). PR openembedded#789 Signed-off-by: Jason Schonberg <[email protected]> Signed-off-by: Khem Raj <[email protected]>

kraj reviewed Feb 13, 2024

View reviewed changes

RNGesus-exe force-pushed the master branch from e8ecfaf to b23362a Compare February 14, 2024 12:14

libdaq: add recipe

b376808

I am adding this recipe as snort3 depends on it. snort3 recipe will be provided in next commit. See more: https://github.com/snort3/libdaq Signed-off-by: Khawaja Shaheryar <[email protected]>

RNGesus-exe force-pushed the master branch from b23362a to ec37377 Compare February 20, 2024 19:11

snort: add snort3 initial recipe

91553f2

I am adding this recipe as snort2 is legacy now. See more: https://github.com/snort3/snort3 Signed-off-by: Khawaja Shaheryar <[email protected]>

RNGesus-exe force-pushed the master branch from ec37377 to 91553f2 Compare February 21, 2024 00:21

kraj closed this Feb 22, 2024

RNGesus-exe mentioned this pull request Mar 12, 2024

general question - Is there anyway to add snort3 in this? kraj/meta-openwrt#140

Closed

Added snort3 & libdaq recipes #789

Added snort3 & libdaq recipes #789

Conversation

RNGesus-exe commented Feb 13, 2024

kraj Feb 13, 2024

Choose a reason for hiding this comment

RNGesus-exe Feb 14, 2024

Choose a reason for hiding this comment

kraj Feb 14, 2024

Choose a reason for hiding this comment

RNGesus-exe Feb 14, 2024

Choose a reason for hiding this comment

kraj Feb 14, 2024

Choose a reason for hiding this comment

kraj commented Feb 14, 2024

kraj commented Feb 14, 2024

RNGesus-exe commented Feb 15, 2024 • edited Loading

kraj commented Feb 17, 2024

RNGesus-exe commented Feb 19, 2024 • edited Loading

kraj commented Feb 19, 2024

RNGesus-exe commented Feb 20, 2024

RNGesus-exe commented Feb 20, 2024 • edited Loading

RNGesus-exe commented Feb 20, 2024 • edited Loading

kraj commented Feb 20, 2024

RNGesus-exe commented Feb 20, 2024

kraj commented Feb 20, 2024

RNGesus-exe commented Feb 20, 2024 • edited Loading

kraj commented Feb 20, 2024

RNGesus-exe commented Feb 20, 2024

RNGesus-exe commented Feb 20, 2024

kraj commented Feb 20, 2024

kraj commented Feb 20, 2024

RNGesus-exe commented Feb 20, 2024 • edited Loading

kraj commented Feb 20, 2024

RNGesus-exe commented Feb 20, 2024

RNGesus-exe commented Feb 20, 2024

kraj commented Feb 20, 2024

kraj commented Feb 20, 2024 • edited Loading

RNGesus-exe commented Feb 21, 2024

RNGesus-exe commented Feb 21, 2024 • edited Loading

kraj commented Feb 22, 2024

shr-project commented Feb 24, 2024

shr-project commented Feb 24, 2024

RNGesus-exe commented Apr 16, 2024

shr-project commented Apr 16, 2024

RNGesus-exe commented Feb 15, 2024 •

edited

Loading

RNGesus-exe commented Feb 19, 2024 •

edited

Loading

RNGesus-exe commented Feb 20, 2024 •

edited

Loading

RNGesus-exe commented Feb 20, 2024 •

edited

Loading

RNGesus-exe commented Feb 20, 2024 •

edited

Loading

RNGesus-exe commented Feb 20, 2024 •

edited

Loading

kraj commented Feb 20, 2024 •

edited

Loading

RNGesus-exe commented Feb 21, 2024 •

edited

Loading