[OSCD Initiative] Vulners analyzer

[uchakin]

1. Vulners_IOC: As a result of collaboration between Vulners and RST Threat Feed, the idea was to send IOC analysis results through theHive analyzer: blog post
2. Vulners_CVE: Vulners have a strong vulnerability database. This data is useful if:
   "if the case (incident) is related to the exploitation of a vulnerability, then the analyst (manually / automatically) can add it to observables and quickly get all the basic information on it in order to continue analyzing the case."
   By default theHive does not have a "cve" type to be observables, so we have to add it to Administrator Settings:

Vulners API KEY available in the user's profile after registration: vulners.com

P.S. part of OSCD sprint

[uchakin]

More details: https://github.com/uchakin/Vulners-analyzer

[dadokkio]

I've an issue with Vulners_IOC:

"errorMessage": "Traceback (most recent call last): File \"/opt/cortex/analyzers/Vulners/vulners_analyzer.py\", line 111, in <module> VulnersAnalyzer().run() File \"/opt/cortex/analyzers/Vulners/vulners_analyzer.py\", line 47, in run if document_id or document_id['type'] == 'rst':TypeError: list indices must be integers or slices, not str",

Probably the check is not correct or there is an error if the return list in empty. Can you check?

It's still missing templates and documentation info but seems good to me.

[uchakin]

@dadokkio

Hello! Thank you for your review.
Fixed an identified error in the last commit and tested the code again on different data.

I did not fully understand your comment about documentation and templates. Or do you mean to add README to the pool request?
Templates according to documentation are placed in 2 folders: thehive-templates/Vulners_CVE_1_0/* and thehive-templates/Vulners_IOC_1_0/*.

[dadokkio]

For the documentation you can now add a README file, subscription and logo informations in json files to have them deployed automatically in the new docs.

For the code I still have issues.. it seems that search returns a list so I have to iterate on the results. Something like this:

        if self.service == 'ioc':
            if self.data_type in ['ip', 'domain', 'url']:
                data = self.get_param('data', None, 'Data is missing')
                document_ids = self.vulners.search(f'iocType:{self.data_type} AND {self.data_type}:"{data}"')

                results = []

                for document_id in document_ids:
                    if document_id.get('type', None) == 'rst':
                        full_document_info = self.vulners.document(document_id['id'],  fields=["*"])
                        ioc_report = {
                            'service': self.service,
                            'first_seen': full_document_info['published'],
                            'last_seen': full_document_info['lastseen'],
                            'tags': full_document_info['tags'],
                            'ioc_score': full_document_info['iocScore']['ioc_total'],
                            'ioc_url': full_document_info['id'],
                            'fp_descr': full_document_info['fp']['descr']
                        }
                        if self.data_type == 'ip':
                            ioc_report['geo_info'] = full_document_info['geodata']
                            ioc_report['asn_info'] = full_document_info['asn']

                        results.append(ioc_report)
                else:
                    self.report({'results': 'No data found'})
                self.report({'results': results})
            else:
                self.error('Invalid data type')

To have this result:

[uchakin]

@dadokkio
Thanks a lot for highlighting this issue!

Updated code and all templates. Added rendering of each result, according to your recommendation. + added README with each points.

[uchakin]

@dadokkio Are there any others issues? Any chance this could be included in the next release?

[dadokkio]

I think there is something strange here. The code will not reach the else clause in any case.

[uchakin]

@dadokkio done
You're right. I fixed and simplified this part.

[dadokkio]

Ok, just one little note and one question.

Can you please add proxies support?

class VulnersAnalyzer(Analyzer):
    def __init__(self):
        Analyzer.__init__(self)
        self.service = self.get_param(
            "config.service", None, "Service parameter is missing"
        )
        self.api_key = self.get_param("config.key", None, "Missing vulners api key")
        proxies = {
            "https": self.get_param("config.proxy_https"),
            "http": self.get_param("config.proxy_http"),
        }
        self.vulners = vulners.Vulners(api_key=self.api_key, proxies=proxies)

The question is related to the search. Can you guarantee that query like yours will return always at most 1 result?

[uchakin]

@dadokkio
Added a proxy.
If I understand your question correctly. Searching by IOC can produce more than one result, it is provided in the template and will look like this:
https://github.com/TheHive-Project/Cortex-Analyzers/blob/b58c598d7d406044c56c088e257faca86fa127a0/analyzers/Vulners/assets/ioc_long_template.png

Thanks a lot for your code review!

[dadokkio]

My question was because at the moment if the search for one ioc returns data you pick vulnerabilities only for the first result.
it's like you are missing a for loop for all results as I suggested last time.

                if all_short_results:
                    if (
                        all_short_results[0]["type"] == "rst"
                        or "type" in all_short_results[0]
                    ):

I tried to run your type of query randomly few times and they did return just one record, so this was the reason for the question.
If it will return every time one result it's ok, if not we are missing a loop that iterate on all_short_results.

[uchakin]

The Vulners API has a limit on the number of free requests per month (1000 with possible limit exceeding)
self.vulners.search - short version with few fields
self.vulners.documentList - full version with few field

Therefore, after receiving id documents and passing all the conditions, we pass them to documentList() (instead of document() ) and get their full version .

all_short_results = self.vulners.search(f'type:rst AND iocType:{self.data_type} AND {self.data_type}:"{data}"')

                results = []

                if all_short_results:
                    if all_short_results[0]['type'] == 'rst' or 'type' in all_short_results[0]:

                        full_documents_info = self.vulners.documentList(
                            [doc_id['id'] for doc_id in all_short_results],  fields=["*"])

                        for document_results in full_documents_info:

The loop (for document_results in full_documents_info) is necessary to collect information on each result for further parsing, as you suggested last time.
For example, try the url - test.com and you will receive something like that: https://github.com/TheHive-Project/Cortex-Analyzers/blob/07dc796f47287b3358ba047f80c9395eea4b3097/analyzers/Vulners/assets/ioc_long_template.png

[dadokkio]

Ok, it was my fault! Everything works perfectly.
I made a little change when a search [eg. 4.4.4.4] return no data. Now the analyzer returns success and the wonderful message instead of throw strange errors =)

[jeromeleonard]

Hello,

Thank you for the submission. Could you please update your Readme to follow the following formatting rule:
https://thehive-project.github.io/Cortex-Analyzers/analyzers_definition/#formatting

And also, could you please update your JSON files, so that extra information can be shared in the documentation. You can follow this guide : https://thehive-project.github.io/Cortex-Analyzers/analyzers_definition/#definition-of-an-analyzer

For example:

  "registration_required": true,
  "subscription_required": true,
  "free_subscription": false,
  "service_homepage": "https://www.domaintools.com",
  "service_logo": {"path":"assets/domaintools_logo.png", "caption": "logo"},

Thanks

[uchakin]

@jeromeleonard

Hello!

Updated documentation according to your comment.
Thanks for review.