Raw upload with Scan OOIS #3169

noamblitz · 2024-07-01T15:16:11Z

Changes

This PR has three main changes:

A placeholder Scan OOI ensures that OOIs from a raw upload are eligible for garbage collection.
Raw upload now has both an input OOI and a valid time, making exchanging raw files from ALL boefjes possible!
Moving serialize() to Base OOI class, making it accessible in all services.

Closes: #3251

Demo

Please add some proof in the form of screenshots or screen recordings to show (off) new functionality, if there are interesting new features for end-users.

QA notes

Please test both the raw upload with different valid times, and the garbage collection of Scan OOIs.

Code Checklist

All the commits in this PR are properly PGP-signed and verified.
This PR only contains functionality relevant to the issue.
I have written unit tests for the changes or fixes I made.
I have checked the documentation and made changes where necessary.
I have performed a self-review of my code and refactored it to the best of my abilities.

Tickets have been created for newly discovered issues.
For any non-trivial functionality, I have added integration and/or end-to-end tests.
I have informed others of any required .env changes files if required and changed the .env-dist accordingly.
I have included comments in the code to elaborate on what is not self-evident from the code itself, including references to issues and discussions online, or implicit behavior of an interface.

Checklist for code reviewers:

Copy-paste the checklist from the docs/source/templates folder into your comment.

Checklist for QA:

Copy-paste the checklist from the docs/source/templates folder into your comment.

rocky/rocky/views/upload_raw.py

rocky/tools/forms/upload_raw.py

underdarknl · 2024-07-03T07:58:16Z

related to #3050

# Conflicts: # octopoes/octopoes/models/types.py # rocky/rocky/bytes_client.py

ammar92

Nice work; just a few tiny remarks. Also, please do add some context to the PR next time.

octopoes/octopoes/models/__init__.py

boefjes/boefjes/api.py

rocky/tools/forms/upload_raw.py

noamblitz · 2024-07-03T19:14:42Z

Nice work; just a few tiny remarks. Also, please do add some context to the PR next time.

Heheh, sorry, finished the PR right at the end of the day and didn't have time for context yet. Tomorrow first thing promise ❤️

Co-authored-by: ammar92 <[email protected]>

ammar92 · 2024-07-03T19:32:34Z

Nice work; just a few tiny remarks. Also, please do add some context to the PR next time.

Heheh, sorry, finished the PR right at the end of the day and didn't have time for context yet. Tomorrow first thing promise ❤️

No worries! I noticed thanks to Jan that this was related to Burpsuite normalizer

rocky/rocky/views/upload_raw.py

underdarknl · 2024-07-04T11:29:25Z

rocky/rocky/views/upload_raw.py

+        options = [(o.primary_key, o.get_ooi_type()) for o in objects]
+
+        return options


We could yield these options one at a time, to avoid building up two giant lists in memory when loading many objects.

rocky/rocky/views/upload_raw.py

rocky/tools/forms/upload_raw.py

Co-authored-by: Jan Klopper <[email protected]>

ammar92

Looks good to me

octopoes/octopoes/models/ooi/scans.py

rocky/tools/forms/upload_raw.py

Donnype · 2024-07-30T11:59:36Z

Good stuff! There is a small issue with choosing the input, see the details and suggestions below.

Checklist for QA:

I have checked out this branch, and successfully ran a fresh make reset.
I confirmed that there are no unintended functional regressions in this branch:
- I have managed to pass the onboarding flow
- Objects and Findings are created properly
- Tasks are created and completed properly
I confirmed that the PR's advertised feature or hotfix works as intended.
I checked the logs for errors and/or warnings and made issues where necessary

What works:

I can manually add a Scan OOI
I can add a Raw File with an input OOI that I can fetch from the Bytes API
I cannot add OOIs when they were not created yet in the past, i.e. valid_time set too early
I can add a Raw File with a valid time in the past

UPDATE:

I can create a RawFile with the Scan OOI as input by copy pasting (or typing) the primary key myself

~~### What doesn't work:~~

~~- I cannot create a RawFile with the Scan OOI as input because it doesn't show up in the dropdown.~~
~~- I also cannot write the primary key myself (also not url encoded):~~

~~(It is going to be hard to do this dynamically somehow though..)~~

This wasn't working because of a valid time that was too old.

This suggestion is now just a nice-to-have:
Perhaps to resolve this we could add a button on the OOI detail page that says "Attach RawFile" and pre-fills the input OOI field? That could make it easier to add a file for the same OOI later. (If we could show all attached raw files for an OOI on the object detail page as well in the end, it would be even easier.)

Bug or feature?:

* main: Basic audit trails via logging (#3317) Raw upload with Scan OOIS (#3169) Fix Garbage collection and disappearing ports issue (#3214) Updated `Django` and `opentelemetry` packages (#3324)

* feature/mula/refactor-queue: Fix poetry Updates according to code review Basic audit trails via logging (#3317) Raw upload with Scan OOIS (#3169) Fix Garbage collection and disappearing ports issue (#3214) Formatting Formatting Fix formatting Updated `Django` and `opentelemetry` packages (#3324) Restructure scheduler development scripts (#3293) Change report flow to POST requests (#3174)

poc for raw upload with scan oois

4ebea28

noamblitz mentioned this pull request Jul 1, 2024

Burpsuite normalizer cannot parse xml file #3050

Closed

underdarknl reviewed Jul 2, 2024

View reviewed changes

rocky/rocky/views/upload_raw.py Outdated Show resolved Hide resolved

underdarknl reviewed Jul 2, 2024

View reviewed changes

rocky/rocky/views/upload_raw.py Outdated Show resolved Hide resolved

underdarknl reviewed Jul 2, 2024

View reviewed changes

rocky/tools/forms/upload_raw.py Outdated Show resolved Hide resolved

noamblitz added 11 commits July 3, 2024 12:51

explanation of scan oois

87bd939

remove emphasized

dd7cdb4

remove unneeded octopoes call

dfbc26d

upload raw in past

fac0491

move serialize ooi to base class

f38863c

better type hinting

c4cf101

Merge branch 'main' into scan-oois

ea2f6fa

# Conflicts: # octopoes/octopoes/models/types.py # rocky/rocky/bytes_client.py

cast to set

ace10d8

add report to types

e5afdcd

missing translations

d810505

fix tests

9571b47

noamblitz marked this pull request as ready for review July 3, 2024 14:40

noamblitz requested a review from a team as a code owner July 3, 2024 14:40

ammar92 reviewed Jul 3, 2024

View reviewed changes

octopoes/octopoes/models/__init__.py Show resolved Hide resolved

boefjes/boefjes/api.py Outdated Show resolved Hide resolved

rocky/tools/forms/upload_raw.py Outdated Show resolved Hide resolved

Update boefjes/boefjes/api.py

76577c0

Co-authored-by: ammar92 <[email protected]>

underdarknl and others added 2 commits July 4, 2024 10:25

Merge branch 'main' into scan-oois

20c7f5b

call super first

8646cce

underdarknl reviewed Jul 4, 2024

View reviewed changes

rocky/rocky/views/upload_raw.py Show resolved Hide resolved

underdarknl reviewed Jul 4, 2024

View reviewed changes

underdarknl reviewed Jul 5, 2024

View reviewed changes

rocky/rocky/views/upload_raw.py Show resolved Hide resolved

underdarknl reviewed Jul 5, 2024

View reviewed changes

rocky/rocky/views/upload_raw.py Show resolved Hide resolved

underdarknl reviewed Jul 5, 2024

View reviewed changes

rocky/tools/forms/upload_raw.py Outdated Show resolved Hide resolved

underdarknl added this to the OpenKAT v1.17 milestone Jul 8, 2024

noamblitz and others added 7 commits July 9, 2024 15:25

Update rocky/rocky/views/upload_raw.py

0fbe261

Co-authored-by: Jan Klopper <[email protected]>

Update rocky/tools/forms/upload_raw.py

07e0b33

Co-authored-by: Jan Klopper <[email protected]>

add initial datetime from url

0b4fba8

merge main

13e9c84

make lang

baf7f78

Merge branch 'main' into scan-oois

04ee6a1

Merge branch 'main' into scan-oois

8eb97d4

ammar92 previously approved these changes Jul 23, 2024

View reviewed changes

Merge branch 'main' into scan-oois

2f88c03

Donnype reviewed Jul 30, 2024

View reviewed changes

octopoes/octopoes/models/ooi/scans.py Outdated Show resolved Hide resolved

Donnype reviewed Jul 30, 2024

View reviewed changes

rocky/tools/forms/upload_raw.py Show resolved Hide resolved

Update django.pot

8fcb8ea

noamblitz dismissed ammar92’s stale review via 8fcb8ea July 31, 2024 09:26

noamblitz and others added 5 commits August 6, 2024 14:23

Rename to externalscan

b329896

Update django.pot

8b55ded

Merge branch 'main' into scan-oois

e484212

Merge branch 'main' into scan-oois

f773bef

Merge branch 'main' into scan-oois

a837b2b

underdarknl approved these changes Aug 8, 2024

View reviewed changes

underdarknl merged commit a8feebe into main Aug 8, 2024
21 checks passed

underdarknl deleted the scan-oois branch August 8, 2024 09:01

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Raw upload with Scan OOIS #3169

Raw upload with Scan OOIS #3169

noamblitz commented Jul 1, 2024 •

edited

Loading

underdarknl commented Jul 3, 2024

ammar92 left a comment

noamblitz commented Jul 3, 2024

ammar92 commented Jul 3, 2024

underdarknl Jul 4, 2024

ammar92 left a comment

Donnype commented Jul 30, 2024 •

edited

Loading

		options = [(o.primary_key, o.get_ooi_type()) for o in objects]

		return options

Raw upload with Scan OOIS #3169

Raw upload with Scan OOIS #3169

Conversation

noamblitz commented Jul 1, 2024 • edited Loading

Changes

Demo

QA notes

Code Checklist

Checklist for code reviewers:

Checklist for QA:

underdarknl commented Jul 3, 2024

ammar92 left a comment

Choose a reason for hiding this comment

noamblitz commented Jul 3, 2024

ammar92 commented Jul 3, 2024

underdarknl Jul 4, 2024

Choose a reason for hiding this comment

ammar92 left a comment

Choose a reason for hiding this comment

Donnype commented Jul 30, 2024 • edited Loading

Checklist for QA:

What works:

Bug or feature?:

noamblitz commented Jul 1, 2024 •

edited

Loading

Donnype commented Jul 30, 2024 •

edited

Loading