New Type of Devices "Cluster" for firewalls similar to "Virtual Chassis" for switches

[florian-sus]

Idea to discuss for Feature Request

Following closed issue "Allow ip address to be assigned to multiple devices #1405" I'd like to propose another solution to that problem.

Introduction of a new object "Cluster" under Devices.
Similar to "Virtual Chassis" where it is possible to create a Virtual Chassis (or Switch-Stack) consisting of several members, a "Cluster" also consists of two, or more member devices.
The difference to "Virtual Chassis" is that you can create "Interfaces" for a Cluster Device.

Use Case:
Realistic documentation of a firewall cluster in netbox.

• a firewall cluster consist of typically two nodes (special cases with more than two nodes possible)
• each node is a Device with a physical location, physical network interfaces, power ports, dedicated IP address for node specific management access, etc. (nothing new for netbox)
• the "Cluster" device, similar to the "Virtual Chassis" can be added to netbox with the physical firewall nodes as members (one member being the master, or active device)
• you can add the "Component" "Interface" to the "Cluster" device
  • an "Interface" of a "Cluster" Device ist always of Interface Type Virtual
  • This Interface must be assigned to an Interface of each of the member nodes

Comparison to FHRP Groups:
FHRP cannot be used in this context since the possible protocols for FHRP (HSRP, VRRP, CARP or GLBP) need one IP for each device and one HA-IP
This is not true for firewall clusters. Here you configure one high available IP address for a cluster interface. This IP is active on one node at any specific time, but is assigned to all members of a cluster.
In most cluster implementations, the HA-Cluster Interfaces are virtual Interfaces with a virtual MAC address generated for the cluster, so adding a cluster Device with its own interfaces in netbox reflects reality.

This way you can keep "the direct relationship of IP address to interface" as mentioned by @jeremystretch in #1405 and you avoid unsetting "Unique IP Space" and duplicate IP address entries for each member of clustered devices.

Comments and further ideas or additions are welcome.

Thank you and best regards, Florian

[mtinberg]

I'm not sure this is necessary, FHRP Groups do have a protocol option of "Other" which is what I use for PaloAlto HA firewalls and that works just fine. You don't need to assign additional IPs to the FHRP group, just create a group for each HA interface assign the IP to the group and assign the interfaces to the group and job done. For example there is an FHRP group for all interfaces on our HA PA except for the OOB management port and I have a naming convention in the FHRP Group Description to help keep track, eg. "HA-123-ae1-7" for HA Group 123 port ae1.7 VLAN 7 subinterface, I can assign 192.0.2.66 to the group and assign the two ae1.7 interfaces to it. — Mark Tinberg ***@***.***> Division of Information Technology-Network Services University of Wisconsin-Madison

…

________________________________ From: florian-sus ***@***.***> Sent: Friday, June 10, 2022 5:13 AM To: netbox-community/netbox ***@***.***> Cc: Subscribed ***@***.***> Subject: [netbox-community/netbox] New Type of Devices "Cluster" for firewalls similar to "Virtual Chassis" for switches (Discussion #9515) Idea to discuss for Feature Request Following closed issue "Allow ip address to be assigned to multiple devices #1405<#1405>" I'd like to propose another solution to that problem. Introduction of a new object "Cluster" under Devices. Similar to "Virtual Chassis" where it is possible to create a Virtual Chassis (or Switch-Stack) consisting of several members, a "Cluster" also consists of two, or more member devices. The difference to "Virtual Chassis" is that you can create "Interfaces" for a Cluster Device. Use Case: Realistic documentation of a firewall cluster in netbox. * a firewall cluster consist of typically two nodes (special cases with more than two nodes possible) * each node is a Device with a physical location, physical network interfaces, power ports, dedicated IP address for node specific management access, etc. (nothing new for netbox) * the "Cluster" device, similar to the "Virtual Chassis" can be added to netbox with the physical firewall nodes as members (one member being the master, or active device) * you can add the "Component" "Interface" to the "Cluster" device * an "Interface" of a "Cluster" Device ist always of Interface Type Virtual * This Interface must be assigned to an Interface of each of the member nodes Comparison to FHRP Groups: FHRP cannot be used in this context since the possible protocols for FHRP (HSRP, VRRP, CARP or GLBP) need one IP for each device and one HA-IP This is not true for firewall clusters. Here you configure one high available IP address for a cluster interface. This IP is active on one node at any specific time, but is assigned to all members of a cluster. In most cluster implementations, the HA-Cluster Interfaces are virtual Interfaces with a virtual MAC address generated for the cluster, so adding a cluster Device with its own interfaces in netbox reflects reality. This way you can keep "the direct relationship of IP address to interface" as mentioned by @jeremystretch<https://github.com/jeremystretch> in #1405<#1405> and you avoid unsetting "Unique IP Space" and duplicate IP address entries for each member of clustered devices. Comments and further ideas or additions are welcome. Thank you and best regards, Florian — Reply to this email directly, view it on GitHub<#9515>, or unsubscribe<https://github.com/notifications/unsubscribe-auth/AAS7UM4TA4CT556RB5YTW7TVOMILTANCNFSM5YNFNEKA>. You are receiving this because you are subscribed to this thread.Message ID: ***@***.***>

[florian-sus]

Hi Mark,
thank you for the hint using FHRP Groups.
I tried that and it works like you described.
The problem I have using FHRP Groups is that I don't get an overview of all IPs and networks directly connected to my clustered devices.
I can adjust the Interface view table and add FHRP Group column but it only shows "other:" which is not very useful to get an overview.

This workaround also seems quite odd to me.
One thing I like a lot using netbox is, that if you use things as intended it reflects reality very well.
Being able to create clustered network devices like firewalls is one of the few things missing.
Especially considering that you can create virtual chassis and virtualization clusters.

Florian

[mtinberg]

Oh I see what you mean, when looking at /dcim/devices/$device_id/interfaces you can have an IP Addresses column but that doesn't show addresses assigned through FHRP Groups, and if you add the FHRP Group column it just shows the protocol and group_id, eg. "HSRP: 0" and not the addresses assigned via FHRP Groups to the interface. I tried querying /ipam/ip-addresses/?interface_id=$if_id&fhrpgroup_id=$fhrp_id but that didn't work, I could query each individually but when combined returned zero results, presumably it's not making the WHERE clause a parenthetical OR so it's trying to look up records where the assigned_object_type is both dcim.interfaces AND ipam.fhrpgroup or something, which doesn't make sense.

>> IPAddress.objects.filter(Q(interface=1234))

<RestrictedQuerySet [<IPAddress: 192.0.2.2/24>]>

>> IPAddress.objects.filter(Q(fhrpgroup=1))<RestrictedQuerySet [<IPAddress: 192.0.2.1/32>]> >> IPAddress.objects.filter(Q(interface=1234), Q(fhrpgroup=1))

<RestrictedQuerySet []>

>> IPAddress.objects.filter(Q(interface=1234) | Q(fhrpgroup=1))

<RestrictedQuerySet [<IPAddress: 192.0.2.2/24>, <IPAddress: 192.0.2.1/32>]> — Mark Tinberg ***@***.***> Division of Information Technology-Network Services University of Wisconsin-Madison

…

________________________________ From: Florian Sus ***@***.***> Sent: Tuesday, June 14, 2022 1:14 AM To: netbox-community/netbox ***@***.***> Cc: Mark Tinberg ***@***.***>; Comment ***@***.***> Subject: Re: [netbox-community/netbox] New Type of Devices "Cluster" for firewalls similar to "Virtual Chassis" for switches (Discussion #9515) Hi Mark, thank you for the hint using FHRP Groups. I tried that and it works like you described. The problem I have using FHRP Groups is that I don't get an overview of all IPs and networks directly connected to my clustered devices. I can adjust the Interface view table and add FHRP Group column but it only shows "other:" which is not very useful to get an overview. This workaround also seems quite odd to me. One thing I like a lot using netbox is, that if you use things as intended it reflects reality very well. Being able to create clustered network devices like firewalls is one of the few things missing. Especially considering that you can create virtual chassis and virtualization clusters. Florian — Reply to this email directly, view it on GitHub<#9515 (reply in thread)>, or unsubscribe<https://github.com/notifications/unsubscribe-auth/AAS7UM6BPTL3TYWDCEO2423VPAPOBANCNFSM5YNFNEKA>. You are receiving this because you commented.Message ID: ***@***.***>

[dataolle]

FHRP groups seems like a bad workaround since no FHRP configuration is typically done nor used for a active-passive HA firewall cluster in this style (like juniper srx, fortigate etc). it does not reflect reality.

[mtinberg]

it does not reflect reality.

Maybe, but the more important question is "is it useful", the model doesn't have to be perfect. At least for me FHRP is a fine abstraction for failover IPs regardless of whether they use HSRP/VRRP per-interface/subnet or a whole-device (excluding management interfaces) active/passive failover of IPs. I do also use Virtualization Clusters to associate together Devices which are in an HA relationship (Nexus vPC, PaloAlto failover, application clusters, etc.). How this works for me is that I can audit the config on each failover member which references a particular IP in its config and successfully map it to the device/interface, without having intentionally duplicate IP records in IPAM. I think the remaining issue was that when adding FHRP column to the Interface table view that it shows the FHRP Name and not the IP(s) which might be more useful, to see both direct-assigned and FHRP-assigned IPs in the Interface table view. — Mark Tinberg ***@***.***> Division of Information Technology-Network Services University of Wisconsin-Madison

…

________________________________ From: dataolle ***@***.***> Sent: Wednesday, October 26, 2022 9:48 AM To: netbox-community/netbox ***@***.***> Cc: Mark Tinberg ***@***.***>; Comment ***@***.***> Subject: Re: [netbox-community/netbox] New Type of Devices "Cluster" for firewalls similar to "Virtual Chassis" for switches (Discussion #9515) FHRP groups seems like a bad workaround since no FHRP configuration is typically done nor used for a active-passive HA firewall cluster in this style (like juniper srx, fortigate etc). it does not reflect reality. — Reply to this email directly, view it on GitHub<#9515 (reply in thread)>, or unsubscribe<https://github.com/notifications/unsubscribe-auth/AAS7UM76TLALG34QL4J7IHLWFFACPANCNFSM5YNFNEKA>. You are receiving this because you commented.Message ID: ***@***.***>

[dataolle]

"is it useful".
Point taken, but i would like something stricter than just fhrp / ip address assignments.. I guess i can do a workaround using validations to prevent a virtual interface for example on only one of the devices, which is not / should not be possible in a ha setup.
Maybe just having a filed marking a device as standby node and pointing to a "active" node and do all ip / interface config just on the active node might be a workaround but still feels really ugly.

[mtinberg]

I hear you. Anything you make to help with this issue, either a Netbox Script to provision IPs/Interfaces which pushes them down to both HA members, or a Report to find discrepancies, or validation rules, might be helpful to others if you can share them or maybe add them to the https://github.com/netbox-community/reports repo. If a bunch of people have the same problem and solve it in the same way, at the worst case they can share tools/process and in the best case the feature becomes supported by core Netbox with a clear model of how it should work. — Mark Tinberg ***@***.***> Division of Information Technology-Network Services University of Wisconsin-Madison

…

________________________________ From: dataolle ***@***.***> Sent: Friday, October 28, 2022 1:51 AM To: netbox-community/netbox ***@***.***> Cc: Mark Tinberg ***@***.***>; Comment ***@***.***> Subject: Re: [netbox-community/netbox] New Type of Devices "Cluster" for firewalls similar to "Virtual Chassis" for switches (Discussion #9515) "is it useful". Point taken, but i would like something stricter than just fhrp / ip address assignments.. I guess i can do a workaround using validations to prevent a virtual interface for example on only one of the devices, which is not / should not be possible in a ha setup. Maybe just having a filed marking a device as standby node and pointing to a "active" node and do all ip / interface config just on the active node might be a workaround but still feels really ugly. — Reply to this email directly, view it on GitHub<#9515 (reply in thread)>, or unsubscribe<https://github.com/notifications/unsubscribe-auth/AAS7UMZ7T67LP2IWECGTUM3WFNZXHANCNFSM5YNFNEKA>. You are receiving this because you commented.Message ID: ***@***.***>

[quequiel]

Another drawback of modeling clusters with FHRP is that you can´t set an FHRP IP as primary IP for the device. So no easy way of seeing addresses in use by a device, no easy way to retrieve device primary address for integrations (eg. backup, monitoring, etc.)

[mtinberg]

That must be a different kind of cluster because when I've done it I usually have a standalone management IP for each cluster member that is not part of an HA VIP, so that each cluster member can be managed and monitored independently, and for any HA sync protocol to communicate with other cluster members. If the cluster is really that integrated that it presents as one device, maybe Virtual Chassis is a better abstraction to tie the members together, like a switch stack. — Mark Tinberg ***@***.***> Division of Information Technology-Network Services University of Wisconsin-Madison

…

________________________________ From: quequiel ***@***.***> Sent: Wednesday, November 9, 2022 8:01 AM To: netbox-community/netbox ***@***.***> Cc: Mark Tinberg ***@***.***>; Comment ***@***.***> Subject: Re: [netbox-community/netbox] New Type of Devices "Cluster" for firewalls similar to "Virtual Chassis" for switches (Discussion #9515) Another drawback of modeling clusters with FHRP is that you can´t set an FHRP IP as primary IP for the device. So no easy way of seeing addresses in use by a device, no easy way to retrieve device primary address for integrations (eg. backup, monitoring, etc.) — Reply to this email directly, view it on GitHub<#9515 (comment)>, or unsubscribe<https://github.com/notifications/unsubscribe-auth/AAS7UM25G6BLNTRYJQWER3LWHOVDZANCNFSM5YNFNEKA>. You are receiving this because you commented.Message ID: ***@***.***>

[quequiel]

Fortigate clusters can have OOB management but it is optional and not always used specially if you don´t have an OOB network. All monitoring / management (of all nodes) can be done through the active node.

Virtual chassis is not well suited as interfaces of each member are "independent" which is not the case of a cluster. For example Port 1 of all cluster members will always have the same configuration (IP, 802.1Q, etc.) while on a switch stack, all interfaces can (and will probably) have different configuration.
On my opinion, VC is a better approach than FHRP but does not represent the truth which is the main idea of netbox. Maybe an extension could be made to the VC model to support this cluster scenarios.

[shepherdjay]

Personally I think Virtual Chassis should just be folded into this request. While I agree with you that virtual chassis are not well suited because member interfaces are independent I think that's a strong argument against virtual chassis in general. Even in a switchstack the config is not independent. The Interface numbering is not independent... its entirely dependent on a members position in a virtual switch stack.

Would really like to see Virtual Chassis become first class devices in Netbox. They and clusters are identical to stand alone members in nearly every way other than the physical space they take up which is inherited from the members themselves. As it is stack members are very cumbersome to use. Their interfaces need to be renamed when they are created or change position. You HAVE to set a master (needs a rename) otherwise you can't create virtual interfaces properly for the virtual chassis like assigning ips.

So I would actually like to see #9515 (comment) suggestion implemented for all devices.

• All devices should have some boolean field to identify whether they standalone or clustered.
• If clustered the type of cluster could control whether interfaces are "mirrored" or "independent"... ie a "Stacked" or "Active/Active" cluster type could define mirrored interfaces that are renumbered when viewed from the cluster device. Whereas a "Active/Passive" or "Mirrored" cluster type could set all members equally when viewed from the cluster device.

[mtinberg]

I don't know Fortigate but I'm still hung up on how the Active node communicates HA state and config with the non-active nodes unless they all have their own IPs, or is there a dedicated physical link between an active and a single standby node that talks whatever protocol to keep the state in sync, as it seems if there is a unique IP on each node, whether you use the OOB management or not, that would be the primary_ip for Netbox purposes. I guess only the Active device could have a primary_ip if the others are not addressable and don't have a relationship to the IP Address model. While Netbox is a source of truth or intent for the network config, you can use the model in whatever way is useful for your config management/documentation, it doesn't have to be perfectly true and accurate to every facet of the device as long as it facilitates the right config management outcome. You can use abstractions and conventions to solve problems, or just skip modeling details that aren't critical, eg. if you wanted to try using VCs for the Fortigates instead of FHRP groups, maybe in your config management you only care about the primary VC Device record and its interfaces, and you don't track or model the interfaces on other members, or maybe only model physical interfaces for cabling purposes with none of the config detail, or write a Script to sync the config detail from the Active node to the others as a nightly cron job so that you only need to make changes in one place. — Mark Tinberg ***@***.***> Division of Information Technology-Network Services University of Wisconsin-Madison

…

________________________________ From: quequiel ***@***.***> Sent: Wednesday, November 9, 2022 12:04 PM To: netbox-community/netbox ***@***.***> Cc: Mark Tinberg ***@***.***>; Comment ***@***.***> Subject: Re: [netbox-community/netbox] New Type of Devices "Cluster" for firewalls similar to "Virtual Chassis" for switches (Discussion #9515) Fortigate clusters can have OOB management but it is optional and not always used specially if you don´t have an OOB network. All monitoring / management (of all nodes) can be done through the active node. Virtual chassis is not well suited as interfaces of each member are "independent" which is not the case of a cluster. For example Port 1 of all cluster members will always have the same configuration (IP, 802.1Q, etc.) while on a switch stack, all interfaces can (and will probably) have different configuration. On my opinion, VC is a better approach than FHRP but does not represent the truth which is the main idea of netbox. Maybe an extension could be made to the VC model to support this cluster scenarios. — Reply to this email directly, view it on GitHub<#9515 (reply in thread)>, or unsubscribe<https://github.com/notifications/unsubscribe-auth/AAS7UM5SJ6OR6V4OG6ACACTWHPRTPANCNFSM5YNFNEKA>. You are receiving this because you commented.Message ID: ***@***.***>

[quequiel]

Cluster heartbeat and session / config sync is done over dedicated (ethernet) interfaces but you don´t configure IP addresses on that links (Internally it uses APIPA addressing) so no, there is no unique IP on each node unless you use OOB.

When clustered all nodes share the same set of MAC and IP addresses. I think the only way to use a script to sync the config would be to disable unique address space enforcement which is definitely a bad idea.

[mtinberg]

Ok, cool. The other device types I have still have in-band management IPs and some level of local config even when they have dedicated HA sync interfaces and the majority of the config is synched or inherited, which is why I was confused, I guess Fortigate is just built different. Anyway, because the HA shares so much system state it may make sense to either pretend these are a stack (Virtual Chassis) or just ignore most of the properties of the cluster members except serial number, location, and interfaces if you are tracking cabling, otherwise your other management/monitoring/config templating systems really do only care about the IPs, interface properties, custom_fields, etc. being documented on the active node as it's the only one that's managed. Does that seem workable and make sense? — Mark Tinberg ***@***.***> Division of Information Technology-Network Services University of Wisconsin-Madison

…

________________________________ From: quequiel ***@***.***> Sent: Thursday, November 10, 2022 9:59 AM To: netbox-community/netbox ***@***.***> Cc: Mark Tinberg ***@***.***>; Comment ***@***.***> Subject: Re: [netbox-community/netbox] New Type of Devices "Cluster" for firewalls similar to "Virtual Chassis" for switches (Discussion #9515) Cluster heartbeat and session / config sync is done over dedicated (ethernet) interfaces but you don´t configure IP addresses on that links (Internally it uses APIPA addressing) so no, there is no unique IP on each node unless you use OOB. When clustered all nodes share the same set of MAC and IP addresses. I think the only way to use a script to sync the config would be to disable unique address space enforcement which is definitely a bad idea. — Reply to this email directly, view it on GitHub<#9515 (reply in thread)>, or unsubscribe<https://github.com/notifications/unsubscribe-auth/AAS7UMZJ3W7PTVAPTSJT4M3WHULXZANCNFSM5YNFNEKA>. You are receiving this because you commented.Message ID: ***@***.***>

[quequiel]

Mark,

Of course it can be modeled that way, it´s a workaround and is the best way with what we have available. I guess the original idea of this post was to discuss if anyone (besides @florian-sus and me) thinks this would be a nice / necessary feature request.

Regards, Alejandro

[florian-sus]

Hi Alejandro,

you are completely right about my intention for this post.
I think its a necessary feature. We've had a lot bad experiences customizing tools and implementing workarounds in the past.
So we try to use all tools and their features as they are intended at least as long as possible.
Otherwise the problems will occur when updating to newer version, which maybe do not support your customization oder workarounds.

In case of netbox you also need to be very clear on how to implement a firewall cluster using the workarounds or things will get messed up.
Adding a firewall cluster feature in netbox will give less room on how things should be done.

[oceans1908]

I would like to see a cluster of devices, especially for firewalls which are Virtual machines in my case.
Reason I wold like the option is not IP addresses, but BGP configuration (prefix-lists which we set now with tags, and ACL rules) Now we have to sync two devices in Netbox with duplicated configuration. 1 Firewall cluster shares 1 config but now I have to administer it twice in Netbox and have to check if both are equal. A cluster device in Netbox will save me that, as I can put the config on the cluster device.

[candlerb]

Maybe "config contexts" will help you.

[dasfliege]

+1 for having a cluster object that is configurable as a normal device object. Pretty much every firewall Active-Passive cluster would benefit from that. It's not only limited to firewalls but also f.e. storage devices that can be configured as clusters with a single management address.

The most proper implementation may would be, that "Clusters" are not limited to servers running VMs but also to other types of hardware like firewalls or storages. But that would require some more conceptional work, as the behavior is quite different.

A more practical implementation would look like the following:

• A checkbox in the the create device dialog, which allows to mark a device as a cluster object. If this checkbox is set, it allows the device to have "child" devices.
• In the device settings of the "child" devices, each interface should be able to be specified as cluster interface. If it is assigned to the cluster, the configuration of the corresponding interface from the parent device (in this case the cluster object) is beeing shown.
• If a interface is not specified as cluster interface, then it is configurable individually. F.e. OOBM interfaces or HA-Link interfaces, if they need a specific IP (f.e. Sophos / SFOS)

[patrickpreuss]

+1 for this.

[Black-FX]

Another +1

[goteamkor]

and another. who's going to make the FR?

[felix-pjm]

+1

[rhysperry111]

We've just deployed new infra which involves a few HA active-passive firewall devices and this is pretty much exactly what we want. The pairs of interfaces in the HA should effectively be treated as a single one logically, apart from the exceptions such as management and HA sync.

[goteamkor]

I see a lot of +1's, but the OP should turn this into a FR.

[jonasnieberle]

+1