Secure SD WAN Workshop 6 - 4 - 4 [PDF]

Zitiervorschau

Managed Secure SD-WAN 6.4.4 Lab Guide

Managed Secure SD-WAN 6.4.4 March 3, 2021 Revision 1.1 Copyright© 2021 Fortinet, Inc. All rights reserved. Fortinet®, FortiGate®, and FortiGuard® are registered trademarks of Fortinet, Inc., and other Fortinet names herein may also be trademarks of Fortinet. This document contains confidential material proprietary to Fortinet, Inc.  This document and information and ideas herein may not be disclosed, copied, reproduced or distributed to anyone outside Fortinet, Inc. without prior written consent of Fortinet, Inc. Fortinet reserves the right to change, modify, transfer, or otherwise revise this publication without notice, and the most current version of the publication shall be applicable.

INTERNAL USE ONLY

Managed Secure SD-WAN 6.4.4

Maintainer(s): Dmitry Perets Contributors: Miguel Munoz, Claudio Salmin (CSE - Telco & MSP) Changelog: Revision

Date

Change Description

1.0

2021-02-07

SME Event Q1 2021

1.1

2021-03-03

Feedback and Postman UI update

INTERNAL USE ONLY

3

Managed Secure SD-WAN 6.4.4

1. Lab Introduction This document describes Managed Secure SD-WAN hands-on lab. The lab can be done either during organized workshops or in a self-paced manner. Although we will be doing our best to guide you through the caveats, this lab guide should NOT be seen as a “step-by-step guide”. It is assumed that you have a prior knowledge about FortiManager, FortiGate and Fortinet Secure SD-WAN Solution. The main objective of this lab is to demonstrate several advanced SD-WAN topologies and go through some of the new SD-WAN features implemented in FortiOS / FortiManager 6.4.x. In particular, we are going to build a generic Multi-Regional SD-WAN topology that features: • Dual-Hub and Single-Hub Regions • ADVPN within and across the Regions • Direct and Remote Internet Access options We are going to cover the following functionality: • IKE Config Mode • ADVPN Shortcut Monitoring • SD-WAN Zones • Cross-Overlay Connectivity • End-to-End Segmentation using VRFs on Hubs and Spokes • Zero-day Malware Outbreak Prevention with FortiSandbox • Zero-Touch Provisioning for Edge devices All the configuration will be done through FortiManager (FMG), in part - manually and in part - using REST API calls. Additionally, we will show how to add FortiPortal to our solution. Note that we are not going into details of the FortiPortal functionality, this is out of scope for this lab.

1.1. Secure SD-WAN Design When we design an SD-WAN solution, we can usually divide the whole process into the following five pillars:

INTERNAL USE ONLY

4

Managed Secure SD-WAN 6.4.4

Keep this model in mind, when you work on this lab. No matter what tools you use to generate the configuration and what abstraction these tools present to you, in the end of the day, the actual configuration of the devices will follow this Five Pillar approach!

1.2. Topology Following is the complete underlay topology provided with this lab:

As can be seen, we have 6 FGT devices (3 Edge devices + 3 Hubs). Each FGT device has 3 WAN links (logically marked as 2 Internet + 1 MPLS). INTERNAL USE ONLY

5

Managed Secure SD-WAN 6.4.4

In this lab we are going to use only two WAN links: • port1 - for Internet connection • port4 - for a private MPLS connection We are going to build the following Dual-Region SD-WAN solution:

The underlay comes pre-configured in this lab. The following table summarizes this configuration:

dc1_fgt

dc2_fgt

dc3_fgt

branch1_fgt

MPLS

INET

P4: 172.16.1.5/24

P1: 100.64.1.1/29

GW: 172.16.1.6

GW: 100.64.1.2

P4: 172.16.2.5/24

P1: 100.64.2.1/29

GW: 172.16.2.6

GW: 100.64.2.2

P4: 172.16.3.5/24

P1: 100.64.3.1/29

GW: 172.16.3.6

GW: 100.64.3.2

P4: 172.16.0.1/29

P1: 192.2.0.1/29

vl_lan: 10.0.1.1/24

GW: 172.16.0.2

GW: 192.2.0.2

vl_lan_ts: 10.0.101.1/24

INTERNAL USE ONLY

LAN P5: 10.1.0.1/24

P5: 10.2.0.1/24

P5: 10.3.0.1/24

6

Managed Secure SD-WAN 6.4.4

branch2_fgt

branch3_fgt

MPLS

INET

LAN

P4: 172.16.0.9/29

P1: 203.0.113.1/29

vl_lan: 10.0.2.1/24

GW: 172.16.0.10

GW: 203.0.113.2

vl_lan_ts: 10.0.102.1/24

P4: 172.16.0.17/29

P1: 203.0.115.1/29

GW: 172.16.0.18

GW: 203.0.115.2

port5: 10.0.3.1/24

1.3. Host Credentials Host

Login

Password

FortiPOC

guest

cseguest

FMG

admin

fortinet

All FGTs

admin

fortinet

All hosts

root

fortinet

FortiPortal

spuser

test123

1.4. Solution Planning To build a good generic design, it is very important to come up with good naming conventions and proper address planning. And it is not always intuitively clear in advance what conventions you should adopt. For example, when you invent an addressing scheme for your IPSEC overlay subnets, do you need them easily summarizable by region? Or by Hub? Or by type of overlay? Making wrong choice might be difficult to fix later, when you already have your scripts and API calls written. Let us propose you some ideas. We will often use the following “variables” throughout this lab: Variable

Description

Value

ul-id

Underlay ID

1 for Internet, 2 for MPLS

branch-id

Branch ID

1-3

dc-id

DC ID

1-3

INTERNAL USE ONLY

7

Managed Secure SD-WAN 6.4.4

Remember the following rules very well - you will understand their importance soon: • Overlay subnets must be easily summarizable by the type of overlay (e.g. “ ALL overlays over Internet, including all Hubs and regions”) • All corporate subnets (all internal networks, connectivity subnets, overlay subnets and so on) must be summarizable together, to easily separate them from external (Internet) destinations • Try making your addressing “human-readable”: by looking at the IP, you can quickly tell to which site and segment it belongs (and in case of an overlay IP - to which Hub this overlay belongs) In this lab, we will use the following summaries and conventions: Subnet

Description

10.0.0.0/8

All corporate subnets

10.201.0.0/16

Overlay subnets over Internet

10.202.0.0/16

Overlay subnets over MPLS

10..0.0/24

DC LAN subnets behind Hubs

10.0..0/24

Branch LAN subnets behind Edge devices

We will break this down further in the respective chapters.

INTERNAL USE ONLY

8

Managed Secure SD-WAN 6.4.4

1.5. Verify Your Environment 1.5.1. Licensing If you are using one of the lab environments managed by Fortinet (such as FortiDemo or VirtualTechExpo), this should not be a concern. But if you are using your own FortiPOC (FPOC) environment, make sure that all the FGTs and the FMG have a valid license ( get system status ). While we provide a complete POC definition file for this lab, we do not provide any licenses! It is up to your FPOC environment to either have a valid license server available or to have manually imported valid licenses. Keep in mind that evaluation licenses are not sufficient, since they do not allow configuring some of the parameters used in this lab!

1.5.2. Initial Configuration The underlay is preconfigured on all the FGTs. As a sanity check (or if you suspect issues), you can verify that all the client hosts (branch*_client, dc*_host) can ping their default gateways (they should all use “eth1” interface for that). Additionally, some of the devices might come powered off by default in the POC. If you cannot access a device, use FPOC dashboard to ensure that it is powered on.

1.5.3. WAN Controller WAN Controller (“wan_simulator”) is an important tool that you will use in this lab. Connect to it via HTTP.

INTERNAL USE ONLY

9

Managed Secure SD-WAN 6.4.4

• First, it gives you granular control over all the WAN links of all the FGT devices. You can manipulate link latency, bring links down and more.

In this lab, we are going to use only ISP1 (port1) and MPLS (port4) connections on each device.

• Second, if you scroll all the way down, you will find the buttons to start/stop Internet traffic generators behind each Branch.

1.6. Setup Your Postman Client In this lab we are going to setup some configuration using JSON RPC API exposed by FMG. Postman is one of the best tools that can be used to deal and interact with HTTP-based API. We have prepared a Postman collection that you will use in this lab. Please note that it is also possible to do all the steps manually from FMG GUI. However the student is encouraged to do this using API. • You should have Postman already installed on your laptop. If not, please go to https:// www.getpostman.com/downloads/ , download and install it.

Important: Please go to Postman Preferences and disable ‘SSL Certificate Verification’.

INTERNAL USE ONLY

10

Managed Secure SD-WAN 6.4.4

• Next step is to import the Postman collection. Click on “Import” in Postman GUI, navigate to “Import From Link” and paste the link to this collection in JSON format.

INTERNAL USE ONLY

11

Managed Secure SD-WAN 6.4.4 You can also browse our online repository on GitHub. Click on a tag icon of the corresponding release to navigate to the source code:

The Postman collection that you have just imported is in file “Managed_SDWAN_6_4_4.postman.json”.

• Now you must create a new environment to define some basic variables that will allow Postman to connect and interact with our FMG. Click on the “eye” icon (upper right corner on macOS), then click “Add” to create a new environment using a name of your choice and including these variables (case-sensitive!): ◦ ip: Use the FMG ip and port given by your FortiPoC (hover the mouse over the HTTPS entry in the FortiPOC menu).

◦ username: admin ◦ password: fortinet ◦ adom: DEMO

INTERNAL USE ONLY

12

Managed Secure SD-WAN 6.4.4 Important note: We also have to enable RPC in FMG. In our lab this has been done for you already. You can check in FMG GUI, by navigating to

System Settings -> Administrators , clicking on

admin user and making sure that the value of JSON API Access is set to “Read-Write”.

After saving the new environment, do not forget to select it in the drop-down list! Finally, check that everything is fine: open Postman collection, click on Login request and then click “Send”. You should receive an answer with “Status: 200 OK”, and in the reply body you should see a JSON response containing code: 0 and message: OK :

INTERNAL USE ONLY

13

Managed Secure SD-WAN 6.4.4

INTERNAL USE ONLY

14

Managed Secure SD-WAN 6.4.4

2. West Region (Dual-Hub) In this chapter we are going to build our West Region topology, with two Hubs and two Edge devices (Spokes). The Edge devices will connect to both Hubs and use them in Active/Passive mode. We are going to start by building the overlay, then we add dynamic routing, then finally we configure SD-WAN rules and test the results.

Before proceeding, please read carefully the Solution Planning section!

2.1. Prepare FortiManager We will start by taking some basic steps in the FortiManager (FMG). Since these steps are not the main focus of our lab, they will be all automated via Postman. To make it even simpler, we have grouped all the relevant requests into one folder, and right now you will learn how to run them all at once. In Postman collection, select the Base folder inside

West Region - Dual-Hub . Then click “Run”

button.

This will open a “Runner” tab, with all the requests from the “Base” folder pre-selected.

INTERNAL USE ONLY

15

Managed Secure SD-WAN 6.4.4 Before continuing, please double-check that you have actually selected the “Base” folder, so that the list of the selected requests is exactly as shown on the above screenshot! If you mistakenly select another folder or the entire collection, you risk completing this lab way too fast, since it will simply configure the entire environment in batch!

Double-check that your environment is still selected in the drop-down list and then simply click the big blue “Run Managed SD-WAN 6.4.4” button.

Once you click the button, Postman will execute all the selected requests, one by one. We have defined success criteria for each request, so Postman will report whether each request fails or succeeds. We expect to see all of them succeeding, of course.

For most requests, success criteria is simply to return “200 OK” status. But some are more sophisticated. For example, when adding devices to FMG, Postman will poll FMG task until it completes and will check the task result.

INTERNAL USE ONLY

16

Managed Secure SD-WAN 6.4.4 Learning Postman is outside the scope of this lab, but we have put some more information in the Appendix, in case you are interested!

So what did all these requests do? Quite a few things actually: • Added the first four FGTs participating in this lab to the FMG and put them into two device groups - “Hubs-West” and “Edge-West”:

• Created some handy meta fields in FMG that we will be using in our CLI templates later:

• Set the values for these meta fields for all the added FGTs:

INTERNAL USE ONLY

17

Managed Secure SD-WAN 6.4.4

• Added normalized interfaces on FMG (known as “dynamic interfaces” prior to FMG 6.4.1), mapping them to the relevant FGT interfaces:

• Created policy packages that will be eventually used for this region:

INTERNAL USE ONLY

18

Managed Secure SD-WAN 6.4.4

• Added CLI Script that configures FGT to send logs to FortiAnalyzer (FAZ) and executed this script on all the FGTs (note that in this lab we are using FAZ functionality running on FMG itself) • Added another CLI Script that configures a special loopback interface on all Hubs, that we will use as our health-check server Each of these steps is, of course, very important, so if you are not sure about any of them, please spend some time on reviewing the respective Postman requests. Also ask yourself a question whether you would be able to make all these changes manually, if you would not have the luxury of having our Postman collection. Throughout this lab we will be referring you to some of these steps again, so you will see exactly at which point they are important.

2.2. Overlay 2.2.1. Define VPN Communities FMG (and in particular VPN Manager) uses the term “VPN Community” for a set of interconnected gateways. Our FGTs will be interconnected over two separate underlay networks, so we will create two separate communities: one over the Internet (W_INET), another one over MPLS (W_MPLS). Here, “W” denotes “West Region”, as you might have guessed. Note that there might be more than one way to build the same topology in FMG. For instance, we could also create one community per Hub per underlay (getting four communities in total). But, as you will see here, two will be enough, because FMG supports having two Hubs within the same community.

The next few tables list some of the parameters that we will configure. Any parameters that do not appear in the tables can be left with their default values. And if there is a parameter listed in the table that you cannot find on FMG, make sure you check under “Advanced Options”!

INTERNAL USE ONLY

19

Managed Secure SD-WAN 6.4.4

Using our addressing scheme, we will assign the following overlay subnets (and .1 will be configured on the respective Hub): Subnet

Community

Hub

10.201.1.0/24

W_INET

dc1_fgt

10.201.2.0/24

W_INET

dc2_fgt

10.202.1.0/24

W_MPLS

dc1_fgt

10.202.2.0/24

W_MPLS

dc2_fgt

The following parameters will be used for each community: Parameter

Value

VPN Topology

Dial-Up

Authentication

Pre-shared Key (auto-generated)

IKE Version

2

IKE SA Proposals

AES256/SHA256, AES256GCM/PRFSHA384

IPSEC SA Proposals

AES256/SHA256, AES256GCM

VPN Zone

OFF

Dead Peer Detection

On Idle

dpd-retrycount

2

dpd-retryinterval

10

While this is surely out of scope in this lab, consider using certificate-based authentication, when you deal with a real-world project! Remember that FMG can act as your CA or you can use FortiAuthenticator for a more sophisticated deployment. Furthermore, it is now even possible to run FortiAuthenticator inside FMG, as a Management Extension Application (MEA)!

The following parameters will be used for each Hub: Parameter

Value

Protected Subnet

all

Role

Hub

Default VPN Interface

underlay port ( port1 for W_INET, port4 for W_MPLS) INTERNAL USE ONLY

20

Managed Secure SD-WAN 6.4.4

Parameter

Value

Hub to Hub Interface

empty

Routing

Manual

Peer Type

Accept any peer ID

IKE Config Mode

ON (Hubs will assign tunnel IPs to Edges)

IPv4 Start/End/Mask

10.20..2-250/24

Add Route

OFF (No static route injection, routing will be handled by BGP)

net-device

OFF

tunnel-search

next-hop

The following parameters will be used for each Edge: Parameter

Value

Protected Subnet

all

Role

Spoke

Default VPN Interface

underlay port ( port1 for W_INET, port4 for W_MPLS)

Routing

Manual

IKE Config Mode

ON (Get tunnel IP from the Hub)

Add Route

OFF (No static route injection, routing will be handled by BGP)

net-device

ON

Now go to VPN Manager and manually create W_INET community, using the above parameters.

INTERNAL USE ONLY

21

Managed Secure SD-WAN 6.4.4

Then add managed gateways to it. You will need to add the Hubs one by one, note that each one of them has different IP ranges defined for the IKE Config Mode (see the above table). For example, for dc1_fgt:

Edges, on the other hand, all have identical configuration. So you can simply add the entire Device Group (“Edge-West”). This also means that you won’t need to change anything here, when you add a new Edge device to your topology (you just need to assign it to the “Edge-West” group). INTERNAL USE ONLY

22

Managed Secure SD-WAN 6.4.4

Look at the list of community members and note the column “Node ID” (you might have to select it in “Column Settings”). This ID is allocated automatically for each gateway that you add to a community, using some running counter. For instance, if you would delete a Hub and add it again, it could get a different ID. Why does it matter? Because this ID influences the naming of the objects that FMG will eventually generate on the FGT (phase1-interface, phase2-interface and tunnel interface). Unfortunately, it is not possible to specify this parameter via FMG GUI. Which means that this ID is unpredictable for you. Why would you want to make it predictable? For example, because you will need to adjust some configuration later, using CLI Templates. And to do that, you have to know the exact name of the object that you want to adjust (be it the phase1-interface or, for example, the tunnel interface to which you want to add an IP). And so, we are showing you a “trick” to make those names predictable! But it will start with a pain… Please go ahead and… delete the gateways that you have just added! Yes, that is correct: both Hubs and Edges. So that your community is empty again. We will now add them again, with Postman requests. And this is not just for fun! Examine the following Postman request: West Region - Dual-Hub / Overlay / 2 - Add VPN Hubs

You will see that every Hub that we add has a parameter id . That’s exactly the ID that we are talking about. REST API allows us to manually specify its value! Now we just need to figure out what value we want…

INTERNAL USE ONLY

23

Managed Secure SD-WAN 6.4.4

When a VPN community has two Hubs, the FMG is using the following formula when generating names for phase1-interfaces and tunnel interfaces: • On Hub: _0 • On Edge (tunnels to to each Hub): _ To make it human-readable, let’s use ID = (we cannot just use , because we must use unique ID inside each community, and the same Hub appears both in W_INET and in W_MPLS). FMG uses hexadecimal value of ID, while in JSON requests we need to put a decimal value. So we come up with the following table: Community

Hub

HEX ID

Decimal ID

W_INET

dc1_fgt

11

17

W_MPLS

dc1_fgt

12

18

W_INET

dc2_fgt

21

33

W_MPLS

dc2_fgt

22

34

And this is exactly what we set with Postman.

Note that this trick is only needed when having two Hubs inside the VPN community. When there is just one Hub, the IDs are not used in naming. All the generated phase1-interfaces and tunnel interfaces are simply called _0 , so they are easily predictable without any tricks. Hence, when you are not using REST API, it might be a better idea for you to split the VPN communities per Hub. That is, instead of two communities, you would end up with four, such as H1_INET, H2_INET, H1_MPLS and H2_MPLS (each community would have just one Hub).

Now when you understand the logic, please go ahead and execute the following requests: West Region - Dual-Hub / Overlay / 1 - Add VPN Communities West Region - Dual-Hub / Overlay / 2 - Add VPN Hubs West Region - Dual-Hub / Overlay / 3 - Add VPN Edge

You should end up having the following picture:

INTERNAL USE ONLY

24

Managed Secure SD-WAN 6.4.4

If you check each community, you will see our Hub IDs set to the expected values:

It is a good time to install policy on all Hubs and Edges, to see how our VPN tunnels come up. But our “real” firewall policy is not ready yet, so let us install the “default” policy, which we have prepared with just a single “Permit Any-Any” rule. • Install default package on all FGTs (simply do it manually with FMG) This will push all the VPN configuration to the devices, and you should now see all IPSEC tunnels coming up:

And of course, do not miss the VPN Manager’s Map View!

INTERNAL USE ONLY

25

Managed Secure SD-WAN 6.4.4

2.2.2. Adjust Overlay Configuration Connect to the FGTs using SSH and examine the generated VPN configuration:

INTERNAL USE ONLY

26

Managed Secure SD-WAN 6.4.4 show vpn ipsec phase1-interface show vpn ipsec phase2-interface show system interface

We are almost there, but the following things are missing: • ADVPN is not enabled • Tunnel interface IPs are not configured Neither of these is currently handled by the VPN Manager. We will have to add it via CLI Templates. ADVPN must be enabled both on Hubs and on Edges. But tunnel IPs must be configured only on the Hubs. That’s because we use IKE Config Mode, so the Hub is assigning tunnel IPs to Edges automatically. In fact, you can check right now that it has been already done (on the Edges):

branch1_fgt # diagnose ip address list | grep W_ IP=10.202.1.3->10.202.1.3/255.255.255.0 index=28 IP=10.202.2.3->10.202.2.3/255.255.255.0 index=29 IP=10.201.2.3->10.201.2.3/255.255.255.0 index=30 IP=10.201.1.3->10.201.1.3/255.255.255.0 index=31

devname=W_MPLS_12 devname=W_MPLS_22 devname=W_INET_21 devname=W_INET_11

Finally, a somewhat less obvious part that is missing: Network Overlay ID. Originally developed for OCVPN, it is nevertheless needed also in our case.

It is not possible to establish two IPSEC tunnels between the same two FGT IPs, unless the Network Overlay ID differs between these two tunnels. In our case, this can happen if DC1 Hub triggers a shortcut between two Edges and then there is a failover to DC2 Hub which also triggers a shortcut between the same two Edges. This second shortcut will fail to establish, as long as the first one is still there. To avoid this problem, we must ensure that the IPSEC tunnels towards each Hub have different Network Overlay IDs.

FMG 6.4.x supports configuring Network Overlay ID per community . Unfortunately, this is not enough for us, because we need a separate ID per Hub . Hence, we will set it using CLI Templates. Again, we will define ID = , same as we used for tunnel interface naming. So let’s fix the missing parts, using CLI Templates. In the latest FMG version they are located under Device Manager -> Provisioning Templates -> CLI Template ).

To avoid typos, you can use the following Postman request: West Region - Dual-Hub / Overlay / 5 - Add CLI Template - Overlay

It creates the following two templates (only partially listed here for brevity, but feel free to see them on the FMG!): • “01-W-Hub-Overlay” for the Hubs:

INTERNAL USE ONLY

27

Managed Secure SD-WAN 6.4.4 # Configure tunnel interface IPs config system interface edit "W_INET_0" set ip 10.201.$(dc-id).1 255.255.255.255 set remote-ip 10.201.$(dc-id).254 255.255.255.0 set allowaccess ping next # ... end # Enable ADVPN config vpn ipsec phase1-interface edit "W_INET_0" set auto-discovery-sender enable set network-overlay enable set network-id $(dc-id)1 next # ... end

◦ auto-discovery-sender enable allows Hubs to send ADVPN shortcut messages to Edges ◦ network-overlay is enabled and network-id is set properly Note how we use the previously defined meta-data ( dc-id ) to set the correct 3rd octet of the tunnel IP per Hub, as well as the Network Overlay ID. Again, consistent IP addressing convention helps a lot here, so you should always pay attention to this!

• “01-W-Edge-Overlay” for the Edges:

# Enable ADVPN config vpn ipsec phase1-interface edit "W_INET_11" set auto-discovery-receiver enable set idle-timeout enable set idle-timeoutinterval 5 set network-overlay enable set network-id 11 next # ... end # Allow shortcut monitoring (ping) config system interface edit "W_INET_11" set allowaccess ping next # ... end

◦ auto-discovery-receiver enable allows Edges to process incoming ADVPN shortcut messages

INTERNAL USE ONLY

28

Managed Secure SD-WAN 6.4.4

◦

idle-timeout feature allows Edges to tear down idle shortcuts

◦ network-overlay is enabled and network-id is set properly ◦ PING is allowed on all tunnel interfaces, to allow ADVPN shortcut monitoring

Note how we can refer to the generated phase1-interfaces easily, because we made their naming predictable.

• Create two CLI Template Groups called “Hub-West-Template” and “Edge-West-Template” • Add the respective templates to the groups • Assign the groups to the respective FGTs You should get the following results:

Finally, install the configuration on all FGTs to apply the templates (“Quick Install (Device DB)”). You should get the following result:

INTERNAL USE ONLY

29

Managed Secure SD-WAN 6.4.4

2.2.3. Examine Overlay Configuration Connect again to FGTs using SSH and examine the final VPN configuration that we have generated.

INTERNAL USE ONLY

30

Managed Secure SD-WAN 6.4.4

2.3. Routing It’s time to add BGP to this topology. We will be doing this with CLI Templates again.

Routing is probably the most complex part of the configuration, especially if we want to make it as generic as possible. Even more so - if we are planning to extend our solution to multiple regions (and we are!). We will do our best to take you through it step by step. Do not forget our goal - as pointed out in the Introduction. All we need is to learn all possible routes to all possible destinations! The burden of picking the most appropriate route out of all the options is not so much for the routing. This intelligence will be handled by SD-WAN!

2.3.1. Design Overview Every Edge will establish IBGP peering with both Hubs over each overlay (4 BGP sessions per Edge), advertising its LAN prefix. The Hubs will act as Route Reflectors, propagating these advertisements to all other Edges. As a result, every Edge will learn all other Edges’ prefixes. This is required for a correct ADVPN operation.

In order to be able to steer the traffic towards ADVPN shortcut, it is required to preserve BGP NH (next-hop) value unchanged all the way between the Edges. Luckily, this is the default behavior for IBGP (as opposed to EBGP), hence, when building ADVPN with BGP, IBGP is the logical choice.

One additional topic that needs attention is cross-overlay route advertisements. As you probably know, by default, when we configure the Hub to be BGP Route Reflector, it reflects each received prefix to each peer (apart from the one who sent it, of course). Since our Edge devices are connected to the Hubs with multiple overlays (W_INET, W_MPLS), this naturally results in duplicate routes. For example, branch2_fgt will advertise two routes towards its LAN prefix - one via W_INET, another via W_MPLS. The Hub will reflect each of them to each of the sessions with branch1_fgt, so the latter will get four routes to the same destination: 1. With BGP NH = branch2_fgt W_INET IP, learnt over branch1_fgt W_INET 2. With BGP NH = branch2_fgt W_INET IP, learnt over branch1_fgt W_MPLS 3. With BGP NH = branch2_fgt W_MPLS IP, learnt over branch1_fgt W_INET 4. With BGP NH = branch2_fgt W_MPLS IP, learnt over branch1_fgt W_MPLS Of course, two of these are duplicate, so the question is whether and how to handle this. There are multiple working approaches, and comparing them is outside the scope of this lab. But what will we do? Focusing only on a single region for now, we will implement the following approach: • On each overlay, Edges will prefer routes originated from the same overlay (W_INET / W_MPLS). They will still accept cross-overlay routes, but only as a choice of last resort, when same-overlay INTERNAL USE ONLY

31

Managed Secure SD-WAN 6.4.4

route is not available (for example, if branch1_fgt is connected only to the Internet and branch2_fgt - only to MPLS). • Static aggregates will be used to guarantee correct recursive resolution of cross-overlay routes.

If the last sentence is not clear, ask yourself the following question. If branch1_fgt is connected only to the Internet and branch2_fgt - only to MPLS, and even if we accept cross-overlay routes (as mentioned in the previous item), the route from branch2_fgt will arrive with BGP NH = branch2_fgt W_MPLS IP. How will branch1_fgt resolve that route, if it is not connected to W_MPLS overlay…? Don’t worry, we will demonstrate that!

To identify the overlay, we will rely on tunnel subnet aggregates. Keep in mind that we will have to come up with an approach that can be easily extended to multiple regions! This is why it was so crucial to choose an IP addressing scheme that allows summarizing all overlays over the Internet or all overlays over MPLS. This summary is enough for now. We will have to dig much deeper when we deal with inter-regional routing!

2.3.2. Configure Hubs We start from the Hubs, because their BGP configuration in a single region is the most straightforward.

It’s a “Zero-Touch Configuration” approach: there is no Edge-specific configuration on the Hubs, hence there is no need to touch their configuration when Edges are added or removed.

Use the following Postman request to create a CLI Template called “02-W-Hub-Routing” with the complete configuration: West Region - Dual-Hub / Routing / 1 - Add CLI Template - Hub Routing

It is a very basic configuration, just defining IBGP session with neighbor-group method. The only items worth noting are: • ibgp-multipath - we must enable it in order to install multiple paths to the same destination into the routing table (one path for each overlay) • additional-path (BGP ADD-PATH) - we allow the Hub to send to the Edges multiple paths to the same destination • The Hub acts as BGP Route Reflector, hence set route-reflector-client enable Also note how we use the meta data to advertise DC LAN subnet (behind Hub):

INTERNAL USE ONLY

32

Managed Secure SD-WAN 6.4.4 config router bgp config network edit 1 set prefix 10.$(dc-id).0.0 255.255.255.0 next end end

In addition to BGP configuration, you will find an important piece that we shouldn’t forget: policy routes for “overlay stickiness”:

# Overlay stickiness config router policy edit 1 set input-device "W_INET_0" set output-device "W_INET_0" next edit 2 set input-device "W_MPLS_0" set output-device "W_MPLS_0" next end

In simple words, these policy routes say: “Prefer to stay within the same overlay”. For example, when traffic from branch1_fgt to branch2_fgt arrives at the Hub via W_INET overlay, there are normally two options for this traffic to continue on its way towards branch2_fgt: it can continue either via W_INET or via W_MPLS. The above policy routes will prefer to stay on W_INET. This is indeed only preference: if, for example, the only available route is via W_MPLS (because Internet link is down on branch2_fgt), it will be used, despite violating the policy route.

These policy routes are crucial for a correct operation of ADVPN. During ADVPN shortcut establishment, the Hub must forward shortcut queries between the two negotiating Edges, and it relies on routing to do so. Hence, it is important that it preserves the overlay whenever possible. Otherwise there could be an attempt to build INET-to-MPLS shortcut, which is physically impossible.

Add the newly added CLI Template to the CLI Template Group “Hub-West-Template”:

INTERNAL USE ONLY

33

Managed Secure SD-WAN 6.4.4

Install the configuration on both Hubs (“Quick Install (Device DB)”).

2.3.3. Configure Edges Use the following Postman request to create a CLI Template called “02-W-Edge-Routing” with the complete configuration: West Region - Dual-Hub / Routing / 2 - Add CLI Template - Edge Routing

Let’s go through it: • First, we define the prefix-lists with our aggregates. To remind you, 10.201.0.0/16 and 10.202.0.0/16 summarize ALL overlays over the Internet and over MPLS respectively, including those from the remote regions that will come later!

config router prefix-list edit "NH_INET" config rule edit 1 set prefix 10.201.0.0 255.255.0.0 set le 32 next end next # ... end

• Second, we build corresponding route-maps, that prefer same-overlay routes, giving them higher weight of 200 (the default weight is 100):

INTERNAL USE ONLY

34

Managed Secure SD-WAN 6.4.4 config router route-map edit "INET_IN" config rule edit 1 # Prefer same-overlay routes (INET) set match-ip-nexthop "NH_INET" set set-weight 200 next edit 100 next end next # ... end

Note that the last empty rule (100) is mandatory, because route-maps all have “default deny” behavior! If we want to accept cross-overlay routes, we have to add that term! It looks empty, because the default rule action is “permit”.

• Third, we configure BGP. This part looks relatively straightforward. We configure 4 IBGP sessions (one per Hub per overlay): ◦ Again, we enable ibgp-multipath in order to install multiple paths to the same destination into the routing table ◦ We also enable additional-path (BGP ADD-PATH) in order to receive multiple paths from the Hubs ◦ We again use the meta data to advertise Edge’s LAN subnet:

# BGP to Hub config router bgp config network edit 1 set prefix 10.0.$(branch-id).0 255.255.255.0 next end end

◦ We use route-map-in to apply the route-maps defined above:

INTERNAL USE ONLY

35

Managed Secure SD-WAN 6.4.4 config router bgp config neighbor edit "10.201.1.10" set interface "W_INET_11" set route-map-in "INET_IN" next # ... end end

• The fourth and the last part of the configuration is probably the most confusing. It includes static aggregates, needed for proper recursive resolution of cross-overlay routes - both inside the region and cross-regional. Do not worry about it now, we will demonstrate it in action later.

And we will also explain these routes have to be static in the current FOS release.

Add the newly added CLI Template to the CLI Template Group “Edge-West-Template”:

Install the configuration on both Edges (“Quick Install (Device DB)”).

2.3.4. Verify BGP Operation Connect to FGTs using SSH and verify that all BGP sessions are up. We expect to see 4 BGP sessions on each Edge (one per Hub per overlay). For example:

INTERNAL USE ONLY

36

Managed Secure SD-WAN 6.4.4 branch1_fgt # get router info bgp summary VRF 0 BGP router identifier 10.0.1.1, local AS number 65000 BGP table version is 4 1 BGP AS-PATH entries 0 BGP community entries Neighbor 10.201.1.1 10.201.2.1 10.202.1.1 10.202.2.1

V 4 4 4 4

AS MsgRcvd MsgSent 65000 10 6 65000 11 6 65000 11 6 65000 10 6

TblVer 2 2 3 1

InQ OutQ Up/Down State/PfxRcd 0 0 00:00:16 3 0 0 00:00:16 3 0 0 00:00:15 3 0 0 00:00:18 3

Total number of neighbors 4

As can be seen, we learn 3 prefixes over each peering. What will be those prefixes? 1. LAN subnet of the remote Edge - learn from each session with two different BGP NHs (from W_INET and W_MPLS) 2. LAN subnet of the DC - learnt only from the respective Hub The following CLI commands will help you investigate further what prefixes have been learnt:

branch1_fgt # get router info bgp neighbors ?

(advertised-routes|received prefix-filter|received-routes|routes)

For instance, note how we give weight=200 to the routes originated from the same overlay. Here is an example over W_INET (see the weight of the routes with NH = 10.201.*.*):

branch1_fgt # get router info bgp neighbors 10.201.1.1 routes VRF 0 BGP table version is 4, local router ID is 10.0.1.1 Status codes: s suppressed, d damped, h history, * valid, > best, i - internal, S Stale Origin codes: i - IGP, e - EGP, ? - incomplete Network * i10.0.2.0/24 *>i *>i10.1.0.0/24

Next Hop 10.202.1.2 10.201.1.2 10.201.1.1

Metric LocPrf Weight RouteTag Path 0 100 0 0 i 0 100 200 0 i 0 100 200 0 i

Total number of prefixes 2

Only the routes with highest weight will be installed in the routing table. The rest will be kept in BGP database ( get router info bgp network ) for the choice of last resort. As a result, we end up with a neat routing table, as follows:

INTERNAL USE ONLY

37

Managed Secure SD-WAN 6.4.4 branch1_fgt # get router info routing-table Routing table for VRF=0 B 10.0.2.0/24 [200/0] via 10.201.1.2, [200/0] via 10.201.2.2, [200/0] via 10.202.1.2, [200/0] via 10.202.2.2, B 10.1.0.0/24 [200/0] via 10.201.1.1, [200/0] via 10.202.1.1, B 10.2.0.0/24 [200/0] via 10.201.2.1, [200/0] via 10.202.2.1,

bgp W_INET_11, W_INET_21, W_MPLS_12, W_MPLS_22, W_INET_11, W_MPLS_12, W_INET_21, W_MPLS_22,

00:01:58 00:01:58 00:01:58 00:01:58 00:02:00 00:02:00 00:02:01 00:02:01

You might wonder why the prefixes learnt via Secondary Hub are active in the routing table. Aren’t we supposed to use only those learnt via the Primary Hub? Well, we could do that. But we prefer to keep all the routes active and use SD-WAN rules to implement the Active/Passive behavior.

Feel free to examine the routing tables on Hubs as well. We are ready for SD-WAN configuration and for sending traffic between Edges.

INTERNAL USE ONLY

38

Managed Secure SD-WAN 6.4.4

2.4. SD-WAN for Internal Traffic In this chapter we are going to configure and test SD-WAN for the traffic within customer’s network. That is, between different Branches and between Branches and DCs. Logically, this traffic will always flow over the overlays that we have built.

2.4.1. Preconfigured Health-Check Loopbacks We have preconfigured a loopback interface on each Hub, both having exactly the same IP address which we will use as our “health-check server”. As a result, we can configure just a single health-check server, while in fact the probes will be reaching different Hubs. The loopback configuration is straightforward:

config system interface edit "lo-HC" set vdom "root" set ip 10.200.99.1 255.255.255.255 set allowaccess ping set type loopback next end

We just need to remember to permit the health probes in the firewall policy, which we will be configuring shortly!

You may be wondering how the Edges will be able to reach those loopback addresses. Shouldn’t the Hubs advertise them via BGP? After all, if we check the routing tables on the Edges, we won’t find any route to those loopbacks. Try pinging them - and it will fail. So how can the health check possibly succeed? Consider it a riddle for now.

2.4.2. Configure Basic Building Blocks First, we must create normalized interfaces on FMG (previously known as “dynamic interfaces”), mapped

to

the

tunnel

interfaces

on

our

FGTs.

Create

them

under

Policy & Objects -> Object Configurations -> Normalized Interface . Use “Per-Platform Mapping”

with platform set to “all” (this effectively creates a default mapping):

INTERNAL USE ONLY

39

Managed Secure SD-WAN 6.4.4

This Postman request will save you some time: West Region - Dual-Hub / Overlay / 4 - Add Normalized Interfaces

For the Edges: Normalized Interface

Default Mapping

W_INET_H1

W_INET_11

W_INET_H2

W_INET_21

W_MPLS_H1

W_MPLS_12

W_MPLS_H2

W_MPLS_22

Normalized Interface

Default Mapping

W_INET

W_INET_0

W_MPLS

W_MPLS_0

For the Hubs:

We are now ready to configure SD-WAN on the Edges. This is done centrally on FMG, under Device Manager -> SD-WAN .

• Configure SD-WAN Interface Members. We need to configure 4 members and map them to the 4 normalized interfaces that we have created above for the Edges. For simplicity, let’s simply use the same names for SD-WAN members and the respective normalized interfaces. For example:

INTERNAL USE ONLY

40

Managed Secure SD-WAN 6.4.4

Note the priority value that we set for all these members. This is a highly recommended best-practice to set higher priority value (that is, lower priority) for all the overlay interfaces. You will see why in the next chapter, when we deal with Internet access. The default priority value is 0, hence we give 10 to our overlays.

• Configure Health-Check Server. Use the special Loopback that we have defined on both Hubs:

INTERNAL USE ONLY

41

Managed Secure SD-WAN 6.4.4

• Create SD-WAN Template called “Edge-West” • Create a new SD-WAN Zone called “overlay” and add all the SD-WAN members to it:

• Define Performance SLA (based on our health-check loopback):

INTERNAL USE ONLY

42

Managed Secure SD-WAN 6.4.4

Always remember to set the values for sla-fail-log-period and sla-pass-log-period under “Advanced Options”. In this lab, set them both to “10” (the value is in seconds). These periodic SLA reports are crucial for proper monitoring in FAZ. Without them, you will see a lot of missing data in reports and widgets.

• Assign this SD-WAN template to the Edges:

Do not install the configuration yet, because we haven’t defined any SD-WAN rules.

2.4.3. SD-WAN Rules: Active/Passive Hub This is the actual “juice”. We are going to define how we would like the traffic to flow. Let’s configure SD-WAN rule for internal (corporate) traffic, so that: 1. The traffic prefers W_INET, as long as it meets the SLA. If it doesn’t, the traffic will switchover to W_MPLS.

INTERNAL USE ONLY

43

Managed Secure SD-WAN 6.4.4

2.

The traffic should always use dc1_fgt, as long as it is available. dc2_fgt should be used only as a backup Hub (that is, only when dc1_fgt is completely out of service).

To achieve that, we will configure two SD-WAN rules, one for the Primary Hub and another one for the Secondary Hub. Here is the first rule:

• Note that here we specify only the interface members connecting to the Primary Hub (DC1) • Also note that W_INET_H1 comes first in the list. When using “Lowest Cost (SLA)” strategy, preference is defined by configuration order, among others. The first interface that matches the SLA will be selected - which is precisely what we want to achieve here. Now configure the second rule in a similar manner, for Secondary Hub (DC2). INTERNAL USE ONLY

44

Managed Secure SD-WAN 6.4.4

2.4.4. Firewall Policy While SD-WAN rules define where the traffic should flow, firewall policy defines whether the traffic is permitted to flow, and if yes, how it should be inspected. Our current “Permit Any-Any” policy is not acceptable, so let’s finally adjust it. Rather than editing the “default” package, we are going to populate the two dedicated packages “Edge-West-PP” and “HubWest-PP”, that we have created using Postman Runner earlier. With the introduction of SD-WAN Zones concept, it is no longer possible to use individual tunnel interfaces in firewall policies! We must group them into SD-WAN Zones and use these zones in the policies. We have assigned all our SD-WAN members to the newly created “overlay” zone. This automatically created a corresponding normalized interface for us. Which we can now use in our policies. If you do not find our SD-WAN Zone in the list of interfaces, try to logout from FMG and login again.

Here are the rules that we must create. For the Edges (“Edge-West-PP”): Name

From

To

Src

Dst

Service

NAT

Action

Corporate

vl_lan

overlay

CORP_LAN

CORP_LAN

ALL

No

Accept

overlay

vl_lan

For the Hubs (“Hub-West-PP”): Name Corporate

Health-check

From

To

Src

Dst

Service

NAT

Action

W_INET

W_INET

all

all

ALL

No

Accept

W_MPLS

W_MPLS

lan

lan

W_INET

any

all

HC

PING

No

Accept

W_MPLS Note that on the Hub we are still using individual interfaces in the firewall policy, because we do not configure SD-WAN on the Hub (and hence, there are no SD-WAN Zones either). Also note that we must explicitly permit the incoming health probes. Traffic towards loopback would not be permitted by default! INTERNAL USE ONLY

45

Managed Secure SD-WAN 6.4.4

Install the policy: • Install “Edge-West-PP” package on Edges • Install “Hub-West-PP” package on Hubs

There is one more caveat to take care of in a real-world scenario, when ADVPN is used. Since it is quite possible for a session to switchover from one overlay to another in the middle (e.g. due to changing health condition of a link), it can happen that a certain TCP session will switchover from ADVPN shortcut to Edgeto-Hub tunnel. Since the Hub is not aware of this TCP session, it will be dropped by the stateful inspection. This is, of course, not desired. Hence, when ADVPN is in use and session switchover is needed, it is important to disable stateful inspection on the Hubs for the Edge-to-Edge traffic. This is done as follows: • Globally enable TCP sessions without SYN:

config system settings set tcp-session-without-syn enable end

• Edit the above “Corporate” rule, setting the following two advanced parameters: ◦ Disable anti-replay (TCP sequence number validation) ◦ Set tcp-session-without-syn to “all” No reason to worry: Edges still provide stateful inspection for all the Edge-to-Edge, DC-to-Edge and Edgeto-DC traffic! And Hubs still provide it for all the other traffic (if there is such), since we have only disabled it on a particular firewall rule. In this lab we are going to use PING for our testing, so you will not experience this issue. But it is good to remember it for a real-world scenario, with TCP traffic.

2.4.5. SD-WAN Testing: Active/Passive Hub Finally, it’s time for testing! Please use the following CLI commands on FGT at each step, to see what’s happening: • Check IPSEC tunnel status and routing table:

get ipsec tunnel list get router info routing-table all

• Check SLA status per link:

diagnose sys sdwan health-check

• Check link preference chosen by SD-WAN (as a result of your configuration + SLA status):

INTERNAL USE ONLY

46

Managed Secure SD-WAN 6.4.4 diagnose sys sdwan service

• And of course… the sniffer:

diagnose sniffer packet any 'icmp and host 10.0.2.101' 4 | grep W_

Follow carefully, because we are going to show some of the not-so-intuitive effects!

Let’s start! • Connect to Branch1_FGT and make sure that the health check passes via all four tunnels: branch1_fgt # diagnose sys sdwan health-check Health Check(DC): Seq(1 W_INET_11): state(alive), packet-loss(0.000%) a_map=0x1 Seq(2 W_INET_21): state(alive), packet-loss(0.000%) a_map=0x1 Seq(3 W_MPLS_12): state(alive), packet-loss(0.000%) a_map=0x1 Seq(4 W_MPLS_22): state(alive), packet-loss(0.000%) a_map=0x1

latency(1.621), jitter(0.157) sl latency(1.522), jitter(0.185) sl latency(1.156), jitter(0.211) sl latency(1.059), jitter(0.166) sl

Now also check the order of the interface members for SD-WAN Rule 1 and make sure that W_INET_11 comes first (which is what we expect): branch1_fgt # diagnose sys sdwan service 1 Service(1): Address Mode(IPV4) flags=0x200   Gen(3), TOS(0x0/0x0), Protocol(0: 1->65535), Mode(sla), sla-compare-order   Members(2):     1: Seq_num(1 W_INET_11), alive, sla(0x1), gid(1), cfg_order(0), cost(0), selected     2: Seq_num(3 W_MPLS_12), alive, sla(0x1), gid(1), cfg_order(1), cost(0), selected   Src address(1):     10.0.0.0-10.255.255.255   Dst address(1):     10.0.0.0-10.255.255.255

So here we go… branch1_fgt can successfully reach the health-check loopbacks defined on the Hubs. But how?! The answer is that health probes do not follow conventional routing! A special kernel route is injected towards each of the configured health-servers, via each of the SD-WAN member interfaces defined for a particular health check. You can find those kernel routes here (note the special proto=17 marking, this has nothing to do with UDP really):

INTERNAL USE ONLY

47

Managed Secure SD-WAN 6.4.4 branch1_fgt # get router info kernel | grep "proto=17" tab=254 vf=0 scope=0 type=1 proto=17 prio=0 10.202.2.3/255.255.255.255/0>10.200.99.1/32 pref=0.0.0.0 gwy=0.0.0.0 dev=29(W_MPLS_22) tab=254 vf=0 scope=0 type=1 proto=17 prio=0 10.202.1.3/255.255.255.255/0>10.200.99.1/32 pref=0.0.0.0 gwy=0.0.0.0 dev=28(W_MPLS_12) tab=254 vf=0 scope=0 type=1 proto=17 prio=0 10.201.2.3/255.255.255.255/0>10.200.99.1/32 pref=0.0.0.0 gwy=0.0.0.0 dev=30(W_INET_21) tab=254 vf=0 scope=0 type=1 proto=17 prio=0 10.201.1.3/255.255.255.255/0>10.200.99.1/32 pref=0.0.0.0 gwy=0.0.0.0 dev=31(W_INET_11)

The next-hop gateway IP is taken from the SD-WAN member interface configuration. In our case, it is set to 0.0.0.0, since we did not configure any gateway for our SD-WAN members. That’s fine, because we have IPSEC tunnels, which are point-to-point, so that a next-hop gateway IP is not really required. But pay attention to this, if you want to include your underlay ports in the SD-WAN! In that case, setting gateway IP might become mandatory, in order for the health check to succeed! Although this is also not always true, since FOS can automatically use the gateway learnt via DHCP, for example.

• Now connect to Branch1_Client using SSH and start pinging Branch2_Client:

ping 10.0.2.101

On branch1_fgt you should notice a new shortcut created over W_INET_11 - namely, W_INET_11_0:

branch1_fgt # get ipsec tunnel list NAME REMOTE-GW PROXY-ID-SOURCE STATUS TIMEOUT W_INET_11_0 203.0.113.1:0 10.201.1.1/10.201.1.1 up 1781 ...

PROXY-ID-DESTINATION 0.0.0.0/0.0.0.0

Use packet sniffer to confirm that the ping is now flowing via that shortcut. • Use “WAN Controller” to raise the latency on the DC1-ISP1 link, so that it exceeds the threshold that you have set in the SLA. Check if the traffic has switched over… Can you explain what you see? In fact, what you see is expected. We have raised the latency on the dc1_fgt, but the traffic is flowing over the direct shortcut between the Edges. Thanks to the ADVPN shortcut monitoring feature, branch1_fgt is now probing the shortcut itself, and not just the path to the Hub. You can see the newly added probe, as well as the failing probe to the Hub:

INTERNAL USE ONLY

48

Managed Secure SD-WAN 6.4.4 branch1_fgt # diagnose sys sdwan health-check Health Check(DC): Seq(1 W_INET_11): state(alive), packet-loss(0.000%) latency(111.643), jitter(0.184) sla_map=0x0 Seq(1 W_INET_11_0): state(alive), packet-loss(0.000%) latency(1.553), jitter(0.244) sla_map=0x1 Seq(2 W_INET_21): state(alive), packet-loss(0.000%) latency(1.441), jitter(0.125) sla_map=0x1 Seq(3 W_MPLS_12): state(alive), packet-loss(0.000%) latency(1.101), jitter(0.191) sla_map=0x1 Seq(4 W_MPLS_22): state(alive), packet-loss(0.000%) latency(1.062), jitter(0.159) sla_map=0x1

But what exactly is it probing? Surely, not the Loopback IP that we have configured as our health-check server, because that wouldn’t work… That IP doesn’t exist on branch2_fgt. Use the sniffer to find out…

• Now raise the latency on the BR2-ISP1 link. This time all the incoming traffic from Internet to branch2_fgt will be affected, so we expect the shortcut to get affected too. You should see the traffic switching over to the MPLS network, still using DC1 Hub (W_MPLS_12). And the new shortcut will be built over that overlay (W_MPLS_12_0). • Reduce the latency on both links and make sure the traffic switches back to W_INET_11_0. You are in the initial state again. • Now use “WAN Controller” to bring DC1-ISP1 link down. Wait for about 15-20 seconds and check the traffic, the probes and the tunnels. Did you expect seeing what you see? branch1_fgt # diagnose sys sdwan health-check Health Check(DC): Seq(1 W_INET_11): state(dead), packet-loss(51.000%) sla_map=0x0 Seq(1 W_INET_11_0): state(alive), packet-loss(0.000%) latency(1.650), jitter(0.199) sla_map=0x1 Seq(2 W_INET_21): state(alive), packet-loss(0.000%) latency(1.543), jitter(0.277) sla_map=0x1 Seq(3 W_MPLS_12): state(alive), packet-loss(0.000%) latency(1.145), jitter(0.210) sla_map=0x1 Seq(3 W_MPLS_12_0): state(alive), packet-loss(0.000%) latency(1.053), jitter(0.118) sla_map=0x1 Seq(4 W_MPLS_22): state(alive), packet-loss(0.000%) latency(1.157), jitter(0.200) sla_map=0x1

This raises not one, but two questions at least: 1. As expected, W_INET_11 is marked as “dead”. But why do we still see the shortcut W_INET_11_0 (and it even meets the SLA!)? The answer is that the default FortiOS behavior is NOT to tear down shortcuts when parent tunnels go down. So even though the parent tunnel to the Hub went down (or will soon go down due to DPD), the shortcut remains active. And so, it can even meet the SLA. INTERNAL USE ONLY

49

Managed Secure SD-WAN 6.4.4 Since FOS 6.4.3, it is possible to change this behavior with the following configuration option under phase1-interface:

set auto-discovery-shortcuts dependent . This will automatically bring

down shortcuts, when their parents are down.

2. OK, but then why aren’t we still sending the traffic over W_INET_11_0 shortcut? That could be logical. Previously we saw that raising the latency around the Hub did not affect the direct shortcut, thanks to ADVPN shortcut monitoring feature. But remember that we do not have BGP peering over the shortcut! The only way for branch1_fgt to learn branch2_fgt prefixes over W_INET_11 or W_INET_11_0 is from the Hub. That BGP session is now down, because the Hub is unreachable. Hence, if you check the routing table on branch1_fgt, you will not find any route pointing to the shortcut anymore. The default (and most used) behavior is that SD-WAN will not forward the traffic without a valid route!

branch1_fgt # get router info bgp summary VRF 0 BGP router identifier 10.0.1.1, local AS number 65000 BGP table version is 4 1 BGP AS-PATH entries 0 BGP community entries Neighbor 10.201.1.1 10.201.2.1 10.202.1.1 10.202.2.1

V 4 4 4 4

AS MsgRcvd MsgSent 65000 11232 11225 65000 11556 11531 65000 11562 11558 65000 11562 11535

TblVer 0 2 1 3

InQ OutQ Up/Down State/PfxRcd 0 0 never Idle 0 0 02:00:35 3 0 0 02:00:36 2 0 0 02:00:34 3

Total number of neighbors 4 branch1_fgt # get router info routing-table details 10.0.2.0/24 Routing table for VRF=0 Routing entry for 10.0.2.0/24 Known via "bgp", distance 200, metric 0, best Last update 00:21:42 ago * 10.201.2.2, via W_INET_21 distance 0 * 10.202.1.2, via W_MPLS_12_0 distance 0 * 10.202.2.2, via W_MPLS_22 distance 0

Hence, the shortcut is up, but it is not used. And it will soon expire due to idle timeout. • Now let’s raise the latency of BR2-MPLS link. What do we expect to happen in this case? You will notice that the traffic remains on W_MPLS_12_0 and suffers a very high latency. Looking at the probe status raises a question again:

INTERNAL USE ONLY

50

Managed Secure SD-WAN 6.4.4 branch1_fgt # diagnose sys sdwan health-check Health Check(DC): Seq(1 W_INET_11): state(dead), packet-loss(100.000%) sla_map=0x0 Seq(2 W_INET_21): state(alive), packet-loss(0.000%) latency(1.512), jitter(0.159) sl a_map=0x1 Seq(3 W_MPLS_12): state(alive), packet-loss(0.000%) latency(1.103), jitter(0.197) sla_map=0x1 Seq(3 W_MPLS_12_0): state(alive), packet-loss(0.000%) latency(111.206), jitter(0.169) sla_map=0x0 Seq(4 W_MPLS_22): state(alive), packet-loss(0.000%) latency(1.057), jitter(0.182) sla_map=0x1

Why not switchover to W_INET_21 (via DC2 Hub), which still meets the SLA? Why stay with W_MPLS_12_0 and suffer this high latency? The answer is: it is expected. As you remember, we have configured two separate SD-WAN rules, the first for DC1 Hub, the second for DC2 Hub. SD-WAN rules are evaluated in the order of configuration, similar to firewall rules. And just like in firewall rules, you have the “traffic matching” part (e.g. by source/destination IP, application and so on) and the “action” part. The SD-WAN strategy belongs to the latter. Hence, the first rule that matches the traffic will be used, and its action will be applied. In our case, the DC1 Hub rule is the first one to match the traffic, so we pick it and stop looking any further. In its “action” it has only two interface members, so we will always try to send traffic to one of them, even if both are out of SLA! That is the true Active/Passive behavior. DC2 Hub will only be used if DC1 Hub is completely out of service, so that the first rule cannot forward traffic at all.

Since FOS 6.4.3, it is possible to change this behavior with the following configuration option under SDWAN rule:

set minimum-sla-meet-members . This defines the minimum number of

members that must meet the SLA in order for the rule to be used. So, if we set it to 1, when all members of the rule are out of SLA, the rule will be skipped, and we will keep looking further. In other words, we will use DC2 Hub not only when DC1 Hub is completely down, but also when it is completely out of SLA.

• Let’s nevertheless try to failover to DC2 Hub. Bring down the DC1-MPLS link. Now DC1 Hub is completely unreachable, both over Internet and over MPLS. You should see how the traffic switches to DC2 Hub (W_INET_21) and builds a new shortcut over it. We are now using the second SD-WAN rule. • By the way, you spent probably more than 5 minutes reading all the explanations above. If you remember, we have configured an idle timeout on Edges which was exactly 5 minutes ( set idle-timeoutinterval 5 ). Check the output of get ipsec tunnel list . The shortcut W_INET_11_0 should have disappeared by now. The shortcut W_MPLS_12_0 might still be there (not enough time passed). Please clear it manually:

INTERNAL USE ONLY

51

Managed Secure SD-WAN 6.4.4 diagnose vpn ike gateway clear name W_MLPS_12_0

• Bring all the links up and reduce all the latency that you have raised in “WAN Controller”. The traffic should go back to DC1 Hub Internet overlay and build a new shortcut. You are in the initial state again. • One last bit. Raise the latency of DC1-MPLS link, wait for the health probes to detect this, then raise also the latency of BR2-ISP1 link. You will notice that the traffic stays on W_INET_11_0. Check the probe status: branch1_fgt # diagnose sys sdwan health-check Health Check(DC): Seq(1 W_INET_11): state(alive), packet-loss(0.000%) latency(1.575), jitter(0.158) sl a_map=0x1 Seq(1 W_INET_11_0): state(alive), packet-loss(0.000%) latency(111.693), jitter(0.204) sla_map=0x0 Seq(2 W_INET_21): state(alive), packet-loss(0.000%) latency(1.495), jitter(0.201) sla_map=0x1 Seq(3 W_MPLS_12): state(alive), packet-loss(0.000%) latency(111.218), jitter(0.186) sla_map=0x0 Seq(4 W_MPLS_22): state(alive), packet-loss(0.000%) latency(1.133), jitter(0.249) sla_map=0x1

We already know why there is no switchover to W_INET_21 - we won’t use DC2 Hub, as long as DC1 Hub is operational. But shouldn’t we try to switch to W_INET_11? That is, shouldn’t we try sending traffic via the Hub? After all, that link seems to be still within SLA, while our currently selected

link

is

out

of

SLA.

diagnose sys sdwan service 1 ,

Even

if

we

check

the

order

of

the

interfaces

in

we can see that W_INET_11 comes before the shortcut

W_INET_11_0! The answer is: we shouldn’t. Let’s check the route to our destination:

branch1_fgt # get router info routing-table details

10.0.2.0/24

Routing table for VRF=0 Routing entry for 10.0.2.0/24 Known via "bgp", distance 200, metric 0, best Last update 00:05:58 ago * 10.201.1.2, via W_INET_11_0 distance 0 * 10.201.2.2, via W_INET_21 distance 0 * 10.202.1.1, via W_MPLS_12 distance 0 * 10.202.2.2, via W_MPLS_22 distance 0

INTERNAL USE ONLY

52

Managed Secure SD-WAN 6.4.4

As you can see, there is no valid route via W_INET_11! This is because shortcuts always hide the routes via their parents. This is simply how ADVPN with BGP works. And remember: SD-WAN will not forward the traffic without a valid route! Hence, we will skip W_INET_11 and keep using W_INET_11_0. And this case illustrates that such behavior is probably justified: even though it seems that it would be better to go via the Hub, this is just an illusion. By probing the Hub, we simply do not detect the latency raised at the ingress to branch2_fgt. The results seen over the shortcut are thus more accurate, at least in this particular example.

We must point out that we are cheating here a little bit. What we checked above was the best route to 10.0.2.0/24. In some cases, there could be another valid route towards 10.0.2.0/24, even though not best. For example, it could be a default route via W_INET_11. As the statement above correctly says, our SDWAN requires a valid route to destination, not necessarily the best one! In the next section you will learn about some not-so-intuitive consequences of this. But at the moment, our best route towards 10.0.2.0/24 is also the only one available, so our example is good enough.

• Bring all the links up and reduce all the latency that you have raised in “WAN Controller”. You are in the initial state again.

2.4.6. SD-WAN Rules: Active/Passive Underlay We saw exactly what Active/Passive Hub behavior means in our context. The traffic will not use DC2 Hub, as long as DC1 Hub is alive. This means, for example, that if an Internet link is down on DC1 Hub, the traffic will flow over the MPLS network, even if an Internet link on DC2 Hub is up. And even if MPLS network suffers from high latency. What if we want to implement a different behavior? We want Internet to be always preferred, even if it means changing the Hub.

Note that this is still not an Active/Active behavior: we do not expect sessions to be load-balanced between the two Hubs. We only expect that DC2 Hub Internet connection will be considered before switching to MPLS network. So we decided to call it “Active/Passive Underlay”, as opposed to “Active/Passive Hub”.

All we need to do is to delete the second SD-WAN rule and add DC2 Hub interface members to the first rule (pay attention to the interface order!):

INTERNAL USE ONLY

53

Managed Secure SD-WAN 6.4.4

Install this configuration on both Edges.

We cannot resist pointing your attention to the beauty of it: you did not have to touch any BGP configuration. The traffic policy is implemented solely via SD-WAN rules.

2.4.7. SD-WAN Testing: Active/Passive Underlay We repeat one of the tests that we did before. • Connect to Branch1_FGT and make sure that the health check passes via all four tunnels:

INTERNAL USE ONLY

54

Managed Secure SD-WAN 6.4.4 branch1_fgt # diagnose sys sdwan health-check Health Check(DC): Seq(1 W_INET_11): state(alive), packet-loss(6.000%) a_map=0x1 Seq(2 W_INET_21): state(alive), packet-loss(0.000%) a_map=0x1 Seq(3 W_MPLS_12): state(alive), packet-loss(0.000%) a_map=0x1 Seq(4 W_MPLS_22): state(alive), packet-loss(0.000%) a_map=0x1

latency(1.659), jitter(0.111) sl latency(1.536), jitter(0.216) sl latency(1.210), jitter(0.223) sl latency(1.195), jitter(0.183) sl

Also make sure that there are no active shortcuts at the moment (that is, we are in the initial state). The interface preference order now looks like this:

branch1_fgt # diagnose sys sdwan service Service(1): Address Mode(IPV4) flags=0x200 Gen(5), TOS(0x0/0x0), Protocol(0: 1->65535), Mode(sla), sla-compare-order Members(4): 1: Seq_num(1 W_INET_11), alive, sla(0x1), gid(1), cfg_order(0), cost(0), selected 2: Seq_num(2 W_INET_21), alive, sla(0x1), gid(1), cfg_order(1), cost(0), selected 3: Seq_num(3 W_MPLS_12), alive, sla(0x1), gid(1), cfg_order(2), cost(0), selected 4: Seq_num(4 W_MPLS_22), alive, sla(0x1), gid(1), cfg_order(3), cost(0), selected Src address(1): 10.0.0.0-10.255.255.255 Dst address(1): 10.0.0.0-10.255.255.255

• Connect to branch1_client using SSH and start pinging branch2_client:

ping 10.0.2.101

As before, you should notice a new shortcut created on branch1_fgt over W_INET_11. • Bring down the DC1-ISP1 link. You remember that last time this triggered switchover to W_MPLS_12. However, this time the traffic has switched to W_INET_21 and a new shortcut has been triggered by DC2 Hub! And that is expected, because DC2 Hub Internet connection was still operational! • Now raise the latency on BR2-ISP1 link. The traffic will switch to MPLS network and will use DC1 Hub, because its interface came first in the order of preference.

INTERNAL USE ONLY

55

Managed Secure SD-WAN 6.4.4

•

Bring all the links up and reduce all the latency that you have raised. The traffic should go back to W_INET_11_0.

INTERNAL USE ONLY

56

Managed Secure SD-WAN 6.4.4

2.5. SD-WAN for Internet Traffic In this chapter we are going to extend SD-WAN to the Internet access as well. We are going to implement a hybrid Local + Remote Breakout, also known as Direct Internet Access (DIA) + Remote Internet Access (RIA). Our overlays over MPLS network will be used for RIA, when the DIA quality is not acceptable.

2.5.1. Configure Basic Building Blocks We are going to add our underlay port (Internet-facing WAN link) to the list of SD-WAN members. In our topology, this is port1. • Add a new SD-WAN member called ul-inet , mapped to the port1 normalized interface that has been already preconfigured for you:

• Add a new health-check server: www.fortinet.com • Edit our existing SD-WAN template (“Edge-West”). Create a new SD-WAN Zone “underlay”, adding the new member to it:

INTERNAL USE ONLY

57

Managed Secure SD-WAN 6.4.4

• Create a new Performance SLA, measured over ul-inet, W_MPLS_H1 and W_MPLS_H2. Define two separate SLA targets: ◦ Target #1: 200 ms latency ◦ Target #2: 300 ms latency We are going to use these two targets to demonstrate how you can apply different thresholds to different applications.

Do not forget to set the values of

sla-fail-log-period and

sla-pass-log-period under

“Advanced Options”)!

INTERNAL USE ONLY

58

Managed Secure SD-WAN 6.4.4

Do not install the configuration yet, as we still need to configure SD-WAN rules.

2.5.2. SD-WAN Rules: DIA/RIA Our strategy for Internet access will be as follows: • We will prefer the Local Breakout (DIA) via the underlay port (ul-inet). • For business-critical traffic, we will use OL_MPLS overlays as a secondary path (RIA), when DIA quality is not acceptable. The thresholds will be different for different applications. • For non-business-critical traffic, we will keep using DIA as long as it is alive, disregarding its current quality.

INTERNAL USE ONLY

59

Managed Secure SD-WAN 6.4.4

Let’s configure three SD-WAN rules for this: • Business-Critical-HighPriority: This rule will be matching GoToMeeting and Salesforce traffic, with SLA Target #1. • Business-Critical-MedPriority: This rule will be matching Office365 traffic, with SLA Target #2. • Non-Business-Critical: This rule will be matching all other Internet traffic. You should end up with something like this:

2.5.3. Firewall Policy Let’s now add firewall policies to permit Internet traffic. Different treatment is required, depending whether it is DIA or RIA: • For DIA, since the traffic leaves our network directly from the Branch, we must ensure advanced protection. In this lab, we are going to apply full SSL inspection (while this alone is not particularly advanced, it is enough to demonstrate our point). In addition, this traffic must be NATed on the Edges.

INTERNAL USE ONLY

60

Managed Secure SD-WAN 6.4.4

• For RIA, the traffic will be backhauled via the DC, hence advanced security can be applied there, as it would happen in a legacy network. In this lab, DIA traffic will leave our network via the Hub, so we are going to apply full SSL inspection on the Hub, but not on Edges. In addition, this traffic must be NATed on the Hubs, but not on the Edges.

INTERNAL USE ONLY

61

Managed Secure SD-WAN 6.4.4

The following firewall policy is needed to achieve these goals. For the Edges: Name

From

To

Service

NAT

Action

DIA

vl_lan

underlay

ALL

Yes

Accept + App Control + SSL deep-inspection

RIA

vl_lan

overlay

ALL

No

Accept + App Control

For the Hubs: Name Edge RIA

From

To

W_MPLS

port1

Service

NAT

Action

ALL

Yes

Accept + App Control + SSL deep-inspection

Note how we are using SD-WAN Zones on Edges, to configure different treatment for the traffic flowing to underlay vs overlay. Remember that this will be exactly the same type of traffic, and the actual path for each session will be determined by the SD-WAN rules.

SD-WAN rules define where the traffic should flow, and the firewall policy defines whether it should flow and what inspection should be applied to it!

Further notes: • We must enable Application Control on all these firewall rules, since our SD-WAN rules rely on it! • Make sure you also enable logging for all sessions - we’ll need to see them in our testing!

2.5.4. Routing Caveat 1 (Default Route) It seems that we have finished our configuration, but there are a few important caveats left. In the previous section, we have reiterated several times that SD-WAN will not forward the traffic without a valid route! So far, we have used BGP to exchange Edge and DC prefixes between all the sites. But what about Internet access? Check the default gateway on branch1_fgt:

INTERNAL USE ONLY

62

Managed Secure SD-WAN 6.4.4 branch1_fgt # get router info routing-table details 0.0.0.0/0 Routing table for VRF=0 Routing entry for 0.0.0.0/0 Known via "static", distance 5, metric 0, best * 192.2.0.2, via port1 distance 0 * 192.2.0.10, via port2 distance 0

Our default gateway is learnt using DHCP from the underlay ports. There is no default route via the overlays. Hence, RIA will not work: our SD-WAN rules will never select the overlays for Internet traffic. But wait, it gets worse! Once we add “port1” to our SD-WAN, something important changes. Namely, SD-WAN members do not use default gateways learnt from DHCP by default - we must explicitly enable this. Otherwise, if we push the configuration right now, the default route via “port1” will completely disappear. So how do we solve these two routing caveats? We have several options: 1. We can configure all Internet SD-WAN rules with set default enable +

set gateway enable .

This will alter the default behavior and will completely bypass route lookup. All the members will be assumed always acceptable, no matter if there is a valid route to the destination through them. This is a somewhat non-standard option, but it can work. 2. We can manually configure default routes via all interfaces that we need: ◦ We will add static routes via W_MPLS_12 and W_MPLS_22 interfaces ◦ We will also add a static route via “port1”, with

set dynamic-gateway enable . This explicit

setting will tell SD-WAN to use the default gateway learnt from DHCP. 3. But there is a nicer and a shorter syntax to accomplish our goals:

config router static edit 100 set sdwan enable next end

This static route instructs FOS to consider all the SD-WAN members as “default route”. Effectively, it means that SD-WAN alone will decide where all the traffic will flow. Because there will always be a valid (default) route via any SD-WAN member, to any destination. We will opt for this last option, as it is the most generic. It allows you to switch between DIA, RIA or a mix of the two without touching the routing configuration. Create a new CLI template called “03-ALL-Edge-Default” and paste the above static route into it. Here “ALL” implies that we will reuse this template for all our regions. For now - add it to the “EdgeWest-Template” group:

INTERNAL USE ONLY

63

Managed Secure SD-WAN 6.4.4

Install the policy on all FGTs. Check the default route on branch1_fgt now:

branch1_fgt # get router info routing-table all ... Routing table for VRF=0 S* 0.0.0.0/0 [1/0] via 192.2.0.2, port1 [1/0] via 100.64.1.1, W_INET_11, [1/0] via 100.64.2.1, W_INET_21, [1/0] via 172.16.1.5, W_MPLS_12, [1/0] via 172.16.2.5, W_MPLS_22,

[10/0] [10/0] [10/0] [10/0]

You can see a default route via all our SD-WAN members now.

And now you know why we set “priority 10” for all the overlay members. We would like to ensure that port1 will still be preferred for Internet access by any traffic that is not handled by SD-WAN. The best example of such traffic is the one locally originated by FGT itself (e.g. IKE or Fortiguard or even a ping that you initiate from CLI). We would like this traffic to always use port1. The default route via port1 has the default priority (0), so it will be preferred over the overlays. Note that you cannot use other attributes, such as “distance”. Higher distance would result in routes not being installed into the routing table at all. That would not solve our original routing caveat.

2.5.5. Routing Caveat 2 (Again… Default Route) Surely, now our configuration is complete…?! Sorry, not yet. Apparently, our useful default route breaks something that worked before. Let us show you… • Connect to branch1_client using SSH and start pinging branch2_client:

INTERNAL USE ONLY

64

Managed Secure SD-WAN 6.4.4 ping 10.0.2.101

As before, you should notice a new shortcut created on branch1_fgt over W_INET_11. • Now raise the latency on the BR2-ISP1 link. We expect the traffic to switchover to W_MPLS overlay. But instead, it stays on W_INET and suffers a very high latency. But how..?! You have already tested it before, it worked just fine! If you start investigating, you will quickly find out that the traffic flows via… W_INET_11. If we check the route to our destination, we still see what we saw in the previous chapter:

branch1_fgt # get router info routing-table details

10.0.2.0/24

Routing table for VRF=0 Routing entry for 10.0.2.0/24 Known via "bgp", distance 200, metric 0, best Last update 00:02:44 ago * 10.201.1.2, via W_INET_11_0 distance 0 * 10.201.2.2, via W_INET_21 distance 0 * 10.202.1.1, via W_MPLS_12 distance 0 * 10.202.2.2, via W_MPLS_22 distance 0

The shortcut W_INET_11_0 still successfully hides the route via W_INET_11. But do you remember how we stressed the fact that SD-WAN was looking for a valid route, rather than just for the best one? Do we have another valid route towards 10.0.2.0/24? Of course we do! That’s our default route: branch1_fgt # get router info routing-table details 0.0.0.0/0 Routing table for VRF=0 Routing entry for 0.0.0.0/0   Known via “static”, distance 1, metric 0, best   * 192.2.0.2, via port1 distance 0   * 100.64.1.1, via W_INET_11 distance 0   * 100.64.2.1, via W_INET_21 distance 0   * 172.16.1.5, via W_MPLS_12 distance 0   * 172.16.2.5, via W_MPLS_22 distance 0

And so, SD-WAN selects W_INET_11, which is the first member in the list that meets the SLA:

INTERNAL USE ONLY

65

Managed Secure SD-WAN 6.4.4 branch1_fgt # diagnose sys sdwan service 1 Service(1): Address Mode(IPV4) flags=0x200   Gen(6), TOS(0x0/0x0), Protocol(0: 1->65535), Mode(sla), sla-compare-order   Member sub interface(3):     1: seq_num(1), interface(W_INET_11):        1: W_INET_11_0(32)   Members(3):     1: Seq_num(1 W_INET_11), alive, sla(0x1), gid(1), cfg_order(0), cost(0), selected     2: Seq_num(3 W_MPLS_12), alive, sla(0x1), gid(1), cfg_order(1), cost(0), selected     3: Seq_num(1 W_INET_11_0), alive, sla(0x0), gid(2), cfg_order(0), cost(0), selected   Src address(1):   10.0.0.0-10.255.255.255   Dst address(1):   10.0.0.0-10.255.255.255

Clearly, this is not what we want. So here we come with another highly-recommended best-practice that you should always configure, when you deal with a mix of DIA and RIA or even just with a mix of underlays and overlays. In other words: just do it always! Add a blackhole route summarizing all your corporate prefixes (e.g. with a RFC1918 summary). In our case:

config router static edit 101 set dst 10.0.0.0 255.0.0.0 set blackhole enable next end

Blackhole routes can be seen as “interrupting” your route lookup: when you try to find a route to the destination, once you hit the blackhole route, you stop and report failure. So, when SD-WAN will check whether there is a valid route to 10.0.2.0/24, it will hit the blackhole route, and the answer will be “no”. But of course, any Internet destination is outside of the 10.0.0.0/8 summary, so it will find the default route (0.0.0.0/0) without hitting the blackhole route, and Internet access will work just fine. Add the above blackhole route to your CLI template “03-ALL-Edge-Default”. You can use the following Postman requests: West Region - Dual-Hub / SD-WAN / 8 - Add CLI Template - Defaults

This request also creates another CLI template - “03-ALL-Hub-Default”, just with the blackhole route. We will apply it to the Hubs too, because it’s really a good habit to have it everywhere.

If this routing caveat didn’t convince you, there is also a security-related reason to add blackhole routes. Imagine that all your overlay tunnels are temporarily down, and at that time some corporate traffic arrives. Since none of your BGP routes is available, this traffic will match the default route and will flow unencrypted via one of your underlay links. Most likely, it will not flow too far - it will probably be discarded by the ISP INTERNAL USE ONLY

66

Managed Secure SD-WAN 6.4.4 router at the egress. But nevertheless, it is not ideal. Adding a blackhole route will guarantee that your corporate traffic will never leak to the underlays.

Make sure these templates are assigned to “Edge-West-Template” and “Hub-West-Template” respectively and then install policy on all FGTs. And now, finally, it’s time for testing!

2.5.6. SD-WAN Testing: DIA/RIA • Connect to branch1_fgt and make sure that all the Internet health-checks are meeting the SLA: branch1_fgt # diagnose sys sdwan health-check Internet Health Check(Internet): Seq(3 W_MPLS_12): state(alive), packet-loss(0.000%) latency(88.899), jitter(9.570) s la_map=0x3 Seq(4 W_MPLS_22): state(alive), packet-loss(0.000%) latency(8.051), jitter(1.581) sl a_map=0x3 Seq(5 port1): state(alive), packet-loss(0.000%) latency(6.410), jitter(0.198) sla_m ap=0x3

• Scroll down to the very bottom of “WAN Controller” page and click “Start Internet Traffic” button under BR1-Host, to generate Internet traffic sourced from branch1_client. • Open FMG’s Log View and switch to Real-time log, limit it just to branch1_fgt. You are going to use this view to see which outgoing interface is selected for different types of traffic, as highlighted on the below screenshot (you may want to customize displayed columns, to match the below view):

At the moment, you should see DIA in action: all Internet traffic is nicely flowing via port1. • Use WAN Controller to increase Internet latency on Branch1 beyond SLA Target #1 and confirm on branch1_fgt CLI that the degradation has been detected. INTERNAL USE ONLY

67

Managed Secure SD-WAN 6.4.4 Note that the latency measured by the FGT is higher than what you set in WAN Controller, which is, of course, expected: our SLA probe is sent to a real server in the Internet, so the actual Internet latency is added to the one simulated in our lab. Make sure you adjust the simulated latency so, that only the SLA Target #1 is breached. SLA Target #2 should still be met!

• Now focus on the traffic matching “Business-Critical-HighPriority” SD-WAN Rule (GoToMeeting or Salesforce). Did it switchover to RIA? We hope it did! • What about the traffic matching “Business-Critical-MedPriority” SD-WAN Rule (Office365)? It didn’t! • To force Office365 switchover, raise the simulated latency even further, to breach also the SLA Target #2. Confirm that the traffic has switched over. • Now check what happens to the rest of the Internet traffic, even now that port1 breaches all our SLA targets. As you can see, it still sticks to port1. • Now use WAN Controller to bring Internet connection completely down on branch1_fgt. What will happen now to the non-critical traffic (such as Twitter or Facebook)? As you can see, it will now hit the implicit rule (0)! Which will simply load-balance the traffic across all available members. • Restore Internet connection on branch1_fgt and remove all the simulated latency. Make sure that all Internet traffic is now back on port1.

INTERNAL USE ONLY

68

Managed Secure SD-WAN 6.4.4

2.6. Cross-Overlay Communication Use “WAN Controller” to: • Bring down BR1-MPLS link • Bring down BR2-ISP1 link What we get now is that branch1_fgt is connected only to the Internet, while branch2_fgt is connected only to MPLS. It is no longer possible to build a shortcut between them, but it should still be possible for them to communicate via the Hub! Connect to branch1_client via SSH and ping branch2_client:

root@branch1-client-cli:~# ping 10.0.2.101 PING 10.0.2.101 (10.0.2.101) 56(84) bytes of data. 64 bytes from 10.0.2.101: icmp_seq=1 ttl=61 time=14.8 ms 64 bytes from 10.0.2.101: icmp_seq=2 ttl=61 time=10.4 ms 64 bytes from 10.0.2.101: icmp_seq=3 ttl=61 time=10.2 ms

As expected, the traffic flows (it’s easy to confirm that it flows via the Hub). Take a look at the routing table on branch1_fgt:

branch1_fgt # get router info routing-table all Codes: K - kernel, C - connected, S - static, R - RIP, B - BGP O - OSPF, IA - OSPF inter area N1 - OSPF NSSA external type 1, N2 - OSPF NSSA external type 2 E1 - OSPF external type 1, E2 - OSPF external type 2 i - IS-IS, L1 - IS-IS level-1, L2 - IS-IS level-2, ia - IS-IS inter area * - candidate default Routing table for VRF=0 ... B 10.0.2.0/24 [200/0] via W_INET_11), 00:04:20 [200/0] via W_INET_11), 00:04:20 S 10.200.0.0/14 [10/0] is [10/0] is

10.202.1.3 (recursive is directly connected, 10.202.2.3 (recursive is directly connected, directly connected, W_INET_11 directly connected, W_INET_21

As you can see, the only remaining routes to 10.0.2.0/24 have BGP NHs in 10.202.0.0/16, which indicates that they were originated over W_MPLS (to be precise - one via dc1_fgt, another via dc2_fgt). But at the moment, branch1_fgt is not connected to W_MPLS overlays, since its MPLS link is down. Hence, normally, it wouldn’t be able to resolve these BGP NHs. But here help our static routes that we have added exactly for this case:

INTERNAL USE ONLY

69

Managed Secure SD-WAN 6.4.4 config router static # Cross-overlay BGP NH reachability edit 102 set dst 10.200.0.0 255.252.0.0 set device "W_INET_11" set comment "Cross-overlay BGP NH next edit 103 set dst 10.200.0.0 255.252.0.0 set device "W_MPLS_12" set comment "Cross-overlay BGP NH next edit 104 set dst 10.200.0.0 255.252.0.0 set device "W_INET_21" set comment "Cross-overlay BGP NH next edit 105 set dst 10.200.0.0 255.252.0.0 set device "W_MPLS_22" set comment "Cross-overlay BGP NH next end

reachability"

reachability"

reachability"

reachability"

Now you see how they provide reachability information for the overlays to which our device is not connected (either temporarily or permanently). You can double-check that the BGP NHs are recursively resolved using one of these static routes:

get router info routing-table details 10.202.1.3 get router info routing-table details 10.202.2.3

You might ask why we only see the route via W_INET_11 (which belongs to dc1_fgt). What about W_INET_21 (which belongs to dc2_fgt)? It is still up, and there is a static route for it too! The answer is that the current FOS release does not support ECMP for recursive NH resolution. Hence, only the first available entry is picked from our static route. That being said, if dc1_fgt goes down, its route will disappear, and so dc2_fgt route will be picked. This is inline with Active/Backup Hub behavior, so it fulfills our needs here.

You might also ask why we used static routes, rather then letting the Hubs advertise these aggregates to all Edges using BGP. The answer is that the current FOS release does not support recursive resolution of a BGP route via another BGP route.

Use “WAN Controller” to bring all the links up again.

INTERNAL USE ONLY

70

Managed Secure SD-WAN 6.4.4

2.7. Inter-DC Connectivity So far we have been conveniently avoiding one tiny issue that exists in our setup. Connect to the DC1_Host and try pinging DC2_Host:

root@dc1-host:~# ping 10.2.0.7 PING 10.2.0.7 (10.2.0.7) 56(84) bytes of data. ^C --- 10.2.0.7 ping statistics --4 packets transmitted, 0 received, 100% packet loss, time 3060ms

Unfortunately, it doesn’t work. If you check the routing table on any of the Hubs, you won’t find the LAN subnet of the other Hub there. No wonder, because we do not advertise any prefixes between the Hubs! In the real-world network, there could be an additional inter-DC link for this connectivity, possibly with a separate IGP instance (e.g. OSPF). But another alternative is to interconnect the DCs via our Hubs using the same two underlays! FMG VPN Manager supports building Hub-to-Hub tunnels, when VPN Community contains two Hubs. All we need to do is to specify a Hub-to-Hub underlay port for each Hub, right here: However, in this lab we are going to implement a full-mesh of tunnels between all Regional Hubs, when we expend our topology to multiple regions. So we will get a tunnel between DC1 Hub and DC2 Hub as part of that full-mesh. Therefore, let’s park this issue for now…

INTERNAL USE ONLY

71

Managed Secure SD-WAN 6.4.4

3. East Region In this chapter, we are going to add the East Region to our solution. There is nothing new to learn here, because East Region follows exactly the same design and conventions as the West Region, it is just simpler (because it has only one Hub). At first, the East Region will be completely separated from the West Region. We will interconnect them in the next chapter - and that’s where the real fun will come… So this chapter is going to be very short, and we are going to use our Postman requests to automate it all. Use “Runner” to run this entire folder: East Region - Single-Hub

Quick summary of what it did: • Added two new FGTs for the East Region to the FMG, to the two new Device Groups - “Edge-East” and “Hubs-East” • Created two new Dial-Up VPN Communities - E_INET and E_MPLS, adding the new devices there • Created two new CLI Template Groups - Edge-East-Template and Hubs-East-Template, as well as a set of new CLI Templates for overlay and routing configuration (this includes also the default route with set sdwan enable and the blackhole route from the previous chapter) • Created a new SD-WAN Template - “Edge-East”, added the rules for the new region • Created two new policy packages with the necessary firewall rules - “Edge-East-PP” and “HubEast-PP” Installing the policy is all that is left to do: • Install “Edge-East-PP” package on the new Edge • Install “Hub-East-PP” package on the new Hub Congratulations! You have successfully completed configuring the new region!

INTERNAL USE ONLY

72

Managed Secure SD-WAN 6.4.4

INTERNAL USE ONLY

73

Managed Secure SD-WAN 6.4.4

4. Inter-Region In this chapter we are going to interconnect our regions into a single SD-WAN/ADVPN topology. We will build a full mesh of tunnels between all the Regional Hubs, exchange the routes and demonstrate cross-regional shortcuts - direct tunnels between Edge devices from different regions.

4.1. Overlay 4.1.1. Define VPN Communities We are going to build a full mesh of IPSEC tunnels between all our three Hubs, via both our underlays (Internet and MPLS). For this purpose, we will add two new full-mesh VPN Communities, adding all our Hubs to each one of them. But first, create a new Normalized Interface called “hub2hub-overlay”:

We will use it for our convenience, when configuring firewall policies. Now let’s create the VPN communities. Let’s call them “WE_INET” and “WE_MPLS” (“WE” stands for “West-East”):

INTERNAL USE ONLY

74

Managed Secure SD-WAN 6.4.4

Use the following parameters: Parameter

Value

VPN Topology

Full Meshed

Authentication

Pre-shared Key (auto-generated)

IKE Version

2

IKE SA Proposals

AES256/SHA256, AES256GCM/PRFSHA384

IPSEC SA Proposals

AES256/SHA256, AES256GCM

VPN Zone

ON, Full Meshed = hub2hub-overlay

Dead Peer Detection

On Idle

dpd-retrycount

2

dpd-retryinterval

10

Note the “VPN Zone” parameter, we haven’t used it before. VPN Manager will automatically assign all the generated tunnel interfaces to the specified Zone. Then we can use that Zone in the firewall policies, without the need to specify individual tunnel interfaces. This is particularly handy in full-mesh INTERNAL USE ONLY

75

Managed Secure SD-WAN 6.4.4

topologies: if we add more regions to our solution, we will not need to update the firewall policies, because the new tunnel interfaces will be automatically added to the same Zone! After creating the VPN Communities, we will need to add our Hubs to them. We will have to apply our trick with Hub IDs, because also here we must ensure that the tunnel interface names are predictable - we will need to adjust their configuration via CLI templates. Hence, we will add the Hubs using API. Please execute the following Postman requests: West-East Inter-Region / 2 - Add Inter-Region VPN Communities West-East Inter-Region / 3 - Add Hubs to Inter-Region VPN Communities

If you examine the second request, you will see that we set Hub IDs as follows: Community

Hub

HEX ID

Decimal ID

WE_INET

dc1_fgt

1A

26

WE_MPLS

dc1_fgt

1B

27

WE_INET

dc2_fgt

2A

42

WE_MPLS

dc2_fgt

2B

43

WE_INET

dc3_fgt

3A

58

WE_MPLS

dc3_fgt

3B

59

We hope you appreciate our creativity!

4.1.2. Adjust Overlay Configuration As you already know, there are two things that we must adjust using CLI Templates: • Configure tunnel interface IPs. • Enable ADVPN Please execute the following Postman request, adding two templates “04-E-Hub-Overlay-EW” and “04-W-Hub-Overlay-WE” - one for the Hubs of each region: West-East Inter-Region / 4 - Add CLI Template - Inter-Region Overlay

We set all the tunnel interface IPs manually, there is no IKE Config Mode to help us here. The convention is not particularly important, but remember that we must stick to our summary: all tunnel subnets over the Internet must fall under 10.201.0.0/16, all those over MPLS - under 10.202.0.0/16.

If you look carefully, you will notice that we also disable RPF check on all the inter-regional tunnels: set src-check disable . This is done in order to make your life a bit simpler, when you deal with crossoverlay communication between regions. We will return to this in the respective section.

INTERNAL USE ONLY

76

Managed Secure SD-WAN 6.4.4

As for the ADVPN - we enable all the options, including auto-discovery-forwarder :

config vpn ipsec phase1-interface edit "WE_INET_1a" set auto-discovery-sender enable set auto-discovery-receiver enable set auto-discovery-forwarder enable next # ...

Add the new templates to the groups “Hub-West-Template” and “Hub-East-Template” respectively.

4.2. Routing 4.2.1. Design Overview We will extend our BGP topology across the regions, so that Regional Hubs will exchange the Edges’ prefixes. It may sound logical to use EBGP, assigning each region a separate autonomous system number (AS). We will do it for two reasons: • BGP NH preservation. To support inter-regional ADVPN shortcuts, we must preserve BGP NH endto-end between the Edge devices from different regions. While it is possible to achieve that with EBGP, it would require certain not-so-standard tweaks. On the other hand, IBGP does it naturally. • BGP ADD-PATH. You already know that we must enable BGP ADD-PATH on Hubs, to advertise multiple routes to the same destination to Edges. But the same requirement exists also between Hubs of different regions. For instance, an Edge device in East Region (branch3_fgt) originates two routes towards its LAN prefix - one via E_INET, another via E_MPLS, each having its own BGP NH (Edge’s tunnel IP on the respective overlay). The East Hub (dc3_fgt) must advertise both these routes to the Hubs of West Region. So we must enable BGP ADD-PATH also on the inter-regional BGP sessions. At the moment, FOS does not support ADD-PATH for EBGP. For the above reasons, we will extend IBGP across the regions, so that our entire SD-WAN/ADVPN topology will be a single AS (a single IBGP domain). This means that the Regional Hubs will reflect routes not only to their Edges, but also to the remote Regional Hubs:

INTERNAL USE ONLY

77

Managed Secure SD-WAN 6.4.4

But the choice of IBGP brings also difficulties with it. One important disadvantage of using IBGP is that we lose built-in loop prevention mechanism on regional level. Looking at the above diagram, consider Branch1 LAN prefix: • branch1_fgt advertises it to dc1_fgt • dc1_fgt reflects it to dc3_fgt • dc3_fgt reflects it to all its clients, hence also to dc2_fgt! This is not depicted on the diagram, because it shows the design that we want to have. Unfortunately, if we don’t take measures, this unwanted reflection will happen. Clearly, we do not want East Region Hub to reflect routes back into West Region! With EBGP, this problem would not exist, since EBGP would not advertise anything back to the same AS from where it came. But with IBGP, we will have to use BGP Communities to identify each region, so that the Hubs only advertise their local regional routes outside, but never act as “transit Hubs” between regions. As you will see shortly, the Hubs will not need to know about BGP Communities of all remote regions. They will only need to know their own ones. In fact, these communities will have only local significance on each Hub: it will attach them to all prefixes received from within its region and then match them when advertising routes to remote regions. This way, any route learnt from a remote region will neither be reflected back, nor will it be reflected to any other remote region - simply because it will not have the local region’s community attached to it. We will repeat this below with a specific example.

4.2.2. Configure Hubs Use the following Postman request to add CLI Tempaltes “05-W-Hub-Routing-WE” and “05-E-HubRouting-EW” - one for the Hubs of each region:

INTERNAL USE ONLY

78

Managed Secure SD-WAN 6.4.4 West-East Inter-Region / 5 - Add CLI Template - Inter-Region Routing

Here is what you will find inside “05-W-Hub-Routing-WE”: • Each Hub defines two BGP Communities for its own region, one for the Edge prefixes (“WEST_EDGE”) and one for the DC prefixes behind Hub itself (“WEST_DC”). • We also define our overlay summaries - NH_INET and NH_MPLS; you are already familiar with them. • We define two ingress route-maps - “WEST_EDGE_IN” and “WEST_DC_IN” - that will attach the local community to all routes that we learn from within the region, as follows: ◦ The routes from the Edge devices of West Region arrive via the neighbor-groups “EDGE_INET” and “EDGE_MPLS”, hence we apply route-map-in as follows:

config router bgp config neighbor-group # Mark Edge prefixes with regional community edit "EDGE_INET" set route-map-in "WEST_EDGE_IN" next edit "EDGE_MPLS" set route-map-in "WEST_EDGE_IN" next end end

◦ The routes for the local DC LAN are defined under config network , hence we apply the route-map there, as follows:

config router bgp config network edit 1 # Mark DC prefixes with regional community set route-map "WEST_DC_IN" next end end

If we had any additional BGP peering within the local region, such as with an existing customer DC router, we would apply

route-map-in WEST_DC_IN also on that peering. Our goal is that ALL

routes originated within West Region will be now marked with either WEST_DC or WEST_EDGE communitiy.

INTERNAL USE ONLY

79

Managed Secure SD-WAN 6.4.4

◦

We define two egress route-maps - “INET_OUT” and “MPLS_OUT” - that will filter routes that we advertise to remote Regional Hubs (in this case, to dc3_fgt): ▪ We advertise routes with WEST_EDGE community, but only within the same overlay type; for example, in route-map “INET_OUT”:

edit 1 set match-community "WEST_EDGE" set match-ip-nexthop "NH_INET" next

▪ We advertise all routes with WEST_DC community:

edit 2 set match-community "WEST_DC" next

▪ We do not advertise anything else. In other words, the only routes that we advertise to remote regions are those originated in the West Region: either Edge routes or local DC routes. And for the former, we also ensure that we avoid cross-overlay advertisements: routes from W_INET get reflected only to WE_INET, routes from W_MPLS get reflected only to WE_MPLS. ◦ We define BGP neighbors over each of the inter-regional Hub-to-Hub tunnels and apply our egress route-maps:

config router bgp config neighbor # Inter-regional tunnels edit "10.201.10$(dc-id).2" set soft-reconfiguration enable set advertisement-interval 1 set link-down-failover enable set connect-timer 1 set remote-as 65000 set additional-path both set adv-additional-path 2 set next-hop-self enable set interface "WE_INET_3a" set route-map-out "INET_OUT" set route-reflector-client enable next # ... end end

INTERNAL USE ONLY

80

Managed Secure SD-WAN 6.4.4

What else can we find in the CLI Template? • We add “overlay stickiness” rules for all the inter-regional traffic. We must include all possible combinations

between

our

W_*

and

WE_*

interfaces.

This

you

can

see

under

config router policy section.

• We must also add static routes towards the tunnel overlays of the remote region, otherwise we will not be able to resolve the BGP NHs learnt from there:

config router static edit 104 set dst 10.201.3.0 255.255.255.0 set device "WE_INET_3a" set comment "Cross-region BGP NH reachability (INET)" next edit 105 set dst 10.202.3.0 255.255.255.0 set device "WE_MPLS_3b" set comment "Cross-region BGP NH reachability (MPLS)" next end

It would be better to learn these subnets from dc3_fgt via BGP, but you already know that the current FOS release does not support recursive resolution of BGP route via another BGP route.

• Last but not least, we recall our promise and handle route advertisement between the two DCs of the West Region - namely, between dc1_fgt and dc2_fgt. This peering MUST NOT be used for any inter-regional or Edge route advertisement. The only purpose of this peering is to provide connectivity between DC1 and DC2! Hence, we apply a strict route-map, using our “WEST_DC” community:

INTERNAL USE ONLY

81

Managed Secure SD-WAN 6.4.4 config router route-map edit "PEERHUB_OUT" config rule edit 1 set match-community "WEST_DC" # Prevent duplicate readvertisement to Edges and remote regions set set-community no-advertise set set-community-additive enable next end next end config router bgp config neighbor # Peer tunnels edit "10.201.10.$(peer-dc-id)" set interface "WE_INET_$(peer-dc-id)a" set route-map-out "PEERHUB_OUT" next # ... end end

Note a special

no-advertise

community that we add in the route-map! This instructs the

receiving Hub that it should NOT reflect this route anywhere. To recap: ◦ dc1_fgt advertises DC1 LAN routes to dc2_fgt ◦ We DO NOT want dc2_fgt to complicate things by reflecting it towards dc3_fgt! ◦ We do want DC1 LAN to be reachable from dc3_fgt, but we let dc1_fgt handle it by itself; as we saw before, it will advertise DC1 LAN to dc3_fgt over its own Hub-to-Hub tunnel with dc3_fgt

Yes, we know that it is complex. We recommend that you spend some time looking at the routing tables in more detail, after you push the configuration!

Take a look also at “05-E-Hub-Routing-EW”, it follows the same design. Add both CLI Templates to the respective groups. If you did it right all the way, you will end up with the following:

INTERNAL USE ONLY

82

Managed Secure SD-WAN 6.4.4

4.3. Firewall Policy The last piece of the configuration is the firewall policy. We do not need to add any new rules this time. We only need to update the existing ones, adding

hub2hub-overlay

Zone to the rules

“Corporate” and “Heath-Check”. For example, this is how these rules will look like in West Region:

INTERNAL USE ONLY

83

Managed Secure SD-WAN 6.4.4

Name Corporate

From

To

Src

Dst

Service

NAT

Action

W_INET

W_INET

all

all

ALL

No

Accept

W_MPLS lan

W_MPLS

hub2hub-

lan

overlay

hub2hub-

all

HC

PING

No

Accept

overlay Health-check

W_INET

any

W_MPLS hub2huboverlay After making this small change in both “Hub-West-PP” and “Hub-East-PP” packages, you can finally reinstall policy on the Hubs of both regions. After that, connect to all the Hubs and run execute router restart , in order for our ingress routemaps to take effect on the previously established BGP sessions with the Edges.

Note that you do not need to push any new configuration to the Edge devices! Inter-regional connectivity is done solely on the Hubs. Edge configuration does not depend on remote regions!

4.4. Inter-regional ADVPN In this section we will test inter-regional connectivity with ADVPN shortcuts. There are not one, but tw

o types of shortcuts that we can demonstrate between the regions!

4.4.1. Edge-to-Edge Connect to branch1_client via SSH and start pinging branch3_client:

root@branch1-client-cli:~# ping 10.0.3.101 PING 10.0.3.101 (10.0.3.101) 56(84) bytes of data. 64 bytes from 10.0.3.101: icmp_seq=1 ttl=60 time=16.2 64 bytes from 10.0.3.101: icmp_seq=2 ttl=62 time=5.63 64 bytes from 10.0.3.101: icmp_seq=3 ttl=62 time=6.05 64 bytes from 10.0.3.101: icmp_seq=4 ttl=62 time=5.81 64 bytes from 10.0.3.101: icmp_seq=5 ttl=62 time=6.02 ...

ms ms ms ms ms

From the increased TTL value after the first (few) packet(s), you can guess that there is now ADVPN shortcut. Indeed, you can find it between branch1_fgt and branch3_fgt:

INTERNAL USE ONLY

84

Managed Secure SD-WAN 6.4.4 branch1_fgt # get ipsec tunnel list NAME REMOTE-GW PROXY-ID-SOURCE TIMEOUT W_INET_11_0 203.0.115.1:0 10.201.1.3/10.201.1.3 1558 ...

branch3_fgt # get ipsec tunnel list NAME REMOTE-GW PROXY-ID-SOURCE TIMEOUT E_INET_0_0 192.2.0.1:0 0.0.0.0/255.255.255.255 1583 ...

PROXY-ID-DESTINATION

STATUS

0.0.0.0/0.0.0.0

up

PROXY-ID-DESTINATION

STATUS

10.201.1.3/10.201.1.3

up

Hence, the traffic is now flowing over a direct tunnel between the two Edge devices in different regions! Let’s investigate how the routing has propagated. • Stop the ping and then clear the shortcut on branch1_fgt, to go back to the initial state

diagnose vpn ike gateway clear name W_INET_11_0

• We know that branch3_fgt advertises its local LAN subnet (10.0.3.0/24) to its Regional Hub dc3_fgt over both overlays (E_INET and E_MPLS). We can confirm this on dc3_fgt:

dc3_fgt # get router info routing-table bgp Routing table for VRF=0 ... B 10.0.3.0/24 [200/0] via 10.201.3.2, E_INET_0, 10:32:35 [200/0] via 10.202.3.2, E_MPLS_0, 10:32:35

• Thanks to our ingress route-map on dc3_fgt, EAST_EDGE community is attached to both these routes:

dc3_fgt # get router info bgp community-list EAST_EDGE VRF 0 BGP table version is 19, local router ID is 10.3.0.1 Status codes: s suppressed, d damped, h history, * valid, > best, i - internal, S Stale Origin codes: i - IGP, e - EGP, ? - incomplete Network *>i10.0.3.0/24 *>i

Next Hop 10.201.3.2 10.202.3.2

Metric LocPrf Weight RouteTag Path 0 100 0 0 i 0 100 0 0 i

Total number of prefixes 1

INTERNAL USE ONLY

85

Managed Secure SD-WAN 6.4.4

•

With this community attached, the routes match our egress route-maps - INET_OUT and MPLS_OUT, one BGP NH matching each of them (remember that we match not only by the community, but also but the overlay subnet):

dc3_fgt # get router info bgp route-map INET_OUT VRF 0 BGP table version is 19, local router ID is 10.3.0.1 Status codes: s suppressed, d damped, h history, * valid, > best, i - internal, S Stale Origin codes: i - IGP, e - EGP, ? - incomplete Network *>i10.0.3.0/24 ...

Next Hop 10.201.3.2

Metric LocPrf Weight RouteTag Path 0 100 0 0 i

dc3_fgt # get router info bgp route-map MPLS_OUT VRF 0 BGP table version is 19, local router ID is 10.3.0.1 Status codes: s suppressed, d damped, h history, * valid, > best, i - internal, S Stale Origin codes: i - IGP, e - EGP, ? - incomplete Network *>i10.0.3.0/24 ...

Next Hop 10.202.3.2

Metric LocPrf Weight RouteTag Path 0 100 0 0 i

• And so, the route to 10.0.3.0/24 is advertised to dc1_fgt, as follows: ◦ Over WE_INET tunnel with BGP NH = 10.201.3.2 ◦ Over WE_MPLS tunnel with BGP NH = 10.202.3.2 • We confirm this on dc1_fgt:

dc1_fgt # get router info bgp neighbors 10.201.101.2 routes VRF 0 BGP table version is 11, local router ID is 10.1.0.1 Status codes: s suppressed, d damped, h history, * valid, > best, i - internal, S Stale Origin codes: i - IGP, e - EGP, ? - incomplete Network *>i10.0.3.0/24 ...

Next Hop 10.201.3.2

Metric LocPrf Weight RouteTag Path 0 100 0 0 i

dc1_fgt # get router info bgp neighbors 10.202.101.2 routes VRF 0 BGP table version is 11, local router ID is 10.1.0.1 Status codes: s suppressed, d damped, h history, * valid, > best, i - internal, S Stale Origin codes: i - IGP, e - EGP, ? - incomplete Network *>i10.0.3.0/24 ...

Next Hop 10.202.3.2

Metric LocPrf Weight RouteTag Path 0 100 0 0 i

INTERNAL USE ONLY

86

Managed Secure SD-WAN 6.4.4

•

Since dc1_fgt is not directly connected neither to E_INET nor to E_MPLS overlay, it would not be able to resolve the BGP NHs, neither 10.201.3.2 nor 10.202.3.2. But luckily, we have added our static routes for cross-regional resolution, so you can see the result below:

dc1_fgt # get router info routing-table all Codes: K - kernel, C - connected, S - static, R - RIP, B - BGP O - OSPF, IA - OSPF inter area N1 - OSPF NSSA external type 1, N2 - OSPF NSSA external type 2 E1 - OSPF external type 1, E2 - OSPF external type 2 i - IS-IS, L1 - IS-IS level-1, L2 - IS-IS level-2, ia - IS-IS inter area * - candidate default Routing table for VRF=0 ... B 10.0.3.0/24 [200/0] via 10.201.3.2 (recursive via 10.201.101.2, WE_INET_3a), 10:39:32 [200/0] via 10.202.3.2 (recursive via 10.202.101.2, WE_MPLS_3b), 10:39:32 S 10.201.3.0/24 [10/0] via 10.201.101.2, WE_INET_3a S 10.202.3.0/24 [10/0] via 10.202.101.2, WE_MPLS_3b ...

• Now that dc1_fgt can resolve the routes properly, it will reflect them to its Edge devices, and branch1_fgt among them. Note that we do not filter cross-overlay advertisements on Hubs, hence both BGP NHs will be reflected via both W_INET and W_MPLS overlays. But the Edges will give weight=200 to the routes from the same overlay type. We can confirm this on branch1_fgt (two routes learnt via each overlay, one with weight=200 in each pair):

branch1_fgt # get router info bgp neighbors 10.201.1.1 routes VRF 0 BGP table version is 11, local router ID is 10.0.1.1 Status codes: s suppressed, d damped, h history, * valid, > best, i - internal, S Stale Origin codes: i - IGP, e - EGP, ? - incomplete Network * i10.0.3.0/24 *>i ...

Next Hop 10.202.3.2 10.201.3.2

Metric LocPrf Weight RouteTag Path 0 100 0 0 i 0 100 200 0 i

branch1_fgt # get router info bgp neighbors 10.202.1.1 routes VRF 0 BGP table version is 11, local router ID is 10.0.1.1 Status codes: s suppressed, d damped, h history, * valid, > best, i - internal, S Stale Origin codes: i - IGP, e - EGP, ? - incomplete Network *>i10.0.3.0/24 * i ...

Next Hop 10.202.3.2 10.201.3.2

Metric LocPrf Weight RouteTag Path 0 100 200 0 i 0 100 0 0 i

• Since branch1_fgt is also not connected directly neither to E_INET nor to E_MPLS, it would not be able to resolve the received BGP NHs. But it helps again having our overlay subnets summarized INTERNAL USE ONLY

87

Managed Secure SD-WAN 6.4.4

by the type of overlay (10.201.0.0/16 and 10.202.0.0/16), so the static routes that we’ve added long time ago help us also this time:

branch1_fgt # get router info routing-table all Codes: K - kernel, C - connected, S - static, R - RIP, B - BGP O - OSPF, IA - OSPF inter area N1 - OSPF NSSA external type 1, N2 - OSPF NSSA external type 2 E1 - OSPF external type 1, E2 - OSPF external type 2 i - IS-IS, L1 - IS-IS level-1, L2 - IS-IS level-2, ia - IS-IS inter area * - candidate default Routing table for VRF=0 B 10.0.3.0/24 [200/0] via W_INET_11), 02:22:30 [200/0] via W_INET_11), 02:22:30 [200/0] via W_MPLS_12), 02:22:30 [200/0] via W_MPLS_12), 02:22:30 S 10.201.0.0/16 [10/0] is [10/0] is S 10.202.0.0/16 [10/0] is [10/0] is

10.201.3.2 (recursive is directly connected, 10.201.3.2 (recursive is directly connected, 10.202.3.2 (recursive is directly connected, 10.202.3.2 (recursive is directly connected, directly directly directly directly

connected, connected, connected, connected,

W_INET_11 W_INET_21 W_MPLS_12 W_MPLS_22

You can see that 10.0.3.0/24 is recursively resolved via DC1 overlays (W_INET_11, W_MPLS_12). You might ask why we do not see the path via DC2 overlays - W_INET_21 and W_MPLS_22. Our static routes list them, so it sounds like we should get 10.0.3.0/24 recursively resolved via DC2 overlays as well. The answer is that the current FOS release does not support ECMP for recursive NH resolution. Hence, only the first entry is picked from our static route. That being said, if DC1 Hub goes down, its routes will disappear from the static route, and so DC2 routes will be picked. This is inline with Active/Backup Hub behavior, so it fulfills our needs here.

So now you know how the route to 10.0.3.0/24 has propagated from the East Region all the way to branch1_fgt in the West Region, while preserving its original BGP NH values. If you restart the ping and let the shortcut come up again, you will see how the route gets resolved via the newly established shortcut, thanks to the /32 route injected by ADVPN as part of its standard operation (this is no different from a “usual” ADVPN shortcut within the region):

INTERNAL USE ONLY

88

Managed Secure SD-WAN 6.4.4 branch1_fgt # get router info routing-table all Codes: K - kernel, C - connected, S - static, R - RIP, B - BGP O - OSPF, IA - OSPF inter area N1 - OSPF NSSA external type 1, N2 - OSPF NSSA external type 2 E1 - OSPF external type 1, E2 - OSPF external type 2 i - IS-IS, L1 - IS-IS level-1, L2 - IS-IS level-2, ia - IS-IS inter area * - candidate default Routing table for VRF=0 B 10.0.3.0/24 [200/0] via 10.201.3.2, W_INET_11_0, 00:00:20 [200/0] via 10.201.3.2, W_INET_11_0, 00:00:20 [200/0] via 10.202.3.2 (recursive is directly connected, W_MPLS_12), 00:00:20 [200/0] via 10.202.3.2 (recursive is directly connected, W_MPLS_12), 00:00:20 C 10.201.3.2/32 is directly connected, W_INET_11_0 S 10.201.0.0/16 [10/0] is directly connected, W_INET_11 [10/0] is directly connected, W_INET_21 S 10.202.0.0/16 [10/0] is directly connected, W_MPLS_12 [10/0] is directly connected, W_MPLS_22

4.4.2. Edge-to-Hub From the same branch1_client, ping a host behind DC3:

root@branch1-client-cli:~# ping 10.3.0.7 PING 10.3.0.7 (10.3.0.7) 56(84) bytes of data. 64 bytes from 10.3.0.7: icmp_seq=1 ttl=61 time=8.54 64 bytes from 10.3.0.7: icmp_seq=2 ttl=62 time=5.87 64 bytes from 10.3.0.7: icmp_seq=3 ttl=62 time=6.10 64 bytes from 10.3.0.7: icmp_seq=4 ttl=62 time=5.97

ms ms ms ms

Also this time we notice a shortcut, judging by the increased TTL value. This time, it’s a shortcut going to dc3_fgt (the East Region Hub!):

branch1_fgt # diagnose vpn ike gateway clear name W_INET_11_2 branch1_fgt # get ipsec tunnel list NAME REMOTE-GW PROXY-ID-SOURCE PROXY-ID-DESTINATION TIMEOUT W_INET_11_1 100.64.3.1:0 10.201.1.3/10.201.1.3 0.0.0.0/0.0.0.0 1740 ...

STATUS up

You can easily confirm that the traffic is now flowing via a direct tunnel between branch1_fgt and dc3_fgt. Feel free to repeat the routing analysis, similar to what we did for Edge-to-Edge case. This time it will be even shorter: effectively, dc3_fgt acts pretty much as an Edge device from dc1_fgt perspective.

INTERNAL USE ONLY

89

Managed Secure SD-WAN 6.4.4

4.5. Cross-Overlay Communication Use “WAN Controller” to: • Bring down BR1-MPLS link • Bring down BR3-ISP1 link We have done something similar before, but this time we will see it working between different regions! What we get now is that branch1_fgt is connected only to the Internet, while branch3_fgt is connected only to MPLS. It is no longer possible to build a shortcut between them, but it should still be possible for them to communicate… via two Hubs! Connect to branch1_client via SSH and ping branch3_client:

ping 10.0.3.101

As expected, the traffic will flow via two Regional Hubs: dc1_fgt and dc3_fgt. Let’s dig into it: • branch1_fgt is only connected to the Internet, so no wonder that it can reach our destination (10.0.3.101) only via its Internet overlay:

branch1_fgt # get router info routing-table all Codes: K - kernel, C - connected, S - static, R - RIP, B - BGP O - OSPF, IA - OSPF inter area N1 - OSPF NSSA external type 1, N2 - OSPF NSSA external type 2 E1 - OSPF external type 1, E2 - OSPF external type 2 i - IS-IS, L1 - IS-IS level-1, L2 - IS-IS level-2, ia - IS-IS inter area * - candidate default Routing table for VRF=0 B 10.0.3.0/24 [200/0] via W_INET_11), 00:37:28 [200/0] via W_INET_11), 00:37:28 S 10.200.0.0/14 [10/0] is [10/0] is ...

10.202.3.2 (recursive is directly connected, 10.202.3.2 (recursive is directly connected, directly connected, W_INET_11 directly connected, W_INET_21

The only available route towards 10.0.3.0/24 is from BGP NH = 10.202.3.2, which is branch3_fgt IP on its E_MPLS overlay. That makes sense. You already know how we resolve BGP NHs when we are not connected to the same overlay using our special static aggregates. They are still valid for this case! And you also remember why dc2_fgt routes are not shown, but that is not bothering us much. You can confirm with the sniffer that the traffic flows via W_INET_11. • On dc1_fgt:

INTERNAL USE ONLY

90

Managed Secure SD-WAN 6.4.4 dc1_fgt # get router info routing-table all Codes: K - kernel, C - connected, S - static, R - RIP, B - BGP O - OSPF, IA - OSPF inter area N1 - OSPF NSSA external type 1, N2 - OSPF NSSA external type 2 E1 - OSPF external type 1, E2 - OSPF external type 2 i - IS-IS, L1 - IS-IS level-1, L2 - IS-IS level-2, ia - IS-IS inter area * - candidate default Routing table for VRF=0 B 10.0.3.0/24 [200/0] via 10.202.3.2 (recursive via 10.202.101.2, WE_MPLS_3b), 00:40:15 S 10.202.3.0/24 [10/0] via 10.202.101.2, WE_MPLS_3b ...

Here, the route is recursively resolved only via WE_MPLS_3b. Why not via WE_INET_3a as well? Because we’ve configured the static route to 10.202.3.0/24 to point only to the MPLS tunnel. You can confirm with the sniffer that the traffic flows via WE_MPLS_3b (yes, we have switched the overlay here!). • On dc3_fgt:

dc3_fgt # get router info routing-table all Codes: K - kernel, C - connected, S - static, R - RIP, B - BGP O - OSPF, IA - OSPF inter area N1 - OSPF NSSA external type 1, N2 - OSPF NSSA external type 2 E1 - OSPF external type 1, E2 - OSPF external type 2 i - IS-IS, L1 - IS-IS level-1, L2 - IS-IS level-2, ia - IS-IS inter area * - candidate default Routing table for VRF=0 B 10.0.3.0/24 [200/0] via 10.202.3.2, E_MPLS_0, 00:44:49 ...

Nothing surprising here, it learns the destination from branch3_fgt via E_MPLS, the only available overlay on that Branch. You can confirm with the sniffer that the traffic flows via E_MPLS_0. • Still on dc3_fgt, let’s check the route back to the source (10.0.1.101):

INTERNAL USE ONLY

91

Managed Secure SD-WAN 6.4.4 dc3_fgt # get router info routing-table all Codes: K - kernel, C - connected, S - static, R - RIP, B - BGP O - OSPF, IA - OSPF inter area N1 - OSPF NSSA external type 1, N2 - OSPF NSSA external type 2 E1 - OSPF external type 1, E2 - OSPF external type 2 i - IS-IS, L1 - IS-IS level-1, L2 - IS-IS level-2, ia - IS-IS inter area * - candidate default Routing table for VRF=0 B 10.0.1.0/24 [200/0] via 10.201.1.2 (recursive via 10.201.101.1, WE_INET_1a), 00:44:56 [200/0] via 10.201.2.2 (recursive via 10.201.102.1, WE_INET_2a), 00:44:56 S 10.201.1.0/24 [10/0] via 10.201.101.1, WE_INET_1a S 10.201.2.0/24 [10/0] via 10.201.102.1, WE_INET_2a ...

Now, this is interesting. The traffic arrived from dc1_fgt via WE_MPLS, as we saw above. But the route back is only via WE_INET! That’s because the BGP NH advertised from branch1_fgt is within 10.201.1.0/24! Our static route back to 10.201.1.0/24 is only pointing to WE_INET. This means that the traffic would be dropped due to RPF check failure, if we would not disable that check on all the inter-regional tunnels. This also means that the reply traffic will cross the regions via WE_INET overlay! So we get a small asymmetry here, which you can confirm in the sniffer:

dc3_fgt # diagnose sniffer packet any "host 10.0.3.101" 4 Using Original Sniffing Mode interfaces=[any] filters=[host 10.0.3.101] 1.023412 WE_MPLS_1b in 10.0.1.101 -> 10.0.3.101: icmp: echo request 1.023431 E_MPLS_0 out 10.0.1.101 -> 10.0.3.101: icmp: echo request 1.024391 E_MPLS_0 in 10.0.3.101 -> 10.0.1.101: icmp: echo reply 1.024406 WE_INET_1a out 10.0.3.101 -> 10.0.1.101: icmp: echo reply ...

To summarize: • Forward direction: branch1_fgt -->W_INET--> dc1_fgt -->WE_MPLS--> dc3_fgt -->E_MPLS--> branch3_fgt

• Reverse direction: branch1_fgt Monitor

and ensure that Branch3_FGT is now fully

functional in our SD-WAN solution. You should see all its SD-WAN members reporting healthy state:

INTERNAL USE ONLY

127

Managed Secure SD-WAN 6.4.4

8. Secure SD-WAN Use Cases 8.1. Zero-Day Malware Outbreak Prevention with FortiSandbox In this chapter we will demonstrate the following use case: 1. A user in Branch 1 downloads a file from the Internet. 2. The file contains malware that is not detected by the branch1_fgt antivirus, but the file is sent to the FortiSandbox. 3. The FortiSandbox rates the file as suspicious/high. 4. The FortiSandbox adds both the file hash and the URL to the corresponding packages and updates all the subscribed FortiGates. 5. Now, a user in Branch 2 tries to download the same file. It is blocked by branch2_fgt based on the URL. 6. The Branch 2 user downloads the same file from a different URL. It is still blocked by branch2_fgt, this time based on the file hash. This use case demonstrates how zero-day malware outbreak can be successfully prevented: malware submitted for the analysis from one branch will be then blocked in all other branches as well.

8.1.1. Configuring the Branches • Configure both Branch FortiGates to use FortiSandbox with IP 90.83.10.150:

• In “Policy Packages”, modify “default” AV profile as follows: ◦ Send Files to FortiSandbox Appliance for Inspection: All Supported Files ◦ Use FortiSandbox Database: Yes

INTERNAL USE ONLY

128

Managed Secure SD-WAN 6.4.4

• Then modify the “default” Web Filter profile as follows: ◦ Block Malicious URLs Discovered by FortiSandbox: Yes ◦ Action for “Newly Observed Domain”": Allow (so that our private domain is not blocked)

INTERNAL USE ONLY

129

Managed Secure SD-WAN 6.4.4

• Apply the two modified profiles to the policy that permits Internet access in Edge-West policy (it’s enough to add it just to the “Internet (DIA)” rule the test). • Install policy on both Edge-West FGTs.

8.1.2. Preparing the Malware We will use “webserver.lab” VM as the “Internet” source of malware. SSH to it and download a fresh fsa_downloader test file to the webserver:

INTERNAL USE ONLY

130

Managed Secure SD-WAN 6.4.4 root@www-webserver-lab:~# cd /var/www/html root@www-webserver-lab:/var/www/html~# rm fsa_downloader* root@www-webserver-lab:/var/www/html~# wget http://rb3.ftnt.io/downloader -O fsa_downloader1.exe

The service that we are using here generates a slightly modified copy of Eicar file every time you try to download from the above URL. As a result, each downloaded file will have a different hash, so we can guarantee that our malware will not be known to FortiSandbox/FortiGates in advance - otherwise it would not be that exciting. Feel free to download a couple of files from the above URL and make sure that their hash is every time different.

8.1.3. Action! Step 1: A user in Branch 1 downloads a file from the Internet. SSH to branch1_client and download the malware from webserver.lab:

root@branch1-client-cli:~# wget http://webserver.lab/fsa_downloader1.exe --2020-02-21 16:20:50-- http://webserver.lab/fsa_downloader1.exe Resolving webserver.lab (webserver.lab)... 128.66.0.1 Connecting to webserver.lab (webserver.lab)|128.66.0.1|:80... connected. HTTP request sent, awaiting response... 200 OK Length: 4096 (4.0K) [application/octet-stream] Saving to: 'fsa_downloader1.exe' fsa_downloader1.exe

100%[==========================>]

4.00K

7.46KB/s

in 0.5s

2020-02-21 16:20:52 (7.46 KB/s) - 'fsa_downloader1.exe' saved [4096/4096]

Note that the file is downloaded successfully. Step 2: The malware is not detected by the branch1_fgt antivirus, but the file is sent to the FortiSandbox. On FMG, go to the Log View and check Antivirus logs. You should shortly see the following:

INTERNAL USE ONLY

131

Managed Secure SD-WAN 6.4.4

As you can see, the file has been submitted to FortiSandbox. Step 3: The FortiSandbox rates the file as suspicious/high. Some time later (might take a few minutes!), additional log entry will appear, confirming FortiSandbox verdict:

The above two steps can also be seen on “FortiView -> Threats -> FortiSandbox Detection” dashboard. Double-click on the entry with the name “fsa_downloader1.exe”:

INTERNAL USE ONLY

132

Managed Secure SD-WAN 6.4.4

Step 4: The FortiSandbox updates all the subscribed FortiGates. For example, browse to the Web UI of branch2_fgt, navigate to “Security Fabric -> Fabric Connectors” and double-click “FortiSandbox”. You will find updated versions of the threat databases and will be able to see even the URL that has been detected on another branch.

INTERNAL USE ONLY

133

Managed Secure SD-WAN 6.4.4

Step 5: On Branch 2, the same file is now blocked based on the URL. Of course, at this point we could try to download the same file again on branch1_client, and it would get blocked (you can try it!). But what is more exciting is that it will be blocked even when we try downloading it on another branch! SSH to branch2_client and try downloading the same file:

root@branch2-client-cli:~# wget http://webserver.lab/fsa_downloader1.exe --2020-04-14 13:47:15-- http://webserver.lab/fsa_downloader1.exe Resolving webserver.lab (webserver.lab)... 128.66.0.1 Connecting to webserver.lab (webserver.lab)|128.66.0.1|:80... connected. HTTP request sent, awaiting response... 403 Forbidden 2020-04-14 13:47:15 ERROR 403: Forbidden.

So, the file has been blocked by branch2_fgt, because it also received the updates from FortiSandbox. The Log View explains the reason - the file has been blocked by the Web Filter, based on the URL:

Check also “FortiView -> Threats -> Top Threats” to find our malicious “webserver.lab” in the list. Step 6: On Branch 2, the same file is also blocked based on the hash. The file has been blocked based on the URL. This is exciting, but not exciting enough. Let us alter the URL. Go back to the “webserver.lab” and rename the file:

root@www-webserver-lab:/var/www/html# mv fsa_downloader1.exe fsa_downloader1_copy.exe

Now go to branch2_client and try downloading the new file: INTERNAL USE ONLY

134

Managed Secure SD-WAN 6.4.4 root@branch2-client-cli:~# wget http://webserver.lab/fsa_downloader1_copy.exe --2020-04-14 13:58:10-- http://webserver.lab/fsa_downloader1_copy.exe Resolving webserver.lab (webserver.lab)... 128.66.0.1 Connecting to webserver.lab (webserver.lab)|128.66.0.1|:80... connected. HTTP request sent, awaiting response... 200 OK Length: 4096 (4.0K) [application/octet-stream] Saving to: 'fsa_downloader1_copy.exe' fsa_downloader1_copy.exe s in 0s

29%[============================>

]

1.17K

--.-KB/

2020-04-14 13:58:11 (126 MB/s) - Read error at byte 1193/4096 (Connection reset by peer). Retrying. --2020-04-14 13:58:12-- (try: 2) http://webserver.lab/fsa_downloader1_copy.exe Connecting to webserver.lab (webserver.lab)|128.66.0.1|:80... connected. HTTP request sent, awaiting response... 403 Forbidden 2020-04-14 13:58:12 ERROR 403: Forbidden.

We can see that download fails, even though the URL is now different. Furthermore, it looks like the client has tried to download the file twice: we can see that the first download attempt was cut in the middle and the second attempt has failed immediately. The Log View shows multiple log entries this time. Reading the Antivirus logs, we can see what happened: • First download attempt could not be blocked based on the URL. But it was interrupted based on the hash of the file being downloaded.

As you can see, the file has been recognized as the one previously found infected by the FortiSandbox.

INTERNAL USE ONLY

135

Managed Secure SD-WAN 6.4.4

•

The next log entry above (“analytics”) indicates that branch2_fgt has submitted the new suspicious data to FortiSandbox (feel free to examine the log entry details)

• The next log entry is identical to the first one - that’s where branch2_fgt blocks the second download attempt • Finally, some time later (but much faster than before!) we receive the response from FortiSandbox, confirming that the new submitted URL is also malicious. Feel free to examine also the FortiView dashboards: “Top Threats” and “FortiSandbox Detection”.

INTERNAL USE ONLY

136

Managed Secure SD-WAN 6.4.4

9. Conclusion Congratulations! You have completed the workshop!

INTERNAL USE ONLY

137

Managed Secure SD-WAN 6.4.4

10. Appendix 10.1. Postman: Dig Into It If you look at each Postman request in our collection, you will notice that writing the request itself is not enough. Sometimes there is something on the “Tests” tab, and sometimes there is something also on the “Pre-request Script” tab.

And if you right-click the collection and then “Edit”, you will even find one simple test that is applied to ALL the requests of the collection:

INTERNAL USE ONLY

138

Managed Secure SD-WAN 6.4.4

Tests and pre-request scripts are particularly important, when you want to run the collection in bulk like we do in Chapter 2. In fact, you can reach the level when you setup the entire environment using Postman Runner - so you can quickly spin up a fully configured demo, for example! The following post on Fortinet Developer Netowrk (FNDN) explains in more detail how we wrote those scripts: Scripting and Bulk Run in Postman By the way, FNDN is also the right place to go for API reference!

INTERNAL USE ONLY

139

Managed Secure SD-WAN 6.4.4

10.2. Reference Configuration: SD-WAN / ADVPN Please refer to this GitHub repository for a complete FOS configuration, templated using Jinja2.

10.2.1. CLI Templates Here we list the CLI Templates used in this lab. 10.2.1.1. Edge-West-Template 01-W-Edge-Overlay:

INTERNAL USE ONLY

140

Managed Secure SD-WAN 6.4.4 # Enable ADVPN config vpn ipsec phase1-interface edit "W_INET_11" set auto-discovery-receiver enable set idle-timeout enable set idle-timeoutinterval 5 set network-overlay enable set network-id 11 next edit "W_INET_21" set auto-discovery-receiver enable set idle-timeout enable set idle-timeoutinterval 5 set network-overlay enable set network-id 21 next edit "W_MPLS_12" set auto-discovery-receiver enable set idle-timeout enable set idle-timeoutinterval 5 set network-overlay enable set network-id 12 next edit "W_MPLS_22" set auto-discovery-receiver enable set idle-timeout enable set idle-timeoutinterval 5 set network-overlay enable set network-id 22 next end # Allow shortcut monitoring (ping) config system interface edit "W_INET_11" set allowaccess ping next edit "W_INET_21" set allowaccess ping next edit "W_MPLS_12" set allowaccess ping next edit "W_MPLS_22" set allowaccess ping next end

02-W-Edge-Routing:

INTERNAL USE ONLY

141

Managed Secure SD-WAN 6.4.4 # Use BGP NH aggregates to identify overlays config router prefix-list edit "NH_INET" config rule edit 1 set prefix 10.201.0.0 255.255.0.0 set le 32 next end next edit "NH_MPLS" config rule edit 1 set prefix 10.202.0.0 255.255.0.0 set le 32 next end next end config router route-map edit "INET_IN" config rule edit 1 # Prefer same-overlay routes (INET) set match-ip-nexthop "NH_INET" set set-weight 200 next edit 100 next end next edit "MPLS_IN" config rule edit 1 # Prefer same-overlay routes (MPLS) set match-ip-nexthop "NH_MPLS" set set-weight 200 next edit 100 next end next end # BGP to Hub config router bgp set as 65000 set router-id 10.0.$(branch-id).1 set keepalive-timer 5 set holdtime-timer 15 set ibgp-multipath enable set additional-path enable set additional-path-select 4 config neighbor edit "10.201.1.1" set soft-reconfiguration enable set advertisement-interval 1 set link-down-failover enable INTERNAL USE ONLY

142

Managed Secure SD-WAN 6.4.4 set interface "W_INET_11" set connect-timer 1 set remote-as 65000 set route-map-in "INET_IN" set additional-path receive next edit "10.201.2.1" set soft-reconfiguration enable set advertisement-interval 1 set link-down-failover enable set interface "W_INET_21" set connect-timer 1 set remote-as 65000 set route-map-in "INET_IN" set additional-path receive next edit "10.202.1.1" set soft-reconfiguration enable set advertisement-interval 1 set link-down-failover enable set interface "W_MPLS_12" set connect-timer 1 set remote-as 65000 set route-map-in "MPLS_IN" set additional-path receive next edit "10.202.2.1" set soft-reconfiguration enable set advertisement-interval 1 set link-down-failover enable set interface "W_MPLS_22" set connect-timer 1 set remote-as 65000 set route-map-in "MPLS_IN" set additional-path receive next end config network edit 1 set prefix 10.0.$(branch-id).0 255.255.255.0 next end end config router static # Cross-overlay BGP NH reachability edit 102 set dst 10.200.0.0 255.252.0.0 set device "W_INET_11" set comment "Cross-overlay BGP NH reachability" next edit 103 set dst 10.200.0.0 255.252.0.0 set device "W_MPLS_12" set comment "Cross-overlay BGP NH reachability" next edit 104 set dst 10.200.0.0 255.252.0.0 set device "W_INET_21" INTERNAL USE ONLY

143

Managed Secure SD-WAN 6.4.4 set comment "Cross-overlay BGP NH reachability" next edit 105 set dst 10.200.0.0 255.252.0.0 set device "W_MPLS_22" set comment "Cross-overlay BGP NH reachability" next # Cross-region same-overlay BGP NH reachability edit 106 set dst 10.201.0.0 255.255.0.0 set device "W_INET_11" set comment "Cross-region BGP NH reachability (INET)" next edit 107 set dst 10.202.0.0 255.255.0.0 set device "W_MPLS_12" set comment "Cross-region BGP NH reachability (MPLS)" next edit 108 set dst 10.201.0.0 255.255.0.0 set device "W_INET_21" set comment "Cross-region BGP NH reachability (INET)" next edit 109 set dst 10.202.0.0 255.255.0.0 set device "W_MPLS_22" set comment "Cross-region BGP NH reachability (MPLS)" next end

03-ALL-Edge-Default:

config router static edit 100 set sdwan enable next edit 101 set dst 10.0.0.0 255.0.0.0 set blackhole enable set comment "Avoid potential leak of corporate traffic to underlay" next end

04-W-Edge-Overlay-TS:

INTERNAL USE ONLY

144

Managed Secure SD-WAN 6.4.4 # Enable ADVPN config vpn ipsec phase1-interface edit "WTS_INET_111" set auto-discovery-receiver enable set idle-timeout enable set idle-timeoutinterval 5 set network-overlay enable set network-id 111 next edit "WTS_INET_121" set auto-discovery-receiver enable set idle-timeout enable set idle-timeoutinterval 5 set network-overlay enable set network-id 121 next edit "WTS_MPLS_112" set auto-discovery-receiver enable set idle-timeout enable set idle-timeoutinterval 5 set network-overlay enable set network-id 112 next edit "WTS_MPLS_122" set auto-discovery-receiver enable set idle-timeout enable set idle-timeoutinterval 5 set network-overlay enable set network-id 122 next end # Allow shortcut monitoring (ping) config system interface edit "WTS_INET_111" set allowaccess ping next edit "WTS_INET_121" set allowaccess ping next edit "WTS_MPLS_112" set allowaccess ping next edit "WTS_MPLS_122" set allowaccess ping next end

05-W-Edge-Routing-TS:

INTERNAL USE ONLY

145

Managed Secure SD-WAN 6.4.4 # Configure VRFs config system interface edit "WTS_INET_111" set vrf 10 next edit "WTS_INET_121" set vrf 10 next edit "WTS_MPLS_112" set vrf 10 next edit "WTS_MPLS_122" set vrf 10 next end # BGP to Hub config router bgp config neighbor edit "10.201.11.1" set soft-reconfiguration enable set advertisement-interval 1 set link-down-failover enable set interface "WTS_INET_111" set connect-timer 1 set remote-as 65000 set route-map-in "INET_IN" set additional-path receive next edit "10.201.12.1" set soft-reconfiguration enable set advertisement-interval 1 set link-down-failover enable set interface "WTS_INET_121" set connect-timer 1 set remote-as 65000 set route-map-in "INET_IN" set additional-path receive next edit "10.202.11.1" set soft-reconfiguration enable set advertisement-interval 1 set link-down-failover enable set interface "WTS_MPLS_112" set connect-timer 1 set remote-as 65000 set route-map-in "MPLS_IN" set additional-path receive next edit "10.202.12.1" set soft-reconfiguration enable set advertisement-interval 1 set link-down-failover enable set interface "WTS_MPLS_122" set connect-timer 1 set remote-as 65000 set route-map-in "MPLS_IN" set additional-path receive INTERNAL USE ONLY

146

Managed Secure SD-WAN 6.4.4 next end config network edit 101 set prefix 10.0.10$(branch-id).0 255.255.255.0 next end end config router static edit 201 set vrf 10 set dst 10.0.0.0 255.0.0.0 set blackhole enable set comment "Avoid potential leak of corporate traffic to underlay" next end

10.2.1.2. Hubs-West-Template 01-W-Hub-Overlay:

# Configure tunnel interface IPs config system interface edit "W_INET_0" set ip 10.201.$(dc-id).1 255.255.255.255 set remote-ip 10.201.$(dc-id).254 255.255.255.0 set allowaccess ping next edit "W_MPLS_0" set ip 10.202.$(dc-id).1 255.255.255.255 set remote-ip 10.202.$(dc-id).254 255.255.255.0 set allowaccess ping next end # Enable ADVPN config vpn ipsec phase1-interface edit "W_INET_0" set auto-discovery-sender enable set network-overlay enable set network-id $(dc-id)1 next edit "W_MPLS_0" set auto-discovery-sender enable set network-overlay enable set network-id $(dc-id)2 next end

02-W-Hub-Routing:

INTERNAL USE ONLY

147

Managed Secure SD-WAN 6.4.4 # Configure BGP neighbors config router bgp set as 65000 set router-id 10.$(dc-id).0.1 set keepalive-timer 5 set holdtime-timer 15 set ibgp-multipath enable set additional-path enable set additional-path-select 4 config neighbor-group edit "EDGE_INET" set soft-reconfiguration enable set advertisement-interval 1 set link-down-failover enable set remote-as 65000 set additional-path send set adv-additional-path 4 set next-hop-self enable set interface "W_INET_0" set update-source "W_INET_0" set route-reflector-client enable next edit "EDGE_MPLS" set soft-reconfiguration enable set advertisement-interval 1 set link-down-failover enable set remote-as 65000 set additional-path send set adv-additional-path 4 set next-hop-self enable set interface "W_MPLS_0" set update-source "W_MPLS_0" set route-reflector-client enable next end config neighbor-range edit 1 set prefix 10.201.$(dc-id).0 255.255.255.0 set neighbor-group "EDGE_INET" next edit 2 set prefix 10.202.$(dc-id).0 255.255.255.0 set neighbor-group "EDGE_MPLS" next end config network edit 1 set prefix 10.$(dc-id).0.0 255.255.255.0 next end end # Overlay stickiness config router policy edit 1 set input-device "W_INET_0" set output-device "W_INET_0" next INTERNAL USE ONLY

148

Managed Secure SD-WAN 6.4.4 edit 2 set input-device "W_MPLS_0" set output-device "W_MPLS_0" next end

03-ALL-Hub-Default:

config router static edit 101 set dst 10.0.0.0 255.0.0.0 set blackhole enable set comment "Avoid potential leak of corporate traffic to underlay" next end

04-W-Hub-Overlay-WE:

INTERNAL USE ONLY

149

Managed Secure SD-WAN 6.4.4 # Configure Hub-to-Hub tunnel interface IPs config system interface edit "WE_INET_$(peer-dc-id)a" set ip 10.201.10.$(dc-id) 255.255.255.255 set remote-ip 10.201.10.$(peer-dc-id) 255.255.255.252 set allowaccess ping set src-check disable next edit "WE_MPLS_$(peer-dc-id)b" set ip 10.202.10.$(dc-id) 255.255.255.255 set remote-ip 10.202.10.$(peer-dc-id) 255.255.255.252 set allowaccess ping set src-check disable next edit "WE_INET_3a" set ip 10.201.10$(dc-id).1 255.255.255.255 set remote-ip 10.201.10$(dc-id).2 255.255.255.252 set allowaccess ping set src-check disable next edit "WE_MPLS_3b" set ip 10.202.10$(dc-id).1 255.255.255.255 set remote-ip 10.202.10$(dc-id).2 255.255.255.252 set allowaccess ping set src-check disable next end # Enable cross-regional aDVPN config vpn ipsec phase1-interface edit "WE_INET_3a" set auto-discovery-sender enable set auto-discovery-receiver enable set auto-discovery-forwarder enable next edit "WE_MPLS_3b" set auto-discovery-sender enable set auto-discovery-receiver enable set auto-discovery-forwarder enable next end

05-W-Hub-Routing-WE:

INTERNAL USE ONLY

150

Managed Secure SD-WAN 6.4.4 # Use BGP communities to identify local region config router community-list edit "WEST_EDGE" config rule edit 1 set action permit set match "65000:10" next end next edit "WEST_DC" config rule edit 1 set action permit set match "65000:11" next end next end # Use BGP NH aggregates to identify overlays config router prefix-list edit "NH_INET" config rule edit 1 set prefix 10.201.0.0 255.255.0.0 set le 32 next end next edit "NH_MPLS" config rule edit 1 set prefix 10.202.0.0 255.255.0.0 set le 32 next end next end config router route-map # Avoid duplicate advertisement to remote regions edit "INET_OUT" config rule edit 1 set match-community "WEST_EDGE" set match-ip-nexthop "NH_INET" next edit 2 set match-community "WEST_DC" next end next edit "MPLS_OUT" config rule edit 1 set match-community "WEST_EDGE" set match-ip-nexthop "NH_MPLS" next edit 2 INTERNAL USE ONLY

151

Managed Secure SD-WAN 6.4.4 set match-community "WEST_DC" next end next edit "PEERHUB_OUT" config rule edit 1 set match-community "WEST_DC" # Prevent duplicate readvertisement to Edges and remote regions set set-community no-advertise set set-community-additive enable next end next # Set regional communities edit "WEST_EDGE_IN" config rule edit 1 set set-community "65000:10" next end next edit "WEST_DC_IN" config rule edit 1 set set-community "65000:11" next end next end config router bgp config neighbor # Peer tunnels edit "10.201.10.$(peer-dc-id)" set soft-reconfiguration enable set advertisement-interval 1 set link-down-failover enable set connect-timer 1 set remote-as 65000 set next-hop-self enable set interface "WE_INET_$(peer-dc-id)a" set route-map-out "PEERHUB_OUT" next edit "10.202.10.$(peer-dc-id)" set soft-reconfiguration enable set advertisement-interval 1 set link-down-failover enable set connect-timer 1 set remote-as 65000 set next-hop-self enable set interface "WE_MPLS_$(peer-dc-id)b" set route-map-out "PEERHUB_OUT" next # Inter-regional tunnels edit "10.201.10$(dc-id).2" set soft-reconfiguration enable set advertisement-interval 1 set link-down-failover enable INTERNAL USE ONLY

152

Managed Secure SD-WAN 6.4.4 set connect-timer 1 set remote-as 65000 set additional-path both set adv-additional-path 2 set next-hop-self enable set interface "WE_INET_3a" set route-map-out "INET_OUT" set route-reflector-client enable next edit "10.202.10$(dc-id).2" set soft-reconfiguration enable set advertisement-interval 1 set link-down-failover enable set connect-timer 1 set remote-as 65000 set additional-path both set adv-additional-path 2 set next-hop-self enable set interface "WE_MPLS_3b" set route-map-out "MPLS_OUT" set route-reflector-client enable next end config neighbor-group # Mark Edge prefixes with regional community edit "EDGE_INET" set route-map-in "WEST_EDGE_IN" next edit "EDGE_MPLS" set route-map-in "WEST_EDGE_IN" next end config network edit 1 # Mark DC prefixes with regional community set route-map "WEST_DC_IN" next end end # Cross-region overlay stickiness config router policy edit 4 set input-device "W_INET_0" set output-device "WE_INET_3a" next edit 5 set input-device "W_MPLS_0" set output-device "WE_MPLS_3b" next edit 6 set input-device "WE_INET_3a" set output-device "W_INET_0" next edit 7 set input-device "WE_MPLS_3b" set output-device "W_MPLS_0" next end INTERNAL USE ONLY

153

Managed Secure SD-WAN 6.4.4 config router static edit 104 set dst 10.201.3.0 255.255.255.0 set device "WE_INET_3a" set comment "Cross-region BGP NH reachability (INET)" next edit 105 set dst 10.202.3.0 255.255.255.0 set device "WE_MPLS_3b" set comment "Cross-region BGP NH reachability (MPLS)" next end

06-W-Hub-Overlay-TS:

# Configure tunnel interface IPs config system interface edit "WTS_INET_0" set ip 10.201.1$(dc-id).1 255.255.255.255 set remote-ip 10.201.1$(dc-id).254 255.255.255.0 set allowaccess ping next edit "WTS_MPLS_0" set ip 10.202.1$(dc-id).1 255.255.255.255 set remote-ip 10.202.1$(dc-id).254 255.255.255.0 set allowaccess ping next end # Enable ADVPN config vpn ipsec phase1-interface edit "WTS_INET_0" set auto-discovery-sender enable set network-overlay enable set network-id 1$(dc-id)1 next edit "WTS_MPLS_0" set auto-discovery-sender enable set network-overlay enable set network-id 1$(dc-id)2 next end

07-W-Hub-Routing-TS:

INTERNAL USE ONLY

154

Managed Secure SD-WAN 6.4.4 # Configure VRFs config system interface edit "WTS_INET_0" set vrf 10 next edit "WTS_MPLS_0" set vrf 10 next end config router bgp config neighbor-group edit "TS_EDGE_INET" set soft-reconfiguration enable set advertisement-interval 1 set link-down-failover enable set remote-as 65000 set additional-path send set adv-additional-path 2 set next-hop-self enable set interface "WTS_INET_0" set update-source "WTS_INET_0" set route-reflector-client enable next edit "TS_EDGE_MPLS" set soft-reconfiguration enable set advertisement-interval 1 set link-down-failover enable set remote-as 65000 set additional-path send set adv-additional-path 2 set next-hop-self enable set interface "WTS_MPLS_0" set update-source "WTS_MPLS_0" set route-reflector-client enable next end config neighbor-range edit 11 set prefix 10.201.1$(dc-id).0 255.255.255.0 set neighbor-group "TS_EDGE_INET" next edit 12 set prefix 10.202.1$(dc-id).0 255.255.255.0 set neighbor-group "TS_EDGE_MPLS" next end end # Overlay stickiness config router policy edit 111 set input-device "WTS_INET_0" set output-device "WTS_INET_0" next edit 112 set input-device "WTS_MPLS_0" set output-device "WTS_MPLS_0" INTERNAL USE ONLY

155

Managed Secure SD-WAN 6.4.4 next end config router static edit 201 set vrf 10 set dst 10.0.0.0 255.0.0.0 set blackhole enable set comment "Avoid potential leak of corporate traffic to underlay" next end

10.2.1.3. Edge-East-Template 01-E-Edge-Overlay:

# Enable ADVPN config vpn ipsec phase1-interface edit "E_INET_0" set auto-discovery-receiver enable set idle-timeout enable set idle-timeoutinterval 5 set network-overlay enable set network-id 31 next edit "E_MPLS_0" set auto-discovery-receiver enable set idle-timeout enable set idle-timeoutinterval 5 set network-overlay enable set network-id 32 next end # Allow shortcut monitoring (ping) config system interface edit "E_INET_0" set allowaccess ping next edit "E_MPLS_0" set allowaccess ping next end

02-E-Edge-Routing:

INTERNAL USE ONLY

156

Managed Secure SD-WAN 6.4.4 # Use BGP NH aggregates to identify overlays config router prefix-list edit "NH_INET" config rule edit 1 set prefix 10.201.0.0 255.255.0.0 set le 32 next end next edit "NH_MPLS" config rule edit 1 set prefix 10.202.0.0 255.255.0.0 set le 32 next end next end config router route-map edit "INET_IN" config rule edit 1 # Prefer same-overlay routes (INET) set match-ip-nexthop "NH_INET" set set-weight 200 next edit 100 next end next edit "MPLS_IN" config rule edit 1 # Prefer same-overlay routes (MPLS) set match-ip-nexthop "NH_MPLS" set set-weight 200 next edit 100 next end next end # BGP to Hub config router bgp set as 65000 set router-id 10.0.$(branch-id).1 set keepalive-timer 5 set holdtime-timer 15 set ibgp-multipath enable set additional-path enable set additional-path-select 4 config neighbor edit "10.201.3.1" set soft-reconfiguration enable set advertisement-interval 1 set link-down-failover enable INTERNAL USE ONLY

157

Managed Secure SD-WAN 6.4.4 set interface "E_INET_0" set connect-timer 1 set remote-as 65000 set route-map-in "INET_IN" set additional-path receive next edit "10.202.3.1" set soft-reconfiguration enable set advertisement-interval 1 set link-down-failover enable set interface "E_MPLS_0" set connect-timer 1 set remote-as 65000 set route-map-in "MPLS_IN" set additional-path receive next end config network edit 1 set prefix 10.0.$(branch-id).0 255.255.255.0 next end end config router static # Cross-overlay BGP NH reachability edit 102 set dst 10.200.0.0 255.252.0.0 set device "E_INET_0" set comment "Cross-overlay BGP NH reachability" next edit 103 set dst 10.200.0.0 255.252.0.0 set device "E_MPLS_0" set comment "Cross-overlay BGP NH reachability" next # Cross-region same-overlay BGP NH reachability edit 106 set dst 10.201.0.0 255.255.0.0 set device "E_INET_0" set comment "Cross-region BGP NH reachability (INET)" next edit 107 set dst 10.202.0.0 255.255.0.0 set device "E_MPLS_0" set comment "Cross-region BGP NH reachability (MPLS)" next end

03-ALL-Edge-Default:

INTERNAL USE ONLY

158

Managed Secure SD-WAN 6.4.4 config router static edit 100 set sdwan enable next edit 101 set dst 10.0.0.0 255.0.0.0 set blackhole enable set comment "Avoid potential leak of corporate traffic to underlay" next end

10.2.1.4. Hubs-East-Template 01-E-Hub-Overlay:

# Configure tunnel interface IPs config system interface edit "E_INET_0" set ip 10.201.$(dc-id).1 255.255.255.255 set remote-ip 10.201.$(dc-id).254 255.255.255.0 set allowaccess ping next edit "E_MPLS_0" set ip 10.202.$(dc-id).1 255.255.255.255 set remote-ip 10.202.$(dc-id).254 255.255.255.0 set allowaccess ping next end # Enable ADVPN config vpn ipsec phase1-interface edit "E_INET_0" set auto-discovery-sender enable set network-overlay enable set network-id $(dc-id)1 next edit "E_MPLS_0" set auto-discovery-sender enable set network-overlay enable set network-id $(dc-id)2 next end

02-E-Hub-Routing:

INTERNAL USE ONLY

159

Managed Secure SD-WAN 6.4.4 # Configure BGP neighbors config router bgp set as 65000 set router-id 10.$(dc-id).0.1 set keepalive-timer 5 set holdtime-timer 15 set ibgp-multipath enable set additional-path enable set additional-path-select 4 config neighbor-group edit "EDGE" set soft-reconfiguration enable set advertisement-interval 1 set link-down-failover enable set remote-as 65000 set additional-path send set adv-additional-path 4 set next-hop-self enable set route-reflector-client enable next end config neighbor-range edit 1 set prefix 10.200.0.0 255.252.0.0 set neighbor-group "EDGE" next end config network edit 1 set prefix 10.$(dc-id).0.0 255.255.255.0 next end end # Overlay stickiness config router policy edit 1 set input-device "E_INET_0" set output-device "E_INET_0" next edit 2 set input-device "E_MPLS_0" set output-device "E_MPLS_0" next end

03-ALL-Hub-Default:

INTERNAL USE ONLY

160

Managed Secure SD-WAN 6.4.4 config router static edit 101 set dst 10.0.0.0 255.0.0.0 set blackhole enable set comment "Avoid potential leak of corporate traffic to underlay" next end

04-E-Hub-Overlay-EW:

INTERNAL USE ONLY

161

Managed Secure SD-WAN 6.4.4 # Configure Hub-to-Hub tunnel interface IPs config system interface edit "WE_INET_1a" set ip 10.201.101.2 255.255.255.255 set remote-ip 10.201.101.1 255.255.255.252 set allowaccess ping set src-check disable next edit "WE_MPLS_1b" set ip 10.202.101.2 255.255.255.255 set remote-ip 10.202.101.1 255.255.255.252 set allowaccess ping set src-check disable next edit "WE_INET_2a" set ip 10.201.102.2 255.255.255.255 set remote-ip 10.201.102.1 255.255.255.252 set allowaccess ping set src-check disable next edit "WE_MPLS_2b" set ip 10.202.102.2 255.255.255.255 set remote-ip 10.202.102.1 255.255.255.252 set allowaccess ping set src-check disable next end # Enable cross-regional aDVPN config vpn ipsec phase1-interface edit "WE_INET_1a" set auto-discovery-sender enable set auto-discovery-receiver enable set auto-discovery-forwarder enable next edit "WE_MPLS_1b" set auto-discovery-sender enable set auto-discovery-receiver enable set auto-discovery-forwarder enable next edit "WE_INET_2a" set auto-discovery-sender enable set auto-discovery-receiver enable set auto-discovery-forwarder enable next edit "WE_MPLS_2b" set auto-discovery-sender enable set auto-discovery-receiver enable set auto-discovery-forwarder enable next end

05-E-Hub-Routing-EW:

INTERNAL USE ONLY

162

Managed Secure SD-WAN 6.4.4 ## Use BGP communities to identify local region config router community-list edit "EAST_EDGE" config rule edit 1 set action permit set match "65000:30" next end next edit "EAST_DC" config rule edit 1 set action permit set match "65000:31" next end next end # Use BGP NH aggregates to identify overlays config router prefix-list edit "NH_INET" config rule edit 1 set prefix 10.201.0.0 255.255.0.0 set le 32 next end next edit "NH_MPLS" config rule edit 1 set prefix 10.202.0.0 255.255.0.0 set le 32 next end next end config router route-map # Avoid duplicate advertisement to remote regions edit "INET_OUT" config rule edit 1 set match-community "EAST_EDGE" set match-ip-nexthop "NH_INET" next edit 2 set match-community "EAST_DC" next end next edit "MPLS_OUT" config rule edit 1 set match-community "EAST_EDGE" set match-ip-nexthop "NH_MPLS" next edit 2 INTERNAL USE ONLY

163

Managed Secure SD-WAN 6.4.4 set match-community "EAST_DC" next end next # Set regional communities edit "EAST_EDGE_IN" config rule edit 1 set set-community "65000:30" next end next edit "EAST_DC_IN" config rule edit 1 set set-community "65000:31" next end next end

config router bgp config neighbor edit "10.201.101.1" set soft-reconfiguration enable set advertisement-interval 1 set link-down-failover enable set connect-timer 1 set remote-as 65000 set additional-path both set adv-additional-path 2 set next-hop-self enable set interface "WE_INET_1a" set route-map-out "INET_OUT" set route-reflector-client enable next edit "10.201.102.1" set soft-reconfiguration enable set advertisement-interval 1 set link-down-failover enable set connect-timer 1 set remote-as 65000 set additional-path both set adv-additional-path 2 set next-hop-self enable set interface "WE_INET_2a" set route-map-out "INET_OUT" set route-reflector-client enable next edit "10.202.101.1" set soft-reconfiguration enable set advertisement-interval 1 set link-down-failover enable set connect-timer 1 set remote-as 65000 set additional-path both set adv-additional-path 2 set next-hop-self enable INTERNAL USE ONLY

164

Managed Secure SD-WAN 6.4.4 set interface "WE_MPLS_1b" set route-map-out "MPLS_OUT" set route-reflector-client enable next edit "10.202.102.1" set soft-reconfiguration enable set advertisement-interval 1 set link-down-failover enable set connect-timer 1 set remote-as 65000 set additional-path both set adv-additional-path 2 set next-hop-self enable set interface "WE_MPLS_2b" set route-map-out "MPLS_OUT" set route-reflector-client enable next end config neighbor-group # Mark Edge prefixes with regional community edit "EDGE" set route-map-in "EAST_EDGE_IN" next end config network edit 1 # Mark DC prefixes with regional community set route-map "EAST_DC_IN" next end end # Cross-region overlay stickiness config router policy edit 4 set input-device "E_INET_0" set output-device "WE_INET_1a" next edit 5 set input-device "E_MPLS_0" set output-device "WE_MPLS_1b" next edit 6 set input-device "WE_INET_1a" set output-device "E_INET_0" next edit 7 set input-device "WE_MPLS_1b" set output-device "E_MPLS_0" next edit 8 set input-device "E_INET_0" set output-device "WE_INET_2a" next edit 9 set input-device "E_MPLS_0" set output-device "WE_MPLS_2b" next edit 10 INTERNAL USE ONLY

165

Managed Secure SD-WAN 6.4.4 set input-device "WE_INET_2a" set output-device "E_INET_0" next edit 11 set input-device "WE_MPLS_2b" set output-device "E_MPLS_0" next end config router static edit 104 set dst 10.201.1.0 255.255.255.0 set device "WE_INET_1a" set comment "Cross-region BGP NH next edit 105 set dst 10.202.1.0 255.255.255.0 set device "WE_MPLS_1b" set comment "Cross-region BGP NH next edit 106 set dst 10.201.2.0 255.255.255.0 set device "WE_INET_2a" set comment "Cross-region BGP NH next edit 107 set dst 10.202.2.0 255.255.255.0 set device "WE_MPLS_2b" set comment "Cross-region BGP NH next end

reachability (INET)"

reachability (MPLS)"

reachability (INET)"

reachability (MPLS)"

10.2.2. Firewall Policy The following screenshots show the complete firewall policy for this topology:

INTERNAL USE ONLY

166

Managed Secure SD-WAN 6.4.4

INTERNAL USE ONLY

167