In this post I will just show some examples of packet captures and debugs for reference.
Generally speaking we would use the packet capture "diagnose sniffer packet" for wanting to know about traffic coming or going from the firewall.
If some traffic is coming in and not coming out of the firewall we can use "debug flow filter" to see how the Forti is processing a packet and why it might be being dropped.
PACKET CAPTURE
Lets do an identical packet capture and see what the different verbosity levels can give us.
Verbosity level options:
FG1 # diagnose sniffer packet port2 'host 52.52.208.2 and tcp port 443'
<verbose>
1: print header of packets
2: print header and data from ip of packets
3: print header and data from ethernet of packets (if available)
4: print header of packets with interface name
5: print header and data from ip of packets with interface name
6: print header and data from ethernet of packets (if available) with intf name
Verbosity 1 - print header of packets.
FG1 # diagnose sniffer packet port2 'host 52.52.208.2 and tcp port 443' 1 10 1
interfaces=[port2]
filters=[host 52.52.208.2 and tcp port 443]
17.779603 192.168.100.220.33384 -> 52.52.208.2.443: syn 383515866
17.941839 52.52.208.2.443 -> 192.168.100.220.33384: syn 2080446819 ack 383515867
17.942386 192.168.100.220.33384 -> 52.52.208.2.443: ack 2080446820
17.945834 192.168.100.220.33384 -> 52.52.208.2.443: psh 383515867 ack 2080446820
18.108347 52.52.208.2.443 -> 192.168.100.220.33384: ack 383516537
18.108405 52.52.208.2.443 -> 192.168.100.220.33384: psh 2080446820 ack 383516537
18.108606 192.168.100.220.33384 -> 52.52.208.2.443: ack 2080447109
18.112843 192.168.100.220.33384 -> 52.52.208.2.443: psh 383516537 ack 2080447109
18.115091 192.168.100.220.33384 -> 52.52.208.2.443: 383516617 ack 2080447109
18.115197 192.168.100.220.33384 -> 52.52.208.2.443: psh 383518057 ack 2080447109
FG1 #
Verbosity 2 - print header and data from ip of packets.
FG1 # diagnose sniffer packet port2 'host 52.52.208.2 and tcp port 443' 2 10 1
interfaces=[port2]
filters=[host 52.52.208.2 and tcp port 443]
4.660159 192.168.100.220.33852 -> 52.52.208.2.443: syn 3833835638
0x0000 4500 003c 5959 4000 4006 b7a7 c0a8 64dc E..<YY@.@.....d.
0x0010 3434 d002 843c 01bb e483 b076 0000 0000 44...<.....v....
0x0020 a002 faf0 a452 0000 0204 05b4 0402 080a .....R..........
0x0030 0d84 568b 0000 0000 0103 0307 ..V.........
4.834071 52.52.208.2.443 -> 192.168.100.220.33852: syn 2802493585 ack 3833835639
0x0000 4500 0034 0000 4000 2f06 2209 3434 d002 E..4..@./.".44..
0x0010 c0a8 64dc 01bb 843c a70a a891 e483 b077 ..d....<.......w
0x0020 8012 7210 68bc 0000 0204 05a0 0101 0402 ..r.h...........
0x0030 0103 0305 ....
4.834515 192.168.100.220.33852 -> 52.52.208.2.443: ack 2802493586
0x0000 4500 0028 595a 4000 4006 b7ba c0a8 64dc E..(YZ@.@.....d.
0x0010 3434 d002 843c 01bb e483 b077 a70a a892 44...<.....w....
0x0020 5010 01f6 1993 0000 0000 0000 0000 P.............
4.836454 192.168.100.220.33852 -> 52.52.208.2.443: psh 3833835639 ack 2802493586
0x0000 4500 02c6 595b 4000 4006 b51b c0a8 64dc E...Y[@.@.....d.
0x0010 3434 d002 843c 01bb e483 b077 a70a a892 44...<.....w....
0x0020 5018 01f6 ed17 0000 1603 0102 9901 0002 P...............
0x0030 9503 03f0 d4b9 ebef 1912 4752 7360 6ae3 ..........GRs`j.
0x0040 2ef5 b6b8 5092 896f 4994 c3c7 804c 2090 ....P..oI....L..
0x0050 1c6b 9a20 4a1e d7c2 b1ca dc23 75e8 48b2 .k..J......#u.H.
0x0060 95c3 b86e 2676 0891 0049 dda0 67e7 3353 ...n&v...I..g.3S
0x0070 e3ff 6a04 0024 1301 1303 1302 c02b c02f ..j..$.......+./
0x0080 cca9 cca8 c02c c030 c00a c009 c013 c014 .....,.0........
0x0090 0033 0039 002f 0035 000a 0100 0228 0000 .3.9./.5.....(..
0x00a0 0015 0013 0000 1077 7777 2e66 6f72 7469 .......www.forti
0x00b0 6e65 742e 636f 6d00 1700 00ff 0100 0100 net.com.........
:
:
etc
Verbosity 3 - print header and data from ethernet of packets (if available).
FG1 # diagnose sniffer packet port2 'host 52.52.208.2 and tcp port 443' 3 10 1
interfaces=[port2]
filters=[host 52.52.208.2 and tcp port 443]
8.493203 192.168.100.220.35050 -> 52.52.208.2.443: syn 304282870
0x0000 000c 291f 14a1 000c 2976 c9ec 0800 4500 ..).....)v....E.
0x0010 003c b0c5 4000 4006 603b c0a8 64dc 3434 .<..@.@.`;..d.44
0x0020 d002 88ea 01bb 1222 fcf6 0000 0000 a002 ......."........
0x0030 faf0 994b 0000 0204 05b4 0402 080a 0dad ...K............
0x0040 e29c 0000 0000 0103 0307 ..........
8.655050 52.52.208.2.443 -> 192.168.100.220.35050: syn 4129167029 ack 304282871
0x0000 000c 2976 c9ec 000c 291f 14a1 0800 4500 ..)v....).....E.
0x0010 0034 0000 4000 2f06 2209 3434 d002 c0a8 .4..@./.".44....
0x0020 64dc 01bb 88ea f61e 16b5 1222 fcf7 8012 d.........."....
0x0030 7210 2cb8 0000 0204 05a0 0101 0402 0103 r.,.............
0x0040 0305 ..
8.655342 192.168.100.220.35050 -> 52.52.208.2.443: ack 4129167030
0x0000 000c 291f 14a1 000c 2976 c9ec 0800 4500 ..).....)v....E.
0x0010 0028 b0c6 4000 4006 604e c0a8 64dc 3434 .(..@.@.`N..d.44
0x0020 d002 88ea 01bb 1222 fcf7 f61e 16b6 5010 ......."......P.
0x0030 01f6 dd8e 0000 0000 0000 0000 ............
We can see with verbosity 3 we do indeed include the ethernet header as an addition vs verbosity 2.
MACs highlighted above and below.
FG1 # get hardware nic port2 | grep Hwaddr:
Hwaddr: 00:0c:29:1f:14:a1
Permanent Hwaddr:00:0c:29:1f:14:a1
And for the source:
user@lubuntu:~$ ifconfig
ens33: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500
inet 192.168.100.220 netmask 255.255.255.0 broadcast 192.168.100.255
inet6 fe80::907b:2e13:3010:b822 prefixlen 64 scopeid 0x20<link>
ether 00:0c:29:76:c9:ec txqueuelen 1000 (Ethernet
Verbosity 4 - print header of packets with interface name.
FG1 # diagnose sniffer packet port2 'host 52.52.208.2 and tcp port 443' 4 10 1
interfaces=[port2]
filters=[host 52.52.208.2 and tcp port 443]
8.626150 port2 -- 192.168.100.220.34194 -> 52.52.208.2.443: syn 30965692
8.788266 port2 -- 52.52.208.2.443 -> 192.168.100.220.34194: syn 3386219767 ack 30965693
8.788639 port2 -- 192.168.100.220.34194 -> 52.52.208.2.443: ack 3386219768
8.790182 port2 -- 192.168.100.220.34194 -> 52.52.208.2.443: psh 30965693 ack 3386219768
8.952384 port2 -- 52.52.208.2.443 -> 192.168.100.220.34194: ack 30966363
8.952696 port2 -- 52.52.208.2.443 -> 192.168.100.220.34194: psh 3386219768 ack 30966363
8.952886 port2 -- 192.168.100.220.34194 -> 52.52.208.2.443: ack 3386220057
8.956138 port2 -- 192.168.100.220.34194 -> 52.52.208.2.443: psh 30966363 ack 3386220057
8.959056 port2 -- 192.168.100.220.34194 -> 52.52.208.2.443: 30966443 ack 3386220057
8.959096 port2 -- 192.168.100.220.34194 -> 52.52.208.2.443: psh 30967883 ack 3386220057
Similar to option1 but with interface names which would be useful for confirming how packets are being routed.
Lets see a Verbosity 4 with "any" as interface to show that point.
FG1 # diagnose sniffer packet any 'host 52.52.208.2 and tcp port 443' 4 10 1
interfaces=[any]
filters=[host 52.52.208.2 and tcp port 443]
10.283604 port2 in 192.168.100.220.34546 -> 52.52.208.2.443: syn 1742588606
10.283655 port1 out 192.168.0.25.34546 -> 52.52.208.2.443: syn 1742588606
10.445357 port1 in 52.52.208.2.443 -> 192.168.0.25.34546: syn 2588941729 ack 1742588607
10.445396 port2 out 52.52.208.2.443 -> 192.168.100.220.34546: syn 2588941729 ack 1742588607
10.445751 port2 in 192.168.100.220.34546 -> 52.52.208.2.443: ack 2588941730
10.445786 port1 out 192.168.0.25.34546 -> 52.52.208.2.443: ack 2588941730
10.447099 port2 in 192.168.100.220.34546 -> 52.52.208.2.443: psh 1742588607 ack 2588941730
10.447113 port1 out 192.168.0.25.34546 -> 52.52.208.2.443: psh 1742588607 ack 2588941730
10.608402 port1 in 52.52.208.2.443 -> 192.168.0.25.34546: ack 1742589277
10.608434 port2 out 52.52.208.2.443 -> 192.168.100.220.34546: ack 1742589277
FG1 #
If you dont need to export to Wireshark, verbosity 4 is troubleshooting gold!
Verbostity 5 - print header and data from ip of packets with interface name.
FG1 # diagnose sniffer packet port2 'host 52.52.208.2 and tcp port 443' 5 10 1
interfaces=[port2]
filters=[host 52.52.208.2 and tcp port 443]
4.863057 port2 -- 192.168.100.220.34374 -> 52.52.208.2.443: syn 2707482517
0x0000 4500 003c 4e9a 4000 4006 c266 c0a8 64dc E..<N.@.@..f..d.
0x0010 3434 d002 8646 01bb a160 e795 0000 0000 44...F...`......
0x0020 a002 faf0 aeb3 0000 0204 05b4 0402 080a ................
0x0030 0d9b 560d 0000 0000 0103 0307 ..V.........
5.025489 port2 -- 52.52.208.2.443 -> 192.168.100.220.34374: syn 627526194 ack 2707482518
0x0000 4500 0034 0000 4000 2f06 2209 3434 d002 E..4..@./.".44..
0x0010 c0a8 64dc 01bb 8646 2567 4a32 a160 e796 ..d....F%gJ2.`..
0x0020 8012 7210 52b9 0000 0204 05a0 0101 0402 ..r.R...........
0x0030 0103 0305 ....
5.025716 port2 -- 192.168.100.220.34374 -> 52.52.208.2.443: ack 627526195
0x0000 4500 0028 4e9b 4000 4006 c279 c0a8 64dc E..(N.@.@..y..d.
0x0010 3434 d002 8646 01bb a160 e796 2567 4a33 44...F...`..%gJ3
0x0020 5010 01f6 0390 0000 0000 0000 0000 P.............
So same as option 2 but with interface names per packet.
Verbosity 6 - print header and data from ethernet of packets (if available) with intf name.
FG1 # diagnose sniffer packet port2 'host 52.52.208.2 and tcp port 443' 6 10 1
interfaces=[port2]
filters=[host 52.52.208.2 and tcp port 443]
6.158669 port2 -- 192.168.100.220.34848 -> 52.52.208.2.443: syn 2317496086
0x0000 000c 291f 14a1 000c 2976 c9ec 0800 4500 ..).....)v....E.
0x0010 003c 18b3 4000 4006 f84d c0a8 64dc 3434 .<..@.@..M..d.44
0x0020 d002 8820 01bb 8a22 2f16 0000 0000 a002 ......."/.......
0x0030 faf0 4234 0000 0204 05b4 0402 080a 0da5 ..B4............
0x0040 9066 0000 0000 0103 0307 .f........
6.323612 port2 -- 52.52.208.2.443 -> 192.168.100.220.34848: syn 3908045332 ack 2317496087
0x0000 000c 2976 c9ec 000c 291f 14a1 0800 4500 ..)v....).....E.
0x0010 0034 0000 4000 2f06 2209 3434 d002 c0a8 .4..@./.".44....
0x0020 64dc 01bb 8820 e8f0 0a14 8a22 2f17 8012 d.........."/...
0x0030 7210 9d31 0000 0204 05a0 0101 0402 0103 r..1............
0x0040 0305 ..
6.323926 port2 -- 192.168.100.220.34848 -> 52.52.208.2.443: ack 3908045333
0x0000 000c 291f 14a1 000c 2976 c9ec 0800 4500 ..).....)v....E.
0x0010 0028 18b4 4000 4006 f860 c0a8 64dc 3434 .(..@.@..`..d.44
0x0020 d002 8820 01bb 8a22 2f17 e8f0 0a15 5010 ......."/.....P.
0x0030 01f6 4e08 0000 0000 0000 0000 ..N.........
Same as option 3 but with interface names per packet.
Interestingly, if you use option 6 and use "any" for the interface choice, the the dest macs show as 00:00:00:00:00:00 or 00:00:00:00:00:01.
FG1 # diagnose sniffer packet any 'host 52.52.208.2 and tcp port 443' 6 10 1
interfaces=[any]
filters=[host 52.52.208.2 and tcp port 443]
7.436821 port2 in 192.168.100.220.35258 -> 52.52.208.2.443: syn 102726635
0x0000 0000 0000 0001 000c 2976 c9ec 0800 4500 ........)v....E.
0x0010 003c 729e 4000 4006 9e62 c0a8 64dc 3434 .<r.@.@..b..d.44
0x0020 d002 89ba 01bb 061f 7beb 0000 0000 a002 ........{.......
0x0030 faf0 307c 0000 0204 05b4 0402 080a 0db4 ..0|............
0x0040 d7a3 0000 0000 0103 0307 ..........
7.436862 port1 out 192.168.0.25.35258 -> 52.52.208.2.443: syn 102726635
0x0000 0000 0000 0000 000c 291f 1497 0800 4500 ........).....E.
0x0010 003c 729e 4000 3f06 0426 c0a8 0019 3434 .<r.@.?..&....44
0x0020 d002 89ba 01bb 061f 7beb 0000 0000 a002 ........{.......
0x0030 faf0 953f 0000 0204 05b4 0402 080a 0db4 ...?............
0x0040 d7a3 0000 0000 0103 0307 ..........
7.598697 port1 in 52.52.208.2.443 -> 192.168.0.25.35258: syn 1137656363 ack 102726636
0x0000 0000 0000 0001 b095 7556 ee94 0800 4500 ........uV....E.
0x0010 0034 0000 4000 3006 85cc 3434 d002 c0a8 .4..@.0...44....
0x0020 0019 01bb 89ba 43cf 422b 061f 7bec 8012 ......C.B+..{...
0x0030 7210 a493 0000 0204 05a0 0101 0402 0103 r...............
0x0040 0305 ..
Its a known "feature"..
https://kb.fortinet.com/kb/documentLink.do?popup=true&externalID=11655&languageId=
FLOW FILTER
We will use thus feature to see how the fortigate is processing a packet. It will tell us why a packet is dropped.
If you are a Checkpoint guy, think "fw monitor" for an analogy.
Check current debug settings:
FG1 # diagnose debug info
debug output: disable
console timestamp: disable
console no user log message: disable
zebos debug level: 306783954 (0x124926d2)
CLI debug level: 3
WAD console log: enable
Clear debug settings:
FG1 # diagnose debug reset
Flow filter choices:
FG1 # diagnose debug flow filter
clear Clear filter.
vd Index of virtual domain.
proto Protocol number.
addr IP address.
saddr Source IP address.
daddr Destination IP address.
port port
sport Source port.
dport Destination port.
negate Inverse filter.
FG1 #
Add a debug for a src:
FG1 # diagnose debug flow filter saddr 192.168.100.220
FG1 # diagnose debug enable
FG1 # diagnose debug flow trace start
Check status again:
FG1 # diagnose debug info
debug output: enable
console timestamp: disable
console no user log message: disable
zebos debug level: 306783954 (0x124926d2)
CLI debug level: 3
WAD console log: enable
Policy block output:
FG1 # id=20085 trace_id=4 func=print_pkt_detail line=5501 msg="vd-root:0 received a packet(proto=1, 192.168.100.220:21190->192.168.0.1:2048) from port2. type=8, code=0, id=21190, seq=52."
id=20085 trace_id=4 func=init_ip_session_common line=5666 msg="allocate a new session-000265f6"
id=20085 trace_id=4 func=vf_ip_route_input_common line=2596 msg="find a route: flag=04000000 gw-192.168.0.1 via port1"
id=20085 trace_id=4 func=fw_forward_handler line=624 msg="Denied by forward policy check (policy 1)"
Policy allowed output with a SNAT.
FG1 # id=20085 trace_id=9 func=print_pkt_detail line=5501 msg="vd-root:0 received a packet(proto=6, 192.168.100.220:37760->3.104.18.112:443) from port2. flag [S], seq 1449070191, ack 0, win 64240"
id=20085 trace_id=9 func=init_ip_session_common line=5666 msg="allocate a new session-00028950"
id=20085 trace_id=9 func=vf_ip_route_input_common line=2596 msg="find a route: flag=04000000 gw-192.168.0.1 via port1"
id=20085 trace_id=9 func=fw_forward_handler line=771 msg="Allowed by Policy-3: SNAT"
id=20085 trace_id=9 func=__ip_session_run_tuple line=3286 msg="SNAT 192.168.100.220->192.168.0.25:37760"
With these examples we can clearly see the steps the forti is taking internally with a packet.
FG1 # get system status | grep Version
Version: FortiGate-VM64 v6.2.3,build1066,191218 (GA)
Release Version Information: GA
