Ipsec troubleshooting

In this part I will be discussing the following problem scenarios----

• IKE SA not established
• IPSec SA’s not established
• MTU/Fragmentation Issues

Problem Scenario 1: No IKE SAs

If we are unable to establish IPSec tunnel from Branch location to  Hub location

Check for Routing. Ping the Branch (using HUB’s IKE endpoint)

HUB# ping ip 40.10.1.1 source 30.3.1.1

Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 40.10.1.1, timeout is 2 seconds:
Packet sent with a source address of 30.3.1.1 
!!!!!
Success rate is 100 percent (5/5), round-trip min/avg/max = 1/1/4 ms

Check for IKE SA

HUB# sh crypto isakmp sa 
IPv4 Crypto ISAKMP SA
dst             src             state          conn-id slot status
30.3.1.1        40.10.1.1       MM_NO_STATE          0    0 ACTIVE (deleted)

IPv6 Crypto ISAKMP SA

Use IKE Debugs to troubleshoot  [ debug crypto isakmp ]

Problem Scenario 1a: 
No IKE SAs

ISAKMP:(0):Checking ISAKMP transform 1 against priority 10 policy
ISAKMP:      encryption 3DES-CBC
ISAKMP:      hash SHA
ISAKMP:      default group 2
ISAKMP:      auth pre-share
ISAKMP:      life type in seconds
ISAKMP:      life duration (basic) of 7200
ISAKMP:(0):Encryption algorithm offered does not match policy!
ISAKMP:(0):atts are not acceptable. Next payload is 0

ISAKMP:(0):Checking ISAKMP transform 1 against priority 65535 policy
ISAKMP:      encryption 3DES-CBC
ISAKMP:      hash SHA
ISAKMP:      default group 2
ISAKMP:      auth pre-share
ISAKMP:      life type in seconds
ISAKMP:      life duration (basic) of 7200
ISAKMP:(0):Encryption algorithm offered does not match policy!ISAKMP:(0):atts are not acceptable. Next payload is 0

ISAKMP:(0):no offers accepted!ISAKMP:(0): phase 1 SA policy not acceptable! (local 30.3.1.1 remote 40.10.1.1)

Check the IKE Policies

HUB# sh crypto isakmp policy 

Global IKE policy
Protection suite of priority 10
         encryption algorithm:    AES - Advanced Encryption Standard (128 bit keys).
         hash algorithm:            Secure Hash Standard
         authentication method:  Pre-Shared Key
         Diffie-Hellman group:   #2 (1024 bit)
         lifetime:                        7200 seconds, no volume limit
Default protection suite
         encryption algorithm:   DES - Data Encryption Standard (56 bit keys).

                                                                                                                                                            

So once we change the encryption algorithm at spoke side to aes, phase 1 will come up.

Problem Scenario 1b:

No IKE SAs

ISAKMP:(1017): sending packet to 40.10.1.1 my_port 500 peer_port 500 (R) MM_KEY_EXCH
ISAKMP:(1017):Input = IKE_MESG_INTERNAL, IKE_PROCESS_COMPLETE
ISAKMP:(1017):Old State = IKE_R_MM3  New State = IKE_R_MM4

ISAKMP (0:1017): received packet from 40.10.1.1 dport 500 sport 500 Global (R) MM_KEY_EXCH
ISAKMP: reserved not zero on ID payload!
%CRYPTO-4-IKMP_BAD_MESSAGE: IKE message from 40.10.1.1 failed its sanity check or is malformed

It means we have a mismatch in pre-shared key, on correcting it our IKE SA should come up.

HUB# sh cry isa sa 
IPv4 Crypto ISAKMP SA
dst             src             state          conn-id slot status
30.3.1.1        40.10.1.1       QM_IDLE           1019    0 ACTIVE

Problem Scenario 2: No IPSec SAs

If you notice that there is  no traffic is being received through the IPSec tunnel

IKE SAs exist, but no IPSec SAs

Check for IPSEC SA (look for inbound and outbound SPI’s)

HUB# sh crypto ipsec sa  peer 40.10.1.1

interface: GigabitEthernet0/1
     Crypto map tag: CMAP, local addr 30.3.1.1

    protected vrf: (none)
    local  ident (addr/mask/prot/port): (3.1.1.0/255.255.255.0/0/0)
    remote ident (addr/mask/prot/port): (4.1.1.0/255.255.255.0/0/0)
    current_peer 40.10.1.1 port 500
      PERMIT, flags={origin_is_acl,}
     #pkts encaps: 0, #pkts encrypt: 0, #pkts digest: 0
     #pkts decaps: 0, #pkts decrypt: 0, #pkts verify: 0
     #pkts compressed: 0, #pkts decompressed: 0
     #pkts not compressed: 0, #pkts compr. failed: 0
     #pkts not decompressed: 0, #pkts decompress failed: 0
     #send errors 0, #recv errors 0

      local crypto endpt.: 30.3.1.1, remote crypto endpt.: 40.10.1.1
      path mtu 1500, ip mtu 1500, ip mtu idb GigabitEthernet0/1
      current outbound spi: 0x0(0)

      inbound esp sas:
      inbound ah sas:

      outbound esp sas:
      outbound ah sas:
HUB#

Use IPSec Debugs to troubleshoot [ debug crypto ipsec ]

Problem Scenario 2a: No IPSec SAs

ISAKMP (0:1022): received packet from 40.10.1.1 dport 500 sport 500 Global (R) QM_IDLE     

ISAKMP:(1022): processing SA payload. message ID = -549695704
ISAKMP:(1022):Checking IPSec proposal 1
ISAKMP: transform 1, ESP_3DES
ISAKMP:   attributes in transform:
ISAKMP:      encaps is 1 (Tunnel)
ISAKMP:      SA life type in seconds
ISAKMP:      SA life duration (basic) of 1800
ISAKMP:      SA life type in kilobytes
ISAKMP:      SA life duration (VPI) of  0x0 0x46 0x50 0x0 
ISAKMP:      authenticator is HMAC-SHA
ISAKMP:(1022):atts are acceptable.

IPSEC(validate_proposal_request): proposal part #1,
   (key eng. msg.) INBOUND local= 30.3.1.1, remote= 40.10.1.1, 
     local_proxy= 3.1.1.0/255.255.255.0/0/0 (type=4), 
     remote_proxy= 4.1.1.0/255.255.255.0/0/0 (type=4),
     protocol= ESP, transform= NONE  (Tunnel), 
     lifedur= 0s and 0kb, 
     spi= 0x0(0), conn_id= 0, keysize= 0, flags= 0x0
Crypto mapdb : proxy_match
         src addr     : 3.1.1.0
         dst addr     : 4.1.1.0
         protocol     : 0
         src port     : 0
         dst port     : 0
IPSEC(ipsec_process_proposal): transform proposal not supported for identity: 
     {esp-3des esp-sha-hmac }
ISAKMP:(1022): IPSec policy invalidated proposal with error 256ISAKMP:(1022): phase 2 SA policy not acceptable! (local 30.3.1.1 remote 40.10.1.1)

Check the IPSec Transform Sets

HUB# sh cry ips transform-set 
Transform set TS: { esp-aes esp-sha-hmac  } 
    will negotiate = { Tunnel,  },

On Correcting encryption algorithm in tranform-set , tunnel should come up.

Problem Scenario 2b: No IPSec SAs

Check the Crypto ACLs

HUB# sh access-list SPOKE-10-ACL
Extended IP access list SPOKE10-ACL
     10 permit ip 3.1.1.0 0.0.0.255 5.1.1.0 0.0.0.255
HUB#

On Correcting crypto access-list , tunnel should come  up.

Problem Scenario 3: Anti-Replay Issues

If you notice that some of the  applications are losing intermittent traffic, or that Voice quality  through tunnel is bad.

Check if the IPSec SA is showing anti-replay drops

HUB# sh cry ips sa peer 40.10.1.1 detail
interface: GigabitEthernet0/1
     Crypto map tag: CMAP, local addr 30.3.1.1

    protected vrf: (none)
    local  ident (addr/mask/prot/port): (3.1.1.0/255.255.255.0/0/0)
    remote ident (addr/mask/prot/port): (4.1.1.0/255.255.255.0/0/0)
    current_peer 40.10.1.1 port 500
      PERMIT, flags={origin_is_acl,}
     #pkts encaps: 2900, #pkts encrypt: 2900, #pkts digest: 2900
     #pkts decaps: 1909, #pkts decrypt: 1909, #pkts verify: 1909
     #pkts compressed: 0, #pkts decompressed: 0
     #pkts not compressed: 0, #pkts compr. failed: 0
     #pkts not decompressed: 0, #pkts decompress failed: 0
     #pkts no sa (send) 0, #pkts invalid sa (rcv) 0
     #pkts encaps failed (send) 0, #pkts decaps failed (rcv) 0
     #pkts invalid prot (recv) 0, #pkts verify failed: 0
     #pkts invalid identity (recv) 0, #pkts invalid len (rcv) 0
     #pkts replay rollover (send): 0, #pkts replay rollover (rcv) 0
     ##pkts replay failed (rcv): 1000    #pkts internal err (send): 0, #pkts internal err (recv) 0

      local crypto endpt.: 30.3.1.1, remote crypto endpt.: 40.10.1.1
      path mtu 1500, ip mtu 1500, ip mtu idb GigabitEthernet0/1
      current outbound spi: 0xC37422AA(3279168170)

      inbound esp sas:
       spi: 0x135E76B1(324957873)
         transform: esp-3des esp-sha-hmac ,
         in use settings ={Tunnel, }
         conn id: 41, flow_id: SW:41, crypto map: CMAP
         sa timing: remaining key lifetime (k/sec): (4419198/860)
         IV size: 8 bytes
         replay detection support: Y
        Status: ACTIVE

Default IPSec Anti-Replay window is 64

Packets received outside the window are dropped

Re-ordering of packets could happen due to QoS on the encrypting router (Spoke) or in the Transit Network

In current Cisco IOS versions, the Anti-Replay window can be increased up to 1024, or diabled altogether

                 crypto ipsec security-association window-size

                 crypto ipsec security-association replay disable

Not recommended to disable anti-replay; first try to fix the QoS issue in the network or encrypting router; give better QoS to Voice traffic, or use crypto LLQ; then try to increase the anti-replay window size.