Burstbytes

There are an abundance of ISAKMP errors that can occur during tunnel negotiation and this is widely due to the fact that the RFC is not very restrictive, it allows for a wide range of configurations and allows the peers to work it out amongst themselves.
Over multiple posts I’m going to cover most of the errors that I’ve seen out in the wild at troubleshooting IKEv2 tunnels with Strongswan, Cisco Meraki, Fortinet and many other vendors, and their likely/guaranteed fix.

It’s important to note that logging in ISAKMP exchanges are significantly different whether or not you are the responder, or the initiator of a tunnel. The RFC states that the Responder does not have to specifically say why they are denying a tunnel, they can give a generic NOTIFY reply with an error that could have multiple root causes. When investigating IKEv2 tunnel formation issues, try to always be the responder of the tunnel so that you can see exactly what you’re being sent, and why you device is rejecting the tunnel.

Most of these tests are done with Strongswan, but apply to any IKEv2 peer, including Cisco Meraki, Fortinet, Palo Alto, and more.

For part one of this guide, please see Common IKEv2 ISAKMP Negotiation errors (and how to fix them) – Part 1

If you’re just here to find a clean set of config that will work with an Enterprise device (such as forming an IKEv2 tunnel to Cisco Meraki), then check out this blog post – Forming an IKEv2 tunnel with a Cisco Meraki MX using Strongswan – swanctl.

If you haven’t set up Strongswan yet at all? Take a look at Configuring Strongswan 5.9.8 with swanctl on Raspberry Pi OS!

IDr payload missing

The IDr payload missing error sounds like a Local or Remote ID mismatch, but it actually usually is hiding a different error – the remote peer might see it as AUTHENTICATION_FAILED , but if you manually form the tunnel, it’ll display more of the error behind it:

initiating IKE_SA Phase1-Meraki[1] to MERAKI
generating IKE_SA_INIT request 0 [ SA KE No N(NATD_S_IP) N(NATD_D_IP) V ]
sending packet: from Local[500] to MerakiPeer[500] (1268 bytes)
received packet: from MerakiPeer[500] to Local[500] (248 bytes)
parsed IKE_SA_INIT response 0 [ SA KE No ]
authentication of 'Local' (myself) with pre-shared key
establishing CHILD_SA Phase2-Meraki
generating IKE_AUTH request 1 [ IDi N(INIT_CONTACT) IDr AUTH SA TSi TSr N(EAP_ONLY) ]
sending packet: from Local[500] to MerakiPeer[500] (1324 bytes)
received packet: from MerakiPeer[500] to Local[500] (76 bytes)
parsed IKE_AUTH response 1 [ N(TS_UNACCEPT) ]
IDr payload missing
generating INFORMATIONAL request 2 [ N(AUTH_FAILED) ]
sending packet: from Local[500] to MerakiPeer[500] (76 bytes)
establishing connection 'Phase2-Meraki' failed

Here you can see that the real error is TS_UNACCEPT (TS_UNACCEPTABLE) which is a subnet mismatch, instead. If you get this error, try to manually initiate the tunnel using swanctl -i -i Phase1-Meraki -c Phase2-Meraki and see the output of the parsed IKE_AUTH response.

INVALID_IKE_VERSION

This error is pretty self explanatory – this usually occurs when the remote peer is misconfigured and sends an Invalid IKE version number. This can be seen in the Packet Capture of this exchange (as it happens in Phase 1 before the packets are encrypted) – some examples I’ve seen involve the IKE version being set to 4.2 or 9.9, or just the wrong version for the config (IKEv1 instead of IKEv2, and vice versa). Also check the source and destination ports being used for the exchange, there are rare cases where this can come as a result of the wrong Source/Dst ports being used.

Transport Mode vs Tunnel Mode

If you’re forming a Site to Site VPN tunnel, and not doing a Client VPN style connection, we need to ensure the remote peer is configured in Tunnel Mode, not Transport Mode. When configured in Transport Mode, it’ll switch the traffic selectors to the /32 of the Public IP of that peer. A log might look something like this:

Feb 5 21:29:53 14[IKE] changing received traffic selectors 192.168.54.0/24=== 192.168.35.0/24 due to NAT
Feb 5 21:29:53 14[CFG] looking for a child config for 142.17.75.156/32 === 40.82.185.115/32

The tunnel will then fail to form with a TS_UNACCEPTABLE or equivalent error (since the subnets will now mismatch).

message parsing failed

invalid HASH_V1 payload length, decryption failed

The error message parsing failed generally refers to the fact that the peer is sending a packet that we can’t decrypt. Setting enc=1 to strongswan.conf will let you further analyse this log and see what specific cause it was.

I would note, a large majority of the time it’s a PSK mismatch, but it’s not uncommon for it to also be a Local ID or Remote ID mismatch depending on the remote vendors configuration. Some vendors will use this same error for different reasons, but these are by far the most common.

Jan 30 01:11:37 04[NET] <382> sending packet: from 170.39.161.111[500] to 170.39.161.108[500] (244 bytes)
Jan 30 01:11:37 09[NET] <382> received packet: from 170.39.161.108[4500] to 170.39.161.111[4500] (100 bytes)
Jan 30 01:11:37 09[IKE] <382> message parsing failed
Jan 30 01:11:37 09[NET] <382> sending packet: from 170.39.161.111[500] to 170.39.161.108[500] (68 bytes)
Jan 30 01:11:37 09[IKE] <382> ID_PROT request with message ID 0 processing failed

[ENC] invalid HASH_V1 payload length, decryption failed?
[ENC] could not decrypt payloads
[IKE] message parsing failed
[IKE] ignore malformed INFORMATIONAL request
[IKE] INFORMATIONAL_V1 request with message ID 2388990997 processing failed

ATTRIBUTES_NOT_SUPPORTED

ATTRIBUTES_NOT_SUPPORTED generally just means that the peer is sending either unusual syntax or attributes in their requests. Some examples of this could be:

Conflicting Lifetimes (in IKEv1)
Invalid/Unsupported SA Attributes

Some vendors will send their subnet combinations in a strange way, instead of sending 192.168.0.0/24, they’ll send 192.168.0.0:255.255.255.0, for example – which Strongswan can’t process.

For troubleshooting this, you want to pull their Phase1/2 config, and initiate the tunnel from the remote side whilst checking /var/log/strongswan.log, and see what Strongswan isn’t happy about.

exceeds maximum size of 524288, discarded

This error message exists when you’re using Strongswan with vici – vici’s implementation has a maximum size of 512KB (524288 bytes), meaning it can’t process too many of two things:

• A maximum amount of Child_SAs for one particular IKE_SA – trying to form over 512 Child_SAs with one IKE_SA can generally cause this issue.
• Too many Child_SAs in general – in my testing once I hit 1300 Child_SAs I was starting to run into this issue when running commands like swanctl -l, since the reply of swanctl -l had too many peers to show.

My tunnel works fine, but Child_SAs fail to form on Rekey

If you run into this issue where Phase 1 and Phase 2 successfully form at first, but at when Phase 2 reaches its lifetime or lifebyte timer it goes to form a new CREATE_CHILD_SA and then fails due to a config mismatch, it’s likely because you’re using Strongswan (or a VPN service that relies on Strongswan), in which the Perfect Forward Secrecy (PFS) group configured for Phase 2 is not verified until rekey occurs.

In this case, it’s still a configuration mismatch (one side has PFS, the other doesn’t, or both sides have different PFS values), and it’s only becoming apparent once the Child_SA expires for the first time.
You can read more about this here: https://docs.strongswan.org/docs/5.9/config/rekeying.html#_ikev2

received netlink error: Function not implemented (38)

unable to add SAD entry with SPI

If you see either of these errors, it means when your daemon of choice (Strongswan in our case, who is connecting to the kernel via Netlink) is getting a response from the Kernel that it has not implemented the function that you are trying to propose to it. Usually, this means that the Cipher configured is not supported, or installed by your Kernel.

Jul  2 05:07:10 09[CHD] <MerakiPeer|3> CHILD_SA Phase2-Meraki{6} state change: CREATED => INSTALLING
Jul  2 05:07:10 09[CHD] <MerakiPeer|3>   using 3DES_CBC for encryption
Jul  2 05:07:10 09[CHD] <MerakiPeer|3>   using HMAC_SHA1_96 for integrity
Jul  2 05:07:10 09[CHD] <MerakiPeer|3> adding inbound ESP SA
Jul  2 05:07:10 09[CHD] <MerakiPeer|3>   SPI 0xceb43d4d, src 112.112.112.223 dst 172.16.0.4
Jul  2 05:07:10 09[KNL] <MerakiPeer|3> received netlink error: Function not implemented (38)
Jul  2 05:07:10 09[KNL] <MerakiPeer|3> unable to add SAD entry with SPI ceb43d4d (FAILED)
Jul  2 05:07:10 09[CHD] <MerakiPeer|3> adding outbound ESP SA
Jul  2 05:07:10 09[CHD] <MerakiPeer|3>   SPI 0xc80eef9d, src 172.16.0.4 dst 112.112.112.223
Jul  2 05:07:10 09[KNL] <MerakiPeer|3> received netlink error: Function not implemented (38)
Jul  2 05:07:10 09[KNL] <MerakiPeer|3> unable to add SAD entry with SPI c80eef9d (FAILED)

In this case, my Kernel did not have the package installed for 3DES, so it was unable to install the Child_SA.
Depending on your distribution, you can check /proc/crypto to see if the correct kernel module has been installed.

Conclusion

I hope that this has helped to identify most of the main ISAKMP errors that can show up, and helps you with your IPSEC troubleshooting for these errors.

Covered across both guides:

NO_PROPOSAL_CHOSEN
AUTHENTICATION_FAILED
IDir ‘x.x.x.x’ does not match to ‘y.y.y.y’
“no shared key found for”
TS_UNACCEPTABLE
IDr payload missing
INVALID_IKE_VERSION
Transport Mode vs Tunnel Mode
message parsing failed
invalid HASH_V1 payload length, decryption failed
ATTRIBUTES_NOT_SUPPORTED
exceeds maximum size of 524288, discarded
My tunnel works fine, but Child_SAs fail to form on Rekey
received netlink error: Function not implemented (38)
unable to add SAD entry with SPI