Here is what, normal situation, VPN users connect to an ASA endpoint to access internal resources via L2TP/IPSec client (RASclient) embedded on their Windows XP machines.
Now the users upgrade their machines to Windows 7 (64 bits) and guess what? The VPN tunnel didnt come up anymore (Error 789).
Debugging on the ASA device show up that the error came up during the IKE Phase 2 part, and the reason was:
All IPSec SA proposals found unacceptable!
My firsts though (after some advises from a security specialist) was to look at the encryption and hashing algorithm used for this tunnel.
My configuration was the following:
Encryption algorithm: 3DES
Integrity algorithm: MD5
(Encryption give you the confidentiality and hashing functions the integrity of the message, Diffie-Hellman is used to generate a secure channel where exchange further connection details during phase 2)
Both endpoint must support the same encryption/hashing/DH-Group to sucessfully generate an IPSec tunnel...
I wondered a little about this data and MD5, sounds me obsolete, maybe they removed it from the RASclient of new Windows releases?
Let's try, I create a new transform-set, with the same parameters (3DES and DH-Group 2) changing just MD5 into SHA-1, added to the crypto-map and guess what... it WORKED!
My conclusion is that, cause of the weaknesses of MD5, Microsoft removed it from the newer L2TP/IPSec Client (RASclient).
Hope this may help someone who had found this error too.