I have setup a VPN from the Cisco VPN client v5 to my PIX, but when I try to setup a VPN from my Linksys, the PIX reports "reserved not zero on payload 5", which according to Cisco documentation means that the key doesn't match. I've tried a single digit key, but get the same results.
Has anyone else done what I'm trying to do, and if so, could you offer me suggestions or configuration examples, including the Advanced button on the Linksys?
That is a VPN group configuration on the PIX. This is not supported on the BEF. The BEF uses a simple plain IPSec tunnel without the Cisco additions like the groups or XAUTH. You have to set up the PIX like this with a simple IPSec tunnel using a shared key for ISAKMP.
What you have set up is the VPN client connection for the Cisco VPN client. That works differently.
Those darn guys at Cisco. Anyway, I setup the PIX according to that article you pointed me to (thanks), and now the Linksys says that the VPN is "Connected", but I can't pass any traffic (ping, map a drive, etc.). Any suggestions as to what I should look at?
Check the logs. Does the BEF have a VPN log somewhere? It may show how far it gets. The PIX should have some debug logs for ipsec and isakmp. On IOS I would do an "debug crypto ipsec" and a "debug crypto isakmp". Then you see how far you get there, in particular whether the IPSec SA gets established or not. Once the IPSec SA is established the IPSec tunnel is up and running. If the IPSec SA is up then it is probably more a problem of some ACL/access list, NAT (i.e. the tunneled traffic is NATed although it should not), or some firewall (remember, that the source and destination IP addresses of the tunneled traffic is not rewritten, i.e. your computers must now accept incoming packets from the other LAN subnet).
The PIX should also have some "show" commands showing the status of the IPSec connection. You should find a lot more information how to troubleshoot IPSec connection on the cisco website. The PIX is probably the easier side to do the debugging...