I don't know that you can do it on the router, but if you're using SSH, you may be able to do it on the box you are trying to connect to. If I remember correctly you can configure SSH to ignore everything but IP's you specify.
I could be wrong, but it seems like I remember doing this on one of my Linux boxes in the past.
That kinda sucks and seems kinda dumb...but yeah, I can either use firewall rules or tcp wrappers but was hoping to prevent it at the linksys firewall level so i didn't have allow possible intruders into my internal network. Thanks for the response