How fast should my website load?

Im trying to save my web template in Photoshop and optimize it for fast loading. The home page is very simple. Some high res images in the header and 2 small store pictures in the body. The lowest load times I seem to be able to get (after turning down the image qualities) is 36 seconds/56.6kbps, 5 seconds/512kbps, 3 seconds/1mbps. Are these fairly average load times? They seem a little high to me.