Bayesian observer models have been quite successful in accounting for perceptual behavior. However, it is a common challenge to specify the two fundamental components of a Bayesian model, the prior distribution and the likelihood function, because they are formally unconstrained. We argue that a perceptual system that is adapted to the statistical structure of its environment naturally imposes constraints on its corresponding Bayesian model description. In particular, we assume the prior to reflect the stimulus distribution and the likelihood to be constrained by a sensory representation that is efficient. We show that these assumptions lead to an observer model that makes two counter-intuitive predictions: First, perceptual biases can be repulsive (i.e. biased away from the peak of the prior), which is in stark contrast to the traditional Bayesian view. Second, the model predicts that neural and stimulus noise are differentially affecting perceptual bias, with larger neural noise leading to an increase in repulsive bias while larger stimulus noise leading to a decrease. We tested our model against reported experimental data regarding two perceptual variables for which the natural stimulus statistics are known (orientation and spatial frequency of visual stimuli). We found that the model predicts the reported repulsive biases from the cardinal orientations and low spatial frequencies, respectively. Furthermore, it also accounts for the observed increase in bias with increasing levels of neural noise, as well as the relative attractive bias when comparing stimuli with high versus low stimulus noise. The model is capable of making quantitative predictions up to a scaling factor for any perceptual variable for which the stimulus statistics are known. Our results suggest that efficient coding provides a powerful constraint in specifying Bayesian observer models, and leads to successful predictions of perceptual effects that have been considered incompatible with the Bayesian framework.