Computer vision tasks often require the robust fit of a model to some data. In a robust fit, two major steps should be taken: i) robustly estimate the parameters of a model, and ii) differentiate inliers from outliers. We propose a new estimator called Adaptive-Scale Residual Consensus (ASRC). ASRC scores a model based on both the residuals of inliers and the corresponding scale estimate determined by those inliers. ASRC is very robust to multiple-structural data containing a high percentage of outliers. Compared with RANSAC, ASRC requires no pre-determined inlier threshold as it can simultaneously estimate the parameters of a model and the scale of inliers belonging to that model. Experiments show that ASRC has better robustness to heavily corrupted data than other robust methods. Our experiments address two important computer vision tasks: range image segmentation and fundamental matrix calculation. However, the range of potential applications is much broader than these.