In this paper, we adaptively model the appearance of objects based on Mixture of Gaussians in a joint spatial-color space (the approach is called SMOG). We propose a new SMOG-based similarity measure. SMOG captures richer information than the general color histogram because it incorporates spatial layout in addition to color. This appearance model and the similarity measure are used in a framework of Bayesian probability for tracking natural objects. In the second part of the paper, we propose an Integral Gaussian Mixture (IGM) technique, as a fast way to extract the parameters of SMOG for target candi-date. With IGM, the parameters of SMOG can be computed efficiently by using only simple arithmetic operations (addition, subtraction, division) and thus the computation is reduced to linear complexity. Experiments show that our method can successfully track objects despite changes in foreground ap-pearance, clutter, occlusion, etc.; and that it outperforms several color-histogram based methods.