We propose a generalization of a PAC-learning algorithm known as the Low Degree Learning Algorithm for non-uniform distributions. We show that our algorithm works under a non-uniform distribution D if the smallest eigenvalue of the Fourier coefficient matrix of the distribution is not too small. We also show that this condition is guaranteed if there are few examples which appear with very small probability under the distribution D.