Citrus fruits are major agricultural products of China and they are rich sources of health beneficial substances. In this study, Raman spectroscopy as a rapid and non-destructive tool was employed to classify eight different citrus fruits. Baseline drift caused by fluorescence of organic compounds in the citrus samples interferes with the Raman signals. A polynomial fitting based method was adopted for baseline correction, which is a key factor both for Raman peaks assignment and subsequent pattern recognition. Principal component analysis (PCA) and hierarchical cluster analysis (HCA) were the two selected pattern recognition techniques. PCA showed the distribution of sweet oranges and mandarins, and HCA was a useful guide for detailed relationship between various citrus samples. The results demonstrated that Raman spectroscopy combined with pattern recognition techniques has substantial potential for discriminating varieties of citrus fruits.