Traffic sign recognition is critical for advanced driver assistant system and road infrastructure survey. Traditional traffic sign recognition algorithms can't efficiently recognize traffic signs due to its limitation, yet deep learning-based technique requires huge amount of training data before its use, which is time consuming and labor intensive. In this study, transfer learning-based method is introduced for traffic sign recognition and classification, which significantly reduces the amount of training data and alleviates computation expense using Inception-v3 model. In our experiment, Belgium Traffic Sign Database is chosen and augmented by data pre-processing technique. Subsequently the layer-wise features extracted using different convolution and pooling operations are compared and analyzed. Finally transfer learning-based model is repetitively retrained several times with fine-tuning parameters at different learning rate, and excellent reliability and repeatability are observed based on statistical analysis. The results show that transfer learning model can achieve a high-level recognition performance in traffic sign recognition, which is up to 99.18 % of recognition accuracy at 0.05 learning rate (average accuracy of 99.09 %). This study would be beneficial in other traffic infrastructure recognition such as road lane marking and roadside protection facilities, and so on.