The project proffers an intelligent two factor speaker authentication system. The word intelligent is used as the project aims to combine Artificial Intelligence with Feature Extraction Techniques (MFCC). In first stage voice authentication system that incorporates Mel Frequency Cepstral Coefficients and Vector Quantization performs Speaker Identification and Verification based on an audio signal. In second stage further verification and authentication of speaker take place using an algorithm of artificial intelligence-machine learning (AI System) namely decision tree. The AI system asks random questions related to the authorized user and verify that the reply is relevant and came from the same user as determined by the voice authentication system, i.e., AI system checks that the voice signal is not breached.