Studies on Computer-Assisted Pronunciation

Learning System for Non-native Learners based on

Speech Recognition Techniques

Chul-Ho JO

January 1999

Department of Information Science
Kyoto University

• Contents
• List of Figures
• List of Tables
• 1 Introduction
  • 1.1 Computer-Assisted Pronunciation Learning
    • 1.1.1 Advantages
    • 1.1.2 Interdisciplinary Problems
    • 1.1.3 Our Approach
      • Training Methods
      • Theoretical Basis
  • 1.2 Technical Considerations
    • 1.2.1 Speech Recognition
      • Large Vocabularies
      • Speaker Independence
      • Continuous Speech
    • 1.2.2 Multi-Media
    • 1.2.3 Platform
      • Portable LL on NoteBook PC
      • On-site LL with Virtual Reality
      • Cyber LL on the World Wide Web
  • 1.3 Thesis Outline
• 2 Linguistic Phonetics and Speech Recognition Techniques for CAPL
  • 2.1 Speech Sounds and Organs of Speech : Articulatory Phonetics
    • 2.1.1 Place and Manner of Articulation
      • Place of Articulation
      • Manner of Articulation
      • Voicing
    • 2.1.2 Vowel Diagrams
    • 2.1.3 Suprasegmentals
  • 2.2 Speech Sounds and Features : Acoustic Phonetics
    • 2.2.1 Sound Waves
    • 2.2.2 Pitch Frequency
    • 2.2.3 Loudness and Intensity
    • 2.2.4 Spectrogram and Formant
      • Spectrogram Reading
      • Formants and Vowels
  • 2.3 Speech Recognition Methods
    • 2.3.1 Signal Processing
    • 2.3.2 Hidden Markov Models (HMMs)
      • Scoring (The Forward Algorithm)
      • Segmentation (The Viterbi Algorithm)
      • Training (The Forward-Backward Algorithm)
      • Phoneme Models
    • 2.3.3 Pair-Wise Classifiers
      • Linear Discriminant Functions
      • Discriminant Functions based on Decision Theory
      • Pair-Wise Discriminant Functions
  • 2.4 The CAPL System Overview
    • 2.4.1 Component Modules
    • 2.4.2 Feedback Mechanism
    • 2.4.3 System Demonstration
      • Applications for CAPL Courseware
      • Auxiliary Tools
• 3 Automatic Pronunciation Assessment Using Hidden Markov Models
  • 3.1 Introduction
  • 3.2 Approach to Automatic Assessment
    • 3.2.1 Pronunciation Quality
      • Perceptual Measure
      • Mathematical Measure
    • 3.2.2 Assessment Strategy
  • 3.3 Visual Representation of Acoustic Features
  • 3.4 Automatic Segmentation and Scoring using HMMs
    • 3.4.1 Segmentation
    • 3.4.2 Scoring
  • 3.5 Pronunciation Error Detection using Statistical Threshold Scores
  • 3.6 Experiments
    • 3.6.1 Task 1 : Specific Mistakes (M-set)
      • Speech Material
      • Automatic Scoring
      • Human Judgement
    • 3.6.2 Task 2 : Linguistic Disparity (P-set)
      • Speech Material
      • Automatic Scoring
      • Human Judgement
    • 3.6.3 Task 3 : Prosodic Transfer (T-set)
      • Speech Material
      • Automatic Scoring
      • Human Judgement
    • 3.6.4 Correlation between Human Ratings and Combined Thresholds
  • 3.7 Conclusion
• 4 Automatic Pronunciation Instruction based on Classification of the Place and Manner of Articulation
  • 4.1 Introduction
  • 4.2 Approach to Automatic Instruction
    • 4.2.1 Pedagogical Background
    • 4.2.2 Related Works
      • Contrastive Visual and Auditory Feedback
      • Formant-Tracking Feedback
      • Real-Time Feedback of Tongue Movement in 3D
    • 4.2.3 Methodological Consideration
      • Minimal Pairs
      • Distinctive Feature Contrast
      • Tips on Correction by L1 Background
  • 4.3 Pedagogical Design for Automatic Instruction
    • 4.3.1 Consonant Categorization for the Place and Manner of Articulation
      • Categorization for the Manner of Articulation
      • Categorization for the Place of Articulation
    • 4.3.2 Vowel Diagrams mapped onto Tongue Position
      • Formants and Articulatory Information
      • Formants and Vowel Diagrams
    • 4.3.3 Automatic Pronunciation Instruction
  • 4.4 Articulatory Model for Feedback
    • 4.4.1 Pair-Wise Classifiers for Consonant Categorization
    • 4.4.2 Distributions of Formant Frequency (F1,F2) for Vowels
  • 4.5 Experiments
    • 4.5.1 Feedback Instruction for Consonants
    • 4.5.2 Feedback Instruction for Vowels
    • 4.5.3 Effect of Feedback
  • 4.6 Conclusion
• 5 Speech Rhythm Training Using Rhythm Pattern Templates
  • 5.1 Introduction
  • 5.2 Approach to Speech Rhythm Training
    • 5.2.1 Related Works
      • Fundamental Auditory Base
      • Tokushuhaku Intelligibility
    • 5.2.2 Our Training Approach
  • 5.3 Analysis of Japanese Speech Rhythm
    • 5.3.1 Characteristics of Japanese Speech Rhythm
    • 5.3.2 Effect Factors of Mora Duration
      • Number and Position of Mora
      • Inherent Phoneme Length
      • Effect of Context
      • Others
    • 5.3.3 Japanese Peculiar Speech Rhythm : Tokushuhaku
      • Moraic Obstruent (/q/)
      • Moraic Nasal (/N/)
      • Long Vowels (/a:/,/i:/,/u:/,/e:/,/o:/)
  • 5.4 Rhythm Pattern Templates
    • 5.4.1 Mora Measurement Method
    • 5.4.2 Hard Templates based on the Manner of Articulation
    • 5.4.3 Soft Templates based on the Voicing
  • 5.5 Experiments
    • 5.5.1 Speech Rhythm Error Detection
    • 5.5.2 Training based on Rhythm Pattern Templates
    • 5.5.3 Training based on Tokushuhaku Model Speech
  • 5.6 Conclusion
• 6 Conclusion
  • Design Evaluation
  • Performance Evaluation
  • Pedagogic Evaluation
• Bibliography