Pyfeat.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Padova Emotional Dataset for FER\n",
    "\n",
    "[Padova Emotional Dataset Article](https://link.springer.com/article/10.3758/s13428-022-01914-4)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- **Objective**: To offer a rich dataset of both genuine (N = 707) and posed (N = 751) emotional expressions for the six universal emotions, using 56 participants.\n",
    "- **Versions**: \n",
    "Available in original clips (with participants' body and background) and modified clips (focusing solely on the face).\n",
    "- **Validation**: \n",
    "Original dataset validated by 122 human raters and the modified dataset by 280 human raters. Includes hit rates for emotion and genuineness, mean standard deviation of genuineness, and intensity perception for each clip.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Dataset Creation\n",
    "- **Participants**: 57 participants aged 20 to 30, with one withdrawing consent, resulting in contributions from 56 participants.\n",
    "- **Experimental Setup**: Designed to record spontaneous emotions with minimal participant awareness of being filmed to preserve natural expressions.\n",
    "- **Emotion Elicitation Procedure**: Utilized a multi-modal protocol with videos, games, and other tasks to elicit a range of emotions."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Supplemental Materials\n",
    "\n",
    "### T1\n",
    "- **Identification Number**: Each subject is assigned a unique identifier.\n",
    "- **Gender**: Marked as F (female) or M (male).\n",
    "- **PEDFE Code**:\n",
    "  - The first character indicates the subject's number.\n",
    "  - The second character represents the emotion (d=disgust, h=happiness, f=fear, a=anger, s=surprise, t=sadness).\n",
    "  - The third character denotes whether the emotion is genuine (g) or posed (s).\n",
    "- **Hit Rate Accuracy Scores**: Percentage of raters correctly recognizing the emotion and its authenticity.\n",
    "- **Intensity**: Rated on scales from 0-9.\n",
    "- **Genuineness**: Rated on scales from -7 to +7.\n",
    "- **Duration**: Length of each clip in seconds."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "\n",
    "# Load Supplemental Material 1 (T1)\n",
    "t1_desc_path = 'data/Supplemental_Material_T1.csv'\n",
    "t1 = pd.read_csv(t1_desc_path)\n",
    "\n",
    "# Display the first few rows of the dataset\n",
    "print(t1.head(3))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### T2\n",
    "- **Page 1: Emotion Hit Rate**\n",
    "  - Details the average and standard deviation of hit rates for each actor in PEDFE, divided by emotion. This data provides insights into the accuracy of emotion recognition across the dataset.\n",
    "\n",
    "- **Page 2: Genuineness Hit Rate**\n",
    "   - Presents genuineness hit rates for each actor by emotion, including averages and standard deviations. This section highlights how well raters could distinguish between genuine and posed emotions.\n",
    "\n",
    "### T3\n",
    "- **Page 1: Emotion Experience**\n",
    "   - Summarizes the emotions participants reported experiencing for each task, offering a direct view into the emotional impact of the elicitation protocol.\n",
    "\n",
    "- **Page 2: Levels of Genuineness**\n",
    "   - Features ratings from participants on the genuineness of their emotions for each task, ranging from completely not genuine to completely genuine. This self-assessment provides a unique perspective on the authenticity of the expressed emotions.\n",
    "\n",
    "- **Page 3: Intensity Ratings**\n",
    "   - Contains participants' self-rated intensity for each task, on a scale from 0 (none) to 9 (strong). These ratings offer a quantifiable measure of the emotional strength participants felt during each task.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Py-Feat Toolbox\n",
    "[Pyfeat Documentation](https://py-feat.org/pages/intro.html)\n",
    "\n",
    "**Overview**\n",
    "\n",
    "The Py-Feat (Python Facial Expression Analysis Toolbox) is a comprehensive library designed for facial feature extraction, emotion recognition, and more, using advanced image processing and machine learning techniques. This toolbox provides an accessible interface to extract and analyze facial expressions from images and videos, enabling researchers to conduct in-depth studies of emotional expressions."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Available Models\n",
    "\n",
    "### Face Detection\n",
    "\n",
    "- **RetinaFace**: A robust single-stage detector that performs dense face localization in uncontrolled environments, known for its high accuracy and efficiency.\n",
    "\n",
    "- **MTCNN**: Multi-task Cascaded Convolutional Networks that excel in face detection and alignment, combining multiple CNN layers for improved performance.\n",
    "\n",
    "- **FaceBoxes**: Designed for real-time face detection on CPU, offering high accuracy, particularly in resource-constrained environments.\n",
    "\n",
    "- **img2pose**: Innovatively estimates 6 degrees of freedom (6DoF) face pose, enabling simultaneous face detection and head pose estimation in a single shot.\n",
    "\n",
    "- **img2pose-c**: A constrained variant of img2pose, optimized for frontal face images, providing superior face pose estimation accuracy within a specified angle range.\n",
    "\n",
    "Each model has unique strengths, catering to different requirements in terms of accuracy, speed, and computational resources.\n",
    "\n",
    "### Facial Landmark Detection\n",
    "\n",
    "- **MobileFaceNet**: Efficient CNNs for real-time face verification, balancing accuracy and computational demand, suitable for mobile devices.\n",
    "\n",
    "- **MobileNet**: Designed for mobile vision applications, offering fast and efficient facial landmark detection.\n",
    "\n",
    "- **PFLD (Practical Facial Landmark Detector)**: Tailored for real-time performance with high accuracy in landmark detection.\n",
    "\n",
    "## Facial Pose Estimation\n",
    "- **img2pose**: Integrates face detection and pose estimation, providing 6DoF face pose estimation in one shot.\n",
    "\n",
    "- **img2pose-c**: Specialized for frontal faces, optimized for accurate pose estimation within a limited angle range.\n",
    "\n",
    "## Action Unit Detection\n",
    "- **xgb**: XGBoost model with hinge-loss for AU07, giving binary-like outputs, indicating the proportion of trees detecting the AU.\n",
    "\n",
    "- **svm**: Uses LinearSVC for binary AU detection. For continuous-valued detections, xgb is recommended.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Initialize a Detector\n",
    "- Class that is a combination of a Face, Landmark, Action Unit, and Emotion detection model into a single object<br>\n",
    "- Available models: https://py-feat.org/pages/models.html"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from feat import Detector\n",
    "\n",
    "detector = Detector(\n",
    "    face_model=\"img2pose\",\n",
    "    landmark_model=\"mobilefacenet\",\n",
    "    au_model='xgb',\n",
    "    emotion_model=\"resmasknet\",\n",
    "    facepose_model=\"img2pose\",\n",
    "    device='cuda' #Use GPU Acceleration\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Video Recognition"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Detector\n",
    "- `face_model=\"img2pose\"`: Specifies the use of the img2pose model for face detection. Performs simultaneous (one-shot) face detection and head pose estimation.\n",
    "\n",
    "- `landmark_model=\"mobilefacenet\"`: Chooses MobileFaceNet as the model for detecting facial landmarks. MobileFaceNet is efficient and designed for mobile and embedded vision applications, balancing accuracy and computational resource use.\n",
    "\n",
    "- `au_model='xgb'`: Utilizes an XGBoost (Extreme Gradient Boosting) model for recognizing facial action units. Action units are fundamental components of facial expressions, corresponding to contractions of specific facial muscles.\n",
    "\n",
    "- `emotion_model=\"resmasknet\"`: Employs the ResMaskNet model for emotion detection. \n",
    "\n",
    "- `facepose_model=\"img2pose\"`: Uses the img2pose model to estimate the pose of the face. This includes the orientation of the face in terms of pitch, yaw, and roll angles, providing context for the facial expressions in relation to the camera perspective.\n",
    "\n",
    "- `device='cuda'`: Specifies that the GPU should be used for computation, leveraging CUDA for acceleration. \n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Processing Videos\n",
    " - https://py-feat.org/basic_tutorials/03_detector_vids.html"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Function that display a video given path\n",
    "from IPython.core.display import Video\n",
    "def play_video(video_path):\n",
    "    return Video(video_path, embed=True)\n",
    "video_directory = 'data/PEDFE_set_clips/'\n",
    "play_video(video_directory + \"1_dg_1.avi\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "\n",
    "import os\n",
    "import pandas as pd\n",
    "\n",
    "# Directory containing videos\n",
    "video_directory = 'data/PEDFE_set_clips'\n",
    "# List all video files in the directory\n",
    "video_files = [f for f in os.listdir(video_directory) if f.endswith(('.avi'))]\n",
    "\n",
    "# Process each video\n",
    "for video_file in video_files:\n",
    "    video_path = os.path.join(video_directory, video_file)\n",
    "    try:\n",
    "        # Process the video, skipping frames for efficiency, and aggregate the results\n",
    "        results = detector.detect_video(video_path, skip_frames=4, aggregate=True)\n",
    "        \n",
    "        if not results.empty:\n",
    "            # Since results are aggregated, we get a single row of results summarizing the video\n",
    "            #aggregated_results = results.iloc[0]  # Access the aggregated results       \n",
    "            #print(f\"Aggregated results for {video_file}:\")\n",
    "            # display how many predictions have been produced\n",
    "            #print(results.shape)\n",
    "            #print(aggregated_results)\n",
    "            # results = results.extract_mean()\n",
    "            selected_columns = ['mean_anger', 'mean_disgust', 'mean_fear', \"mean_happiness\", \"mean_sadness\", \"mean_surprise\", \"mean_neutral\"]\n",
    "            by_video = results.update_sessions(results[\"input\"])\n",
    "            video_means = by_video.extract_mean() \n",
    "            filtered_video_means = video_means[selected_columns]\n",
    "            video_file = video_file[:-4]\n",
    "            filtered_video_means.insert(0, 'PEDFE_code', video_file)\n",
    "            csv_file_path = \"combined_results.csv\"\n",
    "            file_exists = os.path.isfile(csv_file_path)\n",
    "            filtered_video_means.to_csv(csv_file_path, mode='a', header=not file_exists, index=False)  \n",
    "        else:\n",
    "            print(f\"No faces detected in {video_file}.\")      \n",
    "    except Exception as e:\n",
    "        print(f\"Error processing {video_path}: {e}\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "video_prediction = detector.detect_video(video_directory+ \"1_dg_1.avi\", skip_frames=10, batch_size=35)\n",
    "video_prediction=video_prediction.extract_mean()\n",
    "video_prediction.to_csv(\"result\", index=False)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Emotion Prediction"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "video_prediction.loc[[24, 72]].plot_detections(faceboxes=False, add_titles=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Emotion Development"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "axes = video_prediction.emotions.plot()"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Pyfeat",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.18"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}