spark_DataFrames.ipynb

{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "colab": {
      "provenance": [],
      "include_colab_link": true
    },
    "kernelspec": {
      "name": "python3",
      "display_name": "Python 3"
    },
    "language_info": {
      "name": "python"
    }
  },
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "view-in-github",
        "colab_type": "text"
      },
      "source": [
        "<a href=\"https://colab.research.google.com/github/harnalashok/hadoop/blob/main/spark_DataFrames.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "4q64Z8IYb8fr"
      },
      "source": [
        "# Last amended: 16th April,2023\n",
        "# Myfolder: github/hadoop\n",
        "# Objectives:\n",
        "#             i)  Install pyspark on colab\n",
        "#             ii) Install koalas on colab\n",
        "#                 (version installed is spark-3.3.2)\n",
        "#\n",
        "# Java 8 install: https://stackoverflow.com/a/58191107\n",
        "# Hadoop install: https://hadoop.apache.org/docs/stable/hadoop-project-dist/hadoop-common/SingleCluster.html\n",
        "# Spark install:  https://stackoverflow.com/a/64183749\n",
        "#                 https://www.analyticsvidhya.com/blog/2020/11/a-must-read-guide-on-how-to-work-with-pyspark-on-google-colab-for-data-scientists/"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "GuuFSunM_rL2"
      },
      "source": [
        "# Spark Reference API\n",
        "a. [Quickstart](https://spark.apache.org/docs/latest/quick-start.html) <br>\n",
        "b. Dataframe [APIs list](https://spark.apache.org/docs/latest/api/python/reference/pyspark.sql/dataframe.html) at a glance<br>\n",
        "c. Pandas [API](https://spark.apache.org/docs/latest/api/python/reference/pyspark.pandas/index.html) on spark <br>\n",
        "c. ALso look at useful [this source code](https://spark.apache.org/docs/latest/api/python/_modules/pyspark/sql/functions.html) of functions that has examples<br>\n",
        "\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Y3vw_quGSgrW"
      },
      "source": [
        "# A. Full spark install\n",
        "Installs `pyspark (spark-3.3.2-bin-hadoop3)` .<br>\n",
        "Takes less than a minute to install"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "yC40UHIKPJEL"
      },
      "source": [
        "### 1.0 Libraries"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "HBVUrlkidyaE"
      },
      "source": [
        "# 1.0 How to set environment variable\n",
        "import os  \n",
        "import time  "
      ],
      "execution_count": 55,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "w8cVWRNoPOS6"
      },
      "source": [
        "## 2.0 Define some functions"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "mfFBQ0FLcolo"
      },
      "source": [
        "#### ssh_install()"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "mgS9HNCyR7C0"
      },
      "source": [
        "# 2.0 Function to install ssh client and sshd (Server)\n",
        "def ssh_install():\n",
        "  print(\"\\n--1. Download and install ssh server----\\n\")\n",
        "  ! sudo apt-get remove openssh-client openssh-server\n",
        "  ! sudo apt install openssh-client openssh-server\n",
        "  \n",
        "  print(\"\\n--2. Restart ssh server----\\n\")\n",
        "  ! service ssh restart"
      ],
      "execution_count": 56,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "IJNKSScPcsDS"
      },
      "source": [
        "#### Java install"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "BsFu84PSR9jR"
      },
      "source": [
        "# 3.0 Function to download and install java 8\n",
        "def install_java():\n",
        "  ! rm -rf /usr/java\n",
        "\n",
        "  print(\"\\n--Download and install Java 8----\\n\")\n",
        "  !apt-get install -y openjdk-8-jdk-headless -qq > /dev/null        # install openjdk\n",
        "  os.environ[\"JAVA_HOME\"] = \"/usr/lib/jvm/java-8-openjdk-amd64\"     # set environment variable\n",
        "\n",
        "  !update-alternatives --set java /usr/lib/jvm/java-8-openjdk-amd64/jre/bin/java\n",
        "  !update-alternatives --set javac /usr/lib/jvm/java-8-openjdk-amd64/bin/javac\n",
        "  \n",
        "  !mkdir -p /usr/java\n",
        "  ! ln -s \"/usr/lib/jvm/java-8-openjdk-amd64\"  \"/usr/java\"\n",
        "  ! mv \"/usr/java/java-8-openjdk-amd64\"  \"/usr/java/latest\"\n",
        "  \n",
        "  !java -version       #check java version\n",
        "  !javac -version"
      ],
      "execution_count": 57,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Y159J8TDc3wS"
      },
      "source": [
        "#### setup ssh passphrase"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "EOFbfw7n0Pps"
      },
      "source": [
        "# 6.0 Function tp setup ssh passphrase\n",
        "def set_keys():\n",
        "  print(\"\\n---22. Generate SSH keys----\\n\")\n",
        "  ! cd ~ ; pwd \n",
        "  ! cd ~ ; ssh-keygen -t rsa -P '' -f ~/.ssh/id_rsa\n",
        "  ! cd ~ ; cat ~/.ssh/id_rsa.pub >> ~/.ssh/authorized_keys\n",
        "  ! cd ~ ; chmod 0600 ~/.ssh/authorized_keys\n"
      ],
      "execution_count": 58,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "7M2kWg3dc6FT"
      },
      "source": [
        "#### Set environment"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "CRSn9XAV4rsR"
      },
      "source": [
        "# 7.0 Function to set up environmental variables\n",
        "def set_env():\n",
        "  print(\"\\n---23. Set Environment variables----\\n\")\n",
        "  # 'export' command does not work in colab\n",
        "  # https://stackoverflow.com/a/57240319\n",
        "  os.environ[\"JAVA_HOME\"] = \"/usr/lib/jvm/java-8-openjdk-amd64\"     #set environment variable\n",
        "  os.environ[\"JRE_HOME\"] = \"/usr/lib/jvm/java-8-openjdk-amd64/jre\"   \n",
        "  "
      ],
      "execution_count": 59,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "WII-UNCzc9qJ"
      },
      "source": [
        "#### function to install prerequisites\n",
        "java and ssh<br>\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "oh1Mi0rHFpkU"
      },
      "source": [
        "# 8.0 Function to call all functions\n",
        "def install_components():\n",
        "  print(\"\\n--Install java----\\n\")\n",
        "  ssh_install()\n",
        "  install_java()  \n",
        "  #set_keys()\n",
        "  set_env()\n"
      ],
      "execution_count": 60,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "iHJyHMhUdCRZ"
      },
      "source": [
        "## 3.0 Install components\n",
        "Start downloading, install and configure. Takes around 2 minutes<br>\n",
        "Your <u>input *'y'* is required </u>at one place while overwriting earlier ssh keys"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "77YQikvsJiTm",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "3ee7a693-ef27-451c-8a92-bb4562c00285"
      },
      "source": [
        "# 9.0 Start installation\n",
        "start = time.time()\n",
        "install_components()\n",
        "end = time.time()\n",
        "print(\"\\n---Time taken----\\n\")\n",
        "print((end- start)/60)"
      ],
      "execution_count": 61,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "\n",
            "--Install java----\n",
            "\n",
            "\n",
            "--1. Download and install ssh server----\n",
            "\n",
            "Reading package lists... Done\n",
            "Building dependency tree       \n",
            "Reading state information... Done\n",
            "The following packages were automatically installed and are no longer required:\n",
            "  libboost-atomic-dev libboost-atomic1.71-dev libboost-atomic1.71.0\n",
            "  libboost-chrono-dev libboost-chrono1.71-dev libboost-chrono1.71.0\n",
            "  libboost-container-dev libboost-container1.71-dev libboost-container1.71.0\n",
            "  libboost-context-dev libboost-context1.71-dev libboost-context1.71.0\n",
            "  libboost-coroutine-dev libboost-coroutine1.71-dev libboost-coroutine1.71.0\n",
            "  libboost-date-time-dev libboost-date-time1.71-dev libboost-date-time1.71.0\n",
            "  libboost-exception-dev libboost-exception1.71-dev libboost-fiber-dev\n",
            "  libboost-fiber1.71-dev libboost-fiber1.71.0 libboost-filesystem-dev\n",
            "  libboost-filesystem1.71-dev libboost-filesystem1.71.0 libboost-graph-dev\n",
            "  libboost-graph-parallel-dev libboost-graph-parallel1.71-dev\n",
            "  libboost-graph-parallel1.71.0 libboost-graph1.71-dev libboost-graph1.71.0\n",
            "  libboost-iostreams-dev libboost-iostreams1.71-dev libboost-locale-dev\n",
            "  libboost-locale1.71-dev libboost-locale1.71.0 libboost-log-dev\n",
            "  libboost-log1.71-dev libboost-log1.71.0 libboost-math-dev\n",
            "  libboost-math1.71-dev libboost-math1.71.0 libboost-mpi1.71.0\n",
            "  libboost-numpy-dev libboost-numpy1.71-dev libboost-numpy1.71.0\n",
            "  libboost-program-options-dev libboost-program-options1.71-dev\n",
            "  libboost-program-options1.71.0 libboost-python-dev libboost-python1.71-dev\n",
            "  libboost-python1.71.0 libboost-random-dev libboost-random1.71-dev\n",
            "  libboost-random1.71.0 libboost-regex-dev libboost-regex1.71-dev\n",
            "  libboost-regex1.71.0 libboost-serialization-dev\n",
            "  libboost-serialization1.71-dev libboost-serialization1.71.0\n",
            "  libboost-stacktrace-dev libboost-stacktrace1.71-dev\n",
            "  libboost-stacktrace1.71.0 libboost-system-dev libboost-system1.71-dev\n",
            "  libboost-system1.71.0 libboost-test-dev libboost-test1.71-dev\n",
            "  libboost-test1.71.0 libboost-thread-dev libboost-thread1.71-dev\n",
            "  libboost-timer-dev libboost-timer1.71-dev libboost-timer1.71.0\n",
            "  libboost-tools-dev libboost-type-erasure-dev libboost-type-erasure1.71-dev\n",
            "  libboost-type-erasure1.71.0 libboost-wave-dev libboost-wave1.71-dev\n",
            "  libboost-wave1.71.0 libboost1.71-tools-dev libcbor0.6 libevent-dev\n",
            "  libevent-extra-2.1-7 libevent-openssl-2.1-7 libfido2-1 libhwloc-dev\n",
            "  libibverbs-dev libnl-3-dev libnl-route-3-dev libnuma-dev ncurses-term\n",
            "  openmpi-common python3-dev python3-distro python3-distutils python3-lib2to3\n",
            "  python3.8-dev\n",
            "Use 'sudo apt autoremove' to remove them.\n",
            "The following packages will be REMOVED:\n",
            "  openssh-client openssh-server openssh-sftp-server ssh-import-id\n",
            "0 upgraded, 0 newly installed, 4 to remove and 24 not upgraded.\n",
            "After this operation, 5,942 kB disk space will be freed.\n",
            "(Reading database ... 124761 files and directories currently installed.)\n",
            "Removing openssh-server (1:8.2p1-4ubuntu0.5) ...\n",
            "invoke-rc.d: could not determine current runlevel\n",
            "invoke-rc.d: policy-rc.d denied execution of stop.\n",
            "Removing openssh-sftp-server (1:8.2p1-4ubuntu0.5) ...\n",
            "Removing ssh-import-id (5.10-0ubuntu1) ...\n",
            "Removing openssh-client (1:8.2p1-4ubuntu0.5) ...\n",
            "Processing triggers for man-db (2.9.1-1) ...\n",
            "Reading package lists... Done\n",
            "Building dependency tree       \n",
            "Reading state information... Done\n",
            "The following packages were automatically installed and are no longer required:\n",
            "  libboost-atomic-dev libboost-atomic1.71-dev libboost-atomic1.71.0\n",
            "  libboost-chrono-dev libboost-chrono1.71-dev libboost-chrono1.71.0\n",
            "  libboost-container-dev libboost-container1.71-dev libboost-container1.71.0\n",
            "  libboost-context-dev libboost-context1.71-dev libboost-context1.71.0\n",
            "  libboost-coroutine-dev libboost-coroutine1.71-dev libboost-coroutine1.71.0\n",
            "  libboost-date-time-dev libboost-date-time1.71-dev libboost-date-time1.71.0\n",
            "  libboost-exception-dev libboost-exception1.71-dev libboost-fiber-dev\n",
            "  libboost-fiber1.71-dev libboost-fiber1.71.0 libboost-filesystem-dev\n",
            "  libboost-filesystem1.71-dev libboost-filesystem1.71.0 libboost-graph-dev\n",
            "  libboost-graph-parallel-dev libboost-graph-parallel1.71-dev\n",
            "  libboost-graph-parallel1.71.0 libboost-graph1.71-dev libboost-graph1.71.0\n",
            "  libboost-iostreams-dev libboost-iostreams1.71-dev libboost-locale-dev\n",
            "  libboost-locale1.71-dev libboost-locale1.71.0 libboost-log-dev\n",
            "  libboost-log1.71-dev libboost-log1.71.0 libboost-math-dev\n",
            "  libboost-math1.71-dev libboost-math1.71.0 libboost-mpi1.71.0\n",
            "  libboost-numpy-dev libboost-numpy1.71-dev libboost-numpy1.71.0\n",
            "  libboost-program-options-dev libboost-program-options1.71-dev\n",
            "  libboost-program-options1.71.0 libboost-python-dev libboost-python1.71-dev\n",
            "  libboost-python1.71.0 libboost-random-dev libboost-random1.71-dev\n",
            "  libboost-random1.71.0 libboost-regex-dev libboost-regex1.71-dev\n",
            "  libboost-regex1.71.0 libboost-serialization-dev\n",
            "  libboost-serialization1.71-dev libboost-serialization1.71.0\n",
            "  libboost-stacktrace-dev libboost-stacktrace1.71-dev\n",
            "  libboost-stacktrace1.71.0 libboost-system-dev libboost-system1.71-dev\n",
            "  libboost-system1.71.0 libboost-test-dev libboost-test1.71-dev\n",
            "  libboost-test1.71.0 libboost-thread-dev libboost-thread1.71-dev\n",
            "  libboost-timer-dev libboost-timer1.71-dev libboost-timer1.71.0\n",
            "  libboost-tools-dev libboost-type-erasure-dev libboost-type-erasure1.71-dev\n",
            "  libboost-type-erasure1.71.0 libboost-wave-dev libboost-wave1.71-dev\n",
            "  libboost-wave1.71.0 libboost1.71-tools-dev libevent-dev libevent-extra-2.1-7\n",
            "  libevent-openssl-2.1-7 libhwloc-dev libibverbs-dev libnl-3-dev\n",
            "  libnl-route-3-dev libnuma-dev openmpi-common python3-dev python3-distutils\n",
            "  python3-lib2to3 python3.8-dev\n",
            "Use 'sudo apt autoremove' to remove them.\n",
            "The following additional packages will be installed:\n",
            "  openssh-sftp-server ssh-import-id\n",
            "Suggested packages:\n",
            "  keychain libpam-ssh monkeysphere ssh-askpass molly-guard ufw\n",
            "The following NEW packages will be installed:\n",
            "  openssh-client openssh-server openssh-sftp-server ssh-import-id\n",
            "0 upgraded, 4 newly installed, 0 to remove and 24 not upgraded.\n",
            "Need to get 1,109 kB of archives.\n",
            "After this operation, 5,942 kB of additional disk space will be used.\n",
            "Get:1 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 openssh-client amd64 1:8.2p1-4ubuntu0.5 [671 kB]\n",
            "Get:2 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 openssh-sftp-server amd64 1:8.2p1-4ubuntu0.5 [51.5 kB]\n",
            "Get:3 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 openssh-server amd64 1:8.2p1-4ubuntu0.5 [377 kB]\n",
            "Get:4 http://archive.ubuntu.com/ubuntu focal/main amd64 ssh-import-id all 5.10-0ubuntu1 [10.0 kB]\n",
            "Fetched 1,109 kB in 2s (655 kB/s)\n",
            "debconf: unable to initialize frontend: Dialog\n",
            "debconf: (No usable dialog-like program is installed, so the dialog based frontend cannot be used. at /usr/share/perl5/Debconf/FrontEnd/Dialog.pm line 76, <> line 4.)\n",
            "debconf: falling back to frontend: Readline\n",
            "debconf: unable to initialize frontend: Readline\n",
            "debconf: (This frontend requires a controlling tty.)\n",
            "debconf: falling back to frontend: Teletype\n",
            "dpkg-preconfigure: unable to re-open stdin: \n",
            "Selecting previously unselected package openssh-client.\n",
            "(Reading database ... 124675 files and directories currently installed.)\n",
            "Preparing to unpack .../openssh-client_1%3a8.2p1-4ubuntu0.5_amd64.deb ...\n",
            "Unpacking openssh-client (1:8.2p1-4ubuntu0.5) ...\n",
            "Selecting previously unselected package openssh-sftp-server.\n",
            "Preparing to unpack .../openssh-sftp-server_1%3a8.2p1-4ubuntu0.5_amd64.deb ...\n",
            "Unpacking openssh-sftp-server (1:8.2p1-4ubuntu0.5) ...\n",
            "Selecting previously unselected package openssh-server.\n",
            "Preparing to unpack .../openssh-server_1%3a8.2p1-4ubuntu0.5_amd64.deb ...\n",
            "Unpacking openssh-server (1:8.2p1-4ubuntu0.5) ...\n",
            "Selecting previously unselected package ssh-import-id.\n",
            "Preparing to unpack .../ssh-import-id_5.10-0ubuntu1_all.deb ...\n",
            "Unpacking ssh-import-id (5.10-0ubuntu1) ...\n",
            "Setting up openssh-client (1:8.2p1-4ubuntu0.5) ...\n",
            "Setting up ssh-import-id (5.10-0ubuntu1) ...\n",
            "Setting up openssh-sftp-server (1:8.2p1-4ubuntu0.5) ...\n",
            "Setting up openssh-server (1:8.2p1-4ubuntu0.5) ...\n",
            "debconf: unable to initialize frontend: Dialog\n",
            "debconf: (No usable dialog-like program is installed, so the dialog based frontend cannot be used. at /usr/share/perl5/Debconf/FrontEnd/Dialog.pm line 76.)\n",
            "debconf: falling back to frontend: Readline\n",
            "invoke-rc.d: could not determine current runlevel\n",
            "invoke-rc.d: policy-rc.d denied execution of restart.\n",
            "Processing triggers for man-db (2.9.1-1) ...\n",
            "Processing triggers for systemd (245.4-4ubuntu3.21) ...\n",
            "\n",
            "--2. Restart ssh server----\n",
            "\n",
            " * Restarting OpenBSD Secure Shell server sshd\n",
            "   ...done.\n",
            "\n",
            "--Download and install Java 8----\n",
            "\n",
            "openjdk version \"1.8.0_362\"\n",
            "OpenJDK Runtime Environment (build 1.8.0_362-8u362-ga-0ubuntu1~20.04.1-b09)\n",
            "OpenJDK 64-Bit Server VM (build 25.362-b09, mixed mode)\n",
            "javac 1.8.0_362\n",
            "\n",
            "---23. Set Environment variables----\n",
            "\n",
            "\n",
            "---Time taken----\n",
            "\n",
            "0.29629563887914023\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "XBFVWWuafLBL"
      },
      "source": [
        "## 4.0 Install spark\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "CyuLCFRJsvW3"
      },
      "source": [
        "### Define functions"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "kk3BP0OfYpT1"
      },
      "source": [
        "`findspark`: PySpark isn't on `sys.path` by default, but that doesn't mean it can't be used as a regular library. You can address this by either symlinking pyspark into your site-packages, or adding `pyspark` to `sys.path` at runtime. `findspark` does the latter."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "opLGqtPRfM_5"
      },
      "source": [
        "# 1.0 Function to download and unzip spark\n",
        "def spark_koalas_install():\n",
        "  print(\"\\n--1.1 Install findspark----\\n\")\n",
        "  !pip install -q findspark\n",
        "\n",
        "  #print(\"\\n--1.2 Install databricks Koalas----\\n\")\n",
        "  #!pip install koalas\n",
        "  \n",
        "  # This download link NEEDS TO BE CHECKED AGAIN\n",
        "  print(\"\\n--1.3 Download Apache tar.gz----\\n\")\n",
        "            \n",
        "  ! wget -c https://dlcdn.apache.org/spark/spark-3.3.2/spark-3.3.2-bin-hadoop3.tgz \n",
        "\n",
        "  print(\"\\n--1.4 Transfer downloaded content and unzip tar.gz----\\n\")\n",
        "  !  mv /content/spark*   /opt/\n",
        "  ! tar -xzf /opt/spark-3.3.2-bin-hadoop3.tgz  --directory /opt/\n",
        "\n",
        "  print(\"\\n--1.5 Check folder for files----\\n\")\n",
        "  ! ls -la /opt\n"
      ],
      "execution_count": 62,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "ebXfvQ1qiQHz"
      },
      "source": [
        "# 1.1 Function to set environment\n",
        "def set_spark_env():\n",
        "  print(\"\\n---2. Set Environment variables----\\n\")\n",
        "  os.environ[\"JAVA_HOME\"] = \"/usr/lib/jvm/java-8-openjdk-amd64\" \n",
        "  os.environ[\"JRE_HOME\"] = \"/usr/lib/jvm/java-8-openjdk-amd64/jre\" \n",
        "  os.environ[\"SPARK_HOME\"] = \"/opt/spark-3.3.2-bin-hadoop3\" \n",
        "  os.environ[\"SPARK_CONF_DIR\"] = \"/opt/spark-3.3.2-bin-hadoop3/conf\"     \n",
        "  os.environ[\"LD_LIBRARY_PATH\"] += \":/opt/spark-3.3.2-bin-hadoop3/lib/native\"\n",
        "  os.environ[\"PATH\"] += \":/opt/spark-3.3.2-bin-hadoop3/bin:/opt/spark-3.3.2-bin-hadoop3/sbin\"\n",
        "  print(\"\\n---2.1. Check Environment variables----\\n\")\n",
        "  # Check\n",
        "  ! echo $PATH\n",
        "  ! echo $LD_LIBRARY_PATH"
      ],
      "execution_count": 63,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "qUsZggDHj13U"
      },
      "source": [
        "# 1.2 Function to configure spark \n",
        "def spark_conf():\n",
        "  print(\"\\n---3. Configure spark to access hadoop----\\n\")\n",
        "  !mv /opt/spark-3.3.2-bin-hadoop3/conf/spark-env.sh.template  /opt/spark-3.3.2-bin-hadoop3/conf/spark-env.sh\n",
        "  #!echo \"HADOOP_CONF_DIR=/opt/hadoop-3.2.2/etc/hadoop/\" >> /opt/spark-3.1.2-bin-hadoop3.2/conf/spark-env.sh\n",
        "  print(\"\\n---3.1 Check ----\\n\")\n",
        "  #!cat /opt/spark-3.1.1-bin-hadoop3.2/conf/spark-env.sh"
      ],
      "execution_count": 64,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "M7kLbAFLszN3"
      },
      "source": [
        "### Install spark\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "z_eaLhtPktHJ"
      },
      "source": [
        "# 2.0 Call all the three functions\n",
        "def install_spark():\n",
        "  spark_koalas_install()\n",
        "  set_spark_env()\n",
        "  spark_conf()\n"
      ],
      "execution_count": 65,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "emaHs1XxRt5z",
        "outputId": "ca1f353f-eb03-42d8-b235-32b84984da63",
        "colab": {
          "base_uri": "https://localhost:8080/"
        }
      },
      "source": [
        "# 2.1 \n",
        "install_spark()"
      ],
      "execution_count": 66,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "\n",
            "--1.1 Install findspark----\n",
            "\n",
            "\n",
            "--1.3 Download Apache tar.gz----\n",
            "\n",
            "--2023-04-16 05:08:27--  https://dlcdn.apache.org/spark/spark-3.3.2/spark-3.3.2-bin-hadoop3.tgz\n",
            "Resolving dlcdn.apache.org (dlcdn.apache.org)... 151.101.2.132, 2a04:4e42::644\n",
            "Connecting to dlcdn.apache.org (dlcdn.apache.org)|151.101.2.132|:443... connected.\n",
            "HTTP request sent, awaiting response... 200 OK\n",
            "Length: 299360284 (285M) [application/x-gzip]\n",
            "Saving to: ‘spark-3.3.2-bin-hadoop3.tgz’\n",
            "\n",
            "spark-3.3.2-bin-had 100%[===================>] 285.49M   159MB/s    in 1.8s    \n",
            "\n",
            "2023-04-16 05:08:29 (159 MB/s) - ‘spark-3.3.2-bin-hadoop3.tgz’ saved [299360284/299360284]\n",
            "\n",
            "\n",
            "--1.4 Transfer downloaded content and unzip tar.gz----\n",
            "\n",
            "\n",
            "--1.5 Check folder for files----\n",
            "\n",
            "total 292368\n",
            "drwxr-xr-x  1 root root      4096 Apr 16 05:08 .\n",
            "drwxr-xr-x  1 root root      4096 Apr 16 04:33 ..\n",
            "drwxr-xr-x  1 root root      4096 Apr 13 13:46 google\n",
            "drwxr-xr-x  1 root root      4096 Feb  2 05:25 nvidia\n",
            "drwxr-xr-x 13  501 1000      4096 Feb 10 20:40 spark-3.3.2-bin-hadoop3\n",
            "-rw-r--r--  1 root root 299360284 Feb 10 21:28 spark-3.3.2-bin-hadoop3.tgz\n",
            "\n",
            "---2. Set Environment variables----\n",
            "\n",
            "\n",
            "---2.1. Check Environment variables----\n",
            "\n",
            "/opt/bin:/usr/local/nvidia/bin:/usr/local/cuda/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/tools/node/bin:/tools/google-cloud-sdk/bin:/opt/spark-3.3.2-bin-hadoop3/bin:/opt/spark-3.3.2-bin-hadoop3/sbin:/opt/spark-3.3.2-bin-hadoop3/bin:/opt/spark-3.3.2-bin-hadoop3/sbin\n",
            "/usr/local/nvidia/lib:/usr/local/nvidia/lib64:/opt/spark-3.3.2-bin-hadoop3/lib/native:/opt/spark-3.3.2-bin-hadoop3/lib/native\n",
            "\n",
            "---3. Configure spark to access hadoop----\n",
            "\n",
            "\n",
            "---3.1 Check ----\n",
            "\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "MYRPnQhTRdOn"
      },
      "source": [
        "# B. Call libraries\n",
        "We do not import `pandas` but `pyspark.pandas` as `ps`."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "AiO4shcjRjNO"
      },
      "source": [
        "# 3.0 Just call some libraries to test\n",
        "import numpy as np\n",
        "import os\n",
        "import time \n",
        "\n",
        "# 3.1 Get spark in sys.path\n",
        "import findspark\n",
        "findspark.init()\n",
        "\n",
        "# 3.2 Call other spark libraries\n",
        "#     Just to test\n",
        "import pyspark.pandas as ps\n",
        "from pyspark.sql import SparkSession\n",
        "#import databricks.koalas as ks"
      ],
      "execution_count": 67,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "i7lb_6xCwrZm"
      },
      "source": [
        "# 3.3\n",
        "from IPython.core.interactiveshell import InteractiveShell\n",
        "InteractiveShell.ast_node_interactivity = \"all\""
      ],
      "execution_count": 68,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "VCOPgKGO60_I",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 17
        },
        "outputId": "d0c44ee2-c59c-4e6c-85e6-423d64035873"
      },
      "source": [
        "# 3.4 Increase cell width to display wide columnar output\n",
        "from IPython.core.display import display, HTML\n",
        "display(HTML(\"<style>.container { width:100% !important; }</style>\"))"
      ],
      "execution_count": 69,
      "outputs": [
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ],
            "text/html": [
              "<style>.container { width:100% !important; }</style>"
            ]
          },
          "metadata": {}
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "iHd3YFPHO1SV"
      },
      "source": [
        "# C. Build spark session\n",
        "You can modify spark driver/executor memory here<br>\n",
        "SparkSession name is `spark`<br>\n",
        "databricks.koalas is imported as `ks`\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "k5Kb-o82T0oQ"
      },
      "source": [
        "## Modifying spark configuraion\n",
        "Increase driver and executor memory"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "3M6d5xfNJSwx",
        "outputId": "a28b3b4e-f71d-4b93-be64-bd7ab1b538c3",
        "colab": {
          "base_uri": "https://localhost:8080/"
        }
      },
      "source": [
        "# 4.0 Check template file\n",
        "! cat /opt/spark-3.3.2-bin-hadoop3/conf/spark-defaults.conf.template"
      ],
      "execution_count": 70,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "#\n",
            "# Licensed to the Apache Software Foundation (ASF) under one or more\n",
            "# contributor license agreements.  See the NOTICE file distributed with\n",
            "# this work for additional information regarding copyright ownership.\n",
            "# The ASF licenses this file to You under the Apache License, Version 2.0\n",
            "# (the \"License\"); you may not use this file except in compliance with\n",
            "# the License.  You may obtain a copy of the License at\n",
            "#\n",
            "#    http://www.apache.org/licenses/LICENSE-2.0\n",
            "#\n",
            "# Unless required by applicable law or agreed to in writing, software\n",
            "# distributed under the License is distributed on an \"AS IS\" BASIS,\n",
            "# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n",
            "# See the License for the specific language governing permissions and\n",
            "# limitations under the License.\n",
            "#\n",
            "\n",
            "# Default system properties included when running spark-submit.\n",
            "# This is useful for setting default environmental settings.\n",
            "\n",
            "# Example:\n",
            "# spark.master                     spark://master:7077\n",
            "# spark.eventLog.enabled           true\n",
            "# spark.eventLog.dir               hdfs://namenode:8021/directory\n",
            "# spark.serializer                 org.apache.spark.serializer.KryoSerializer\n",
            "# spark.driver.memory              5g\n",
            "# spark.executor.extraJavaOptions  -XX:+PrintGCDetails -Dkey=value -Dnumbers=\"one two three\"\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "SXEkJxzyNYn4"
      },
      "source": [
        "# 4.1 Create spark-defaults.conf \n",
        "! cp /opt/spark-3.3.2-bin-hadoop3/conf/spark-defaults.conf.template  /opt/spark-3.3.2-bin-hadoop3/conf/spark-defaults.conf\n"
      ],
      "execution_count": 71,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "oZD3i2MXNacY"
      },
      "source": [
        "# 4.2 Amend properties\n",
        "! echo \"spark.driver.memory 6g\" >> /opt/spark-3.3.2-bin-hadoop3/conf/spark-defaults.conf\n",
        "! echo \"spark.executor.memory 3g\" >> /opt/spark-3.3.2-bin-hadoop3/conf/spark-defaults.conf"
      ],
      "execution_count": 72,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "JtFbJ5UEObIg",
        "outputId": "f7acc383-c78e-430f-df32-22852aa75587",
        "colab": {
          "base_uri": "https://localhost:8080/"
        }
      },
      "source": [
        "# 4.3 Check now\n",
        "! cat /opt/spark-3.3.2-bin-hadoop3/conf/spark-defaults.conf"
      ],
      "execution_count": 73,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "#\n",
            "# Licensed to the Apache Software Foundation (ASF) under one or more\n",
            "# contributor license agreements.  See the NOTICE file distributed with\n",
            "# this work for additional information regarding copyright ownership.\n",
            "# The ASF licenses this file to You under the Apache License, Version 2.0\n",
            "# (the \"License\"); you may not use this file except in compliance with\n",
            "# the License.  You may obtain a copy of the License at\n",
            "#\n",
            "#    http://www.apache.org/licenses/LICENSE-2.0\n",
            "#\n",
            "# Unless required by applicable law or agreed to in writing, software\n",
            "# distributed under the License is distributed on an \"AS IS\" BASIS,\n",
            "# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n",
            "# See the License for the specific language governing permissions and\n",
            "# limitations under the License.\n",
            "#\n",
            "\n",
            "# Default system properties included when running spark-submit.\n",
            "# This is useful for setting default environmental settings.\n",
            "\n",
            "# Example:\n",
            "# spark.master                     spark://master:7077\n",
            "# spark.eventLog.enabled           true\n",
            "# spark.eventLog.dir               hdfs://namenode:8021/directory\n",
            "# spark.serializer                 org.apache.spark.serializer.KryoSerializer\n",
            "# spark.driver.memory              5g\n",
            "# spark.executor.extraJavaOptions  -XX:+PrintGCDetails -Dkey=value -Dnumbers=\"one two three\"\n",
            "spark.driver.memory 6g\n",
            "spark.executor.memory 3g\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Obzrn0jTVslw"
      },
      "source": [
        "## Stop and start SparkSession"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "JDGr6SWsOqox"
      },
      "source": [
        "# 5.0 Build spark session:\n",
        "# Stop spark, if started\n",
        "#spark.stop()\n",
        "# 5.1 Now start spark\n",
        "spark = SparkSession. \\\n",
        "                    builder. \\\n",
        "                    master(\"local[*]\"). \\\n",
        "                    appName(\"myexpt\"). \\\n",
        "                    getOrCreate()"
      ],
      "execution_count": 74,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "dFOjGwWiaNZH",
        "outputId": "37d82f54-21f9-46f1-98ea-0b0d6700ad80",
        "colab": {
          "base_uri": "https://localhost:8080/"
        }
      },
      "source": [
        "sc = spark.sparkContext\n",
        "spark.sparkContext.getConf().getAll()"
      ],
      "execution_count": 75,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "[('spark.driver.extraJavaOptions',\n",
              "  '-XX:+IgnoreUnrecognizedVMOptions --add-opens=java.base/java.lang=ALL-UNNAMED --add-opens=java.base/java.lang.invoke=ALL-UNNAMED --add-opens=java.base/java.lang.reflect=ALL-UNNAMED --add-opens=java.base/java.io=ALL-UNNAMED --add-opens=java.base/java.net=ALL-UNNAMED --add-opens=java.base/java.nio=ALL-UNNAMED --add-opens=java.base/java.util=ALL-UNNAMED --add-opens=java.base/java.util.concurrent=ALL-UNNAMED --add-opens=java.base/java.util.concurrent.atomic=ALL-UNNAMED --add-opens=java.base/sun.nio.ch=ALL-UNNAMED --add-opens=java.base/sun.nio.cs=ALL-UNNAMED --add-opens=java.base/sun.security.action=ALL-UNNAMED --add-opens=java.base/sun.util.calendar=ALL-UNNAMED --add-opens=java.security.jgss/sun.security.krb5=ALL-UNNAMED'),\n",
              " ('spark.executor.memory', '3g'),\n",
              " ('spark.app.name', 'myexpt'),\n",
              " ('spark.executor.id', 'driver'),\n",
              " ('spark.app.submitTime', '1681619872205'),\n",
              " ('spark.sql.warehouse.dir', 'file:/content/spark-warehouse'),\n",
              " ('spark.app.id', 'local-1681619874286'),\n",
              " ('spark.driver.memory', '6g'),\n",
              " ('spark.rdd.compress', 'True'),\n",
              " ('spark.driver.host', 'db2403490e92'),\n",
              " ('spark.executor.extraJavaOptions',\n",
              "  '-XX:+IgnoreUnrecognizedVMOptions --add-opens=java.base/java.lang=ALL-UNNAMED --add-opens=java.base/java.lang.invoke=ALL-UNNAMED --add-opens=java.base/java.lang.reflect=ALL-UNNAMED --add-opens=java.base/java.io=ALL-UNNAMED --add-opens=java.base/java.net=ALL-UNNAMED --add-opens=java.base/java.nio=ALL-UNNAMED --add-opens=java.base/java.util=ALL-UNNAMED --add-opens=java.base/java.util.concurrent=ALL-UNNAMED --add-opens=java.base/java.util.concurrent.atomic=ALL-UNNAMED --add-opens=java.base/sun.nio.ch=ALL-UNNAMED --add-opens=java.base/sun.nio.cs=ALL-UNNAMED --add-opens=java.base/sun.security.action=ALL-UNNAMED --add-opens=java.base/sun.util.calendar=ALL-UNNAMED --add-opens=java.security.jgss/sun.security.krb5=ALL-UNNAMED'),\n",
              " ('spark.serializer.objectStreamReset', '100'),\n",
              " ('spark.master', 'local[*]'),\n",
              " ('spark.submit.pyFiles', ''),\n",
              " ('spark.submit.deployMode', 'client'),\n",
              " ('spark.driver.port', '44131'),\n",
              " ('spark.app.startTime', '1681619872470'),\n",
              " ('spark.ui.showConsoleProgress', 'true')]"
            ]
          },
          "metadata": {},
          "execution_count": 75
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "jimCCywmPJmm",
        "outputId": "f028646f-168f-4e64-9a35-2c1d757037de"
      },
      "source": [
        "# 5.2\n",
        "print(spark.sparkContext._conf.getAll())"
      ],
      "execution_count": 76,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "[('spark.driver.extraJavaOptions', '-XX:+IgnoreUnrecognizedVMOptions --add-opens=java.base/java.lang=ALL-UNNAMED --add-opens=java.base/java.lang.invoke=ALL-UNNAMED --add-opens=java.base/java.lang.reflect=ALL-UNNAMED --add-opens=java.base/java.io=ALL-UNNAMED --add-opens=java.base/java.net=ALL-UNNAMED --add-opens=java.base/java.nio=ALL-UNNAMED --add-opens=java.base/java.util=ALL-UNNAMED --add-opens=java.base/java.util.concurrent=ALL-UNNAMED --add-opens=java.base/java.util.concurrent.atomic=ALL-UNNAMED --add-opens=java.base/sun.nio.ch=ALL-UNNAMED --add-opens=java.base/sun.nio.cs=ALL-UNNAMED --add-opens=java.base/sun.security.action=ALL-UNNAMED --add-opens=java.base/sun.util.calendar=ALL-UNNAMED --add-opens=java.security.jgss/sun.security.krb5=ALL-UNNAMED'), ('spark.executor.memory', '3g'), ('spark.app.name', 'myexpt'), ('spark.executor.id', 'driver'), ('spark.app.submitTime', '1681619872205'), ('spark.sql.warehouse.dir', 'file:/content/spark-warehouse'), ('spark.app.id', 'local-1681619874286'), ('spark.driver.memory', '6g'), ('spark.rdd.compress', 'True'), ('spark.driver.host', 'db2403490e92'), ('spark.executor.extraJavaOptions', '-XX:+IgnoreUnrecognizedVMOptions --add-opens=java.base/java.lang=ALL-UNNAMED --add-opens=java.base/java.lang.invoke=ALL-UNNAMED --add-opens=java.base/java.lang.reflect=ALL-UNNAMED --add-opens=java.base/java.io=ALL-UNNAMED --add-opens=java.base/java.net=ALL-UNNAMED --add-opens=java.base/java.nio=ALL-UNNAMED --add-opens=java.base/java.util=ALL-UNNAMED --add-opens=java.base/java.util.concurrent=ALL-UNNAMED --add-opens=java.base/java.util.concurrent.atomic=ALL-UNNAMED --add-opens=java.base/sun.nio.ch=ALL-UNNAMED --add-opens=java.base/sun.nio.cs=ALL-UNNAMED --add-opens=java.base/sun.security.action=ALL-UNNAMED --add-opens=java.base/sun.util.calendar=ALL-UNNAMED --add-opens=java.security.jgss/sun.security.krb5=ALL-UNNAMED'), ('spark.serializer.objectStreamReset', '100'), ('spark.master', 'local[*]'), ('spark.submit.pyFiles', ''), ('spark.submit.deployMode', 'client'), ('spark.driver.port', '44131'), ('spark.app.startTime', '1681619872470'), ('spark.ui.showConsoleProgress', 'true')]\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "L6XYg1IGs17n"
      },
      "source": [
        "# D. Experimentation with pyspark pandas API\n",
        "pyspark pandas is imported as `ps`"
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "### pyspark pandas Series"
      ],
      "metadata": {
        "id": "abGZCqJZt69y"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "# 1.0\n",
        "ps.Series(np.random.random(size = (10,)))"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "UwoN1fj0pPrj",
        "outputId": "52616348-7585-4506-f382-9ac87cda9b1c"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "0    0.517089\n",
              "1    0.873551\n",
              "2    0.847714\n",
              "3    0.024512\n",
              "4    0.328187\n",
              "5    0.142017\n",
              "6    0.800549\n",
              "7    0.679741\n",
              "8    0.187777\n",
              "9    0.332397\n",
              "dtype: float64"
            ]
          },
          "metadata": {},
          "execution_count": 24
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "# 1.0.1\n",
        "ps.Series([10,20,30])"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "NJbhHwj_zN-D",
        "outputId": "26dc4fa5-0a22-4c05-8b1f-40a35c0e75c4"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "0    10\n",
              "1    20\n",
              "2    30\n",
              "dtype: int64"
            ]
          },
          "metadata": {},
          "execution_count": 25
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "# 1.0.2\n",
        "ps.Series((4,9,1))"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "MgwcD_60zT59",
        "outputId": "8875de5f-4b82-46b7-ccb1-177b76b6db4b"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "0    4\n",
              "1    9\n",
              "2    1\n",
              "dtype: int64"
            ]
          },
          "metadata": {},
          "execution_count": 26
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "# 1.0.3\n",
        "ps.Series(np.random.random(size = (5)), index = ['a', 'b', 'c', 'd', 'e'])"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "gQZGabXkuJ75",
        "outputId": "4b8926b4-9ef2-4c92-cc96-f17f5ec92e68"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "a    0.964787\n",
              "b    0.336336\n",
              "c    0.542865\n",
              "d    0.801836\n",
              "e    0.679892\n",
              "dtype: float64"
            ]
          },
          "metadata": {},
          "execution_count": 27
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "# 1.0.4\n",
        "ps.Series(np.random.random(size = (5)), index = ['a', 'b', 'c', 'd', 'e']).shape"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "DKHfDFCSucrL",
        "outputId": "5803fbd7-e536-4583-d63c-4596aad8a355"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "(5,)"
            ]
          },
          "metadata": {},
          "execution_count": 28
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "# 1.0.5\n",
        "dict(ps.Series(np.random.random(size = (5)), index = ['a', 'b', 'c', 'd', 'e']))"
      ],
      "metadata": {
        "id": "1Mn3eq6bvGfP"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "# 1.0.6\n",
        "# Series to numpy:\n",
        "# Note the warnings\n",
        "ps.Series(np.random.random(size = (5)), index = ['a', 'b', 'c', 'd', 'e']).to_numpy()"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "_dYnHNAhvQKt",
        "outputId": "6be891ec-6604-4119-ac9d-0089515e2f0d"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stderr",
          "text": [
            "/opt/spark-3.3.2-bin-hadoop3/python/pyspark/pandas/utils.py:975: PandasAPIOnSparkAdviceWarning: `to_numpy` loads all data into the driver's memory. It should only be used if the resulting NumPy ndarray is expected to be small.\n",
            "  warnings.warn(message, PandasAPIOnSparkAdviceWarning)\n"
          ]
        },
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "array([0.56726725, 0.32933734, 0.01821741, 0.16558889, 0.14302725])"
            ]
          },
          "metadata": {},
          "execution_count": 30
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "# 1.1\n",
        "abc = ps.Series(np.random.random(size = (5)), index = ['a', 'b', 'c', 'd', 'e'])\n"
      ],
      "metadata": {
        "id": "OE9jxOkovgjp"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "# 1.2 Try to press tab after .(dot)\n",
        "#       and see what all methods are avaiable:\n",
        "\n",
        "abc."
      ],
      "metadata": {
        "id": "gSqlNshlw5SG"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "# 1.3\n",
        "abc.add(abc)"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "dk-asrqovmKl",
        "outputId": "9a3e5628-7f79-470e-d08a-65720c1b8232"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "a    1.766233\n",
              "b    0.250937\n",
              "c    1.370136\n",
              "d    1.962745\n",
              "e    1.675979\n",
              "dtype: float64"
            ]
          },
          "metadata": {},
          "execution_count": 33
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "### pyspark pandas DataFrame"
      ],
      "metadata": {
        "id": "pI3OmHR0o93C"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "# 2.1\n",
        "ps.DataFrame([\n",
        "               [1,2,3],\n",
        "               [4,5,6]\n",
        "              ], columns = list('abc') )"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 112
        },
        "id": "4zfroZo4oYrR",
        "outputId": "c8da26fc-c85f-4947-c471-8b5d88658e02"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "   a  b  c\n",
              "0  1  2  3\n",
              "1  4  5  6"
            ],
            "text/html": [
              "<div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>a</th>\n",
              "      <th>b</th>\n",
              "      <th>c</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>0</th>\n",
              "      <td>1</td>\n",
              "      <td>2</td>\n",
              "      <td>3</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1</th>\n",
              "      <td>4</td>\n",
              "      <td>5</td>\n",
              "      <td>6</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>"
            ]
          },
          "metadata": {},
          "execution_count": 60
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "# 2.1.1\n",
        "ps.DataFrame(\n",
        "              [\n",
        "                [1,2,3],\n",
        "                [4,5,6]\n",
        "              ],\n",
        "              columns = list('abc'),\n",
        "              index = [4,7]\n",
        "             )"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 112
        },
        "id": "7J11P7EzouxY",
        "outputId": "0257be54-8a20-4526-a783-1c1cd3149ac7"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "   a  b  c\n",
              "4  1  2  3\n",
              "7  4  5  6"
            ],
            "text/html": [
              "<div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>a</th>\n",
              "      <th>b</th>\n",
              "      <th>c</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>4</th>\n",
              "      <td>1</td>\n",
              "      <td>2</td>\n",
              "      <td>3</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>7</th>\n",
              "      <td>4</td>\n",
              "      <td>5</td>\n",
              "      <td>6</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>"
            ]
          },
          "metadata": {},
          "execution_count": 61
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "oVlRiGQJmk58",
        "outputId": "7fb87100-4b42-49dd-e905-248d8173888f",
        "colab": {
          "base_uri": "https://localhost:8080/"
        }
      },
      "source": [
        "# 2.1.2\n",
        "#     Note how nulls have been inserted.\n",
        "#     As np.nan and None\n",
        "\n",
        "pdf = ps.DataFrame(\n",
        "                   {\n",
        "                      'x1': ['a','a','b','b',np.nan, 'c', 'd','d'] * 10,\n",
        "                      'x2': ['apple',np.nan, 'orange','orange', 'peach',np.nan,'apple','orange'] * 10,\n",
        "                      'x3': [1, 1, 2, 2, 2, 4, 1, 2] *10,\n",
        "                      'x4': [2.4, None, 3.5, 1.4, 2.1,1.5, 3.0, 2.0] *10,\n",
        "                      'y1': [1, 0, 1, 0, 0, 1, 1, 0] * 10,\n",
        "                      'y2': ['yes', 'no', 'no',np.nan,None, 'yes','', 'yes'] *10,\n",
        "                      'y3': [True, True, False, None, False, True, False,False] *10\n",
        "                    }\n",
        "                   )"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stderr",
          "text": [
            "/opt/spark-3.3.2-bin-hadoop3/python/pyspark/pandas/internal.py:1573: FutureWarning: iteritems is deprecated and will be removed in a future version. Use .items instead.\n",
            "  fields = [\n",
            "/opt/spark-3.3.2-bin-hadoop3/python/pyspark/sql/pandas/conversion.py:486: FutureWarning: iteritems is deprecated and will be removed in a future version. Use .items instead.\n",
            "  for column, series in pdf.iteritems():\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "# 2.1.3\n",
        "type(pdf)\n",
        "\n",
        "print(\"\\n======\\n\")\n",
        "\n",
        "# 2.1.4\n",
        "pdf.head()"
      ],
      "metadata": {
        "id": "B60T70VVUk9t"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "# 2.1.5\n",
        "pdf.columns"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "fYhjAHp4gmtS",
        "outputId": "c90e8f86-ff61-4f01-b41d-dac521e004f6"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "Index(['x1', 'x2', 'x3', 'x4', 'y1', 'y2', 'y3'], dtype='object')"
            ]
          },
          "metadata": {},
          "execution_count": 55
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "L3pSQ7sDmnjt",
        "outputId": "6f969e10-39bd-4fa5-9a4c-57fced33e7a8",
        "colab": {
          "base_uri": "https://localhost:8080/"
        }
      },
      "source": [
        "# 2.1.6\n",
        "pdf['x1'].head(6)\n",
        "\n",
        "print(\"\\n======\\n\")\n",
        "\n",
        "# 2.1.7\n",
        "type(pdf['x1'])    # pyspark.pandas.series.Series"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "0       a\n",
              "1       a\n",
              "2       b\n",
              "3       b\n",
              "4    None\n",
              "5       c\n",
              "Name: x1, dtype: object"
            ]
          },
          "metadata": {},
          "execution_count": 42
        },
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "\n",
            "======\n",
            "\n"
          ]
        },
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "pyspark.pandas.series.Series"
            ]
          },
          "metadata": {},
          "execution_count": 42
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "# 2.2\n",
        "pdf[['x1', 'x2']].head()\n",
        "\n",
        "print(\"\\n======\\n\")\n",
        "\n",
        "# 2.3\n",
        "type(pdf[['x1', 'x2']])   # pyspark.pandas.frame.DataFrame"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 277
        },
        "id": "mSsEL1PkWCct",
        "outputId": "48b68590-34ab-4c02-9368-4bd5ab012fd4"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "  x1      x2\n",
              "0  a   apple\n",
              "1  a        \n",
              "2  b  orange\n",
              "3  b  orange\n",
              "4      peach"
            ],
            "text/html": [
              "<div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>x1</th>\n",
              "      <th>x2</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>0</th>\n",
              "      <td>a</td>\n",
              "      <td>apple</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1</th>\n",
              "      <td>a</td>\n",
              "      <td></td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2</th>\n",
              "      <td>b</td>\n",
              "      <td>orange</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>3</th>\n",
              "      <td>b</td>\n",
              "      <td>orange</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>4</th>\n",
              "      <td></td>\n",
              "      <td>peach</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>"
            ]
          },
          "metadata": {},
          "execution_count": 34
        },
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "\n",
            "======\n",
            "\n"
          ]
        },
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "pyspark.pandas.frame.DataFrame"
            ]
          },
          "metadata": {},
          "execution_count": 34
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "# 2.3.1\n",
        "pdf.shape   # (80,7)\n",
        "\n",
        "print(\"\\n======\\n\")\n",
        "\n",
        "# 2.3.2\n",
        "pdf.dtypes"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "4XdS5RBBbG4E",
        "outputId": "45fee606-5761-4384-b830-db9a850ccce2"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "(80, 7)"
            ]
          },
          "metadata": {},
          "execution_count": 48
        },
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "\n",
            "======\n",
            "\n"
          ]
        },
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "x1     object\n",
              "x2     object\n",
              "x3      int64\n",
              "x4    float64\n",
              "y1      int64\n",
              "y2     object\n",
              "y3     object\n",
              "dtype: object"
            ]
          },
          "metadata": {},
          "execution_count": 48
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "# 2.4 Only numeric features are described:\n",
        "\n",
        "pdf.describe()"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 300
        },
        "id": "sPDEiWzPbVtE",
        "outputId": "956f36c0-039b-45ba-e1e6-55c15262b57a"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "              x3         x4         y1\n",
              "count  80.000000  70.000000  80.000000\n",
              "mean    1.875000   2.271429   0.500000\n",
              "std     0.932874   0.713664   0.503155\n",
              "min     1.000000   1.400000   0.000000\n",
              "25%     1.000000   1.500000   0.000000\n",
              "50%     2.000000   2.100000   0.000000\n",
              "75%     2.000000   3.000000   1.000000\n",
              "max     4.000000   3.500000   1.000000"
            ],
            "text/html": [
              "<div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>x3</th>\n",
              "      <th>x4</th>\n",
              "      <th>y1</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>count</th>\n",
              "      <td>80.000000</td>\n",
              "      <td>70.000000</td>\n",
              "      <td>80.000000</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>mean</th>\n",
              "      <td>1.875000</td>\n",
              "      <td>2.271429</td>\n",
              "      <td>0.500000</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>std</th>\n",
              "      <td>0.932874</td>\n",
              "      <td>0.713664</td>\n",
              "      <td>0.503155</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>min</th>\n",
              "      <td>1.000000</td>\n",
              "      <td>1.400000</td>\n",
              "      <td>0.000000</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>25%</th>\n",
              "      <td>1.000000</td>\n",
              "      <td>1.500000</td>\n",
              "      <td>0.000000</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>50%</th>\n",
              "      <td>2.000000</td>\n",
              "      <td>2.100000</td>\n",
              "      <td>0.000000</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>75%</th>\n",
              "      <td>2.000000</td>\n",
              "      <td>3.000000</td>\n",
              "      <td>1.000000</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>max</th>\n",
              "      <td>4.000000</td>\n",
              "      <td>3.500000</td>\n",
              "      <td>1.000000</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>"
            ]
          },
          "metadata": {},
          "execution_count": 49
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "# 2.5 Describing cat features:\n",
        "\n",
        "pdf[['x1', 'x2']].describe()"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 175
        },
        "id": "rX3Rt5aObgAk",
        "outputId": "51a10b3f-ee84-4237-faf8-1e4b01a293de"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "        x1      x2\n",
              "count   70      60\n",
              "unique   4       3\n",
              "top      d  orange\n",
              "freq    20      30"
            ],
            "text/html": [
              "<div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>x1</th>\n",
              "      <th>x2</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>count</th>\n",
              "      <td>70</td>\n",
              "      <td>60</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>unique</th>\n",
              "      <td>4</td>\n",
              "      <td>3</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>top</th>\n",
              "      <td>d</td>\n",
              "      <td>orange</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>freq</th>\n",
              "      <td>20</td>\n",
              "      <td>30</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>"
            ]
          },
          "metadata": {},
          "execution_count": 50
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "# 2.5.1\n",
        "pdf.info()"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "A84BUGymfWpD",
        "outputId": "1ed04487-9ed0-4d73-fc74-c525e86f14af"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "<class 'pyspark.pandas.frame.DataFrame'>\n",
            "Int64Index: 80 entries, 0 to 79\n",
            "Data columns (total 7 columns):\n",
            " #   Column  Non-Null Count  Dtype  \n",
            "---  ------  --------------  -----  \n",
            " 0   x1      70 non-null     object \n",
            " 1   x2      60 non-null     object \n",
            " 2   x3      80 non-null     int64  \n",
            " 3   x4      70 non-null     float64\n",
            " 4   y1      80 non-null     int64  \n",
            " 5   y2      60 non-null     object \n",
            " 6   y3      70 non-null     object \n",
            "dtypes: float64(1), int64(2), object(4)"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "# 3.0 Slicing:\n",
        "\n",
        "pdf.iloc[:2, :4]"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 112
        },
        "id": "riNF6jiOWfY-",
        "outputId": "c031ffc5-8540-40fe-b9f6-5fd1c8c56d9b"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "  x1     x2  x3   x4\n",
              "0  a  apple   1  2.4\n",
              "1  a   None   1  NaN"
            ],
            "text/html": [
              "<div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>x1</th>\n",
              "      <th>x2</th>\n",
              "      <th>x3</th>\n",
              "      <th>x4</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>0</th>\n",
              "      <td>a</td>\n",
              "      <td>apple</td>\n",
              "      <td>1</td>\n",
              "      <td>2.4</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1</th>\n",
              "      <td>a</td>\n",
              "      <td>None</td>\n",
              "      <td>1</td>\n",
              "      <td>NaN</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>"
            ]
          },
          "metadata": {},
          "execution_count": 52
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "# 3.1 Slicing:\n",
        "\n",
        "pdf.iloc[[1,3,7], [2,4]]"
      ],
      "metadata": {
        "id": "K-r6ksTC3xrW",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 143
        },
        "outputId": "cd4678f5-719a-4da8-c527-a2b3871c2cff"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "   x3  y1\n",
              "1   1   0\n",
              "3   2   0\n",
              "7   2   0"
            ],
            "text/html": [
              "<div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>x3</th>\n",
              "      <th>y1</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>1</th>\n",
              "      <td>1</td>\n",
              "      <td>0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>3</th>\n",
              "      <td>2</td>\n",
              "      <td>0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>7</th>\n",
              "      <td>2</td>\n",
              "      <td>0</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>"
            ]
          },
          "metadata": {},
          "execution_count": 57
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "# 3.2 Slicing:\n",
        "\n",
        "pdf.loc[[1,3,7], ['x1', 'x2']]"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 143
        },
        "id": "Y69CfHQigZK0",
        "outputId": "5634eccc-d368-440b-af83-298dab0e6995"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "  x1      x2\n",
              "1  a    None\n",
              "3  b  orange\n",
              "7  d  orange"
            ],
            "text/html": [
              "<div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>x1</th>\n",
              "      <th>x2</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>1</th>\n",
              "      <td>a</td>\n",
              "      <td>None</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>3</th>\n",
              "      <td>b</td>\n",
              "      <td>orange</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>7</th>\n",
              "      <td>d</td>\n",
              "      <td>orange</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>"
            ]
          },
          "metadata": {},
          "execution_count": 59
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "### Operations on Series"
      ],
      "metadata": {
        "id": "0gU3NGgKq5G5"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "# 4.0 Addition\n",
        "https://stackoverflow.com/a/73701464\n",
        "ps.Series(np.random.random(size = (10,))) + ps.Series(np.random.random(size = (10,)))"
      ],
      "metadata": {
        "id": "yctFCWjkqxqo"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "# 4.1 Set option: Default is False to avoid excessive computation:\n",
        "\n",
        "ps.set_option('compute.ops_on_diff_frames', True)"
      ],
      "metadata": {
        "id": "U26Bc8X8q236"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "# 4.2\n",
        "ps.Series(np.random.random(size = (10,))) + ps.Series(np.random.random(size = (10,)))"
      ],
      "metadata": {
        "id": "nI2NvQILq4IY",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "3db961a4-7b07-436f-d540-2ab499eb0fde"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "0    1.036895\n",
              "1    0.330829\n",
              "2    0.493790\n",
              "3    1.170752\n",
              "4    0.985252\n",
              "5    1.285973\n",
              "6    1.422313\n",
              "7    0.321337\n",
              "8    0.828303\n",
              "9    0.749595\n",
              "dtype: float64"
            ]
          },
          "metadata": {},
          "execution_count": 116
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "# 4.3 Add two series with different Index\n",
        "abc = ps.Series(\n",
        "                 np.random.normal(loc = 1.0, scale = 2.0, size = (5,) ),\n",
        "                 index = ['a', 'c', 'b', 'e', 'f']\n",
        "                )"
      ],
      "metadata": {
        "id": "quPraW4aufxd"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "# 4.3 Add two series with different Index\n",
        "# 4.3 Add two series with different Index\n",
        "cde = ps.Series(\n",
        "                 np.random.normal(loc = 1.0, scale = 2.0, size = (5,) ),\n",
        "                 index = ['a', 'b', 'c', 'e', 'k']\n",
        "                )"
      ],
      "metadata": {
        "id": "e56RpfE91D5r"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "abc\n",
        "print(\"\\n========\\n\")\n",
        "cde\n",
        "print(\"\\n========\\n\")\n",
        "abc + cde"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "T4iRU8H31e9E",
        "outputId": "a9c9fc93-27fb-4f81-bb1d-2fa45a3bb184"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "a   -0.855975\n",
              "c    3.015098\n",
              "b   -0.675212\n",
              "e    6.212318\n",
              "f    1.581511\n",
              "dtype: float64"
            ]
          },
          "metadata": {},
          "execution_count": 122
        },
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "\n",
            "========\n",
            "\n"
          ]
        },
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "a    1.676420\n",
              "b   -0.172374\n",
              "c    3.272379\n",
              "e   -4.057456\n",
              "k    6.525404\n",
              "dtype: float64"
            ]
          },
          "metadata": {},
          "execution_count": 122
        },
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "\n",
            "========\n",
            "\n"
          ]
        },
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "a    0.820444\n",
              "b   -0.847586\n",
              "c    6.287476\n",
              "e    2.154862\n",
              "f         NaN\n",
              "k         NaN\n",
              "dtype: float64"
            ]
          },
          "metadata": {},
          "execution_count": 122
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "### DataFrame operations"
      ],
      "metadata": {
        "id": "5P0-g8oD20O_"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "tr =  ps.DataFrame(  [\n",
        "                        [4,5,6],\n",
        "                        [1.2,1.3,1.4]\n",
        "                     ],\n",
        "                     columns = list('abc')\n",
        "                   )"
      ],
      "metadata": {
        "id": "P0GF2WzNFBPi"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "tr"
      ],
      "metadata": {
        "id": "WMDP8oL1FTtw",
        "outputId": "dac3f757-70bd-43d1-e0d7-cb35026ef6c7",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 112
        }
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "     a    b    c\n",
              "0  4.0  5.0  6.0\n",
              "1  1.2  1.3  1.4"
            ],
            "text/html": [
              "<div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>a</th>\n",
              "      <th>b</th>\n",
              "      <th>c</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>0</th>\n",
              "      <td>4.0</td>\n",
              "      <td>5.0</td>\n",
              "      <td>6.0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1</th>\n",
              "      <td>1.2</td>\n",
              "      <td>1.3</td>\n",
              "      <td>1.4</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>"
            ]
          },
          "metadata": {},
          "execution_count": 151
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "tr['xyz'] = [8,9]"
      ],
      "metadata": {
        "id": "PIiYxIFoFkwo"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "tr"
      ],
      "metadata": {
        "id": "XCsyiuY8F3cI",
        "outputId": "c97a9ced-2450-40c3-cd97-085a140fef59",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 112
        }
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "     a    b    c  xyz\n",
              "0  4.0  5.0  6.0    8\n",
              "1  1.2  1.3  1.4    9"
            ],
            "text/html": [
              "<div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>a</th>\n",
              "      <th>b</th>\n",
              "      <th>c</th>\n",
              "      <th>xyz</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>0</th>\n",
              "      <td>4.0</td>\n",
              "      <td>5.0</td>\n",
              "      <td>6.0</td>\n",
              "      <td>8</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1</th>\n",
              "      <td>1.2</td>\n",
              "      <td>1.3</td>\n",
              "      <td>1.4</td>\n",
              "      <td>9</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>"
            ]
          },
          "metadata": {},
          "execution_count": 154
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "tr.drop(columns = ['xyz'])\n",
        "tr.drop(['xyz'], axis = 1)"
      ],
      "metadata": {
        "id": "l0tkmbnOGabv",
        "outputId": "054cbb54-04e0-44f8-9e57-be1a25243b72",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 207
        }
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "     a    b    c\n",
              "0  4.0  5.0  6.0\n",
              "1  1.2  1.3  1.4"
            ],
            "text/html": [
              "<div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>a</th>\n",
              "      <th>b</th>\n",
              "      <th>c</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>0</th>\n",
              "      <td>4.0</td>\n",
              "      <td>5.0</td>\n",
              "      <td>6.0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1</th>\n",
              "      <td>1.2</td>\n",
              "      <td>1.3</td>\n",
              "      <td>1.4</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>"
            ]
          },
          "metadata": {},
          "execution_count": 157
        },
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "     a    b    c\n",
              "0  4.0  5.0  6.0\n",
              "1  1.2  1.3  1.4"
            ],
            "text/html": [
              "<div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>a</th>\n",
              "      <th>b</th>\n",
              "      <th>c</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>0</th>\n",
              "      <td>4.0</td>\n",
              "      <td>5.0</td>\n",
              "      <td>6.0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1</th>\n",
              "      <td>1.2</td>\n",
              "      <td>1.3</td>\n",
              "      <td>1.4</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>"
            ]
          },
          "metadata": {},
          "execution_count": 157
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "tr.rename(columns = {'xyz' : 'cde'})"
      ],
      "metadata": {
        "id": "HfLoYgzcHU4V",
        "outputId": "96c9b983-7512-467f-e77e-ad64711fbbcb",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 112
        }
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "     a    b    c  cde\n",
              "0  4.0  5.0  6.0    8\n",
              "1  1.2  1.3  1.4    9"
            ],
            "text/html": [
              "<div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>a</th>\n",
              "      <th>b</th>\n",
              "      <th>c</th>\n",
              "      <th>cde</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>0</th>\n",
              "      <td>4.0</td>\n",
              "      <td>5.0</td>\n",
              "      <td>6.0</td>\n",
              "      <td>8</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1</th>\n",
              "      <td>1.2</td>\n",
              "      <td>1.3</td>\n",
              "      <td>1.4</td>\n",
              "      <td>9</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>"
            ]
          },
          "metadata": {},
          "execution_count": 158
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "df = ps.DataFrame([\n",
        "                    ('bird', 389.0),\n",
        "                    ('bird', 24.0),\n",
        "                    ('mammal', 80.5),\n",
        "                    ('mammal', np.nan)],\n",
        "                  index=['falcon', 'parrot', 'lion', 'monkey'],\n",
        "                  columns=('class', 'max_speed')\n",
        "                  )\n",
        "\n",
        "df"
      ],
      "metadata": {
        "id": "bfKexwitHk8f",
        "outputId": "0a80a874-3048-4c23-df0d-a7b609adc438",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 175
        }
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "         class  max_speed\n",
              "falcon    bird      389.0\n",
              "parrot    bird       24.0\n",
              "lion    mammal       80.5\n",
              "monkey  mammal        NaN"
            ],
            "text/html": [
              "<div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>class</th>\n",
              "      <th>max_speed</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>falcon</th>\n",
              "      <td>bird</td>\n",
              "      <td>389.0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>parrot</th>\n",
              "      <td>bird</td>\n",
              "      <td>24.0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>lion</th>\n",
              "      <td>mammal</td>\n",
              "      <td>80.5</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>monkey</th>\n",
              "      <td>mammal</td>\n",
              "      <td>NaN</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>"
            ]
          },
          "metadata": {},
          "execution_count": 164
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "df.reset_index()\n",
        "df.reset_index(drop = True)"
      ],
      "metadata": {
        "id": "1wimjIV6IxMK",
        "outputId": "a335ce35-da47-4fb3-d6ff-58544543f3e0",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 333
        }
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "    index   class  max_speed\n",
              "0  falcon    bird      389.0\n",
              "1  parrot    bird       24.0\n",
              "2    lion  mammal       80.5\n",
              "3  monkey  mammal        NaN"
            ],
            "text/html": [
              "<div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>index</th>\n",
              "      <th>class</th>\n",
              "      <th>max_speed</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>0</th>\n",
              "      <td>falcon</td>\n",
              "      <td>bird</td>\n",
              "      <td>389.0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1</th>\n",
              "      <td>parrot</td>\n",
              "      <td>bird</td>\n",
              "      <td>24.0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2</th>\n",
              "      <td>lion</td>\n",
              "      <td>mammal</td>\n",
              "      <td>80.5</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>3</th>\n",
              "      <td>monkey</td>\n",
              "      <td>mammal</td>\n",
              "      <td>NaN</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>"
            ]
          },
          "metadata": {},
          "execution_count": 166
        },
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "    class  max_speed\n",
              "0    bird      389.0\n",
              "1    bird       24.0\n",
              "2  mammal       80.5\n",
              "3  mammal        NaN"
            ],
            "text/html": [
              "<div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>class</th>\n",
              "      <th>max_speed</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>0</th>\n",
              "      <td>bird</td>\n",
              "      <td>389.0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1</th>\n",
              "      <td>bird</td>\n",
              "      <td>24.0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2</th>\n",
              "      <td>mammal</td>\n",
              "      <td>80.5</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>3</th>\n",
              "      <td>mammal</td>\n",
              "      <td>NaN</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>"
            ]
          },
          "metadata": {},
          "execution_count": 166
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "#### Filteration"
      ],
      "metadata": {
        "id": "23bUsm8124UR"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "ps.date_range(start = '01/01/2001', end = \"01/01/2002\", periods = 24)"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "eeSGX8BJ1f6r",
        "outputId": "2def1204-fdf6-4c47-c330-53d0f2326820"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stderr",
          "text": [
            "/opt/spark-3.3.2-bin-hadoop3/python/pyspark/pandas/namespace.py:1881: FutureWarning: Argument `closed` is deprecated in favor of `inclusive`.\n",
            "  pd.date_range(\n"
          ]
        },
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "DatetimeIndex([       '2001-01-01 00:00:00', '2001-01-16 20:52:10.434782',\n",
              "               '2001-02-01 17:44:20.869565', '2001-02-17 14:36:31.304347',\n",
              "               '2001-03-05 11:28:41.739130', '2001-03-21 08:20:52.173913',\n",
              "               '2001-04-06 05:13:02.608695', '2001-04-22 02:05:13.043478',\n",
              "               '2001-05-07 22:57:23.478260', '2001-05-23 19:49:33.913043',\n",
              "               '2001-06-08 16:41:44.347826', '2001-06-24 13:33:54.782608',\n",
              "               '2001-07-10 10:26:05.217391', '2001-07-26 07:18:15.652173',\n",
              "               '2001-08-11 04:10:26.086956', '2001-08-27 01:02:36.521739',\n",
              "               '2001-09-11 21:54:46.956521', '2001-09-27 18:46:57.391304',\n",
              "               '2001-10-13 15:39:07.826086', '2001-10-29 12:31:18.260869',\n",
              "               '2001-11-14 09:23:28.695652', '2001-11-30 06:15:39.130434',\n",
              "               '2001-12-16 03:07:49.565217',        '2002-01-01 00:00:00'],\n",
              "              dtype='datetime64[ns]', freq=None)"
            ]
          },
          "metadata": {},
          "execution_count": 135
        },
        {
          "output_type": "stream",
          "name": "stderr",
          "text": [
            "/opt/spark-3.3.2-bin-hadoop3/python/pyspark/pandas/namespace.py:1881: FutureWarning: Argument `closed` is deprecated in favor of `inclusive`.\n",
            "  pd.date_range(\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "ux = ps.date_range(start = '01-01-2001', end = '01-01-2002', periods = 30)"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "YWxtbjKC3Uc8",
        "outputId": "c2dd024f-7b2e-4f21-ead9-07d56c8e5398"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stderr",
          "text": [
            "/opt/spark-3.3.2-bin-hadoop3/python/pyspark/pandas/namespace.py:1881: FutureWarning: Argument `closed` is deprecated in favor of `inclusive`.\n",
            "  pd.date_range(\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "ux"
      ],
      "metadata": {
        "id": "2dPOiZPDE6kN",
        "outputId": "1ecb0e73-755d-4780-8a56-e0a6afefc582",
        "colab": {
          "base_uri": "https://localhost:8080/"
        }
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "DatetimeIndex([       '2001-01-01 00:00:00', '2001-01-13 14:04:08.275862',\n",
              "               '2001-01-26 04:08:16.551724', '2001-02-07 18:12:24.827586',\n",
              "               '2001-02-20 08:16:33.103448', '2001-03-04 22:20:41.379310',\n",
              "               '2001-03-17 12:24:49.655172', '2001-03-30 02:28:57.931034',\n",
              "               '2001-04-11 16:33:06.206896', '2001-04-24 06:37:14.482758',\n",
              "               '2001-05-06 20:41:22.758620', '2001-05-19 10:45:31.034482',\n",
              "               '2001-06-01 00:49:39.310344', '2001-06-13 14:53:47.586206',\n",
              "               '2001-06-26 04:57:55.862068', '2001-07-08 19:02:04.137931',\n",
              "               '2001-07-21 09:06:12.413793', '2001-08-02 23:10:20.689655',\n",
              "               '2001-08-15 13:14:28.965517', '2001-08-28 03:18:37.241379',\n",
              "               '2001-09-09 17:22:45.517241', '2001-09-22 07:26:53.793103',\n",
              "               '2001-10-04 21:31:02.068965', '2001-10-17 11:35:10.344827',\n",
              "               '2001-10-30 01:39:18.620689', '2001-11-11 15:43:26.896551',\n",
              "               '2001-11-24 05:47:35.172413', '2001-12-06 19:51:43.448275',\n",
              "               '2001-12-19 09:55:51.724137',        '2002-01-01 00:00:00'],\n",
              "              dtype='datetime64[ns]', freq=None)"
            ]
          },
          "metadata": {},
          "execution_count": 146
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "gx = ps.DataFrame(np.random.random(size = (30,)))\n",
        "gx"
      ],
      "metadata": {
        "id": "xlC3l-zfEBB7"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "gx['tr'] = ux"
      ],
      "metadata": {
        "id": "H8tsTNFSEofT"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "gx.columns"
      ],
      "metadata": {
        "id": "QRqhWq9nEYZI",
        "outputId": "961004fa-91eb-4584-b0d8-3749afcef9bb",
        "colab": {
          "base_uri": "https://localhost:8080/"
        }
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "Index([0, 'tr'], dtype='object')"
            ]
          },
          "metadata": {},
          "execution_count": 145
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "ind.dtype"
      ],
      "metadata": {
        "id": "s1ze3Ec8CF-8",
        "outputId": "74080376-3c86-464c-c312-ef952a050a65",
        "colab": {
          "base_uri": "https://localhost:8080/"
        }
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "dtype('<M8[ns]')"
            ]
          },
          "metadata": {},
          "execution_count": 131
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "ps.DataFrame(np.random.random(size = (30, 5)) )"
      ],
      "metadata": {
        "id": "89FF-MMZ4LZ0"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "# E. SparkSQL engine\n",
        "API is [here](https://spark.apache.org/docs/latest/api/python/reference/pyspark.sql/dataframe.html)"
      ],
      "metadata": {
        "id": "r7XXANDaW8xe"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "# Create a pyspark pandas DataFrame\n",
        "#     Note how nulls have been inserted.\n",
        "#     As np.nan and None\n",
        "\n",
        "pdf = ps.DataFrame(\n",
        "                   {\n",
        "                      'x1': ['a','a','b','b',np.nan, 'c', 'd','d'] * 10,\n",
        "                      'x2': ['apple',np.nan, 'orange','orange', 'peach',np.nan,'apple','orange'] * 10,\n",
        "                      'x3': [1, 1, 2, 2, 2, 4, 1, 2] *10,\n",
        "                      'x4': [2.4, None, 3.5, 1.4, 2.1,1.5, 3.0, 2.0] *10,\n",
        "                      'y1': [1, 0, 1, 0, 0, 1, 1, 0] * 10,\n",
        "                      'y2': ['yes', 'no', 'no',np.nan,None, 'yes','', 'yes'] *10,\n",
        "                      'y3': [True, True, False, None, False, True, False,False] *10\n",
        "                    }\n",
        "                   )"
      ],
      "metadata": {
        "id": "KRBWANL0XIF2",
        "outputId": "7076d879-507e-427e-f0de-632a51da17a5",
        "colab": {
          "base_uri": "https://localhost:8080/"
        }
      },
      "execution_count": 77,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stderr",
          "text": [
            "/opt/spark-3.3.2-bin-hadoop3/python/pyspark/pandas/internal.py:1573: FutureWarning: iteritems is deprecated and will be removed in a future version. Use .items instead.\n",
            "  fields = [\n",
            "/opt/spark-3.3.2-bin-hadoop3/python/pyspark/sql/pandas/conversion.py:486: FutureWarning: iteritems is deprecated and will be removed in a future version. Use .items instead.\n",
            "  for column, series in pdf.iteritems():\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "# Have a look:\n",
        "pdf.head()"
      ],
      "metadata": {
        "id": "EOkOP8I74GjD",
        "outputId": "2ca2a7c3-6b90-4041-a689-7ec38b42a36e",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 206
        }
      },
      "execution_count": 78,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "     x1      x2  x3   x4  y1    y2     y3\n",
              "0     a   apple   1  2.4   1   yes   True\n",
              "1     a    None   1  NaN   0    no   True\n",
              "2     b  orange   2  3.5   1    no  False\n",
              "3     b  orange   2  1.4   0  None   None\n",
              "4  None   peach   2  2.1   0  None  False"
            ],
            "text/html": [
              "<div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>x1</th>\n",
              "      <th>x2</th>\n",
              "      <th>x3</th>\n",
              "      <th>x4</th>\n",
              "      <th>y1</th>\n",
              "      <th>y2</th>\n",
              "      <th>y3</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>0</th>\n",
              "      <td>a</td>\n",
              "      <td>apple</td>\n",
              "      <td>1</td>\n",
              "      <td>2.4</td>\n",
              "      <td>1</td>\n",
              "      <td>yes</td>\n",
              "      <td>True</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1</th>\n",
              "      <td>a</td>\n",
              "      <td>None</td>\n",
              "      <td>1</td>\n",
              "      <td>NaN</td>\n",
              "      <td>0</td>\n",
              "      <td>no</td>\n",
              "      <td>True</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2</th>\n",
              "      <td>b</td>\n",
              "      <td>orange</td>\n",
              "      <td>2</td>\n",
              "      <td>3.5</td>\n",
              "      <td>1</td>\n",
              "      <td>no</td>\n",
              "      <td>False</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>3</th>\n",
              "      <td>b</td>\n",
              "      <td>orange</td>\n",
              "      <td>2</td>\n",
              "      <td>1.4</td>\n",
              "      <td>0</td>\n",
              "      <td>None</td>\n",
              "      <td>None</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>4</th>\n",
              "      <td>None</td>\n",
              "      <td>peach</td>\n",
              "      <td>2</td>\n",
              "      <td>2.1</td>\n",
              "      <td>0</td>\n",
              "      <td>None</td>\n",
              "      <td>False</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>"
            ]
          },
          "metadata": {},
          "execution_count": 78
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "pdf.index"
      ],
      "metadata": {
        "id": "UiMMr5q3Xul2",
        "outputId": "af6c919a-0ac5-44d1-ac7d-368813e10437",
        "colab": {
          "base_uri": "https://localhost:8080/"
        }
      },
      "execution_count": 79,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "Int64Index([ 0,  1,  2,  3,  4,  5,  6,  7,  8,  9, 10, 11, 12, 13, 14, 15, 16,\n",
              "            17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33,\n",
              "            34, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47, 48, 49, 50,\n",
              "            51, 52, 53, 54, 55, 56, 57, 58, 59, 60, 61, 62, 63, 64, 65, 66, 67,\n",
              "            68, 69, 70, 71, 72, 73, 74, 75, 76, 77, 78, 79],\n",
              "           dtype='int64')"
            ]
          },
          "metadata": {},
          "execution_count": 79
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "pdf.iloc[3:5, :].head()"
      ],
      "metadata": {
        "id": "G1C_uOI5fzEC",
        "outputId": "0113f327-fce6-4153-9153-938127248e96",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 112
        }
      },
      "execution_count": 83,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "     x1      x2  x3   x4  y1    y2     y3\n",
              "3     b  orange   2  1.4   0  None   None\n",
              "4  None   peach   2  2.1   0  None  False"
            ],
            "text/html": [
              "<div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>x1</th>\n",
              "      <th>x2</th>\n",
              "      <th>x3</th>\n",
              "      <th>x4</th>\n",
              "      <th>y1</th>\n",
              "      <th>y2</th>\n",
              "      <th>y3</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>3</th>\n",
              "      <td>b</td>\n",
              "      <td>orange</td>\n",
              "      <td>2</td>\n",
              "      <td>1.4</td>\n",
              "      <td>0</td>\n",
              "      <td>None</td>\n",
              "      <td>None</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>4</th>\n",
              "      <td>None</td>\n",
              "      <td>peach</td>\n",
              "      <td>2</td>\n",
              "      <td>2.1</td>\n",
              "      <td>0</td>\n",
              "      <td>None</td>\n",
              "      <td>False</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>"
            ]
          },
          "metadata": {},
          "execution_count": 83
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "# Transform to pyspark SQL Dataframe\n",
        "\n",
        "pdf_sc = pdf.to_spark()\n",
        "type(pdf_sc)"
      ],
      "metadata": {
        "id": "So8hHrvVXALl",
        "outputId": "1d1830e6-8c21-4727-82ed-ff89beb4a9e7",
        "colab": {
          "base_uri": "https://localhost:8080/"
        }
      },
      "execution_count": 80,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stderr",
          "text": [
            "/opt/spark-3.3.2-bin-hadoop3/python/pyspark/pandas/utils.py:975: PandasAPIOnSparkAdviceWarning: If `index_col` is not specified for `to_spark`, the existing index is lost when converting to Spark DataFrame.\n",
            "  warnings.warn(message, PandasAPIOnSparkAdviceWarning)\n"
          ]
        },
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "pyspark.sql.dataframe.DataFrame"
            ]
          },
          "metadata": {},
          "execution_count": 80
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "# Show the dataframe\n",
        "pdf_sc.show(3)\n",
        "print(\"\\n------\\n\")\n",
        "pdf_sc.head(3)\n",
        "print(\"\\n------\\n\")\n",
        "pdf_sc.collect()  # Shows all DataFrame\n"
      ],
      "metadata": {
        "id": "RkUcC8KfXZSk"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "# How many rows and columns\n",
        "pdf_sc.columns\n",
        "print(\"\\n------\\n\")\n",
        "len(pdf_sc.columns)\n",
        "print(\"\\n------\\n\")\n",
        "pdf_sc.count()"
      ],
      "metadata": {
        "id": "AjU1LkwJYDdj",
        "outputId": "ef5de2cc-8d94-4f91-c6e2-8f20a1c92ac5",
        "colab": {
          "base_uri": "https://localhost:8080/"
        }
      },
      "execution_count": 84,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "['x1', 'x2', 'x3', 'x4', 'y1', 'y2', 'y3']"
            ]
          },
          "metadata": {},
          "execution_count": 84
        },
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "\n",
            "------\n",
            "\n"
          ]
        },
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "7"
            ]
          },
          "metadata": {},
          "execution_count": 84
        },
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "\n",
            "------\n",
            "\n"
          ]
        },
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "80"
            ]
          },
          "metadata": {},
          "execution_count": 84
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "pdf[['x1','x2']].head()"
      ],
      "metadata": {
        "id": "h89eYy2MgZGx",
        "outputId": "8710b62a-85d8-4b16-f015-7d49479280c4",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 206
        }
      },
      "execution_count": 85,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "     x1      x2\n",
              "0     a   apple\n",
              "1     a    None\n",
              "2     b  orange\n",
              "3     b  orange\n",
              "4  None   peach"
            ],
            "text/html": [
              "<div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>x1</th>\n",
              "      <th>x2</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>0</th>\n",
              "      <td>a</td>\n",
              "      <td>apple</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1</th>\n",
              "      <td>a</td>\n",
              "      <td>None</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2</th>\n",
              "      <td>b</td>\n",
              "      <td>orange</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>3</th>\n",
              "      <td>b</td>\n",
              "      <td>orange</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>4</th>\n",
              "      <td>None</td>\n",
              "      <td>peach</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>"
            ]
          },
          "metadata": {},
          "execution_count": 85
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "pdf_sc.select('x1', 'x2').show()"
      ],
      "metadata": {
        "id": "1XruLV_Vgjj3"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "pdf.iloc[:, 2:5].head()"
      ],
      "metadata": {
        "id": "nS53zOTDgxYr",
        "outputId": "3fbd0d87-11ef-4432-99ce-7628b765fdc0",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 206
        }
      },
      "execution_count": 90,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "   x3   x4  y1\n",
              "0   1  2.4   1\n",
              "1   1  NaN   0\n",
              "2   2  3.5   1\n",
              "3   2  1.4   0\n",
              "4   2  2.1   0"
            ],
            "text/html": [
              "<div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>x3</th>\n",
              "      <th>x4</th>\n",
              "      <th>y1</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>0</th>\n",
              "      <td>1</td>\n",
              "      <td>2.4</td>\n",
              "      <td>1</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1</th>\n",
              "      <td>1</td>\n",
              "      <td>NaN</td>\n",
              "      <td>0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2</th>\n",
              "      <td>2</td>\n",
              "      <td>3.5</td>\n",
              "      <td>1</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>3</th>\n",
              "      <td>2</td>\n",
              "      <td>1.4</td>\n",
              "      <td>0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>4</th>\n",
              "      <td>2</td>\n",
              "      <td>2.1</td>\n",
              "      <td>0</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>"
            ]
          },
          "metadata": {},
          "execution_count": 90
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "# Show specific columns and rows:\n",
        "pdf_sc.select('x1', 'x2')\n",
        "print(\"\\n------\\n\")\n",
        "pdf_sc.select(pdf_sc.columns[2:4] ).show(3)"
      ],
      "metadata": {
        "id": "PQLg5O9PYat0",
        "outputId": "cb56fd1e-a589-47d8-97b0-cf4719e97c7c",
        "colab": {
          "base_uri": "https://localhost:8080/"
        }
      },
      "execution_count": 103,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "DataFrame[x1: string, x2: string]"
            ]
          },
          "metadata": {},
          "execution_count": 103
        },
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "\n",
            "------\n",
            "\n",
            "+---+----+\n",
            "| x3|  x4|\n",
            "+---+----+\n",
            "|  1| 2.4|\n",
            "|  1|null|\n",
            "|  2| 3.5|\n",
            "+---+----+\n",
            "only showing top 3 rows\n",
            "\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "pdf_sc.describe().show()"
      ],
      "metadata": {
        "id": "TleGhWOWYwdj",
        "outputId": "23b97486-18b2-498c-cc8a-95b81a6fcf0d",
        "colab": {
          "base_uri": "https://localhost:8080/"
        }
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "+-------+----+-----+------------------+------------------+------------------+----+\n",
            "|summary|  x1|   x2|                x3|                x4|                y1|  y2|\n",
            "+-------+----+-----+------------------+------------------+------------------+----+\n",
            "|  count|  70|   60|                80|                70|                80|  60|\n",
            "|   mean|null| null|             1.875|2.2714285714285714|               0.5|null|\n",
            "| stddev|null| null|0.9328736058850428|0.7136643259868996|0.5031546054266275|null|\n",
            "|    min|   a|apple|                 1|               1.4|                 0|    |\n",
            "|    max|   d|peach|                 4|               3.5|                 1| yes|\n",
            "+-------+----+-----+------------------+------------------+------------------+----+\n",
            "\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "pdf.dtypes"
      ],
      "metadata": {
        "id": "RIsShICBiRZ_",
        "outputId": "e5ac45b7-5d57-4894-98bd-abb75fc09695",
        "colab": {
          "base_uri": "https://localhost:8080/"
        }
      },
      "execution_count": 104,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "x1     object\n",
              "x2     object\n",
              "x3      int64\n",
              "x4    float64\n",
              "y1      int64\n",
              "y2     object\n",
              "y3     object\n",
              "dtype: object"
            ]
          },
          "metadata": {},
          "execution_count": 104
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "pdf_sc.schema"
      ],
      "metadata": {
        "id": "poeXnpQ4btLg",
        "outputId": "4ecd6aec-d900-4c58-fc23-ddbc539cbe74",
        "colab": {
          "base_uri": "https://localhost:8080/"
        }
      },
      "execution_count": 106,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "StructType([StructField('x1', StringType(), True), StructField('x2', StringType(), True), StructField('x3', LongType(), False), StructField('x4', DoubleType(), True), StructField('y1', LongType(), False), StructField('y2', StringType(), True), StructField('y3', BooleanType(), True)])"
            ]
          },
          "metadata": {},
          "execution_count": 106
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "See [here](https://sparkbyexamples.com/pyspark/pyspark-where-filter/)"
      ],
      "metadata": {
        "id": "2-xl9qKodE2k"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "# &, |, df.x3.isin(list)\n",
        "# SQL expression\n",
        "pdf_sc.select(\"x3\").where(\"x3 > 2.0\").show()\n",
        "# \n",
        "pdf_sc.select(\"x3\").where(pdf_sc.x3.isin(1.0,2.0)).show()"
      ],
      "metadata": {
        "id": "uDuiP9zNa_lx"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "pdf.head()"
      ],
      "metadata": {
        "id": "bXDJHylWbocw",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 206
        },
        "outputId": "c6002843-f456-4ccf-e559-74f1a03eb542"
      },
      "execution_count": 109,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "     x1      x2  x3   x4  y1    y2     y3\n",
              "0     a   apple   1  2.4   1   yes   True\n",
              "1     a    None   1  NaN   0    no   True\n",
              "2     b  orange   2  3.5   1    no  False\n",
              "3     b  orange   2  1.4   0  None   None\n",
              "4  None   peach   2  2.1   0  None  False"
            ],
            "text/html": [
              "<div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>x1</th>\n",
              "      <th>x2</th>\n",
              "      <th>x3</th>\n",
              "      <th>x4</th>\n",
              "      <th>y1</th>\n",
              "      <th>y2</th>\n",
              "      <th>y3</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>0</th>\n",
              "      <td>a</td>\n",
              "      <td>apple</td>\n",
              "      <td>1</td>\n",
              "      <td>2.4</td>\n",
              "      <td>1</td>\n",
              "      <td>yes</td>\n",
              "      <td>True</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1</th>\n",
              "      <td>a</td>\n",
              "      <td>None</td>\n",
              "      <td>1</td>\n",
              "      <td>NaN</td>\n",
              "      <td>0</td>\n",
              "      <td>no</td>\n",
              "      <td>True</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2</th>\n",
              "      <td>b</td>\n",
              "      <td>orange</td>\n",
              "      <td>2</td>\n",
              "      <td>3.5</td>\n",
              "      <td>1</td>\n",
              "      <td>no</td>\n",
              "      <td>False</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>3</th>\n",
              "      <td>b</td>\n",
              "      <td>orange</td>\n",
              "      <td>2</td>\n",
              "      <td>1.4</td>\n",
              "      <td>0</td>\n",
              "      <td>None</td>\n",
              "      <td>None</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>4</th>\n",
              "      <td>None</td>\n",
              "      <td>peach</td>\n",
              "      <td>2</td>\n",
              "      <td>2.1</td>\n",
              "      <td>0</td>\n",
              "      <td>None</td>\n",
              "      <td>False</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>"
            ]
          },
          "metadata": {},
          "execution_count": 109
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "pdf_sc.groupby('x1').agg({\"x3\" : \"min\", \"y1\" : \"max\"}).show()"
      ],
      "metadata": {
        "id": "9FAtbY5Zjdmp",
        "outputId": "37a1c4c7-1d92-4fb8-ba4c-056dadf43eb6",
        "colab": {
          "base_uri": "https://localhost:8080/"
        }
      },
      "execution_count": 112,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "+----+-------+-------+\n",
            "|  x1|max(y1)|min(x3)|\n",
            "+----+-------+-------+\n",
            "|null|      0|      2|\n",
            "|   d|      1|      1|\n",
            "|   c|      1|      4|\n",
            "|   b|      1|      2|\n",
            "|   a|      1|      1|\n",
            "+----+-------+-------+\n",
            "\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "# Create a pyspark pandas DataFrame\n",
        "#     Note how nulls have been inserted.\n",
        "#     As np.nan and None\n",
        "\n",
        "xyz = ps.DataFrame(\n",
        "                   {\n",
        "                      'x1': ['a','a','b','b',np.nan, 'c', 'd','d'] * 10,\n",
        "                      'x2': ['apple',np.nan, 'orange','orange', 'peach',np.nan,'apple','orange'] * 10,\n",
        "                      'x3': [1, 1, 2, 2, 2, 4, 1, 2] *10,\n",
        "                      'x4': [2.4, None, 3.5, 1.4, 2.1,1.5, 3.0, 2.0] *10,\n",
        "                      'y1': [1, 0, 1, 0, 0, 1, 1, 0] * 10,\n",
        "                      'y2': ['yes', 'no', 'no',np.nan,None, 'yes','', 'yes'] *10,\n",
        "                      'y3': [True, True, False, None, False, True, False,False] *10\n",
        "                    }\n",
        "                   )"
      ],
      "metadata": {
        "id": "oB007Qi3oEED",
        "outputId": "1dfe4c37-0a75-4161-d8ce-9a7d95146d1c",
        "colab": {
          "base_uri": "https://localhost:8080/"
        }
      },
      "execution_count": 123,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stderr",
          "text": [
            "/opt/spark-3.3.2-bin-hadoop3/python/pyspark/pandas/internal.py:1573: FutureWarning: iteritems is deprecated and will be removed in a future version. Use .items instead.\n",
            "  fields = [\n",
            "/opt/spark-3.3.2-bin-hadoop3/python/pyspark/sql/pandas/conversion.py:486: FutureWarning: iteritems is deprecated and will be removed in a future version. Use .items instead.\n",
            "  for column, series in pdf.iteritems():\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "uxt = xyz.to_spark()"
      ],
      "metadata": {
        "id": "oHh9-y2roHtM",
        "outputId": "7bba24f1-c14c-45e7-d9cc-8e4b4a67f758",
        "colab": {
          "base_uri": "https://localhost:8080/"
        }
      },
      "execution_count": 124,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stderr",
          "text": [
            "/opt/spark-3.3.2-bin-hadoop3/python/pyspark/pandas/utils.py:975: PandasAPIOnSparkAdviceWarning: If `index_col` is not specified for `to_spark`, the existing index is lost when converting to Spark DataFrame.\n",
            "  warnings.warn(message, PandasAPIOnSparkAdviceWarning)\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "pdf_sc.createOrReplaceTempView(\"abc\")\n",
        "uxt.createOrReplaceTempView(\"cde\")"
      ],
      "metadata": {
        "id": "FhKVKGS2lsYH"
      },
      "execution_count": 125,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "#spark.sql(\"select * from abc\").show()\n",
        "spark.sql(\"select x1,min(x3)  from abc group by x1\").show()"
      ],
      "metadata": {
        "id": "ynzO1HIKmPzm",
        "outputId": "436038df-0033-40ea-910b-a615822009b0",
        "colab": {
          "base_uri": "https://localhost:8080/"
        }
      },
      "execution_count": 122,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "+----+-------+\n",
            "|  x1|min(x3)|\n",
            "+----+-------+\n",
            "|null|      2|\n",
            "|   d|      1|\n",
            "|   c|      4|\n",
            "|   b|      2|\n",
            "|   a|      1|\n",
            "+----+-------+\n",
            "\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "spark.sql(\"select a.x1 as X1, f.x1 as xz1 from abc a,cde f where a.x3 = f.x3\").show()"
      ],
      "metadata": {
        "id": "4X8oI0wgmV_m",
        "outputId": "298a593d-bd5a-419e-c8ca-97d16db72ddf",
        "colab": {
          "base_uri": "https://localhost:8080/"
        }
      },
      "execution_count": 132,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "+---+---+\n",
            "| X1|xz1|\n",
            "+---+---+\n",
            "|  a|  a|\n",
            "|  a|  a|\n",
            "|  a|  d|\n",
            "|  a|  a|\n",
            "|  a|  a|\n",
            "|  a|  d|\n",
            "|  a|  a|\n",
            "|  a|  a|\n",
            "|  a|  d|\n",
            "|  a|  a|\n",
            "|  a|  a|\n",
            "|  a|  d|\n",
            "|  a|  a|\n",
            "|  a|  a|\n",
            "|  a|  d|\n",
            "|  a|  a|\n",
            "|  a|  a|\n",
            "|  a|  d|\n",
            "|  a|  a|\n",
            "|  a|  a|\n",
            "+---+---+\n",
            "only showing top 20 rows\n",
            "\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [],
      "metadata": {
        "id": "hxtzDg14osJs"
      },
      "execution_count": null,
      "outputs": []
    }
  ]
}