Feature/more params (#4)

* Adds optional intra-sentence summarization
* Fixes max percentile math for group top k
* Cleans up naming
* Removes logging to avoid clutter when used as lib
* Re-evaluate after updates
* Clean up docs
* Add license and release action

.github/workflows/release.yml

@@ -0,0 +1,20 @@
+name: Upload Python Package
+
+on:
+  release:
+    types: [created]
+
+jobs:
+  deploy:
+    runs-on: ubuntu-latest
+    steps:
+    - uses: actions/checkout@v2
+    - name: Set up Python
+      uses: actions/setup-python@v1
+      with:
+        python-version: '3.x'
+    - name: Build and publish
+      uses: d1618033/gh-action-python-publish-using-poetry@master
+      with:
+        pypi_username: ${{ secrets.PYPI_USERNAME }}
+        pypi_password: ${{ secrets.PYPI_PASSWORD }}

LICENSE

@@ -0,0 +1,19 @@
+Copyright (c) 2020 crodriguez1a@github
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND,
+EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF
+MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT.
+IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM,
+DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR
+OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE
+OR OTHER DEALINGS IN THE SOFTWARE.

README.md

@@ -24,6 +24,18 @@ from bpe_summarizer import bpe_summarize
 bpe_summarize(article, percentile=99)
 ```
 
+## Parameters
+
+Parameter|Definition|Default|Type
+--|--|--|--
+`document` | A text blob with sentences delineated by punctuation | `None` | `String`
+`percentile` | Sentences that include tokens in the top kth percentile  will remain after summarization | `99` | `Float`
+`tokenizer` | A [huggingface](https://github.com/huggingface/tokenizers) `PreTrainedTokenizer` instance that relies on byte-pair-encoding | `BartTokenizer.from_pretrained("facebook/bart-large")` | `transformers.PreTrainedTokenizer`
+`apply_intra_sentence` | If `True`, summarization will be applied at both the document level and the sentence level | `False` | `False`
+`intra_sentence_percentile`| When `apply_intra_sentence` is `True`, this percentile will be applied to individual sentences | `50`* | `Float`
+
+* *Note: `intra_sentence_percentile` is ignored if its value represents less than the percentile score of the mean of tokens, otherwise the percentile score of the mean is used.*
+
 ## Examples
 
 **Human Summary**

notebooks/Evaluation Single Sentence.ipynb

@@ -2,13 +2,15 @@
  "cells": [
   {
    "cell_type": "code",
-   "execution_count": 1,
+   "execution_count": 40,
    "metadata": {},
    "outputs": [
     {
      "name": "stdout",
      "output_type": "stream",
      "text": [
+      "The autoreload extension is already loaded. To reload it, use:\n",
+      "  %reload_ext autoreload\n",
       "env: TFHUB_CACHE_DIR=tfhub/\n"
      ]
     }
@@ -21,7 +23,7 @@
   },
   {
    "cell_type": "code",
-   "execution_count": 2,
+   "execution_count": 41,
    "metadata": {},
    "outputs": [],
    "source": [
@@ -34,7 +36,7 @@
   },
   {
    "cell_type": "code",
-   "execution_count": 3,
+   "execution_count": 42,
    "metadata": {},
    "outputs": [],
    "source": [
@@ -45,7 +47,7 @@
   },
   {
    "cell_type": "code",
-   "execution_count": 22,
+   "execution_count": 43,
    "metadata": {},
    "outputs": [],
    "source": [
@@ -64,7 +66,7 @@
   },
   {
    "cell_type": "code",
-   "execution_count": 39,
+   "execution_count": 44,
    "metadata": {},
    "outputs": [
     {
@@ -73,7 +75,7 @@
        "'notification repeated overdraft'"
       ]
      },
-     "execution_count": 39,
+     "execution_count": 44,
      "metadata": {},
      "output_type": "execute_result"
     }
@@ -88,7 +90,7 @@
   },
   {
    "cell_type": "code",
-   "execution_count": 24,
+   "execution_count": 45,
    "metadata": {},
    "outputs": [
     {
@@ -97,7 +99,7 @@
        "'notification subject extended holds repeated overdraft'"
       ]
      },
-     "execution_count": 24,
+     "execution_count": 45,
      "metadata": {},
      "output_type": "execute_result"
     }
@@ -111,7 +113,7 @@
   },
   {
    "cell_type": "code",
-   "execution_count": 25,
+   "execution_count": 46,
    "metadata": {},
    "outputs": [
     {
@@ -120,7 +122,7 @@
        "'difference golden potatoes yellow potatoes'"
       ]
      },
-     "execution_count": 25,
+     "execution_count": 46,
      "metadata": {},
      "output_type": "execute_result"
     }
@@ -136,7 +138,7 @@
   },
   {
    "cell_type": "code",
-   "execution_count": 38,
+   "execution_count": 47,
    "metadata": {},
    "outputs": [
     {
@@ -145,7 +147,7 @@
        "\"what's difference golden potatoes yellow potatoes?\""
       ]
      },
-     "execution_count": 38,
+     "execution_count": 47,
      "metadata": {},
      "output_type": "execute_result"
     }