UI for uploading datasets

[hjwilli]

We need an interface to allow the user to upload a new dataset and specify some information about the dataset, like which field represents the 'label' column, and which fields (if any) contain categorical or ordinal data. The backend lab server will do validation checks on the data and the UI will also need to display validation errors and warnings in an intuitive way. The errors may be about the entire dataset (i.e. the file is unreadable), or some specific part of the data (i.e. there is a NULL value on row 3, col 5). It may also be nice to display a preview of the data before doing the final upload. There should be some unit tests.

New datasets are submitted to the lab server via a PUT to '/api/v1/datasets'. See registerDatafile() method in lab\init\loadInitialDatasets.py for an example of it use.
Additional api calls will need to be added to the backend lab server to support validation.

---

First pass functionality:
Update the ui to have an add datasets button on the datasets page. Clicking will navigate to a pop-up or a new page where the user can choose a file to upload and can enter the name of the 'label' column. Submitting the request will call the existing api/v1/datasets api method which will either result in the dataset being uploaded successfully or an error being returned. Basic unit tests.

Advanced functionality (UI flow might change to something like [user clicks add datasets] -> [user chooses and submits a file] -> [preview UI shows up where user can specify column info and make final submission]):

• Data preview UI before final upload
• Target field is a choice of existing columns, not a text box
• User can specify fields that contain categorical or ordinal data
• Display validation warnings and errors

[hjwilli]

For ordinal variables, we also need user input to know how the values should be ordered... for example, if a column named 'rank' had the values 'first', 'second', and 'third', we need to know that 'first' maps to 1, 'second' to 2, and 'third' to 3. For another example a column 'metal' with 'gold', 'silver', 'bronze' might map 'gold' to 1, 'silver' to 2, and 'bronze' to 3.

[hjwilli]

The api specification for uploading datasets with ordinal and categorical features is defined here: https://github.com/EpistasisLab/pennai/blob/bc0ff561357c8d91ab20224b415d82872d02e7d0/lab/api.raml#L562-L583

[joshc0044]

The introduction of uploading datasets through the website UI, whereas previously they were manually added to the project's directory structure, highlights some necessary modifications which should be made to ensure the existing features provided by the current website remain unaffected.

Using the existing codebase from the main branch (master) as a baseline and comparing against feature branch (file_upload):

• uploading datasets are not dynamically added to the UI

The existing site has a datasets page with a collection of cards where each card/widget is an individual dataset; these currently are not built dynamically in that they are created when the page loads. As such, when adding datasets through a web form/file upload UI, the existing mechanisms creating the dataset card need to be modified in order to load the UI elements dynamically after the user uploads the file.

• as new content becomes available, have UI reflect those changes in typical modern web experience (without user having to manually refresh the page)

The feature branch file_upload provides the following as of 3/4/2019: there is a basic file upload form with a text field for manually entering the dependent column where the new dataset shows up in the UI after manually refreshing the page.

After adding new datasets there is an issue with the controls not affecting the intended dataset; for example, interacting with an existing dataset (located in project directory) by toggling the AI on/off is okay but after uploading a new dataset and interacting with the controls on the newly uploaded dataset will affect the dataset which was previously interacted upon.

• find cause and fix issue of controls not affecting the intended dataset as described above
  (having some trouble reproducing this error, need consistent way to replicate problem)

Complete the rest of the task as specified in the comments above

1. Create UI/form for uploading new datasets and entering relevant information regarding said dataset

• need a way to partially read file contents (at least first row containing column headers) in order to present to the user list of information specific to the dataset in which they are uploading - related to data preview looks like papaparse will do the trick, already in use on dataset preview page

• create mockup/functional prototype in which user can manually enter categorical and ordinal information

2. Create data preview/validation UI

• add categorical & ordinal data to dataset preview

3. Create corresponding unit test

• examine existing javascript unit tests and research potential unit test frameworks for fully rendering UI - something like jest and react

4. Polish UI

• hide form until file selected
• display server response in popup similar to 'launch experiment' button failure
• use blue button in upper left area to link to file upload page instead of menu item

[joshc0044]

Last aspects of this task that are still pending:

Advanced UI and more thorough unit tests

[joshc0044]

• dependent column dropdown - search (remove text field)

• categorical feature - close icon / x icon (the close icon might be inherently an issue with semantic ui - here & here - after updating semantic ui to the latest version other parts of the website UI break) I think issue with close buttons on selected items in dropdown not working has to do with adding onClick handler event for each option, removing onclick handler lets the dropdown menu work as intended, now need alternative way to record selected items w/o click handler for each option, will probably try to use onChange for entire dropdown (as opposed to attaching a clickhandler function for each option)

• ordinal feature, use values from entire dataset

• as selecting column keys remove as options for other fields (remove text fields inputs from cat/ord)

update Sept 16 -

There are some incongruities with the semantic ui dropdown having multiple options or at least I am having trouble getting it to work properly. A regular semantic ui dropdown with multiple items works fine in isolation (from the example at the source - https://react.semantic-ui.com/modules/dropdown/#types-multiple-selection). There is a fair amount of functionality built into the widget in which the internal logic driving the behavior is not necessarily accessible/modifiable.

Specifically, the issue is that the list of selected options is being managed at the react component state level and I am not able to properly override the default list of items the widget is managing independently on its own. For example, when a user attempts to select an option that is already being used in another field, the list of selected items is correct at the component state level but the semantic dropdown widget itself is managing a different list. I tried a few different methods attempting to override the dropdown widget's values but it doesn't work.

As an interim solution I removed the semantic ui dropdowns and replaced them with plain html select (https://developer.mozilla.org/en-US/docs/Web/HTML/Element/select). This allows a user to select multiple entries and works as intended. They are not searchable out of the box though and use the shift and ctrl keys to allow users to select multiple items or add/remove an item respectively.

This is how they look -

I added a basic currently selected box to provide some feedback for the UI.

A major limitation of the select html element is that these things can be notoriously difficult or outright impossible to style (https://developer.mozilla.org/en-US/docs/Learn/HTML/Forms/Styling_HTML_forms).

In the meantime I am using these select elements just to have something working in place.

In order to use the semantic ui dropdown I'd have to reach out for help on stackoverflow and/or a react/semantic ui forum where I can ask about accomplishing this - in order to manage a list of multiple items external to the widget requires somewhat advanced react/semantic ui knowledge that I do no have.

[hjwilli]

Closing this issue. Updates to the upload will be handled in future tickets.