Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Unable to Import CSV files with Strings Containing Commas #509

Closed
Bartimaeus- opened this issue Sep 5, 2021 · 4 comments
Closed

Unable to Import CSV files with Strings Containing Commas #509

Bartimaeus- opened this issue Sep 5, 2021 · 4 comments

Comments

@Bartimaeus-
Copy link
Contributor

(PlotJuggler 3.3.0 on Windows)

PlotJuggler is unable to import CSV files that contain text strings with commas (text strings are surrounded by quotes)

The import fails differently depending on if the strings containing commas is in the header or the following rows.

Header items that have comma-containing strings
Quotes are ignored and additional columns are assumed:
image

This type of heater formatting is used by some data acquisition equipment such as the Onset HOBO 4 Channel Thermocouple Logger, which will include information like the channel number, units or settings in addition to the label.

Data entries that have comma-containing strings
This will result in PlotJuggler giving an error that the number of entries in the data row is larger than expected from the header
image

One example of data that uses this formatting is from the online NOAA climate dataset, where the weather station name may contain a comma: https://www.ncdc.noaa.gov/cdo-web/search
Example data:

"STATION","NAME","LATITUDE","LONGITUDE","ELEVATION","DATE","TAVG","TMAX","TMIN","TOBS"
"USR0000CGOO","GOOSE VALLEY CALIFORNIA, CA US","33.0736","-116.8458","466.3","2021-01-01","13.2","18.3","10.0",
"USR0000CGOO","GOOSE VALLEY CALIFORNIA, CA US","33.0736","-116.8458","466.3","2021-01-02","11.7","17.2","8.3",

In this instance, the "NAME" data which contains a comma is not plottable and could be safely ignored. This does raise questions about support for string entries (if any is intended or would be supported) so that 1) string values could be viewed in the tree in the left hand column (would allow for viewing debug messages for example) and 2) string data could be preserved for csv export

Example file attached (note that some lines have blank entries for temperatures)
2703382.csv

@facontidavide
Copy link
Owner

Ok, I tried my best to address this special case with a custom SplitLine function.

I think it should work now.
For the records, data like the one you posted doesn't make much sense in PlotJuggler, since it seems to be particularly suited for SQL queries, rather than plotting.

About showing strings, this will be possible in the future.

@facontidavide
Copy link
Owner

image

@facontidavide
Copy link
Owner

facontidavide commented Sep 5, 2021

Strings added :)

PJ_CSV-2021-09-06_01.21.55.mp4

@Bartimaeus-
Copy link
Contributor Author

Cheers mate
image

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants