More optimizations after #157

[jpmckinney]

Memoizes the result of CSV#encode_str, CSV#escape_re, and disables CSV's converters feature.

At this point, the two big bottlenecks are build_formats (already optimized, but eliminating some if branches will make it faster), and CSV#init_separators, which is slow because determining the row_sep is slow.

Call graph after memoizing escape_re:

After memoizing encode_str:

)

After disabling converters:

[jpmckinney]

About 40% faster with all optimizations.

[jpmckinney]

Note that after switching to FastCSV, init_parsers can be overridden to skip the building of @parsers, since those are only used by CSV#shift, which FastCSV overrides.

[pezholio]

This is great stuff - thanks so much! Stackprof looks a really useful tool too, I'll definitely be using that as I do more optimisations 👍

[jpmckinney]

I think the next big optimization will involve rewriting validate_stream and parse_line to not do all that line break/quote char work and let the CSV library do as much as possible (i.e. call CSV.new once for the entire file, not once for every row!). Not sure yet how that looks though.

[pezholio]

Yeah, it's a tricky one, because parse_line doesn't really know if it's dealing with a full string (especially when streaming from a URL), so some of the work needs to be manual. I'm assuming that would be faster than calling the CSV library anyway?

[jpmckinney]

The CSV library calls @io.gets(@row_sep) in its shift method. You can pass to the CSV library an IO object whose gets method waits for more data from Typhoeus until it hits a @row_sep (which is the semantics of IO#gets). This sounds like something someone doing streaming would have already implemented elsewhere.

If CSVlint isn't parsing a remote file, then you can just use the CSV library normally as before.