Skip to content
This repository has been archived by the owner on Nov 30, 2023. It is now read-only.

improve memory performance of transitfeed.py #30

Closed
bdferris opened this issue Sep 26, 2014 · 3 comments
Closed

improve memory performance of transitfeed.py #30

bdferris opened this issue Sep 26, 2014 · 3 comments
Labels
bug imported

Comments

@bdferris
Copy link
Contributor

From [email protected] on June 28, 2007 03:23:39

processing feeds takes tons of memory and it seems to get worse with each
release. Internally we have some large GTFS files that won't load on a 2GB
machine.

We need to do some profiling to work out if we are doing something that
prevents python doing the right thing.

It might be worth trying to reload a medium sized feed several times. If we
delete the Schedule object each time then the python process should stop
growing.

Here's someone with debugging a possibly similar problem http://mail.python.org/pipermail/python-list/2007-June/445838.html but he didn't get a reply.

Original issue: http://code.google.com/p/googletransitdatafeed/issues/detail?id=30
@bdferris
Copy link
Contributor Author

From [email protected] on May 29, 2008 02:24:02

I wasn't able to process my entire feed (52000 trips, 810000 stop times) in 360MB of
memory. So I had to split it into 4 roughly even sized feeds, based on route. Then I
could load these into the validator and schedule viewer. Inconvenient, but I think
it's OK for the purposes of testing. Good luck in fixing this.

@bdferris
Copy link
Contributor Author

From [email protected] on July 29, 2008 16:41:17

By the way, I have a fix under review ( http://codereview.appspot.com/2644 ) in the
branch http://googletransitdatafeed.googlecode.com/svn/branches/tom.brown.code-sqlite/python and put Windows binaries at http://googletransitdatafeed.googlecode.com/files/transitfeed-sqlite.zip This has been working for me.

@bdferris
Copy link
Contributor Author

From [email protected] on October 07, 2008 16:06:05

I have released 1.1.6 which uses sqlite for the stop_times table. Slow, but works.

Status: Fixed

EddyIonescu added a commit to EddyIonescu/transitfeed that referenced this issue Sep 17, 2018
@EddyIonescu
# This is a combination of 40 commits.
9a19f29 
# This is the 1st commit message:

add muni ac gtfs

# This is the commit message google#2:

only output routes

# This is the commit message google#3:

add initial version of kml route styler

# This is the commit message google#4:

add sample csv route styles

# This is the commit message google#5:

add route styling functionality to kmlwriter based on csv provided in /route_styles

# This is the commit message google#6:

add generated kml with styling and selection done based on csv

# This is the commit message google#7:

replace K and T with KT muni line in csv input

# This is the commit message google#8:

change zero routes assertion to warning due to naive automation scripts

# This is the commit message google#9:

strip whitespace on color in csv

# This is the commit message google#10:

add script for generating all maps

# This is the commit message google#11:

add extracted gtfs and most output

# This is the commit message google#12:

update excels

# This is the commit message google#13:

update automation script to enable starting midpoint in the list

# This is the commit message google#14:

update requirements file

# This is the commit message google#15:

set default skip ahead to none

# This is the commit message google#16:

add more kml outputs

# This is the commit message google#17:

add parallel processing

# This is the commit message google#18:

add muni bart current 15 kml

# This is the commit message google#19:

add muni current 15 geojson

# This is the commit message google#20:

add conversion to geojson

# This is the commit message google#21:

change pipe to and

# This is the commit message google#22:

add json outputs because google maps kml layer is garbage

# This is the commit message google#23:

remove once daily tripper for samtrans 120 route

# This is the commit message google#24:

convert samtrans current 15 kml to geojson

# This is the commit message google#25:

add regional new plan

# This is the commit message google#26:

updated kml manually

# This is the commit message google#27:

add json of new projects proposed

# This is the commit message google#28:

add Hidden_Shapes param to kmlwriter and route_styler

# This is the commit message google#29:

add Hidden_Shapes column to route style csvs

# This is the commit message google#30:

dict typo

# This is the commit message google#31:

support for none hidden_shape

# This is the commit message google#32:

nan support...

# This is the commit message google#33:

nan stuff

# This is the commit message google#34:

add hidden shapes to vallejo sfbay ferryroute

# This is the commit message google#35:

fix bug

# This is the commit message google#36:

set same stroke width for everyone

# This is the commit message google#37:

sfbay ferry modified

# This is the commit message google#38:

reverse boolean bug

# This is the commit message google#39:

Reload KML/Json output

# This is the commit message google#40:

remake kml json output
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Assignees
No one assigned
Labels
bug imported
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
1 participant
@bdferris