Street data processing toolbox from Esri

Recently I was doing quite a lot of work with street data processing so that one could get network dataset in ArcGIS based on raw data coming as Navstreets shapefiles from HERE (known also as Nokia and previously Navteq).

Just for some years ago, there was no other way to compile those shapefiles into network dataset except from doing a lot of operations manually and use ArcObjects/Python. Fortunately, Esri has published a Street data processing toolbox which would do just that. It is free and supports both Navteq Navstreets data as well as TomTom Multinet.

I was worrying a bit first because I was not sure whether the SDP toolbox is capable of compiling really large datasets covering a bigger part of Europe and North America. However, it went really well and huge datasets were compiled within a week. I have been using some machines I had at my disposal, but I have run out of RAM quite soon. I had to use Amazon EC2 running a rather powerful instance. As a best practice, in addition to running on a 64-bit OS and having lots of RAM, you should also be sure that you have plenty of temporary disk space on the system drive (the temp folder is used heavily for the temp data storage). Note that the temporary space is not the same as the destination for the output file geodatabase where the network dataset will be created, and is often located at C:\Users\%username%\AppData\Local\Temp\. The USA+Canada network dataset was compiled in 70 hours (Amazon EC2 m2.2xlarge instance was used).

Be aware that SDP tools cannot be run with background processing enabled, so you should author and run a tool via a model or Python script tool if you want to run some custom data processing operations first. Remember to switch off the background processing in ArcGIS Desktop application (Geoprocessing > Geoprocessing Options). You can run Dissolve Network GP tool to get a more compiled network dataset which will be faster in operation (remember to build a network dataset after dissolving) and then compress file geodatabase with Compress Geodatabase GP tool to make it smaller, perform faster and be read-only. The network dataset in this format (dissolved network and compressed geodatabase) performs as good as SDC.



2 thoughts on “Street data processing toolbox from Esri

  1. Did you try this with the TomTom data also? I had tried this same tool on a global FGDB and it ran for 4 weeks and didn’t do anything but crash on some values being out of range. I asked ESRI to change/update this code but I still have not seen any updates to allow ranges larger than what they had put in.

Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out /  Change )

Google+ photo

You are commenting using your Google+ account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )


Connecting to %s