Essential extra toolsets for ArcGIS Desktop professionals

If you are an ArcGIS users you could at some point of time find out that you lack a certain tool which you need to perform a certain analysis. Or you just need to hack some datasets really quickly and you are lazy to write your own tool for that. If you are not developer, though, the only option you would have is to see whether a model can be built in ModelBuilder to replicate the required operation (by combining multiple geoprocessing tools).

If ModelBuilder wouldn’t help you, then there is a good chance you would start searching for this kind of tool on the Internet. A great place to start is Google. The GIS Stackexchange web site is indexed very well and you will be able to find many of the tools you are looking for as answers to the questions on this web site too.

Another place to search for tools is ArcGIS.com. This one has replaced arcscripts and if you are on ArcGIS 9.3+ you probably wouldn’t go to arcscripts at all. When at ArcGIS.com, you don’t need to sign in to search for tools. Remember to enable the Show ArcGIS Desktop Content option and filter the results to show Tools only (figure below).

ArcGIS.com searchHere is a great reference page on how to do efficient searches: https://doc.arcgis.com/en/arcgis-online/reference/search.htm

In my “GIS Analyst toolbox”, I have collected many useful tools over last years. Some of them for older versions of ArcGIS, but in some cases they will just work; in others – you can get an idea on how tools are implemented and re-write to run on your version (if you can program of course or use ModelBuilder). Here is the short list that includes some of them:

SampleArcPyMappingScriptTools_10_v1 (ArcGIS toolbox, Python code)
From: http://www.arcgis.com/home/item.html?id=18c19ec00acb4d568c27bc20a72bfdc8
These (~20) tools were created as representative samples for how arcpy.mapping could be used to perform a variety of tasks. A must have for anyone who works with map documents often! It will also let you get started with arcpy.mapping module in no time, awesome!

Cartography Tools

  •    Adjust Layout Text Width (from ArcMap)
  •    Find and Replace a Text String
  •    Page Layout Element Report
  •    Shift Page Layout Elements
  •    Update Symbology

Export and Printing Tools

  •    Append PDF Documents
  •    Export Map Documents to PDF
  •    Print Data Driven Page(s)
  •    Print Map Document(s)

MXD and LYR Management Tools

  •    Add Layer File into MXD (from ArcMap)
  •    Find Broken Data Sources (Report)
  •    Find Data Source (Report)
  •    Find Layers Projected on the Fly (Report)
  •    Multi Layer File Summary (Report)
  •    Multi MXD Summary (Report)
  •    Replace Layer with Layer File
  •    Replace Layer with Layer File (from ArcMap)
  •    Update MXD from pGDB to fGDB
  •    Update MXD tags

Tools that will let you create your own cartographic effects
From: http://mappingcenter.esri.com/index.cfm?fa=arcgisResources.modelsScripts
Lots of useful tools for any cartographer who wants to add some extra mapping features.

Database inspector (ArcGIS toolbox, Python code)
From: https://sites.google.com/site/geospatialonestop/tools/SchemaInspector.zip?attredirects=0&d=1
This is a must have for any ArcGIS analyst doing any Python development for geodatabase maintenance and geodata management. This is a suite of tools for analyzing components of a geodatabase and for finding the differences between geodatabase schemas. Can print properties of fields, feature classes, relationship classes, domains, tables. Can compare particular geodatabase components between two workspaces. Can compare two geodatabases and tell you the difference. A great tool I use daily.

ArcGIS Server ServerAdminToolkit 10.1+ (by Kevin Hibma from Esri) (ArcGIS toolbox, Python)
From: http://www.arcgis.com/home/item.html?id=12dde73e0e784e47818162b4d41ee340
These tools perform some common administrative tasks with an ArcGIS Server machine. All of these tasks can be accomplished through the UI (ArcMap), the Web Manager or the REST Administration page. By using tools you can automate redundant workflows or chain common workflows together. Most of these tasks, turned into tools, have more detailed explanations in the help. This package is composed of three main parts: Tools, Standalone executable, and Code.

ArcREST (ArcGIS toolbox, Python)
From: https://github.com/Esri/ArcREST
A set of python tools to assist working with ArcGIS REST API for ArcGIS Server (AGS), ArcGIS Online (AGOL), and ArcGIS WebMap JSON. An amazing package that any ArcGIS Online or ArcGIS Server admin wants to have! Nearly all things you can do with the REST API, you can do with ArcREST. A must have.

Spatial Analyst Supplemental Tools
From: http://www.arcgis.com/home/item.html?id=3528bd72847c439f88190a137a1d0e67
A collection of script tools to supplement Spatial Analyst Tools.

  • Create Dendrogram
  • Draw Signature
  • Erase Raster Values
  • Filled Contours
  • Maximum Upstream Elevation
  • Peak
  • Tabulate Area 2
  • Viewshed Along Path
  • Zonal Statistics As Table 2

Geomorphometry & Gradient Metrics (ArcGIS toolbox, Python)
From: http://evansmurphy.wix.com/evansspatial#!arcgis-gradient-metrics-toolbox/crro

Urban Network Analysis Toolbox for ArcGIS (Python)
From: http://cityform.mit.edu/projects/urban-network-analysis.html#download
The tools incorporate three important features that make them particularly suited for spatial analysis on urban street networks.

National Water-Quality Assessment (NAWQA) Area-Characterization (ArcGIS toolbox, Python)
From: http://www.arcgis.com/home/item.html?id=cbb59504f59f4e18b23817fb0ef40e56 and http://pubs.usgs.gov/of/2010/1268/
The toolbox is composed of a collection of custom tools that implement geographic information system (GIS) techniques used by the NAWQA Program to characterize aquifer areas, drainage basins, and sampled wells.

X-ray for ArcGIS (ArcMap, ArcCatalog) (add-in)
From: http://www.arcgis.com/home/item.html?id=f0ae73e90c1a4992a1059e7d370966d4 and http://www.arcgis.com/home/item.html?id=9ea218ff575f4a5195e01a2cae03a0ae
The X-Ray for ArcCatalog add-in can be used to develop, refine and document your geodatabase designs. The X-Ray add-in for ArcMap can be used to document the properties of your map documents (MXDs).

Marine Geospatial Ecology Tools (MGET) (ArcGIS toolbox, Python)
From: http://mgel.env.duke.edu/mget/download/
A free, open-source geoprocessing toolbox that can help you solve a wide variety of marine research, conservation, and spatial planning problems. MGET plugs into ArcGIS and can perform tasks such as:

  •    Accessing oceanographic data from ArcGIS
  •    Identifying ecologically-relevant oceanographic features in remote sensing imagery
  •    Building predictive species distribution models
  •    Modeling habitat connectivity by simulating hydrodynamic dispersal of larvae
  •    Detecting spatiotemporal patterns in fisheries and other time series data

Favorite tools and resources for cartographers
From: http://blogs.esri.com/esri/arcgis/2013/06/04/favorites-cartographers/
A compilation of some of the most popular tools and sources of information about maps and cartographic design.

Geospatial Modelling Environment (known as HawthsTools)
From: http://www.spatialecology.com/gme/gmedownload.htm
GME provides you with a suite of analysis and modelling tools, ranging from small ‘building blocks’ that you can use to construct a sophisticated work-flow, to completely self-contained analysis programs. It also uses the extraordinarily powerful open source software R as the statistical engine to drive some of the analysis tools. One of the many strengths of R is that it is open source, completely transparent and well documented: important characteristics for any scientific analytical software.

ArcREST: Python package for administering ArcGIS Server and ArcGIS Online/Portal

ArcREST is a great toolset I have found some time ago. It is for anyone who administers ArcGIS Online, ArcGIS Portal or ArcGIS Server. In short, it is a Python wrapper for the Esri REST API. I had to write many Python scripts that allowed me to update the properties of ArcGIS Server services in batch, but now I don’t need to write anything like this anymore. This is because now I can do everything I did on my own just by using ArcREST. If you are an ArcGIS Online / Portal admin, you should definitely take a look at this module since it can save you a lot of time, and you won’t need to author your own scripts for managing the ArcGIS Online content and organization settings with the scripting techniques.

This Python package is authored by Esri Solutions team and is available in public access on GitHub. You can download the source code, optionally install the package, and then use it on your local machine just like any Python package. If you don’t want to install the package, you can just add the path to arcrest and arcresthelper folders to the Python path by adding this into your Python file:

import sys
sys.path.append(r”path to arcrest folder”) #C:\GIS\Tools

Provided that you have a folder named arcrest in the example Tools folder, when you run the Python file, it will be able to import the arcrest package and access its modules.

To get an overview of this Python package, take a look at this excellent DevSummit 2015 video where developers of ArcREST talked about it.

Even though this is not a full implementation of the Esri REST API, it covers most of it and Esri developers update the code to include latest changes in the REST API. It is a good idea to clone the repository and pull the changes now and then to get the latest code if you will use on the daily basis.

I felt kind of sad first that all the Python code I wrote for administering ArcGIS Server won’t be used any longer, but at the same time so glad the ArcREST was developed. It is a great piece of software that will let you get started in no time at all and access all your server/online resources with Python.

Caveat: it does have some dependencies on arcpy package which is used for converting feature sets into JSON and back, but apart from that you should be able to run the tools on a machine with no ArcGIS software installed whatsoever.

Publishing Python scripts as geoprocessing services: best practices

Why Python instead of models?

If you have been publishing your ModelBuilder models as geoprocessing (further GP) services you have probably realized that it can be quite cumbersome. If you haven’t moved to Python, I think you really should. Authoring Python scripts has serious advantages over authoring models in the context of publishing GP services. This is because during the publishing process, ArcGIS Server will turn data and anything that may be needed to change into variables and this might mess up the model if you haven’t followed the guidelines on authoring GP services. The rule of thumb for me was that if there are more than 10 objects in the model, it is a good time to switch to Python. Another thing is that you can easily make modifications in the Python code without republishing; in contrast, you need to republish the model each time you want to release an updated version of the GP service. Finally, since you don’t need to restart the GP service when updating the Python file (in contrast to republishing the model which requires restarting service), there is no down-time for the service and users won’t notice anything.

What happens after publishing?

Let’s take a look at what is going on under the hood. You have run your script tool in ArcMap and got the result published as a service. Now you can find your service and all the accompanying data inside the arcgisserver folder somewhere on your disk drive. The path would be: C:\arcgisserver\directories\arcgissystem\arcgisinput\%GPServiceName%.GPServer

You will find a bunch of files within the folder. Let’s inspect some of them:

  • serviceconfiguration.json – provides an overview over all the properties of the service including its execution type, enabled capabilities, output directory and many others. Here you will see all the settings you usually see in the Service Editor window.
  • manifest.xml and manifest.json – provides an overview of the system settings that were used while publishing the service. Those are not the files you usually would want to inspect.

Inside the folder esriinfo/metadata there is a file named metadata.xml which is really helpful because there you can see what date a service was published. Two tags you should look at are:

  • <CreaDate>20141204</CreaDate>
  • <CreaTime>15443700</CreaTime>

Since this information is not exposed from the GUI in ArcGIS Desktop or ArcGIS Server Manager, this is the only way to find out what time the service was created. This information may be very handy when you are unsure about the release versions.

Inside the extracted/v101 folder, you will find the result file and the toolbox you have worked with when publishing the GP service. Here you will also find a folder named after the folder where you source Python file was stored and containing the source Python file.

Best practices to organize the Python code and files?

Let’s look inside the Python file. You might have noticed that when publishing some of the variables you’ve declared were renamed to g_ESRI_variable_%id%. The rule of a thumb is that you shouldn’t really use strings; you can turn paths to datasets and names into variables. Of course you don’t have to do this since Esri will update those inline variables, but it is so much harder to refactor with those variable names, so you better organize your code correctly from the beginning.

If running the script tool in ArcGIS, scratch geodatabase is located at C:\Users\%user%\AppData\Local\Temp\scratch.gdb. However, after publishing the tool, the service will get a new scratch geodatabase. If you need to inspect the intermediate data created, go to the scratch geodatabase (the path can be retrieved with the arcpy.env.scratchGDB) which will be a new file geodatabase in each run of GP service with the following notation: c:\arcgisserver\directories\arcgisjobs\%service%_gpserver\%jobid%\scratch\scratch.gdb.

Keep in mind that GP service will always use its local server jobs folder for writing intermediate data and this behavior cannot be changed. But having the service writing to the scratch workspace is actually a lot safer than writing to a designated location on disk. This is because there is no chance of multiple GP service instances trying to write to the same location at the same time which can result in dead-locking and concurrency issues. Remember that each submitted GP job will be assigned to a new unique scratch folder and geodatabase in the arcgisjobs folder.

Make sure you don’t use arcpy.env.workspace in your code; always declare a path variable and assign it to be the folder or a geodatabase connection. For the datasets path, use the os.path.join() instead of concatenating strings. For performance reasons, use in_memory workspace for intermediate data with the following notation:

var = os.path.join(in_memory,"FeatureClassName")

You can take advantage of using in_memory workspace, but for troubleshooting purposes it might be better to write something to disk to inspect later on. In this case, it might be handy to create a variable called something like gpTempPlace which you can change to be either “in_memory” or a local file geodatabase depending whether you run clean code in production or troubleshoot the service in the staging environment.

Make sure you don’t use the same name for variable and feature class/field name. This sometimes leads to unexpected results when running the service. It might be helpful to add “_fc” in the end for feature class variable and “_field” for the field variable. This way, you will also be able to distinguish them much easier. The same is applicable for the feature class and feature layer (created with Make Feature Layer GP tool) names.

Remember that you can adjust the Logging level of ArcGIS Server (in Manger) or GP service only (done in Service Editor window) for troubleshooting purposes. It is often useful to set the Message Level setting to Info level before publishing the GP service into production because this will give you the detailed information what exactly went wrong when running the GP service. You can access this information either from the Results window in ArcMap or from the ArcGIS Server logs in Manager.

How do I update the published service?

A special word should go for those users who needs to publish the GP services often while making changes in the code. It is important to understand that after publishing the GP service, the copied Python code file and the toolbox don’t maintain any connection to the source Python file and the source toolbox you have authored. This implies that after making edits to the script tool, you need to push those changes to the published service on server.

There are two types of changes you can make on your source project: the tool parameters in the script tool properties and the Python code. Keep in mind that you cannot edit the published toolbox; so if you added a new parameter or modified existing parameter data type, you would need to republish the service. However, if have only modified the Python script source code, there is no need to republish the whole service as you only need to replace the contents within the Python file.

Even though you can automate service publishing workflow with Python, it still takes time to move the toolbox and the Python code files. Therefore, you can save a lot of time by finding a way to replace the Python code in the published Python code file. In order to update the Python code file, you have really just two options – you either copy and replace the published file with the updated source file or copy/paste the source code. This approach may work if you have a tiny Python script and all the paths to the data on your machine and on the server are the same. This can be a plausible solution when you have Desktop and Server on the same machine. If you have a configuration file where the Python source code file gets all the paths and dataset names from, you could also safely replace the published Python file. However, this is still an extra thing to do.

The best practice I came to while working for two years on the GP services is to split the code files and the tool itself. Let me explain.

How do I separate the toolbox logic and Python code?

Create a Python file (a caller script) which will contain all the import statements for the Python modules and your files. By appending the path to your Python files, you will be able import the Python files you are working on; this is very useful when your GP service consists not just of one file yet of multiple modules.

import sys
import socket
sys.path.append(r'\\' + socket.gethostname() + "path to code files")
import codefile1 #your Python file with business logic
import codefile2 #your Python file with business logic

This file should also include all the parameters which are exposed in the script tool.

Param1 = arcpy.GetParameterAsText(0)
Param2 = arcpy.GetParameterAsText(1)
Param3 = arcpy.GetParameterAsText(2)

Then you define a main function which will be executed when running the GP service. You call the functions defined within the files you imported.

def mainworkflow(Param1,Param2):
    Result = codefile1.functionName(Param1,Param2)
return Result

It is also handy to add some parameter handling logic; when you run the script tool in ArcMap, you supply some values for the tool which will become default values visible when users will execute GP service from ArcMap or from any other custom interface. In order to avoid that, you can just leave those parameters empty and then return empty output for GP script tool publishing only purposes.

if Param1 == '' and Param2 == "":
    Result = ""
else:
    Result = mainworkflow(Param1,Param2)

Create a script tool from this caller Python file defining the parameters and their data types. After the tool will be published as a GP service, you can work with the Python files which will contain only the code that actually does the job.

After performing and saving the changes in the code, feel free to run the GP service directly – the caller Python file (published as a GP service) will import the codefile1 at the folder you specified and run the code. There is no need to restart the GP service or re-import your update module.

Apache JMeter for ArcGIS Server: using a .csv file

Today let’s try to use JMeter with ArcGIS Server a bit more. Previously, we have set up a test configuration where “users”, or threads, sends a map request to get back an image in a certain extent. As you probably know, map services often contain data that can be distributed unevenly. In built-up areas there will probably be more features with more complex symbology and in rural areas there are usually fewer features. This implies that the server will need to perform a larger amount of work for generating an image over the built-up areas compared to the map image request jobs that will be submitted over rural areas.

That is why it is so important to make your requests as diverse as possible by asking the map service to generate map image in various spatial extents and map scales. In an earlier blog post, we have created an HTTP Request configuration element where we have specified a certain URL to be sent to the server, where the bounding box (or extent) is hard-coded. This means that all the test threads will ask the service to generate the same image all over the time which will lead to a fairly equal burden on the server system resources. In reality, however, the server will likely experience different load since the users will navigate the map service in different scales and extents thus generating different images.

Here is where using a .csv file comes into play. Instead of specifying the same map extent and scale in the request URL, we will ask the JMeter to read these values from a .csv file and insert them dynamically into the URL we will specify. Think of this as inline variable substitution if you are comfortable with geoprocessing in ArcGIS Desktop.

Let’s look at the map image request URL:

http://localhost:6080/arcgis/rest/services/SampleWorldCities/MapServer/export?bbox=0%2C5%2C10%2C30&bboxSR=&layers=&layerDefs=&size=&imageSR=&format=png&transparent=false&dpi=&time=&layerTimeOptions=&dynamicLayers=&gdbVersion=&mapScale=50000000&f=html

Here we are interested in bbox parameter as well as mapScale parameter. For bbox the 0,5,10,30 is used (the coordinates of the extent) and for mapScale the 50000000 is used (1:50,000,000 map scale). We, however, would like to generate images over some other extents and in different map scales. To achieve this, we are going to create a really simple .csv file with some columns and rows. A .txt file would also work fine, but to make it a bit more illustrative let’s stick with the .csv file. We generate different extents by specifying various coordinates and map scale. You can go ahead here and generate as many extents and scales as required!

A sample .csv file would look like this.

Sample .csv file
Sample .csv file

Then you have to right-click the Thread Group, which have called in the previous post Map service image export and choose Add > Config Element > CSV Data Set Config. This will add a new configuration element for which we will set up use of the .csv file.

Add CSV Data Set
Add CSV Data Set

Click the CSV Data Set Config we have just added and fill out the parameters as required.

.csv file parameters
.csv file parameters

Filename: specify the location of your file.
Variable Names: the column names you entered in the .csv file.
Delimiter: the comma is used by default

Now click the HTTP Request element which we have called in the previous post Export a map image with bounding box. You can also create an HTTP Request Defaults element which will apply to all HTTP Requests you will specify within your project.

Next you will fill out the parameters as below. The column name in the .csv is used as ${columnname}. The row value that is being read from the .csv file will be inserted in the URL that is sent to the server. This allows us to send various requests to the server by reading values from an external data source.

Http Request with .csv settings
Http Request with .csv settings

Feel free to add more parameters like image format and size – you will just need to create a couple more columns in your .csv file and specify them in your test project in JMeter as we did with the mapScale and bbox parameters.

Happy testing!

Apache JMeter for ArcGIS Server: simple tutorial

Recently I got a chance to play with Apache JMeter. There is a great blog post from Esri about the JMeter, what it is and what one can do with that.

Please go ahead and read the post to get to know JMeter. I am personally interested in JMeter as a tool that allows me to test the ArcGIS Server 10.1 software by simulating having multiple users performing various requests to the services or server itself. Let’s consider a following scenario:
You have installed ArcGIS Server 10.1 in the staging environment and want to know how the software will handle hundreds of requests coming to your map service. Do you need more hardware? Do you need to specify a bigger maximum number of instances for the map service? That type of questions is something ArcGIS Server administrator deal often and I believe JMeter is a good starting point for those who want to see how well the software operates.

One way to test the ArcGIS Server is to ask a hundred of people get at their computers, open a web browser and come to the web application where the map service is being drawn. Thereafter, you could collect the statistics on the map service and server performance from the ArcGIS Server logs and/or operating system monitoring facilities. However, it is so much easier to get this kind of work done in an automated fashion.

It might be not that easy to learn how JMeter components organized GUI-wise (those who figured out how to use Windows 8 tiles will probably need less time though), so we will start by creating a really simple test project. In this sample project, we will run several identical requests to the sample ArcGIS Server SampleWorldCities map service shipped out-of-the-box in order to get the image of the map back. When we will be done with that, we will gradually make things a bit more complex and there you would probably see the true power of JMeter. This however will happen in the upcoming posts.

The information in the Esri Support Services post is good enough to let you get JMeter up and running. Now there are just some steps left:

  1. Right-click the Test Plan in the tree panel to the left and choose Add > Threads (Users) > Thread group. This is to group users that will perform requests to the map service. Apache1
  2. Rename the Thread Group to the “Map service image export”.
  3. Right-click the Map service image export thread group in the tree panel to the left and choose Add > Sampler > HTTP Request. This will represent the actual request that user will send to the map service.Apache2
  4. Rename the HTTP Request to “Export a map image with bounding box” and set all other properties as on the picture.
    Server name or IP: your machine name (can keep it as localhost for now).
    Port number: 6080 (though you definitely could use the path with web adaptor going via port 80 or some other port your IIS web application is configured to run at).
    Implementation: here we are telling to use HttpClient4 (go and ask some Java guru why the client is called HttpClient4).
    Parameters: f=image is something that is dictated by REST API syntax (to get the image back and not just text)
    Path: /arcgis/rest/services/SampleWorldCities/MapServer/export?bbox=-104.53020709870626,-25.471925275660684,75.35704494206294,51.908146633940035. This is the path you can get by browsing to the SampleWorldCities map service in the Services Directory application (click to browse). By using this URL, we are asking the SampleWorldCities map service to generate a map image for us based on the specified bounding box, or extent.Apache3
  5. If you click the Map service image export thread group we’ve created in the beginning, you can set the properties of the requesting process. Here we set properties for our threads (think users) that will be running. By default, we have just one user (Number of Threads (users) = 1) that will ask the map service to generate and ship the image in the specified extent back to the client from server just once (Loop Count = 1). Ramp-up Period is just a period of time that should past before the number of active users (threads) will reach the number you specify. From the Apache help: “The ramp-up period tells JMeter how long to take to “ramp-up” to the full number of threads chosen. If 10 threads are used, and the ramp-up period is 100 seconds, then JMeter will take 100 seconds to get all 10 threads up and running.”Apache5
  6. Right-click the Export a map image with bounding box sample request we’ve just created and choose Add > Listener > View Results Tree. This is to let us see how the request is being performed, what time it takes to run the request and actually make sure we have got our image back.Apache4
  7. Click the View Results Tree in the left panel and then choose Run > Start from the main menu (or press Ctrl-R). This will perform the export image operation. If you click the Export a map image with bounding box and switch to the Response data tab, you should see the map image received.Apache6

Congratulations! You have just performed a very simple yet automated test to get to know JMeter and how it can be used for performance & testing purposes when using ArcGIS Server. You can now play around a bit more with:

  • The listeners – by adding a couple of other listeners to check how else you might see the results as well as how much time it took (right-click the Export a map image with bounding box in the tree in the left panel and choose Add > Listener > View Results in Table or Add > Listener > Response Time Graph).
  • The threads – go to the Map service image export thread group and increase the number of threads (i.e., users) and set to run the test forever. Remember that you can stop sending requesting by using the Run > Stop in the main menu.

Sometime later I will let you know what other good things (but a bit more complex to implement) one can take advantage of when using JMeter for ArcGIS Server. Happy testing!