Getting geodatabase features with arcpy and heapq Python module

If you have ever needed to merge multiple spatial datasets into a single one using ArcGIS, you have probably used the Merge geoprocessing tool. This tool can take multiple datasets and create a single one by merging all the features together. However, when your datasets are stored on disk as multiple files and you only want to get a subset of features from it, running the Merge tool to get all the features together into a single feature class may not be very smart.

First, merging features will take some time particularly if your datasets are large and there are a few of them. Second, even after you have merged the features together into a single feature class, you still need to iterate it getting the features you really need.

Let’s say you have a number of feature classes and each of them stores cities (as points) in a number of states (one feature class per state). Your task is to find out 10 most populated cities in all of the feature classes. You could definitely run the Merge tool and then use the arcpy.da.SearchCursor with the sql_clause to iterate over sorted cities (the sql_clause argument can have an ORDER BY SQL clause). Alternatively, you could chain multiple cursor objects and then use the sorted built-in function to get only the top 10 items. I have already blogged about using the chains to combine multiple arcpy.da.SearchCursor objects in this post.

However, this can also be done without using the Merge geoprocessing tool or sorted function (which will construct a list object in memory) solely with the help of arcpy.da.SearchCursor and the built-in Python heapq module. Arguably, the most important advantage of using the heapq module lies in ability to avoid constructing lists in memory which can be critical when operating on many large datasets.

The heapq module is present in Python 2.7 which makes it available to ArcGIS Desktop users. However, in Python 3.6, it got two new optional key and reverse arguments which made it very similar to the built-in sorted function. So, ArcGIS Pro users have a certain advantage because they can choose to sort the iterator items in a custom way.

Here is a sample code that showcases efficiency of using the heapq.merge over constructing a sorted list in memory. Please mind that the key and reverse arguments are used, so this code can be run only with Python 3.

 

Advertisements

Using Python start up script for all Python interpreters

This post would be helpful for users of desktop GIS software such as ArcMap who need to use Python inside those applications.

There is a not so well known trick to trigger execution of a Python script before any Python interpreter on your system starts.

Note: If you are a QGIS user, there is a special way of achieving this. Please see the question Script that runs automatically from the QGIS Python Console when QGIS starts, using schedule tasks on Windows 10 for details.

The way to do this is to set up an environment variable called PYTHONSTARTUP in your operating system. You need to do two things:

  1. Create an environment variable that would point to a path of a valid Python script file (.py) with the code that you would like to get executed before any Python interactive interpreter starts. Look at the question [Installing pythonstartup file](https://stackoverflow.com/questions/5837259/installing-pythonstartup-file) for details.
  2. Write Python code that you would like to get executed.

A very important thing to consider is that

The file is executed in the same namespace where interactive commands are executed so that objects defined or imported in it can be used without qualification in the interactive session.

This means that you can do a bunch of imports and define multiple variables which would be available to you directly at the start up of your GIS application. This is very handy because I often need to import the os and sys modules as well as import arcpy.mapping module and create mxd variable pointing to the current map document I have open in ArcMap.

Here is the code of my startup Python script which you can modify to suit your needs. If your workflow relies on having some data at hand, then you might need to add more variables exposed. I have ArcMap and ArcGIS Pro users in mind.

I have included in the example above a more specific workflow where you would like to be able to quickly execute SQL queries against an enterprise geodatabase (SDE). So, when ArcMap has started, you only need to create a variable conn pointing to a database connection file (.sde) and then use the sql() function running your query. Thanks to the tabulate package, the list of lists you get back is drawn in a nice table format.

2018-03-15 12_55_03-PythonWindowArcMap

 

 

Building concave hulls (alpha shapes) with PyQt, shapely, and arcpy

In the previous post, I have shared some information about how to set up PyQt development environment for ArcGIS Pro Python developers. If you are interested in building interactive desktop applications using Python and PyQt and running them within Pro, by all means, do check the post as it will guide you through the process.

In this post, I’d like to share a sample application that I’ve put together to illustrate the process of building concave hulls (also known as alpha shapes). I’ve built a fairly simple PyQt5 application that can be run from Pro or as a standalone application (without the functionality of adding the data to the map of course). With this application, I wanted to show how easy it is to integrate a PyQt window into ArcGIS Pro, but also to show how to build the concave hulls and what kind of results you get when adjusting the alpha value.

This application can be helpful for anyone who would like to create polygons around a set of points but creating convex hulls does not produce polygons that are good enough in terms of the boundary bounds. This work is inspired by two blog posts:

Let’s do some work with the points that form an outline that looks like an “H”.

This is what you would get generating the convex hull:

convex_hull

This is what you would get generating the concave hull with a specific alpha value:

concave_hull_better

And one more concave hull with another alpha value:

concave_hull_good

The basic workflow is that you interactively remove the triangulation edges for the points that are too far from each other. Adjusting the alpha value to tune the boundaries is a tricky process because it may improve the boundaries in one place (areas 1 and 2) but degrade them in another place (areas 3, 4, and 5). Look at the figure below; we are trying to increase the alpha value to improve the figure above:

concave_hull_best

So, as you can see, it is all about finding a sweet spot where the produced results are good enough and trying to improve the boundaries will only make them worse.

The animated figure to demonstrate the process of increasing the alpha value. Removing too many edges will result in having separate disconnected polygons:

alphas

Building concave hulls using multiple alpha values can be very helpful when you have clusters of points you would like to generate separate polygons for. Because the points in these two clusters are located fairly far from each other, they don’t have any connecting triangulation edges. As the end result, you get two polygons – one polygon for each cluster of points.

two_clusters

The source code behind this application in available here:

 

Developing Python GUI in ArcGIS Pro with PyQt

Many ArcGIS analysts and developers working with Python tools in ArcGIS Desktop or ArcGIS Pro would like to be able to build custom Python script tools extending the built-in framework of the GP tools dialog. I have answered a question Recommendations about graphic interface for ArcPy/Python script providing useful links on how to build an own GUI inside ArcGIS desktop application. I have also blogged about this topic in another post – Building custom UI tools for ArcGIS with Python.

Other users have also shown some progress. I have seen that Erik Pimpler has a couple of videos (Building GUI Applications for ArcGIS Desktop with Python – Part 1, Building GUI Applications for ArcGIS Desktop with Python – Part 2) that showcase how one can build custom GUI inside ArcMap using wxPython – a Python framework for GUI development. I have also seen Andrew Chapkowski integrating Tkinter into a Python add-in, but this is basically it. However, I do see that many users would like to be able to build own GUI inside ArcGIS using Python and/or using a designer or form builder to facilitate the process of the layout preparation. There is even an idea Form Builder for Python Tools on the ArcGIS Ideas website with quite a few votes.

When building a GUI app with Python, my personal choice is PyQt. It’s extremely mature and robust framework for GUI development which is a pleasure to work with. I have built some GUI utilities for text editing (think some of the Linux grep functionality in a Windows app) as well as SQL editor to work with file geodatabases. I have been incredibly productive building applications really quickly. There are so many useful tutorials on PyQt, so I have never really have stuck at any problem. You can build a helpful utility application in one evening.

Resources on PyQt development:

QGIS developers are familiar with this framework because they build QGIS plugins using PyQt, however if you are an ArcGIS developer, this is pretty new to you unless you have used PyQt for other projects. So, let’s see how you can set up an PyQt application with ArcGIS Pro.

  1. Install ArcGIS Pro (further just Pro). You will get a default conda environment called arcgispro-py3 which you can copy first into a new environment so you could restore it should something bad happen with the default one which we will be playing with.
  2. With the help of Python Package Manager in Pro, you need to install pyqt 5.6.0 into the default environment. This will quite take some time (5-10 mins).
  3. Now you are ready to write a PyQt application that can be called from a Python script tool.
  4. Run Pro, create and then save a new empty project.
  5. Create a new custom script tool in a toolbox and point it to a Python file with the following content.

Now you can run the script tool just like you would any other custom script tool and you should get this tiny app window open:

2017-11-25 16_45_14-Basic App

Note: if you are getting an error message trying to run the tool:

This application failed to start because it could not find or load the Qt platform plugin “windows”.

then you need to add a system environment variable QT_QPA_PLATFORM_PLUGIN_PATH with the path to the plugins folder used by PyQt5. If you have installed ArcGIS Pro into a default location for all users, you would need to provide this value: C:\Program Files\ArcGIS\Pro\bin\Python\envs\arcgispro-py3\Library\plugins\platforms. Restart ArcGIS Pro to let it grab the environment variable on the next start.

If you did get this window – congratulations, you have just accessed arcpy from a Python program that was run as a PyQt5 application. You have accessed the currently open project in Pro and accessed its property. So, what does this imply?

This means you can write code within a PyQt5 application that will be accessing your current map and its layers as well as basically everything you can reach with arcpy. Now it is up to you to choose what GUI to build as Qt framework provides excellent set of widgets to choose from.

Note: you may have noticed that after you have run the script tool and the PyQt app window was opened, Pro thought that the script tool is still running, so the script tool will complete only upon closing the application. This behavior can be changed if you run your PyQt application in own thread, so as your application starts, the script tool will complete its execution. This can be done using the threading module.

Create a new Python Python module to use as the source code for the script tool:

This is what you would see.

2017-11-25 17_02_54-Basic App

Note: since the application runs in its own thread, it won’t be able to access the current project: it will run as if you were running it as a standalone script. You can of course still use arcpy and other packages you have in your Pro conda environment.

To give you a feeling of how powerful PyQt is and what kind of amazing tools you can build to let Pro users to work with, I’ve created a sample app that can showcase some of the PyQt power. In this application, you can choose an input point feature class and it will generate something called alpha shape (or concave hull). I’ve adopted some of the code you can find in the blog posts Drawing Boundaries In Python by HumanGeo and The fading shape of alpha by Sean Gillies.

In the application, you can choose what edges of the triangulation you would like to keep based on the distance between the edge vertices by using an interactive slider. To visualize the produced data, embedded version of Matplotlib plot is drawn. When you are happy with the result, you can add the produced data to ArcGIS Pro current map as layers. I will post the source code of this application in another post later on.

Below is the screen recording of running the Pro application and starting a custom script tool that in turn starts a PyQt application window.

alphas.gif

Call ArcGIS Pro .NET SDK class library from Python

As some of you may already know, it is possible to interact with ArcObjects COM libraries using Python with the help of comtypes module. This means a whole lot for any GIS analyst who at some point may hit the limitations of arcpy package and will need to do something using ArcObjects which provide a fine-grained API into the ArcGIS components.

If you switch to ArcGIS Pro, though, you won’t be able to run your old ArcObjects code that, for instance, will hide or show map grids (graticules) in map layout before exporting the map (this is not supported with arcpy). Even though ArcGIS Pro is shipped with arcpy, its functionality is not any richer than of ArcGIS Desktop based arcpy (at least for now).

However, because ArcGIS Pro comes with .NET SDK, it becomes relevant to learn how to access .NET .dll files and access the methods of classes available within the class libraries. I’ve tested to use pythonnet module and it worked great to access a class library authored with Visual Studio 2015 and Visual C#.

In order to make calls to .dll files compiled based on .NET code (in my case, it’s C#) using Python, you would need to install a Python package called pythonnet. Learn more at Calling a C# library from python.

However, accessing the ArcGIS Pro .NET libraries isn’t that easy as there are many dependent references involved. There is support for embedding the core functionality of ArcGIS Pro (such as accessing the geodatabase data, iterating features with cursors, constructing geometries etc.), however, one is supposed to access this functionality either using a console application or a WPF application.

There is a Python module called subprocess that would let you get the result of console app execution as a string which you could load into a Python data structure to use later, however, this is not a very elegant way to proceed. You basically author your console apps and then use Python to call those apps supplying the input arguments. This is probably the easiest way to get your .NET code executed from Python, though, as you have everything you need already installed.

An alternative approach is to compile a class library using x64 platform in Visual Studio and then access it from 64 bit Python (your ArcGIS Desktop x86 Python won’t work). Read more about accessing the core of .NET SDK at ProConcepts CoreHost.

Here are the steps involved. Please mind that this can get messy with all those Python modules installations, so you might like spinning a virtual machine or at least a virtualenv for this.

  1. Build your class library in Visual Studio for x64 platform.
  2. Install pythonnet for x64.
  3. Install pywin32 for x64.
  4. Copy two .dll files from:
    ​C:\Python27\ArcGISx6410.4\Lib\site-packages\pywin32_system32 to C:\Python27\ArcGISx6410.4\Lib\site-packages\win32\lib
  5. You should be able to run the code in the snippet below.