Call ArcGIS Pro .NET SDK class library from Python

As some of you may already know, it is possible to interact with ArcObjects COM libraries using Python with the help of comtypes module. This means a whole lot for any GIS analyst who at some point may hit the limitations of arcpy package and will need to do something using ArcObjects which provide a fine-grained API into the ArcGIS components.

If you switch to ArcGIS Pro, though, you won’t be able to run your old ArcObjects code that, for instance, will hide or show map grids (graticules) in map layout before exporting the map (this is not supported with arcpy). Even though ArcGIS Pro is shipped with arcpy, its functionality is not any richer than of ArcGIS Desktop based arcpy (at least for now).

However, because ArcGIS Pro comes with .NET SDK, it becomes relevant to learn how to access .NET .dll files and access the methods of classes available within the class libraries. I’ve tested to use pythonnet module and it worked great to access a class library authored with Visual Studio 2015 and Visual C#.

In order to make calls to .dll files compiled based on .NET code (in my case, it’s C#) using Python, you would need to install a Python package called pythonnet. Learn more at Calling a C# library from python.

However, accessing the ArcGIS Pro .NET libraries isn’t that easy as there are many dependent references involved. There is support for embedding the core functionality of ArcGIS Pro (such as accessing the geodatabase data, iterating features with cursors, constructing geometries etc.), however, one is supposed to access this functionality either using a console application or a WPF application.

There is a Python module called subprocess that would let you get the result of console app execution as a string which you could load into a Python data structure to use later, however, this is not a very elegant way to proceed. You basically author your console apps and then use Python to call those apps supplying the input arguments. This is probably the easiest way to get your .NET code executed from Python, though, as you have everything you need already installed.

An alternative approach is to compile a class library using x64 platform in Visual Studio and then access it from 64 bit Python (your ArcGIS Desktop x86 Python won’t work). Read more about accessing the core of .NET SDK at ProConcepts CoreHost.

Here are the steps involved. Please mind that this can get messy with all those Python modules installations, so you might like spinning a virtual machine or at least a virtualenv for this.

  1. Build your class library in Visual Studio for x64 platform.
  2. Install pythonnet for x64.
  3. Install pywin32 for x64.
  4. Copy two .dll files from:
    ​C:\Python27\ArcGISx6410.4\Lib\site-packages\pywin32_system32 to C:\Python27\ArcGISx6410.4\Lib\site-packages\win32\lib
  5. You should be able to run the code in the snippet below.

 

ArcGIS Pro vs ArcGIS Desktop: geoprocessing tools

Both ArcGIS Desktop and ArcGIS Pro have a suite of toolboxes with hundreds of geoprocessing tools available. However, there are still quite a few tools that are available only in ArcMap and have not been ported to Pro. You can find the list of those tools at the Help page Tools that are not available in ArcGIS Pro.

However, what not so many people may know is that there are actually quite a few tools that are available exclusively in ArcGIS Pro. They have never existed in ArcMap in other words. They may or may not become a core part of the ArcGIS Desktop ArcToolbox. I have seen that many of the tools added to ArcGIS Pro 1.3 have been added to ArcGIS Desktop 10.5. A good part of them is specific to Pro, however, and would not make sense in ArcMap, such as Consolidate Project or Package Project tools. However, there are some tools that are applicable to both environments as they operate on geodatabase, for instance, Alter Domain, or with a network dataset, for instance, Create Network Dataset From Template.

To be able to find out what tools are available in ArcGIS Pro and not available in ArcGIS Desktop, I’ve written a tiny Python script that reads all the toolboxes available in the both installations and then compares them. You would need to run the code for Pro and Desktop Python installations, of course.

As I need to call Python installed in Pro and Desktop, I prefer to use a .bat file for this:

"C:\Python27\ArcGIS10.4\python.exe" "C:\GIS\Temp\ToolLister.py" "Desktop"
"C:\Program Files\ArcGIS\Pro\bin\Python\Scripts\propy.bat" "C:\GIS\Temp\ToolLister.py" "Pro"

After calling this .bat file, you will get two .json files each containing the tools available in these two environments.

Please note that I have run this code for ArcGIS Desktop 10.4 and ArcGIS Pro 1.4.

The code for your reference (ToolLister.py):

The list of tools present in Pro that are not in Desktop

ArcGIS Pro does have some new toolsets we never had in Desktop.

Standard Feature Analysis

The Standard Feature Analysis toolbox contains tools for performing spatial analysis on feature data in your portal.

Raster Analysis

The Raster Analysis toolbox contains a set of powerful tools for performing raster analysis on data in your portal. By distributing the processing between multiple server nodes, you can efficiently process large datasets in less time than ever before.

GeoAnalytics Tools

The GeoAnalytics Tools toolbox contains a set of powerful tools for performing spatial analysis on big data. GeoAnalytics Tools are powered by your ArcGIS GeoAnalytics Server. ArcGIS GeoAnalytics Server distributes the analysis between multiple server nodes. By using distributed processing, you can process large datasets in less time.

And here are the new tools in existing toolboxes:

Geostatistical Analyst Tools
  • EBKRegressionPrediction_ga
  • GALayerToRasters_ga
Editing Tools
  • AlignFeatures_edit (since Desktop 10.5 only)
  • CalculateTransformationErrors_edit (since Desktop 10.5 only)
Analysis Tools
  • EnrichLayer_analysis
  • PairwiseIntersect_analysis
  • SummarizeWithin_analysis
  • PairwiseDissolve_analysis
  • SummarizeNearby_analysis
  • PairwiseBuffer_analysis
  • SplitByAttributes_analysis
  • GraphicBuffer_analysis
Spatial Analyst Tools
  • LocateRegions_sa (since Desktop 10.5 only)
  • RemoveRasterSegmentTilingArtifacts_sa (since Desktop 10.5 only)
3D Analyst Tools
  • ClassifyLasBuilding_3d (since Desktop 10.5 only)
  • TileLas_3d (since Desktop 10.5 only)
Conversion Tools
  • AddRasterToGeoPackage_conversion (since Desktop 10.5 only)
Data Management Tools (some of the tools may be available since Desktop 10.5)
  • AlterDomain_management
  • AddRelate_management
  • RemoveRelate_management
  • CreateMobileMapPackage_management
  • CreateVectorTileIndex_management
  • CreateVectorTilePackage_management
  • MosaicDatasetToMobileMosaicDataset_management
  • CreateSceneLayerPackage_management
  • EnableCOGO_management
  • DisableCOGO_management
  • BuildStereoModel_management
  • InterpolateFromPointCloud_management
  • ComputeCameraModel_management
  • GeneratePointCloud_management
  • ComputeMosaicCandidates_management
Space Time Pattern Mining Tools
  • LocalOutlierAnalysis_stpm
Network Analyst Tools
  • CreateNetworkDatasetFromTemplate_na
  • CreateTemplateFromNetworkDataset_na

Hope you will find this list useful. If your main workflows are still ArcGIS Desktop based (which I believe they are for most of us), keep in mind that you can call ArcGIS Pro Python 3.5 installation to take advantage of those new geoprocessing tools if you have both Pro and Desktop installed. Take a look at this Help page Python in ArcGIS Pro. To learn more about calling Windows applications from Python, take a look at the built-in subprocess module.

ArcPy: Batch export of attachments from geodatabase feature class with custom naming

If you have worked with attachments within an ArGIS geodatabase, you might have needed to take the attachments out of the geodatabase to use in another system or to process them separately. Unfortunately, there is no geoprocessing tool that would allow you to export all attachments to files, however this workflow is supported with arcpy and Python. There is a KB article at Esri site that shows the code necessary to export attachments of a feature class into an output folder.

This workflow is based, however, on using the attributes from the attachments table – %featureclassname%__ATTACH. But what if you would like to export the attachments naming output files using attributes of the source feature that has attachments associated with it? In this case, you would need to find out what feature every attachment is associated to (this is exposed via a relationship class) and then use this value to name the output file. To get all the information needed regarding the relationship class properties a Describe object for the relationship class can be used. Of course when exporting the attachments and naming output files using a source feature attribute, you have to make sure it is unique, otherwise the later exported attachment files will overwrite the previous ones.

This tool will handle Unicode characters in the attribute field you would like to use for naming output files, so it will work for users working in various locales, however, there is no validation for the output file name in terms of operating system restrictions on legal file names, but its easy to fix, for instance, using regex.

The code that will do this with some extra comments to help you follow along:

 

Creating convex hull using arcpy

When working with features in ArcGIS, you might need at some point of time to get a sense of the extent of input features using some kind of visual representation. This is often helpful when generating district polygons from point data or getting a sense of features distribution purely for visualization purposes. An ArcGIS geoprocessing tool called Minimum Bounding Geometry will let you generate those types of geometries. However, the Geometry Type options Convex hull, Circle, and Envelope are only available with an ArcGIS Desktop Advanced license. This makes it impossible to use this tool out-of-the-box if you are on Standard or Basic license level yet need occasionally generate convex hulls.

Fortunately, one can use arcpy Geometry.convexHull() method which

constructs the geometry that is the minimal bounding polygon such that all outer angles are convex.

This can be used to generate a convex hull polygon from an input Geometry object which can be a point, polyline, or a polygon. Using a very simple Python code, you can mimic the Minimum Bounding Geometry tool operation while having only Basic/Standard license.

For point data:

2017-04-17 15_13_31-Untitled - ArcMap

For line data:

2017-04-17 15_15_14-Untitled - ArcMap

For polygon data:

2017-04-17 15_16_27-Untitled - ArcMap

The Python code snippet:

Modify properties of scale bar element in ArcMap map document layout using Python and ArcObjects

This is yet another post on how to use ArcObjects and Python comtypes package to call ArcObjects COM objects using Python. In this example, we have a map document with a couple of layout elements pre-created such as a scale bar, a north arrow, and a scale text. We would like to modify the scale bar label. Currently, it is ‘Miles’, but we have decided that we would like to change this label to be ‘Miles (1.6 km)’ for all our map documents we are going to export to pdf later on.

arcpy site-package doesn’t provide a way to either read or write this information when accessing a MapsurroundElement object. This means we have to use ArcObjects. Keep in mind that through ArcObjects, you can read/write many other properties of the scale bar such as divisions, symbols, text formatting and many others.
Here is the code that will update scale bar unit label:

Access to ArcMap map document grid using Python and ArcObjects

When automating mapping production and exporting your map document layouts, you may need to manage map grids. They are grids or graticules which show the network of latitude and longitude lines in the layout.

For the very least, you would like to hide/show map grids in a map document’s data frame before exporting the map. The reason for this is that if your grid cell is of 1×1 km size, when exporting the map layout being in a small extent (for instance, 1: 1,000,000 you will get a pretty much no map except a black image because the grid will take all the place.

Being able to access map grids can also be handy when you have multiple grids with different grid cell sizes and you want to be able to control at what map scales each of the map grids should be visible.

Unfortunately, you cannot use arcpy to access grids defined for a data frame. However, the grids are exposed via an ArcGIS extension called Production Mapping which according to the Esri Help page

streamlines your GIS data and map production by providing tools that facilitate data creation, maintenance, and validation, as well as tools for producing high-quality cartographic products.

You might be surprised to learn this, but Production Mapping extension has its own arcpy-like site-package called arcpyproduction. Within this package, there is a class Grid class which provides access to grid properties for a grids and graticules layer. However, if you don’t have Production Mapping extension (it might be hard to justify buying this extension if you are not involved in heavy map production), as map grids are not exposed via arcpy, you have to use ArcObjects.

Since many would like to automate showing/hiding map grids when exporting map layouts, I’ve written a tiny function which can do that using Python. I have blogged earlier about how to start using ArcObjects from Python, so make sure to read this post first. Using this code, you will be able to choose which map grid you would like to use (if you have multiple grids with different cell size defined) when exporting a map layout.

Python progression path for GIS professionals

Over last years, I was working with Python almost full time either scripting some desktop GIS workflows or developing code for the back-end geoprocessing services using arcpy. I learned all kinds of Python packages, everything from data science packages such as pandas and numpy to more widely applicable ones such as xlsxwriter and reportlab. Being able to find a package and start using it producing the outputs needed in a matter of minutes is one of the key selling points of Python, I think.

However, due to the presence of such a large number of resources that are related to Python (just check this repository on GitHub – A curated list of awesome Python frameworks, libraries, software and resources)- one might feel a bit lost. There are so many things to learn, which are the most important ones? It also makes things a bit more complicated for niche developers or GIS analysts who do Python programming just occasionally. I have also experienced frustration being unable to identify the key competence areas to focus on and how to track my progress. Am I learning Python packages that are relevant for geospatial operations? What else should I learn after I’ve managed a certain feature of the language or a framework?

The result of this thought process is a public repository on GitHub which I am working on. It’s called Progression path for a GIS analyst who wants to become proficient in using Python for GIS: from apprentice to guru which is inspired partially by the awesome-python and partially by a SO post Python progression path – From apprentice to guru.

This is an attempt to provide a structured collection of resources that could help a GIS professional to learn how to use Python when working with spatial data management, mapping, and analysis. The resources are organized by progress category so basically everyone should be able to learn something new along the way. The resources will include books, web pages and blog posts, online courses, videos, Q/A from GIS.SE, links to code snippets, and some bedtime readings.

Be sure to check this one out, pick a topic of interest and start working on it. Also, feel free to star the repository if you have a GitHub account 🙂