Printing pretty tables with Python in ArcGIS

This post would of interest to ArcGIS users authoring custom Python script tools who need to print out tables in the tool dialog box. You would also benefit from the following information if you need to print out some information in the Python window of ArcMap doing some ad hoc data exploration.

Fairly often your only way to communicate the results of the tool execution is to print out a table that the user could look at. It is possible to create an Excel file using a Python package such as xlsxwriter or by exporting an existing data structure such as a pandas data frame into an Excel or .csv file which user could open. Keep in mind that it is possible to start Excel with the file open using the os.system command:

os.system('start excel.exe {0}'.format(excel_path))

However, if you only need to print out some simple information into a table format within the dialog box of the running tool, you could construct such a table using built-in Python. This is particularly helpful in those cases where you cannot guarantee that the end user will have the 3rd party Python packages installed or where the output table is really small and it is not supposed to be analyzed or processed further.

However, as soon as you would try to build something flexible with the varying column width or when you don’t know beforehand what output columns and what data the table will be printed with, it gets very tedious. You need to manipulate multiple strings and tuples making sure everything draws properly.

In these cases, it is so much nicer to be able to take advantage of the external Python packages where all these concerns have been already taken care of. I have been using the tabulate, but there are a few others such as PrettyTable and texttable both of which will generate a formatted text table using ASCII characters.

To give you a sense of the tabulate package, look at the code necessary to produce a nice table using the ugly formatted strings (the first part) and using the tabulate package (the second part):

The output of the table produced using the built-in modules only:

builtin

The output of the table produced using the tabulate module:

tabulate

 

 

Advertisements

Using Python start up script for all Python interpreters

This post would be helpful for users of desktop GIS software such as ArcMap who need to use Python inside those applications.

There is a not so well known trick to trigger execution of a Python script before any Python interpreter on your system starts.

Note: If you are a QGIS user, there is a special way of achieving this. Please see the question Script that runs automatically from the QGIS Python Console when QGIS starts, using schedule tasks on Windows 10 for details.

The way to do this is to set up an environment variable called PYTHONSTARTUP in your operating system. You need to do two things:

  1. Create an environment variable that would point to a path of a valid Python script file (.py) with the code that you would like to get executed before any Python interactive interpreter starts. Look at the question [Installing pythonstartup file](https://stackoverflow.com/questions/5837259/installing-pythonstartup-file) for details.
  2. Write Python code that you would like to get executed.

A very important thing to consider is that

The file is executed in the same namespace where interactive commands are executed so that objects defined or imported in it can be used without qualification in the interactive session.

This means that you can do a bunch of imports and define multiple variables which would be available to you directly at the start up of your GIS application. This is very handy because I often need to import the os and sys modules as well as import arcpy.mapping module and create mxd variable pointing to the current map document I have open in ArcMap.

Here is the code of my startup Python script which you can modify to suit your needs. If your workflow relies on having some data at hand, then you might need to add more variables exposed. I have ArcMap and ArcGIS Pro users in mind.

I have included in the example above a more specific workflow where you would like to be able to quickly execute SQL queries against an enterprise geodatabase (SDE). So, when ArcMap has started, you only need to create a variable conn pointing to a database connection file (.sde) and then use the sql() function running your query. Thanks to the tabulate package, the list of lists you get back is drawn in a nice table format.

2018-03-15 12_55_03-PythonWindowArcMap

 

 

Generating a database schema report using SchemaSpy

I recently wanted to generate a visual report over a database to see what tables, relationships, and other objects are present. I also wanted to see the schema of all the tables. The registrant Python package I’ve written does give me most of the things I want but it was not designed to fetch any relational structure such as primary/foreign keys and constraints.

I found an excellent free command line tool tool, SchemaSpy. It requires Java so it may be an extra thing to fix if you do not have it installed. The syntax for Microsoft SQL Server:

java.exe -jar .\schemaspy-6.0.0-rc2.jar -t mssql05 -dp .\sqljdbc42.jar -db non_spat -host localhost -port 1433 -u user -p password -o C:\Temp\sqlschema

You would need to download the Microsoft JDBC Driver for SQL Server if you work with MS SQL Server (I used the version 6).

Python progression path for GIS professionals

Over last years, I was working with Python almost full time either scripting some desktop GIS workflows or developing code for the back-end geoprocessing services using arcpy. I learned all kinds of Python packages, everything from data science packages such as pandas and numpy to more widely applicable ones such as xlsxwriter and reportlab. Being able to find a package and start using it producing the outputs needed in a matter of minutes is one of the key selling points of Python, I think.

However, due to the presence of such a large number of resources that are related to Python (just check this repository on GitHub – A curated list of awesome Python frameworks, libraries, software and resources)- one might feel a bit lost. There are so many things to learn, which are the most important ones? It also makes things a bit more complicated for niche developers or GIS analysts who do Python programming just occasionally. I have also experienced frustration being unable to identify the key competence areas to focus on and how to track my progress. Am I learning Python packages that are relevant for geospatial operations? What else should I learn after I’ve managed a certain feature of the language or a framework?

The result of this thought process is a public repository on GitHub which I am working on. It’s called Progression path for a GIS analyst who wants to become proficient in using Python for GIS: from apprentice to guru which is inspired partially by the awesome-python and partially by a SO post Python progression path – From apprentice to guru.

This is an attempt to provide a structured collection of resources that could help a GIS professional to learn how to use Python when working with spatial data management, mapping, and analysis. The resources are organized by progress category so basically everyone should be able to learn something new along the way. The resources will include books, web pages and blog posts, online courses, videos, Q/A from GIS.SE, links to code snippets, and some bedtime readings.

Be sure to check this one out, pick a topic of interest and start working on it. Also, feel free to star the repository if you have a GitHub account 🙂

4 years with GIS at StackExchange

It’s been 4 years since I’ve joined the GIS.SE community – one of the sites under the StackExchange umbrella. It’s a good time to reflect a bit I guess. I have learned a lot during those years reading answers provided by peer GIS professionals. I have also learned quite a lot while answering questions for which I thought I could provide a working solution. I have never thought I would get so many reputation points though. Yet I have now just over 20K (which places me in the top 20 users of the GIS.SE site), which is quite amazing as I have never thought that my answers could get so many up votes.

In one of the episodes of one of my favorite Python podcasts – Talk Python To Me – Martijn Pieters provided his thoughts on why he enjoys answering questions on StackOverflow. I’ve decided to share my arguments why being involved in GIS.SE is so great:

  • Answering questions is a great exercise which you can do just for fun. For instance, someone has asked a question – Displaying attributes upon their polygons within ArcMap legend?. I have never done it myself, so I got curious whether this could be done. After searching around in the UI for some time, I finally found the sequence of steps to perform and shared it with the questioner. Great.

 

  • An answer can be a form of organizing own knowledge about a subject. For instance, someone has asked a question – Formatting street network data for ArcGIS Network Analyst?. I have been working with Network Analyst for many years but never have compiled a short “getting started guide” on where to find data sources, what skills are necessary etc. Answering this question gave me a chance to put all my reference information into one place which other users can also find and read.

 

  • An answer can help someone who has faced a very peculiar problem to get a solution quickly without spending hours if not days. For example, someone has asked *.lyr to *.style conversion in ArcGIS. I needed to do exactly this just a few weeks earlier and was searching for the tool to do this for some days. Being able to answer a question in 10 minutes after it was posted knowing that someone else wouldn’t need to go through all possible UI tools in ArcMap is incomparable.

 

  • An answer I give to others can be an answer I will be looking for in the future myself. It is funny, but sometimes, when I have a question I am trying to find an answer to, when using a search engine, I find the GIS.SE page with the answer I just needed and a sample code snippet I can use. I am so pleased someone has found the time to post it. Looking at the author of the answer makes me laugh because it was me.

 

  • Answering questions makes me a better writer as I need to formulate my thoughts very accurately and convey the information in a clear and consistent way. I should also be concise yet comprehensive. I believe answering on GIS.SE helped me to become a better writer which I think is a great skill to have.

 

  • Answering questions makes me a more competent professional as I learn a lot about the GIS software and GIS programming. I have answered many arcpy related questions while I was learning arcpy myself. In addition, some questions gave me a chance to play with a software I don’t use daily which helps me to stay up-to-date on what software is being used by other people and how.

    To summarize, over last 4 years, I learned a lot about GIS software features, helped many people with their tasks, improved my writing skills, and became a more competent GIS expert. Not too bad!

    PS. Go ahead and read a great meta post How do high reputation (>5,000) users find time to commit to GIS Stack Exchange?

LICEcap – recording the screen to animated GIF

If you have ever needed to explain to someone a sequence of steps to perform in an application, you have probably ended up with writing a long sequence of steps. This takes a lot of time and when even slight changes are made in the interface, you need to rewrite the whole thing.

I remember having nice visual animations in ArcGIS Desktop Help files where one could follow the clicks recorded in the ArcMap GUI. One of the ways to record the screen without creating large video files is to use a GIF recorder. I’ve looked for different ones and the LICEcap seems to suit me well.


LICEcap is GPL free software, so the source is available. It’s fast, has an option to adjust the FPS value and make pauses and add the text boxes. A great piece of software you could use to make a quick record of a certain workflow – you could use it yourself for any future reference or to send a GIF to someone when helping out with the software.

Stream Android screen to a PC with TeamViewer

I was looking for quite some time for an easy way to stream an Android tablet screen onto a PC (Windows) to do demos on how applications run on Android (such as demos on Collector and other ArcGIS apps). Of course you can use a portable cam or webcam, but what if you just want to show something really quickly without setting up the cameras?

In case you will ever need to do demos with the mobile apps – read below how to do this.

There are tons of ways to do this (vnc servers, paid apps, give root access on the device), but the most efficient I’ve found so far is to use the TeamViewer. There is also a free version of this software.

  1. Download TeamViewer on your Windows machine. You don’t need to install it, it is possible to just run the .exe once.
  2. Install TeamViewer QuickSupport on your tablet.
  3. When running the TeamViewer on your tablet, you will be asked if you want to install add-on for remote control of the device. The tablet will automatically find the add-on for your manufacture (Lenovo, Sony, Samsung, HTC, HP etc) and install it.
  4. Then just connect from the Windows machine to the Android tablet (both should be connected to the Internet of course).

There may be some lagging when having a poor Internet connection, but generally people have the understanding for this and tend focus on the functionality and not on the performance of the screen sharing. Keep in mind that you can also do the remote control, so troubleshooting directly on the customer’s tablet is an option if you work in tech support.

So, you can not only see your screen, but actually perform clicks to simulate touches on the tablet. An amazing piece of functionality I’ll use often now for making demonstrations to our customers.