Stream Android screen to a PC with TeamViewer

I was looking for quite some time for an easy way to stream an Android tablet screen onto a PC (Windows) to do demos on how applications run on Android (such as demos on Collector and other ArcGIS apps). Of course you can use a portable cam or webcam, but what if you just want to show something really quickly without setting up the cameras?

In case you will ever need to do demos with the mobile apps – read below how to do this.

There are tons of ways to do this (vnc servers, paid apps, give root access on the device), but the most efficient I’ve found so far is to use the TeamViewer. There is also a free version of this software.

  1. Download TeamViewer on your Windows machine. You don’t need to install it, it is possible to just run the .exe once.
  2. Install TeamViewer QuickSupport on your tablet.
  3. When running the TeamViewer on your tablet, you will be asked if you want to install add-on for remote control of the device. The tablet will automatically find the add-on for your manufacture (Lenovo, Sony, Samsung, HTC, HP etc) and install it.
  4. Then just connect from the Windows machine to the Android tablet (both should be connected to the Internet of course).

There may be some lagging when having a poor Internet connection, but generally people have the understanding for this and tend focus on the functionality and not on the performance of the screen sharing. Keep in mind that you can also do the remote control, so troubleshooting directly on the customer’s tablet is an option if you work in tech support.

So, you can not only see your screen, but actually perform clicks to simulate touches on the tablet. An amazing piece of functionality I’ll use often now for making demonstrations to our customers.

Find fields in a feature class that have only nulls with arcpy

A friend asked to help to write a tiny script that would list fields that contain only nulls (i.e., no data at all). So, basically any columns that were populated with any value for any feature since their creation time.

Quite easy and quick to do with arcpy.


import arcpy
fc = r"C:\Users\user\Documents\ArcGIS\Default.gdb\_DeleteColumns"

#getting list of fields with nulls allowed (those that don't allow nulls won't have
#nulls so we can skip them already now
not_null_fields = [field.name for field in arcpy.ListFields(fc) if field.isNullable != "False"]

#getting a dict {field_name : [list of all values]}
fields_dict = {field: list(set([feature[not_null_fields.index(field)] for feature in arcpy.da.SearchCursor(fc,not_null_fields)])) for field in not_null_fields}

#finding out which fields contain only None (null in Python terms)
fields_to_remove = [k for k,v in fields_dict.iteritems() if v == [None]]

#remove fields from the fc that contain only Null values
for field in fields_to_remove:
arcpy.DeleteField_management(fc,field)

Essential extra toolsets for ArcGIS Desktop professionals

If you are an ArcGIS users you could at some point of time find out that you lack a certain tool which you need to perform a certain analysis. Or you just need to hack some datasets really quickly and you are lazy to write your own tool for that. If you are not developer, though, the only option you would have is to see whether a model can be built in ModelBuilder to replicate the required operation (by combining multiple geoprocessing tools).

If ModelBuilder wouldn’t help you, then there is a good chance you would start searching for this kind of tool on the Internet. A great place to start is Google. The GIS Stackexchange web site is indexed very well and you will be able to find many of the tools you are looking for as answers to the questions on this web site too.

Another place to search for tools is ArcGIS.com. This one has replaced arcscripts and if you are on ArcGIS 9.3+ you probably wouldn’t go to arcscripts at all. When at ArcGIS.com, you don’t need to sign in to search for tools. Remember to enable the Show ArcGIS Desktop Content option and filter the results to show Tools only (figure below).

ArcGIS.com searchHere is a great reference page on how to do efficient searches: https://doc.arcgis.com/en/arcgis-online/reference/search.htm

In my “GIS Analyst toolbox”, I have collected many useful tools over last years. Some of them for older versions of ArcGIS, but in some cases they will just work; in others – you can get an idea on how tools are implemented and re-write to run on your version (if you can program of course or use ModelBuilder). Here is the short list that includes some of them:

SampleArcPyMappingScriptTools_10_v1 (ArcGIS toolbox, Python code)
From: http://www.arcgis.com/home/item.html?id=18c19ec00acb4d568c27bc20a72bfdc8
These (~20) tools were created as representative samples for how arcpy.mapping could be used to perform a variety of tasks. A must have for anyone who works with map documents often! It will also let you get started with arcpy.mapping module in no time, awesome!

Cartography Tools

  •    Adjust Layout Text Width (from ArcMap)
  •    Find and Replace a Text String
  •    Page Layout Element Report
  •    Shift Page Layout Elements
  •    Update Symbology

Export and Printing Tools

  •    Append PDF Documents
  •    Export Map Documents to PDF
  •    Print Data Driven Page(s)
  •    Print Map Document(s)

MXD and LYR Management Tools

  •    Add Layer File into MXD (from ArcMap)
  •    Find Broken Data Sources (Report)
  •    Find Data Source (Report)
  •    Find Layers Projected on the Fly (Report)
  •    Multi Layer File Summary (Report)
  •    Multi MXD Summary (Report)
  •    Replace Layer with Layer File
  •    Replace Layer with Layer File (from ArcMap)
  •    Update MXD from pGDB to fGDB
  •    Update MXD tags

Tools that will let you create your own cartographic effects
From: http://mappingcenter.esri.com/index.cfm?fa=arcgisResources.modelsScripts
Lots of useful tools for any cartographer who wants to add some extra mapping features.

Database inspector (ArcGIS toolbox, Python code)
From: https://sites.google.com/site/geospatialonestop/tools/SchemaInspector.zip?attredirects=0&d=1
This is a must have for any ArcGIS analyst doing any Python development for geodatabase maintenance and geodata management. This is a suite of tools for analyzing components of a geodatabase and for finding the differences between geodatabase schemas. Can print properties of fields, feature classes, relationship classes, domains, tables. Can compare particular geodatabase components between two workspaces. Can compare two geodatabases and tell you the difference. A great tool I use daily.

ArcGIS Server ServerAdminToolkit 10.1+ (by Kevin Hibma from Esri) (ArcGIS toolbox, Python)
From: http://www.arcgis.com/home/item.html?id=12dde73e0e784e47818162b4d41ee340
These tools perform some common administrative tasks with an ArcGIS Server machine. All of these tasks can be accomplished through the UI (ArcMap), the Web Manager or the REST Administration page. By using tools you can automate redundant workflows or chain common workflows together. Most of these tasks, turned into tools, have more detailed explanations in the help. This package is composed of three main parts: Tools, Standalone executable, and Code.

ArcREST (ArcGIS toolbox, Python)
From: https://github.com/Esri/ArcREST
A set of python tools to assist working with ArcGIS REST API for ArcGIS Server (AGS), ArcGIS Online (AGOL), and ArcGIS WebMap JSON. An amazing package that any ArcGIS Online or ArcGIS Server admin wants to have! Nearly all things you can do with the REST API, you can do with ArcREST. A must have.

Spatial Analyst Supplemental Tools
From: http://www.arcgis.com/home/item.html?id=3528bd72847c439f88190a137a1d0e67
A collection of script tools to supplement Spatial Analyst Tools.

  • Create Dendrogram
  • Draw Signature
  • Erase Raster Values
  • Filled Contours
  • Maximum Upstream Elevation
  • Peak
  • Tabulate Area 2
  • Viewshed Along Path
  • Zonal Statistics As Table 2

Geomorphometry & Gradient Metrics (ArcGIS toolbox, Python)
From: http://evansmurphy.wix.com/evansspatial#!arcgis-gradient-metrics-toolbox/crro

Urban Network Analysis Toolbox for ArcGIS (Python)
From: http://cityform.mit.edu/projects/urban-network-analysis.html#download
The tools incorporate three important features that make them particularly suited for spatial analysis on urban street networks.

National Water-Quality Assessment (NAWQA) Area-Characterization (ArcGIS toolbox, Python)
From: http://www.arcgis.com/home/item.html?id=cbb59504f59f4e18b23817fb0ef40e56 and http://pubs.usgs.gov/of/2010/1268/
The toolbox is composed of a collection of custom tools that implement geographic information system (GIS) techniques used by the NAWQA Program to characterize aquifer areas, drainage basins, and sampled wells.

X-ray for ArcGIS (ArcMap, ArcCatalog) (add-in)
From: http://www.arcgis.com/home/item.html?id=f0ae73e90c1a4992a1059e7d370966d4 and http://www.arcgis.com/home/item.html?id=9ea218ff575f4a5195e01a2cae03a0ae
The X-Ray for ArcCatalog add-in can be used to develop, refine and document your geodatabase designs. The X-Ray add-in for ArcMap can be used to document the properties of your map documents (MXDs).

Marine Geospatial Ecology Tools (MGET) (ArcGIS toolbox, Python)
From: http://mgel.env.duke.edu/mget/download/
A free, open-source geoprocessing toolbox that can help you solve a wide variety of marine research, conservation, and spatial planning problems. MGET plugs into ArcGIS and can perform tasks such as:

  •    Accessing oceanographic data from ArcGIS
  •    Identifying ecologically-relevant oceanographic features in remote sensing imagery
  •    Building predictive species distribution models
  •    Modeling habitat connectivity by simulating hydrodynamic dispersal of larvae
  •    Detecting spatiotemporal patterns in fisheries and other time series data

Favorite tools and resources for cartographers
From: http://blogs.esri.com/esri/arcgis/2013/06/04/favorites-cartographers/
A compilation of some of the most popular tools and sources of information about maps and cartographic design.

Geospatial Modelling Environment (known as HawthsTools)
From: http://www.spatialecology.com/gme/gmedownload.htm
GME provides you with a suite of analysis and modelling tools, ranging from small ‘building blocks’ that you can use to construct a sophisticated work-flow, to completely self-contained analysis programs. It also uses the extraordinarily powerful open source software R as the statistical engine to drive some of the analysis tools. One of the many strengths of R is that it is open source, completely transparent and well documented: important characteristics for any scientific analytical software.

Named tuples from Python collections with arcpy

I am reading a book by Jeff Knupp Writing Idiomatic Python and thought to share a great tip for everyone who works with arcpy and Python.

If you are scripting in ArcGIS, you have probably done it a hundred times. Set up a workspace, create a cursor object for a feature class or a table and then retrieve the rows.

Rows are represented as a list of tuples:


rows = [(57525, 57526, 179.4063375296146),
(57525, 57529, 107.75624872381789),
(57525, 57530, 73.06809412350371),
(57526, 57529, 71.6500888057982)]

Then if you need to access individual attributes of a feature, you would use an index value starting from zero:


for row in rows:
print row[0]

It’s OK with just a few columns but what if your business logic involves having dozens of columns and you need to access them individually? You would end up having lots of row[14] and row[23]. Then you would either comment the row with the clarification on what field this index value actually represents, count manually to see what column you are actually accessing or create a variable for each value with the name replicating the column name or alias.

In either case you would end up having a bunch of the code that is hard to maintain and read. This is where named tuples come into play. I wish I’ve learned about them earlier.

The rows I added earlier represents the source and destination features ID (polylines) and the angle at which they intersect. It would be so much easier to read the code if we could use column names instead of indexes.

from collections import namedtuple
featureRow = namedtuple("featureRow",['sourceId','destId','angle'])
for row in rows:
feature = featureRow._make(row)
print feature

>>> featureRow(sourceId=57525, destId=57526, angle=179.4063375296146)

You could also access the feature attributes by accessing the attributes of the feature object:

row_info_string = 'Angle from feature {sourceId} to feature {destId} is {angle}'
print(row_info_string.format(sourceId=feature.sourceId, destId=feature.destId,angle=feature.angle))

If you have many fields, then instead of retyping them or coming up with alternative names, you could just get a list of the fields in the table.

fields = [field.name for field in arcpy.ListFields(feature_class)]
featureRow = namedtuple("featureRow",fields)
print feature

>>> featureRow(SourceID=57525, DestinationID=57526, Angle=179.4063375296146)

Print feature.SourceID

>>> 57525

Using the named tuple with the rowObject.property notation is so much easier and nicer to read. Try it and you will love it!

Design of WebGIS back-end: architecture considerations

I have spent last two years doing a lot of Python development and designing and implementing Web GIS which included ArcGIS Server, geoprocessing services and ArcGIS API for JavaScript (further JS) web client. What I would like to do is to share an idea which I got to like.

If you need to do something, try doing it at the back-end

Imagine you have a JS web application where users will work with some feature services via a web map. They can select multiple features and calculate the sum of the values features have in a field (or fields). Let’s go through alternatives you have now.

  1. Pre-calculate the values you think your users will query and store them in the database.
    This would work fine actually when you know that your users are going to generate reports on a certain fields often and the performance is crucial. It might actually make sense to calculate certain values beforehand and store them. The disadvantage of this is additional storage and that you need to keep the values updated – the calculated field depends on other fields and their values can change. This would imply re-calculating the report field often as a part of the daily or weekly routine depending on the workflow.
  1. Get the feature’s data from the ArcGIS Server feature service and calculate the requested value on-the-fly in the client.
    Unless you are retrieving complex geometry, this operation wouldn’t cost you much. The problem is that the volume of JS code (or TypeScript) will increase and every upcoming modification in the code would imply new release which can be a painful process if you need to compress your code and move things around. Another thing is that if the amount of data you work with is rather large, there is a good chance the web browser might get slow and the performance will degrade significantly.
  1. Use the database server to calculate the values.
    This became my favorite over last years. This approach has multiple advantages.
    First, this operation runs on the database server machine with enough RAM and CPU resources. So you are not limited by the web browser capacity. The database servers are very good at calculating the values: this kind of operation is very inexpensive because in most cases it does not involve use of cursors. You have a privilege to work in transaction which provides a higher level of data integrity (it would be hard to mess up the database since you can roll back).
    Second, you can use SQL. It might not sound as an advantage first, but remember that code is written once, but is read many times. Readability counts. SQL is a clean way of communicating the workflow and the database code (such as stored procedures) is very easy to maintain. Unlike JS, you work with just one database object and don’t really have any dependencies on the system provided that you have a database server of a certain version and privileges required to create and execute stored procedures.
    Finally, allowing the database server do the work for you, you expose a certain procedure to other clients which could work with it. You don’t need to modify the client code and by updating the SQL code at one place, you automatically make it available for all the applications that work with it.

ArcREST: Python package for administering ArcGIS Server and ArcGIS Online/Portal

ArcREST is a great toolset I have found some time ago. It is for anyone who administers ArcGIS Online, ArcGIS Portal or ArcGIS Server. In short, it is a Python wrapper for the Esri REST API. I had to write many Python scripts that allowed me to update the properties of ArcGIS Server services in batch, but now I don’t need to write anything like this anymore. This is because now I can do everything I did on my own just by using ArcREST. If you are an ArcGIS Online / Portal admin, you should definitely take a look at this module since it can save you a lot of time, and you won’t need to author your own scripts for managing the ArcGIS Online content and organization settings with the scripting techniques.

This Python package is authored by Esri Solutions team and is available in public access on GitHub. You can download the source code, optionally install the package, and then use it on your local machine just like any Python package. If you don’t want to install the package, you can just add the path to arcrest and arcresthelper folders to the Python path by adding this into your Python file:

import sys
sys.path.append(r”path to arcrest folder”) #C:\GIS\Tools

Provided that you have a folder named arcrest in the example Tools folder, when you run the Python file, it will be able to import the arcrest package and access its modules.

To get an overview of this Python package, take a look at this excellent DevSummit 2015 video where developers of ArcREST talked about it.

Even though this is not a full implementation of the Esri REST API, it covers most of it and Esri developers update the code to include latest changes in the REST API. It is a good idea to clone the repository and pull the changes now and then to get the latest code if you will use on the daily basis.

I felt kind of sad first that all the Python code I wrote for administering ArcGIS Server won’t be used any longer, but at the same time so glad the ArcREST was developed. It is a great piece of software that will let you get started in no time at all and access all your server/online resources with Python.

Caveat: it does have some dependencies on arcpy package which is used for converting feature sets into JSON and back, but apart from that you should be able to run the tools on a machine with no ArcGIS software installed whatsoever.

Useful resources in computer science/math for GIS Analysts

A lot of people who are studying GIS at school or already working as GIS analysts or GIS consultants often wonder what kind of competence will help to be attractive for employers and what domains of expertise are going to be in demand in the foreseeable future.

Usually the kind of questions GIS professionals ask is how much a GIS analyst should learn from other domains. So, we are wondering how much math, statistics, programming, and computer science should GIS analysts learn. Naturally, knowing what kind of GIS specific expertise is in demand is also very helpful. I have several posts on how get better at GIS here, here, and here.

To know what kind of GIS tools can do what kind of job is definitely helpful. This is much like a woodworker should know what kind of tools he has in his toolbox and what tools are available in the woodworking shop. Finding an appropriate tool for a certain job is not so hard nowadays with the Internet search engine and QA sites. However, the ability to understand both how data processing tools work and what happens behind the scenes to be able to interpret the analysis results is indispensable.

What is often true for many GIS analysts is that during their studies the main focus was on the GIS techniques and tools while math and CS courses were supplementary. This makes sense and the graduates are indeed most often competent GIS professionals capable of operating various GIS software suites, provide user support, and perform all kind of spatial analysis. However, it is also possible that in a career change, a person who hasn’t done any studies on GIS, is working as a GIS analyst and needs to catch up a bit. For those people who feel that they lack background GIS competence that they should had a chance to learn during their studies, or for you who just want to learn something that could help to have a broader view and give a deeper understanding of the GIS, I have compiled a list of useful links and books. Please enjoy!

There are lots of great questions answered on the GIS.SE web site; here is just a few:

Great books:

Spatial Mathematics: Theory and Practice through Mapping (2013)
This book provides gentle introduction into some mathematical concepts with focus on mapping and might be a good book to start learning math in GIS. No advanced background in math is required and high-school math competence will be sufficient.

Table of contents

  • Geometry of the Sphere
  • Location, Trigonometry, and Measurement of the Sphere
  • Transformations: Analysis and Raster/Vector Formats
  • Replication of Results: Color and Number
  • Scale
  • Partitioning of Data: Classification and Analysis
  • Visualizing Hierarchies
  • Distribution of Data: Selected Concepts
  • Map Projections
  • Integrating Past, Present, and Future Approaches

Mathematical Techniques in GIS, Second Edition (2014)
This book gives you a fairly deep understanding of the math concepts that are applicable in GIS. To follow the first 5 chapters, you don’t need any math except high school math. Later on, the book assumes that you have good knowledge of math at the level of a college Algebra II course. If you feel that it gets hard to read, take an Algebra II course online at Khan Academy or watch some videos from MIT to catch up first and then get back to the book. What I really liked about this book is that there are plenty of applicable examples on how to implement certain mathematical algorithms to solve the basic GIS problems such as point in polygon problem, finding if lines are intersecting and calculating area of overlap between two polygons. This could be particularly useful for GIS analysts who are trying to develop own GIS tools and are looking for some background on where to get started with the theory behind the spatial algorithms.

Table of contents

  • Characteristics of Geographic Information
  • Numbers and Numerical Analysis
  • Algebra: Treating Numbers as Symbols
  • The Geometry of Common Shapes
  • Plane and Spherical Trigonometry
  • Differential and Integral Calculus
  • Matrices and Determinants
  • Vectors
  • Curves and Surfaces
  • 2D/3D Transformations
  • Map Projections
  • Basic Statistics
  • Correlation and Regression
  • Best-Fit Solutions

GIS: A Computing Perspective, Second Edition (2004)
The book is a bit dated, but it is probably the best book in computer science for a GIS professional. It provides very deep understanding of the computational aspects that are used in GIS.

Table of contents

  • Introduction
  • Fundamental database concepts
  • Fundamental spatial concepts
  • Models of geospatial information
  • Representation and algorithms
  • Structures and access methods
  • Architectures
  • Interfaces
  • Spatial reasoning and uncertainty
  • Time

Practical GIS Analysis (2002)
This book is a unique example of a book for GIS professionals who want to see how the basic GIS algorithms and tools work. The exercises that follow give readers a chance to execute many common GIS algorithms by hand which let truly understand even some complex operations such as generating TIN or finding the shortest path on a street network. The software used as a reference is ArcView GIS 3, but it is still relevant as the GIS concepts haven’t changed much since then.

Table of contents

  • GIS Data Models
  • GIS Tabular Analysis
  • Point Analysis
  • Line Analysis
  • Network Analysis
  • Dynamic Segmentation
  • Polygon Analysis
  • Grid Analysis
  • Image Analysis Basics
  • Vector Exercises
  • Grid Exercises
  • Saving Time in GIS Analysis

Maths for Map Makers (2004)
I haven’t read this book so don’t have anything to comment on this. Sorry!

Table of contents

  • Plane Geometry
  • Trigonometry
  • Plane Coordinates
  • Problems in Three Dimensions
  • Areas and Volumes
  • Matrices
  • Vectors
  • Conic Sections
  • Spherical Trigonometry
  • Solution of Equations
  • Least Squares Estimation
  • References
  • Least Squares models for the general case
  • Notation for Least Squares

Exploring Spatial Analysis in GIS (1996)
I haven’t read this book either. I guess this one might be hard to find, but have listed it here just in case.

Good luck with the readings!