Using Python to close an ArcMap map document down

This article isn’t particularly mainstream but at the recent Esri UK user conference I was asked the following question:

How do I close down an open ArcMap map document through a Python script? If I have a number of map documents open I just want a particular map document to close and leave all of the other ones open….

“Hmmmmm – good question. Email me and I will see what I can think of…”

Well the customer emailed earlier this week asking the same question. How am I going to solve this?

Normally I might have a look in the excellent ArcGIS for Desktop help for ArcPy but there was nothing there as this is really not an ArcPy problem.

An internet search found that there were a number of others who had asked the same question but did not really receive a satisfactory reply.

For example:

Use the os.system(“TASKKILL /F /IM ArcMap.exe”)

Which was suggested in this thread https://geonet.esri.com/thread/58161

Or use the Python library psutil to get a list of processes and if the name of a process (Arcmap.exe) is found then stop it using the terminate() method which was suggested on this thread https://geonet.esri.com/thread/80474

There are countless others asking pretty much the same sort of thing (not necessarily to do with a map document) on http://gis.stackexchange.com

As you can imagine both options will close all open instances of ArcMap.

But what is you wanted to be a little more choosy about which instance of ArcMap you wanted to close down? What about if you wanted to close ArcMap down based upon the name of its open map document?

 

Using the psutil Python module

The answer to this is to use the Python module psutil. This module is used to obtain information from running processes – basically anything you have running in Task Manager can be obtained using this module, and a whole lot more.

It is not part of Python’s Standard Library which means you will need to download it and install it onto your machine which you want to run the script on. Thankfully it is incredibly easy to do this as it is part of the Python Package Index (PyPI) repository and modules found here can be installed easily as long as you have an internet connection.

Let’s go and install the psutil

1: Open up a command line prompt

2: At the command prompt, Change Directory (CD) into your Python \Scripts directory, for example:

cd C:\Python27\ArcGISx6410.3\Scripts

Inside this folder are a number of python utility scripts – the one we want to use is pip.exe as this will install the named module within the PyPI repository.

3: At the command prompt type in:

pip.exe install psutil

and press <return>

This will start to install psutil into your Python \Site-Packages folder. As it installs you will see messages along the lines of:


Downloading psutil-4.3.0-cp27-none-win_amd64.whl (169kB)

100% |################################| 172kB 1.9MB/s

Installing collected packages: psutil

Successfully installed psutil-4.3.0



And that is all you have to do to install it! You can now import the module into your script and take advantage of its functionality.

Information about psutil can be found here:

https://pypi.python.org/pypi/psutil

It is well documented and the nice thing about such an unfamiliar (to me) module is that it contains many ‘easy to follow’ samples. If only all Python modules were as well documented….

 

Let’s consider this problem….

I have a number of ArcMap map documents open and part of my workflow is to close a particular map document (Editing.mxd) once I have made some changes to using ArcPy functionality.

Inspecting the Task Manager shows that a number of ArcMap applications are open and each application has its own process ID (View menu > Select Columns and tick PID (Process Identifier) )

01_TaskManager

But which one is the Editing.mxd process? Task Manager can not provide that level of information.

This is where the psutil module comes in handy. This can be used to close down the Editing.mxd map document.

Let’s write some code to do this:

1: Import psutil and get a list of process IDs

mxdName = "Editing.mxd"
import psutil
pidList = psutil.pids()

2: Process the PID list to obtain the corresponding operating system process and get its name

for pidID in pidList:
  process = psutil.Process(pidID)
  pidName = process.name()

3: Find the ArcMap processes and get their memory maps.

  if (pidName == "ArcMap.exe"):
    tMemMaps = process.memory_maps()

The memory_maps() method retrieves information about the processes (ArcMap.exe) as a Python list of tuples.

The information contained in the list of tuples can be inspected using a for .. in loop and one of the items contained within it is the name (and location) of the current ArcMap object which is being processed, for example,  Editing.mxd.

02_MamMaps

4: Parse each memory map, obtain path information and see if the desired mxd is present

      for eachMemMap in tMemMaps:
        mxdPresent = eachMemMap[0].find(mxdName)

The find() method belongs on the string object and returns the index number at which the value contained in mxdName (i.e. Editing.mxd) is found.

5: If the index number is greater than -1 kill the process

        if mxdPresent > -1:
          process.kill()   # Close the desired ArcMap
          break

If the string is not found then a value of -1 is returned from the find() method. If the number is greater than -1 then the map document name is the one we want to close, so let’s kill the process. The break statement is there as it is assumed there is only one instance of the map document called Editing.mxd open so there is no point processing any other process’ memory maps.

The finished script is displayed below:

mxdName = "Editing.mxd"
import psutil
pidList = psutil.pids()

for pidID in pidList:
    process = psutil.Process(pidID)
    pidName = process.name()
    if (pidName == "ArcMap.exe"):   # Process only the
                                    # ArcMap.exe
        #Get the process memory map
        tMemMaps = process.memory_maps() # A list of
                                         # tuples
        for eachMemMap in tMemMaps:
            mxdPresent = eachMemMap[0].find(mxdName)
            if mxdPresent > -1:
                process.kill()      # Close the desired
                                    # ArcMap process
                break
print(mxdName + " has been closed down")

It is available in GitHub at:

https://github.com/epjmorris/ArcPySnippets/tree/master/CloseArcMapScript

 

Let’s go Pro!

Saying that though, this script will not work with ArcGIS Pro – it’s a bit of a Misfire.

Yes – there is an ArcGISPro.exe process which can be seen in Task Manager and the process does have a memory map but within the memory map there is no reference to the currently opened map project (.aprx file).

This isn’t the end of the world though as you can only ever have one instance of ArcGIS Pro running. If you try and launch another ArcGIS Pro project while one is already open then this is the message which will be displayed:

03_InstanceOfPro

In which case you can use the solution as suggested in one of the GeoNet posts, with a very slight amendment:

import os
os.system(“TASKKILL /F /IM ArcGISPro.exe”)

And there you have it! The Python module psutil is a pretty useful module if you want to see what is associated with a particular process. I am thinking of this module as the Python equivalent of Sysinternals very handy “Process Explorer” application.

I think I have only really scratched the surface of this module and it is one I suspect I will use more in the future.

Useful links
Python Standard Library
https://docs.python.org/2.7/library/

Python Package Index (PyPI) repository
https://pypi.python.org/pypi

psutil module homepage
https://pypi.python.org/pypi/psutil/4.3.0

Sysinternals Process Explorer
https://technet.microsoft.com/en-us/sysinternals/processexplorer.aspx

Working with script templates in PyScripter

In a previous post I went over some PyScripter configuration which the attendees to the “Introduction to Geoprocessing Scripts using Python” course carry out. These things include adding line numbers to a script and making sure full code completion was available for the ArcPy site package.

This article will explain how to create script templates – scripts with ready to go code. This is another nice feature of PyScripter.

Script templates can be used to provide a new script with code already stubbed out such as:

  • Add frequent comments.
  • Import commonly used modules / site packages.
  • Provide a structure for runtime error handling and log file generation.

An example of a script template is shown below:

TemplateCode
Default Script Template

This default code appears when you press the New Python Module button NewButton or if you go to the File menu and then choose New > New Python Module.

So how do you change this code template?

By default you are provided with the following code (plus comments):

MainModule

This can be easily changed by:

1: Select from the Tools menu > Options > File Templates menu item which will display the File Templates dialog box.

This dialog contains a number of different types of templates which you can use in case if you ever need to create an XML or HTML document within PyScripter.

2: Under Name, choose Python Script.

3: In the File Template section locate the Template area.

01ScriptTemplate

This is where you can add your template code which will appear when you create a new script module.

You might want to include the following:

  • Comments
  • import statements
  • Error handling and logging

**NB** Please be aware that when you update this template there is no way of regaining the original code so it might be a good idea to copy the code into a file just in case….

For example:

#------------------------------------------------------
# Name:        $[ActiveDoc-Name]
# Purpose:
# Author:      $[UserName]
# Created:     $[DateTime-'DD/MM/YYYY'-DateFormat]
#------------------------------------------------------
import arcpy, os, sys
try:
    # Add implementation here
    arcpy.env.workspace = &amp;amp;quot;&amp;amp;quot;
except arcpy.ExecuteError():
    # Capture GP error
    print (arcpy.GetMessages(2))
except Exception as e:
    # Capture general Python error
    print (e)

Notice that in the comments part of the template code there are a number of $[variable] entries, for example $[UserName]. You can probably guess that these custom parameters are similar to inline variables – you can find more information about these parameters via Help > PyScripter > Parameters help documentation.

4: Press the Update UpdateButton button to save your updated code back to the template.

5: Press the OK button

NewPythonFileDialog

When you press the New NewButton button your new module will have your updated template code.

Can I create multiple script templates?

Yes! You can choose from a list of code templates which you have created.

So “How do I create these other templates?

Creating a new script template is pretty easy. It is all achieved, once again, via the File Template dialog – the same dialog which you used a moment ago to change the default code template.

So:

1: From the Tools menu select Options > File Templates menu item.

This will display the File Templates dialog box.

2: Click on Python Script in the Name section to display the template code for a new Python module.

FileTemplateDlg

This is the code which is displayed when you create a new module via the New NewButton button or File menu > New > New Python Module menu item.

3: In the Name text box type in a new name for your new template, for example GPScriptTemplate.

Note that your name can not have any spaces within it.

4: Press the Add AddButton button

This will add your new template to the bottom of the template list.

5: Add your template code by amending what is already there or by writing your own code.

6: Once you have added your code press the Update UpdateButton button to save your changes to the template.

7: Press the Up UpButton button to move your template up the template list so all the all the Python scripts are grouped together.

8: Press the OK button to commit your changes and close the dialog.

 

To choose from a list of code templates you have created you should:

1: Go to the File menu choose New > New File. This will display the New File dialog box.

2: Under Categories, choose Python and you will see your named template, for example GPScriptTemplate

NewFileDlg

3: Press the Create button and your code contained in the template will be added to your newly created script module.

 

So, if you use PyScripter, this could be Heaven for Everyone – it’s worth giving them a go. Script templates can make writing your code just that little bit quicker, especially if you tend to write the same code over and over again.

Configuring PyScripter for writing Python scripts

PyScripter is a great Interactive Development Environment (IDE) for creating Python scripts as it provides an excellent environment for writing and debugging your code. This is the IDE which attendees to the “Introduction to Geoprocessing Scripts using Python” course use.

PyScripter

There are a couple of things which I get the class to do to configure PyScripter as these additional steps enhance the script writing experience within the course. This blog will lead you through these additional steps.

If you want to use PyScripter then you can access the source files from this location on GitHub, but if you do not have the time or the energy to compile the files then you can download the PyScripter installer from here. It is very easy to do and will take you no more than two minutes to complete. Links to the source files and the installer are provided at the end of this article.

Add Line Numbers

Line numbers should be made visible within your Python script if for no other reason than to aid the debugging process.

When a run-time error is generated a Traceback error is created. Amongst other things it lets you know which module your error occurred in, the type of error and the actual line of code which generated the error. Also included is the line number. I think you’ll agree it is easier searching for the line number when it is displayed than searching for a particular line of code.

To add line numbers to your module in PyScripter you should:

  • From the Tools menu choose Options > Editor Options
  • On the Display tab locate the Gutter section and tick the check box next to Show line number

 AddLineNos02

  • Press the OK button to add line numbers to the left hand side of your script.

   AddLineNos03

Remove the script border

if you use PyScripter then you may have noticed that on the right hand side of your script is a vertical line. This is used as a guide to suggest the maximum length for a line of code. This is easier said than done, especially when writing scripts which involve long workspace pathways or if you like adding comments at the end of a line of code.

Border

So how can it be removed?

In the example above I could remove the hard-coded pathway and use arcpy.GetParamerterAsText() to pass the workspace value into the script.

I could also remove the line by quickly changing an option in PyScripter.

To do this you should:

  • From the Tools menu choose Options > Editor Options
  • On the Display tab locate the Right Edge
  • Whatever the value is for Edge Column change it to “0”.

 Border01

  • Press the OK button and the border edge is no longer visible.

Initiating code completion for ArcPy

You may find that the intelligent code completion (AKA IntelliSense) for the arcpy site package is not always complete. In the example below notice that when arcpy.env is typed the overwriteOutput property is available, but very little else is presented in relation to the members of the env class.

 Intellisence01

It is fairly easy to ensure that you have full code completion for the arcpy site package, or for any third party / user defined module:

  • From the Tools menu, select Options > IDE Options
  • In the Code Completion region, of the IDE Options dialog, locate the Special packages
  • After the scipy entry add a comma and type arcpy as shown below:

Intellisence02

  • Press the OK button to dismiss the dialog box and now when you retype env. you will now see the full code completion:

Intellisence03

In a future post I will have a look at creating “script templates” which are handy if you find that you are writing the same “boiler-plate” code within your newly created scripts – but I will leave that Bijou for another time

Are there any other tips you have come across? If so then let everyone know about it by replying to blog and I will post your comments up!

Link to source files on GitHub:

https://github.com/pyscripter/pyscripter

Link to the PyScripter installer:

http://sourceforge.net/projects/pyscripter/

How do I specify optional arguments in a geoprocessing tool?

Here is another common question I get asked quite a bit within the “Introduction to geoprocessing scripts using Python” course:

When I am calling a geoprocessing tool from within arcpy how can I specify an optional argument?

So, when running a geoprocessing tool within your script you must supply all mandatory arguments, at the very least for the tool to run. You can also supply additional optional arguments if you need to. Traditionally you had to make sure that these optional arguments were supplied in a strict order as stipulated by the tool’s syntax.

Let’s have a look at the Make Feature Layer tool found in the Data Management Toolbox. This is a useful tool that allows you to create a feature layer which can then be used in further geoprocessing tools such as Select Layer By Location. It also has a few optional arguments which you can specify.

Looking at the help for the tool you can see that it is made up of mandatory arguments (arguments which must be specified at the very least for the tool to begin execution), and optional arguments, specified by curly brace brackets {}.

Syntax

MakeFeatureLayer_management (in_features, out_layer, {where_clause}, {workspace}, {field_info})

There are times when I would like to use the field_info optional parameter as I might wish to rename an attribute field found in my input layer to a new attribute field in my resulting feature layer. As you can see from looking at the tool’s syntax, the field_info parameter is the fifth argument in the tool’s parameter list and the third optional parameter.

So how do you make sure that the optional field_info parameter is passed into the tool at its correct position?

There are two main ways in achieving this:

1: Maintain order

The first way of specifying that I wish to use the field_info parameter is by creating placeholders for the previous unrequired optional arguments. The placeholders can be one of the following: an empty set of quotation marks (“”), a hash sign inside of a string (“#”) or by using the None type.

For example:


fldInfo = arcpy.FldInfo()
fldInfo.AddField("PC", "PostCode", "VISIBLE", "NONE")
arcpy.MakeFeatureLayer_management(inputFC,
                                  outputFL,
                                  "",
                                  "",
                                  fldInfo)

The disadvantages of this approach is that additional code is required to display these placeholders and in a tool which has many optional parameters (for example Topo to Raster in the Spatial Analyst toolbox) it is quite easy to lose one’s position within the argument list.

2: Use keyword arguments

A much neater way of overcoming the specifying of the placeholders is to only include the optional arguments you want by fully qualifying the optional argument with its parameter name, for example:

field_info = fldInfo

The syntax for the Make Feature Layer tool would now read:


arcpy.MakeFeatureLayer_management(inputFC,
                                  outputFL,
                                  field_info = fldInfo)

The advantages of this approach are that:

1: You are only including the optional arguments within the tool’s argument list

2: It is easier to see what arguments are included in the argument list

3: Your code is much more readable

Please note that this notation can only be used if you are an ArcGIS for Desktop 10.1 user, or above. If you are working with ArcGIS for Desktop 10.0 then you will have to use the notation in which you specify placeholders.

and there you have it!

** EDIT ** Many thanks to Jason Scheirer ( @jasonscheirer ) for pointing out a previous mistake in this post.

Working with distance (linear) units in a geoprocessing script

In this article I will answer another common question I am often asked in class:

“When, for example, I want to perform a buffer what linear units are available and where can I get a definitive list of them?

There are many geoprocessing tools which allow to specify some kind of distance unit as an input into the tool. These distance units are usually associated with the linear unit data type. Examples of these tools include Buffer, Generate Near Table, Grid Index Features, to name but a few. In fact the more you look the more you realise that this tool input is really quite common.

Using distance (linear) units in a geoprocessing tool is straight-forwards. Let’s use the trusty Buffer tool as an example:

arcpy.Buffer_analysis(“Roads”,
                      “RoadsBuff100”,
                      100)

In the above example we are buffering the Roads layer to a distance of 100. But is this 100 metres? 100 feet? Or 100 miles? It all depends upon the linear unit of the input dataset’s coordinate system. In the United Kingdom the coordinate system tends to be British National Grid whose unit of measurement is the metre so the Roads layer will be buffered to 100 metres.

A common question I get asked is:

“What about if I wanted to buffer the roads to 100 feet? How would I do that?”

In the Buffer tool dialog this pretty easy as I can select my units from a drop down list:

Unitsdropdown

How would I do this in my Python script?

It is relatively straight-forwards. All you have to do is make the buffer distance into a string and include the distance units of choice inside of the string:

arcpy.Buffer_analysis(“Roads”,
                      “RoadsBuff100”,
                      “100 feet”)

If your distance (linear) units are composed of more than one word, such as “Decimal Degrees” then the two words should be combined into one i.e. “DecimalDegrees

The follow-up question invariably is:

“How do I know what distance units are available to me?”

Well, you could look in the drop down list on the dialog for the tool concerned. Or you could look in the ArcGIS for Desktop Help and find the well-hidden help page which lists the linear units. The page is found at this link.

If you do not have an internet connection then you can find the relevant help document here:

Desktop >

Geoprocessing >

Tool errors and warnings >

Tool errors 1 – 10000 >

Tool errors and warnings: 801 – 900 >

000817 : Invalid linear unit type

In the next blog I will have a look another common question: “What is the best way to specify optional arguments for a geoprocessing tool?”

Ride the Wild Wind: Performing Spatial Analysis in ArcGIS Online

At the Esri UK user conference I did a hands-on training session which showed attendees how to create a map journal based story map describing the effect of wind turbines on the landscape of the Lake District. Attendees were able to create a story map in the space of 30 minutes – the finished story map can be found here:

http://arcg.is/1A4rzNG

The story map is based on the map journal template and shows some of the things which can be included within it, such as text, video, photographs, links to other websites and, of course, ArcGIS Online web maps and 3D Scenes.

The third section shows a web map which contains the results of a visibility analysis presented in 2D, with the ability to view the same data as a 3D web scene. Visibility2D

The visibility analysis was performed using ArcGIS Online – no desktop GIS software was used at all to perform the study, but due to time constraints it was not possible for attendees to create this dataset within the session.

So this article will show you how to create this missing piece of information which is an important part of the story map.

Getting ready….

To perform the analysis you will need to get access to an ArcGIS Online Organisational Account.

It is possible that you already have access to one so it is worth checking with your GIS co-ordinator. If you don’t then you can create your own 60 day trial account from this link. https://www.arcgis.com/home/signin.html

Follow the instructions; add your details and you will be off and running very quickly.

The links at the end of the article provide some good resources for activating your trial subscription.

The visibility analysis used a single point layer which represented the locations of operational wind turbines from 2011.

1: Open up a browser and go to sharegeo.ac.uk

2: Type “wind” into the search box and download the “Windfarms in Great Britain – 2011” dataset.

3: Zip up the operational windfarms shapefile. ArcGIS Online will then be able to read the zipped shapefile as a single entity and display it within a web map.

Now you will add the windfarms into a web map which will be used to perform the spatial analysis.

To add the shapefile into the web map you should:

4: Create a new web map

5: Load the zipped shapefile into the web map. Notice that when the data is added Smart mapping symbology is used. You should change the symbology field to MW. SmartMapping

6: You may wish to change the symbology of the layer from the default brown colour to some other colour, such as red.

Now that you have your organisational account and your source dataset you can start to perform some spatial analysis using ArcGIS Online.

Let’s do some spatial analysis in “The Cloud” using ArcGIS Online

Now you can perform the visibility analysis using the criteria which has been defined for you. Let’s make some assumptions about the wind turbines and the height of the observer… let us assume the turbines are 80 meters high and that the height of the observer is 1.5 meters. We will also assume that the maximum viewing distance is 30 km. Your analysis will be based around the nearest 15 points to the Lake District boundary.

1: Click on the drop down arrow to the right of your windfarm layer and choose Perform Analysis

2: Then choose Find Locations > Create Viewshed.

3: Fill in the variables in the Viewshed dialog:

  • We will say that all the turbines are 80m – this is a generalisation
  • We will calculate the viewshed for objects 1.5m above the surface to mimic the average person’s eye-level
  • We will set the max distance to the maximum (30km)

4: Make sure that the Use current map extent check box is ticked as this will limit the input wind farm features to those only in the map extent. Your dialog should look like the following: Visibility

This visibility analysis will cost 0.017 of a credit. A bargain!

5: Press the RUN ANALYSIS button to perform the visibility analysis.

It will take a few moments for the analysis to be performed. The resulting layer will be created using the name you specified in the designated location. The resulting visibility layer will be added to the web map: Visibility2D_Results

6: The final thing you need to do is save the web map! This means that any content you have in your web map will be available when the web map is referenced within the story map.

So, in this short blog entry you have seen how easy it is to perform some advanced spatial analysis which is normally the preserve of desktop GIS. The beauty of performing this analysis in “The Cloud” using ArcGIS Online means it is so easy to share your results to other people or organisations using intuitive and accessible client applications such as a story map.

Useful references:

http://www.esri.com/news/arcuser/1012/get-up-and-running-with-arcgis-online-for-organizations.html

https://doc.arcgis.com/en/arcgis-online/reference/activate-subscription.htm

How can I check to see if an attribute field exists using ArcPy?

Something I get asked in virtually every “Introduction to geoprocessing scripts using Python” course is:

Using Python and ArcPy, how do I know if an attribute field exists?”.

This is an important question to ask if you are data-loading into an existing dataset using an Update Cursor or writing to a brand new dataset using an Insert Cursor because you may need to make sure that, at the very least, a particular attribute field exists or you may need to make a new one to support some data loading of attribute data.

In the ArcPy site package there is no “FieldExists()” function.…. so how do you determine whether an attribute field exists in a dataset?

Possibly the best way is to use a combination of the arcpy.ListFields() function, and count the length of the resulting list.

When you inspect the documentation for the arcpy.ListFields() function, you will see that it returns a Python list of field objects. You will also see that it has an optional wildcard argument. This allows you to find all those attribute fields beginning with or containing the specified search string.

To see if an attribute field exists you should add the full name of the field as the wildcard. If the resulting list has exactly one item inside it then you know the field is present within your table or feature class. If the list is empty then the wild card has not found the attribute field in the dataset which means the field does not exist.

Let’s see how this works in a code sample. I want to find out if the ST_NAME field exists in the Address feature layer as presented below:

FieldExists

If the field does not exist then create a new field using the Add Field geoprocessing tool. If it does exist then great! The script can continue to execute.

# Check to see if the ST_NAME attribute field exists
import arcpy
arcpy.env.workspace = r”C:\Student\PYTH\Database\corvallis.gdb”
fields = arcpy.ListFields(“Address”, “ST_NAME”)
if len(fields) != 1:
    arcpy.AddField_management(“Address”, “ST_NAME”, “TEXT”)

# Perform rest of script workflow once field is present…

And there you have it…!