 |
| Written By: Mike Miller |
"The short answer is no". But it will help you more than you can imagine. The GIS world doesn’t need everyone to be coders. Most GIS software is sufficiently complex that one could easily spend their entire careers working with one program and never use all of it, let alone have to improve on it. It would certainly be possible to specialize in a specific aspect of GIS or a specific type of analysis and never learn to write your own code. If the thought of programming sends tremors of fear down your spine, fear not. You will always be needed.
On the other hand, if you are the kind of person who likes to get under the hood and tinker around to see how things work. Or if you find yourself annoyed by doing the same mundane tasks repeatedly, then you most certainly should consider learning to code. It will greatly increase your value to your employer and save you yearsof frustration. Best of all, it’s FUN!!!! And your co-workers will love you when you tell them “I can automate that for you!”
A desktop GIS package such as ArcGIS or QGIS is a toolset for answering spatial questions. As powerful as they are there is no possible way that the developers can anticipate EVERY question and provide simple, out-of-the-box solutions. If you need a house, you could order a pre-fab house as a kit and put it together with little trouble. That’s fine if it meets your needs. But most builders will quickly encounter situations where adjustments have to be made. It may be due to the size, shape, or topography of the lot or to the requirements of the client. You could make a career of building pre-fab houses and turning down clients that want something beyond what you can do. However, somebody is going to learn how to meet those needs, and that someone will probably be in higher demand and be able to command a higher wage than you can with your pre-fab only business.
Why program?
Reasons for learning to write your own code fall into three broad categories. Granted there is some overlap between them.
- Analytical
- Automation
- User Interfaces
Analytical reasons involve using geospatial tools to solve problems that are either new, or sufficiently complicated that no out-of-the-box solution exists. My first foray into spatial programming came as an undergraduate student in wildlife biology. I worked on a project that was trying to monitor populations of medium-sized forest carnivores in Yellowstone NP using snow-tracking surveys. The field work was simple enough, you go ski around and record the number of tracks that you encounter. What was not so easy, was to understand how large of a change in track counts was required to suggest a real change in actual population that we should be concerned about, or even what the probability was of detecting rare animals, such as wolverines and lynx, given our level of effort.
I decided to create some computer simulations to answer those questions. This involved creating random animal track patterns and a sampling scheme, and then calculating the number of intersections. I was in the middle of a GIS minor at the time, but a GIS computer at the time was very expensive and were not available for such frivolous un-funded undergrad questions, so I wrote the simulations in PASCAL at first. Later when ArcView 3.0 was released and I saw the power of its Avenue programming language I re-wrote my simulations in a real GIS and was able to take advantage of its tools for spatial analysis and display. This greatly simplified my project but it still couldn’t be done without code. It was great to have a bigger toolset, but I had to put the pieces together myself.
Later this line of questioning lead into a PhD program focusing on statistical methods for analyzing animal movement patterns as a linear path, rather than a series of unconnected points. Esri had just released ArcGIS 8.0 and it required learning Visual Basic and ArcObjects, which was a much more powerful set of tools than Avenue. But it was still only tools. Nobody had even asked the question at that time, let alone solved it. This project introduced the element of time as a 3rd coordinate for each vertex, which I was able to do by storing time in the z coordinate. Of course, this also required writing all my own code to do my own analysis.
Throughout this time, I worked with other researchers, many of them brilliant scientists, who had great ideas about research questions in the field of landscape ecology. None of them however, knew how to program in a GIS environment so they could not actually implement those ideas without help. Needless to say, I was very busy for a few years.
Automation
Nobody likes repetitive mundane tasks. Humans evolved brains to solve novel problems and most of us quickly get bored when doing the same thing over and over again. It’s difficult to focus on these tasks and the inability to focus inevitably leads to mistakes. I’ve had employers tell me that it would be cheaper to hire someone at minimum wage to do the boring repetitive stuff then it would be to pay me to automate a process for them. Most of the time their math was wrong. Even when it appeared correct on the surface they failed to understandhow much people dis-liked doing those jobs
how difficult it is to enter data into a spreadsheet for 8 hours a day
how costly it could be when mistakes were inevitably made
All those difficulties are magnified when you are unwilling to pay employees a reasonable wage. The costs of hiring and training someone new every few months when those employees inevitably find better offers can also be substantial.
The bottom line is that if it is fairly simple to automate a process, you should automate it. Skilled employees are far too valuable to risk having one leave because they were bored at their job. One of my first jobs in the environmental consulting industry was for a company working near a major natural gas field. Every new well required a biological assessment, which was basically a check-list. How far to the nearest road? What was the underlying geology? How many existing wells were in a 1 mile radius? Was the project in endangered species habitat? etc. Each one took a couple of hours of research using our GIS data. I told my new boss that I could automate most of that and simply output a report listing all the information that could be determined by GIS. It would not eliminate the need for a field visit but it would certainly eliminate a lot of mundane office work. I was shocked to be told by my boss that it would mean that we couldn’t bill as many hours and he would make less money.
A year later one of the state agencies changed their regulations for disposal of fracking fluid. They required a similar set of information and refused to grandfather in wells that had been permitted prior to the rule change. One of our clients had 7,000 wells permitted that were subject to the new rule and they were freaking out about the potential delays in productivity. My boss asked me if I could help so I worked with the state agency to come up with language and data that would be acceptable to them and I automated the process to run off of a spreadsheet that included just a well name and latitude/longitude coordinates for each well. It took me 2 days to write the code and then it took 3 days to run it for all 7,000 wells but it saved the company almost a year of backlog. My boss looked like a hero.
In another job, I faced a similar situation and a similar boss who didn’t think he needed a “programmer”. Just someone to make maps. This project also involved natural gas wells but it wasn’t on public land. There was, however, a lot of local opposition from private landholders and the company was doing its best to show that they were conducting environmental review for each well and meeting government guidelines even though, legally, they weren’t required in many cases. I told my boss that I could automate that process as well and got a similar response. It wasn’t until some of our field employees began making errors that made our client look bad and we were in danger of losing the client that my boss told me to go ahead and automate the GIS review part.
I was involved in another project involving a proposed mine. Part of this effort was to evaluate the changes in fish habitat resulting from modifications of the flow regime in nearby rivers due to mining operations. Another company was determining what those changes would be. When we finally received those numbers and I worked through evaluating the impacts under one scenario, I realized that I could automate the calculations for any scenario given a set of flow data. My boss told me not to bother because these were “THE final numbers” and wouldn’t change. I ignored him because I would rather write code than perform mundane spreadsheet calculations and I don’t believe in “final” numbers. Two weeks later, he told me that there had been a mistake in the original flow numbers we received and asked me how long it would take to re-do the analysis. It had taken me 2 weeks to do originally and I told him 30 minutes to re-do it. Later on, the numbers changed again several times as the project went through the NEPA process and my boss was very happy that I could run a new set of numbers quickly.
User Interfaces
Most people are not GIS experts, but almost everyone can navigate a map on a computer or phone and they can enter data through a computer form. Programming also allows you to develop customized interfaces to your data. There are several reasons why you might want to do this.
- For in-house staff who are not knowledgeable in GIS but need to be able to find things on a map.
- To ensure that data is carefully validated prior to entry in order to minimize mistakes and decrease data entry time.
- To provide secure access to your clients of the data that is being collected for them.
- To allow users of mobile devices to view and edit your data
In many GIS applications, data is stored on a central server and it can be accessed by many different clients. Clients can be a fully functional desktop GIS software such as ArcMap or QGIS, they can also be web pages, mobile applications, and/or custom interfaces written in Python, Visual Basic, or some other programming language. An interface can also be a custom plug-in to an existing desktop GIS program.
A custom interface for in-house staff and/or clients can make your data available for viewing and other specific tasks to those without special GIS knowledge and without consuming an expensive desktop GIS license. Project managers should be able to search for data, filter it by attributes, and perform simple, canned analysis. Field personnel should be able to add data as well, and that data should be thoroughly validated to reduce errors and might possibly update other feature classes or perform other tasks.
One of the main environmental constraints on one of the natural gas projects I discussed above was raptor nests. When we were given a new project our field personnel would survey the area for raptor nests. This was a dynamic project and we would conduct weekly surveys throughout the raptor nesting season. I developed an interface for adding raptor nests that only required the field personnel to click on the map at the nest location and choose the species and nest status from some drop down lists. The interface would then add a point nest as well as two different size buffers. The diameter of one of the buffers was dependent on the species. It would automatically include information about the employee who found the nest, the date, etc. This reduced the time involved in adding a nest from 5-10 minutes to 5-10 seconds and greatly reduced the number of errors.
This was a large and dynamic project and often a nest found while surveying one project would impact other projects around it, so the project manager was automatically notified via e-mail when a new nest was found or when the status of an existing nest changed so that they could immediately notify the client.
Another recent job involved developing wetlands data for a large area from aerial interpretation. I divided the area up into over 60,000 sectors, each of which represented the area of imagery visible on one computer screen at 1:6000 scale. In order to ensure that each and every part of the area had been viewed I developed an interface that would automatically move from one sector to the next after it had been finished being searched for wetlands. Once marked as complete, the data was updated so that it was not revisited again. The interpreters also had an option of marking it as “uncertain” and enter the question they had about their interpretation. Project managers could then go through the questionable interpretations and see what issues the interpreters were having and make decisions about how to handle them. Project managers could also immediately see how much of the area had been completed and by whom. Each of these wetlands also received a cowardin classification. The cowardin system is a hierarchical coding system where the available codes at each level are dependent on what was coded at the previous level. For examples, rivers got a different set of options than lakes. In order to reduce errors, the interface built the code based on a set of drop-down menus and each menu only included valid options based on the previous selections. This system eliminated a large amount of errors and time in the QA/QC process.
What language should you learn?
If you’ve decided that coding is for you and you want to learn how, the next step is deciding which language you should learn. If you had asked me 5 years ago I would have recommended Visual Basic so that you could develop add-ins for ArcGIS. Although ESRI had released some API’s to their data and geospatial libraries through Python, I found them limiting. ArcGIS was built from ArcObjects and the interfaces available to ArcObjects through visual basic essentially would have allowed you to recreate the entirety of ArcGIS and thus you were virtually unlimited in what you could do. In particular, Visual Basic and ArcObjects allowed you to create your own data entry forms which was something not available in Python, other than through the somewhat clunky geoprocessing tool interface.Python
Several things have changed since then however. ESRI had appeared to be moving away from ArcObjects and towards its ArcPy Python API for some time but ArcGIS was still built from ArcObjects. The release of ArcGIS Pro took that move a step further. ArcGIS Pro only had support for Python and appears to have abandoned ArcObjects altogether. In addition over that time I have discovered the many advantages of QGIS and began using it in many projects. QGIS also allows Python plug-ins with no support for VisualBasic.Python has become a mature, open-source programming language that can be used for many purposes. This includes server-side programming for internet applications through the Django and GeoDjango frameworks. There are also vast numbers of third-party libraries for numerical and geospatial calculations available.
Web-based languages
Another thing that has changed is the movement of the GIS world in general towards web-based and mobile interfaces to GIS data. If you need to have multi-user access to your data through the internet you will almost certainly be viewing that data through a web browser. You will almost certainly need to be able to write code using standard web technologies including HTML, CSS, and JavaScript. If you want your users to be able to edit your data from a web interface, you will also need to know some SQL to interact with the database and a server side language.Today, my recommendation would be… it depends. I would probably not recommend VisualBasic or ArcObjects anymore to anyone. If you are working in research and doing high-powered number crunching of your spatial data, I would recommend Python. Python will allow you to take full advantage your processors 64-bit, multi-threading capabilities and even allow you to cluster computers together if you need massive computing power. If you understand the basics of Python you can write plug-ins for ArcGIS, ArcGIS Pro, and QGIS. You can also write web applications using the Django framework although you will still need to know HTML, CSS, and JavaScript for the front-end.
If you are mainly interested in developing interfaces to your GIS data for remote users via the internet, I would focus on JavaScript. There are several excellent open-source JavaScript mapping API’s available such as Leaflet and OpenLayers. The Google maps API is not open-source but it has several advantages as well. If you are going to be dealing with the ESRI ecosystem, their JavaScript API also provides access to data stored in ESRI compatible formats. The Turf.js library and several others give JavaScript the ability to perform spatial analysis in the browser.
If you need remote editing capability you’ll need a database server and a server-side programming language. PHP is the most popular server-side language. It is widely available and you’ll find plenty of resources available to learn it. You won’t be limited by PHP, although other languages may have advantages. If you want to stay in the Microsoft world you can use one of Microsoft .NET languages on the server. You can write server-side code using Python through the Django frameworks and thus you would have all of Python’s number crunching capability at your disposal. You could also write server-side code using Ruby, or Java, or even JavaScript through Node.js.
PostGIS is the most common spatially enabled database in the open source world. ESRI’s ArcSDE extension works with several common commercial and open-source databases. Most databases have some level of spatial capability although not at the same level as PostGIS or ArcSDE. If your needs are simple, you can even store spatial data in a text field as GeoJSON. NoSQL or document databases such as MongoDB and CouchDB can store GeoJSON natively.
Writer's Recomendation
There is no right answer for 100% of cases however. I’ve written code for analysis in JavaScript because I wanted to make it available to many people and the analysis was not terribly computation intensive. I’ve not written any user interfaces in Python, but you could. If I were starting from scratch today, however, I would start with the web-based technologies of JavaScript and PostGIS. Unless you are really doing intensive number-crunching, this technology stack will give you the most bang for your buck and provide the most benefit to your employer. If you need more capability than can be provided over the internet, you can also write a python plug-in for QGIS that will access the same data. And because QGIS is free anyone can download the software, install your plug-in, and even connect to your PostGIS data if necessary.An additional benefit of writing plug-ins in QGIS is that if you write plug-ins for ESRI products, you will almost inevitably find that your code won’t run in some future release. When this happens, you will need to upgrade your own software to upgrade your plug-in and when you do that you will need a seperate computer with an older version of ArcGIS just to support clients who haven’t upgraded yet and you might run into licensing limitations etc. And clients rarely want to pay you to upgrade the software you wrote for them 3 years ago. It becomes a massive headache as anyone who has written ESRI plug-ins can tell you. If you write QGIS plug-ins all of those problems go away simply because you are allowed to have multiple versions of QGIS running on the same machine.
How to get started.
Whichever language you choose, the first step is to learn the basics of the language. There are many, many, resources available for learning both Python and JavaScript. Books, videos, on-line tutorials, university classes, and traditional commercial courses.Once you have the basics, the next step is to decide which geospatial libraries you will need and/or use. If you choose the Python route, this means you will need to choose between ArcPy, pyQGIS, or GeoDjango (if you want to use Python for web access). If you choose the web-based JavaScript route you will need to choose which mapping API you will use (Leaflet, OpenLayers, Google Maps, or ESRI), whether you will need to use Turf.js for spatial analysis, and whether your application will require a database server. If you require a database server you will need to choose which database and spatial extension you will use, as well as which server-side programming language you will use. To keep things simple, I would recommend learning the basics with PHP and PostGIS before moving onto other technologies.
Reference:
https://millermountain.com/geospatialblog/2017/05/15/coding-for-gis-professionals/#more-80
0 comments:
Post a Comment