Post 3: Data Downloading, Interoperability, and Working with Projections

Objectives and Goals
With the advent of Esri and open-sourced GIS, data acquisition is now easier then ever. Many companies and organisations provide free and easy to access data. However, when data comes from different sources, is often managed and stored differently. The goal of this lab was to become familiarized with the process of downloading of different data from different internet sources, importing the data to ArcGIS, joining the data, projecting the data, and building a geodatabase to contain this data. In addition, the metadata of each downloaded piece of data would be analysed for data accuracy. Data accuracy is critical in the GIS world. Without a confirmed high level of accuracy, downloaded cannot often be trusted. Nowadays, standards exist for data accuracy. To quantify the accuracy of the downloaded data, it would be checked for several criteria: a listed scale, effective resolution, a minimum mapping unit, the planimetric coordinate accuracy, the lineage, the temporal accuracy, and the attribute accuracy when appropriate.
Methods'
First, six pieces of data were downloaded for Trempealeau County, Wisconsin, from six sources:

• NTAD railway data from the US Department of Transportation – Bureau of Transportation website 
• NLCD 2011 data from the USGS National Map Viewer
• Two DEMs of Trempealeau country (as the country is not completely covered by one) from the same USGS National Map Viewer
• A Cropland Data Layer from the USDA Geospatial Data Gateway
• A Trempealeau County file geodatabase from the Trempealeau County Land Records divisional website
• A SSURGO dataset from the USDA NRCS Web Soil Survey

After downloading the data, each one needed to be properly formatted before it could be analysed. All of the compressed files (.zip) were extracted to a folder created for this exercise. Microsoft Access was utilized to import the tabular data for the SSURGO data. The soils soilmu_a_wi121.shp was imported to the Trempealeau County geodatabase, renamed TMP.gdb, along with its component table. A relationship class was created between the component table and the shapefile, and they were joined in ArcMaop based on a relationship class. The NTAD rail-lines data was added to the TMP.gdb and clipped to fit Trempealeau Country. The DEMs were then mosaicked into a single raster. To finish off data preparation, a series of Python code (Post 2: Python Script 1) was written to project clip and load the DEM, NLCD, and NASS (cropland data) into the TMP.gdb. Once completed, this data was utilized to create a series of cartographically pleasing maps of the Trempealeau County (Figure 2).

While these maps are not the primary focus of this assignment, they do describe the spatial relationships and correlation btween elevation, landcover, different croplands, and the railways of Trempealeau Country.

Data Accuracy
Once the data had been properly downloaded and sorted, the metadata was checked for data accuracy in the previously mentioned criteria and sorted into a table for simple and effective analysis (Figure 1).

Figure 1: A table holding data accuracy measurements from the downloaded
data and retrieved from the data's metadata tables.

According to this tabulated data, the NTAD dataset is by far the least trustworthy due to the lack of many key components for accuracy analysis in the metadata. While the lineage and temporal accuracy can be conclusively determined, the scale is listed as 1:24,000 to 1:100,000, and there are no metadata entries for the effective resolution, minimum mapping unit, or planimetric coordinate accuracy. While attribute accuracy may not apply for this type of data, the lack of those three critical measurements makes the data untrustworthy. In contrast, the NASS dataset is by far the most complete in terms of metadata. Metadata entries exist for all of the desired criteria, and the Attribute Accuracy meets the industry standards of 85 % accuracy. However, temporally speaking, this data is quite old, being generated in 2003 and phased out by later data from the same source. In addition, the scale and resolution are not very detailed, at 1:100,000 and 30m, respectively, with a planimetric coordinate accuracy of 25 m. The NLCD dataset, while being temporally more accurate, suffers from an identical scale, resolution, and coordinate accuracy, in addition to not having its attribute accuracy assessed. These datasets can not be used in a detailed map. The most accurate datasets in terms of scale, the DEMs and NASS, are also flawed in other areas of their metadata.  The DEMs are missing their minimum mapping unit and planimetric coordinate accuracy, while the NRCS Web Soil Survey has not defined its effective resolution. The Trempealeau Country Geodatabase is a unique exception from the rest of the downloaded data. Its various accuracy measurements vary, as each of its feature classes have different accuracy measurements.
Conclusion
When looking at all the data, it is clear that everything has something wrong with it in terms of accuracy. Much of the metadata is incomplete, and while some may be accurate in some areas, they may be inaccurate in others. Anytime these datasets would be used, its important to keep in mind their limitations. A less spatially accurate dataset may be implemented in a large scale map, while a temporally inaccurate but spatially accurate dataset may be used if the data it shows is known to change relatively little over time.
Sources
Hupy, C. (2017). NL Exercise 5 – Part I: Data Downloading, Interoperability, and Working with Projections. Eau Claire, Wisconsin

Hupy, C. (2017). Exercise 5 – Part II: Data Downloading, Interoperability, and Working with Projections in Python. Eau Claire, Wisconsin.

Trempealeau County Land Records. Retrieved 3/13/2017, from http://www.tremplocounty.com/landrecords/

USDA Geospatial Data Gateway. Retrieved 3/13/2017, from https://datagateway.nrcs.usda.gov/

USDA NRCS Web Soil Survey. Retrieved 3/13/2017, from http://websoilsurvey.sc.egov.usda.gov/App/HomePage.htm

US Department of Transportation. In Bureau of Transportation. Retrieved 3/13/2017, from https://www.rita.dot.gov/bts/sites/rita.dot.gov.bts/files/publications/national_transportation_atlas_database/index.html

USGS National Map Viewer. Retrieved 3/13/2017, from http://nationalmap.gov/about.html

Post 2: Python Scripts

What is Python
Python is a free, versatile and easy to use programming language that now operates with ArcGIS and its related software. Within scripting software like PyScripter, python codes can be created to run a variety of tools. With python coding, it is possible to generate a lopping script to run multiple feature classes through a series of tools typically only accessible through ArcMap or ArcCatalog. By generating a looping script, it is possible to automate the processing of multiple feature classes. This is particularly handy when there are dozens or even hundreds of files which require processing. This even serves to prevent human error in processing. A properly constructed, automated script is far less likely to perform an error in processing over hundreds of files than a person completing the processing for each file individually. Because of this, Python scripting knowledge is a considerable asset for any working in the GIS world.
Python Script 1

The first script completed was designed to perform the initial processing on several rasters of Trempealeau County so they could be analysed (Post 3). In it, the list of rasters was retrieved from the work space folder and cued into a loop. After defining the output, each raster was put through a series of processing code. This code would reproject the rasters into the Trempealeau county projected coordinate system, taken from a county boundary shapefile in the TMP geodatabase, and then clipped to the boundaries of the same county. Afterwards, the completed outputs would be placed within the TMP geodatabase.

Pythpn Script 1

Python Script 2
This script was designed to select mines of certain criteria from a shapefile containing all of the Wisconsin frac sand mines, as surveyed by the DNR, and export mines meeting this criteria as its own shapefile. This selection of mines shapefile would be used to creating routing data for trucks carrying sand from the mines to available railway terminals (Post 5). Specifically, this script was designed to select all active, mine facilities lacking listed rail facilities. In addition, the script would select only the locations within the state of Wisconsin, as this was the study area, and would additionally remove all mines from the selection within 1.5 kilometers of a railway, as these locations likely have a private railway connection that is not displayed. Finally, it would save this layer selection as its own feature class and add it to the work-space geodatabase.

Python Script 2