idfg-twilliams's blog

GIS - MORE, Better, Faster

We have implemented a system of distributed, synchronized GIS databases across the state.  This should substantially increase the speed with which you can access and navigate GIS data in ArcMap.

There is now a copy of our GIS layers stored on the U:\ drive in each regional office as well as for Nampa Research and Eagle Lab.  The database, U:\GIS_IDFG.gdb will be synchronized nightly with the main database at HQ.

We have also added a directory of Layerfiles to the U:\ drive.  The directory is located at U:\IDFG_Layers.   The Layerfiles reference the data in U:\GIS_IDFG.gdb and provide predefined colors, symbols, and labeling.  You may add them to your map document through ArcCatalog, the AddData button, or the LayerFetcher.

Some of the Layerfiles, such as the aerial imagery and topo base maps, point to data that is delivered via the internet.  The speed at which these layers draw will depend on the speed of the internet connection at your regional office.  You can navigate around your map more quickly if you keep these layers turned off until you need them. The internet base map layers are located in the WebBaseMaps subdirectory of IDFG_Layers.

Other subdirectories include Hunt, Fish, Conservation, and Reference.  Go in to ArcCatalog and take a look around in U:\IDFG_Layers and U:\GIS_IDFG.gdb to see what’s available.

This should be a significant improvement for IDFG GIS users but there will be some pain.  Many of the data connections in your existing Map Documents may be broken.  This is actually a good thing.  It will encourage you to connect to the newer, better, faster data.

If you find a broken connection in your map:

Right-click the layer in your map project.

Select Data --> Repair Data Source --> Navigate to U:\ GIS_IDFG.gdb and select the equivalent layer - OR - use the LayerFetcher to add the new layer and remove the old one.

See this post on repairing broken data sources for a visual step-by-step and more options for fixing broken connections.

Thank you for your patience during this transition.

ArcGIS Server, ArcGIS Online - Questions, Answers, and Resources

ArcGIS Server Services

  • Should we use fewer Services with more layers or more services with fewer layers?
    • From an application performance perspective.
    • From a server demand perspective.
      Is server demand currently an issue?
       
  • Should we use dedicated services with everything an application needs or general services that can be used by many applications or both?
     
  • How many services are too many and is that likely to be an actual problem for us in the foreseeable future?
     
  • Is there any advantage to separating the services that run our applications from services we are sharing?
     
  • What data are we, and should we be, sharing through services?
     
  • What can we do to increase the efficiency of our services?
     
  • What kind monitoring can we set up?
     
  • What testing can we do?
     
  • Should we move some of our data and services to the State server or ArcGIS Online?

Resources:
http://gis.stackexchange.com/questions/17486/lots-of-layers-in-one-or-multiple-services-and-why

http://www.techques.com/question/26-17030/How-many-layers-should-be-in-a-MapService---what-are-the-tradeoffs

http://resources.arcgis.com/content/enterprisegis/10.0/services_performance

 

ArcGIS Server Caching

  • How are our current caches set up?
     
  • What is being cached?
     
  • How will they be used?
     
  • What else do we want to cache?
     
  • For what layers do we want to build caches and for what do we cache dynamically?
     
  • Can we do a combination of predefined caching?
    i.e.  Build a cache to a certain scale then cache dynamically beyond that.  Or, cache the most likely layer combinations and dynamically cash the less likely combinations.
     
  • Does it make sense to cache small simple layers like IDFG Regions beyond the most general zoom levels?
     
  • How many and how large cached services can we support?

Resources:
http://help.arcgis.com/en/arcgisserver/10.0/help/arcgis_server_dotnet_he...
http://resources.arcgis.com/en/help/main/10.1/index.html#/Strategies_for...
 

ArcGIS Online

  • What do we have on there now?
     
  • Where do the data, services, application code actually reside?
     
  • What are we likely to use it for in the future?
     
  • Who will be the “Named Users”?
     
  • How many named users do we want to license?
     
  • Should we move some of our data and services to the State server or ArcGIS Online?
     
  • Do we want to upload data, create services, build caches, now while it’s free? Will they delete everything and make us start over.

Should I use ArcGIS from Citrix or Install it on my computer?

 You can use either or both. The best solution depends on three main factors.

·         What am I doing?

·         Where is my data?

·         Where am I?

 

What am I doing?

Appropriate for Citrix

·         making maps with existing data

·         querying and selecting  spatial data

·         creating GIS points from XY coordinates (GPS files)

·         Sub -setting data by selection and export

·         Creating or editing GIS data feature by feature

·         Buffering small data layers (few features, simple features)

·         Joins on attribute tables

Not Appropriate for Citrix

·         Geoprocessing (Intersect, Union, Identity, …)

·         Sub -setting data by Clipping

·         Spatial joins

·         Spatial Analyst

·         Buffering large or complex data layers

 

If you are doing one of the things on the Appropriate for Citrix list, Citrix may be your best option.  This will also depend on the location of your data.

When you are accessing ArcGIS through Citrix, you may be sharing the resources on a server with other people.   ArcMap shares resources very well with some operations, such as moving around on the map, making selections, and editing features.   Operations such as geoprocessing or buffering a large number of features are very CPU intensive.  When you start one of these operations, ArcMap assumes you are the most important person and this is the most important thing you are doing.  This can render ArcMap unusable to anyone else on that Citrix server.  Even if you are working with ArcMap loaded on your computer, don’t start one of these operations if you need to finish a Power Point for a meeting that starts in 30 minutes.  ArcMap can gobble up all the CPU resources on your computer.   Consider both the number of features and the complexity of the features. Points are simple.  A rectangle is simple.  A line or polygon with lots twists and turns is complex.  It may take less time to buffer 1000 points than one complex polygon.

 

Where is my data?

IDFG Spatial Data Server (LayerFetcher,  K:/ifwis/sdelayers) – This data source is available from both Citrix and desktop installations.  In the Regions, these layers will usually draw faster over Citrix, provided there are no other data source issues in your map document.

IDFG Network Drives (Q, K, T) – If you access ArcMap through Citrix, or Citrix and your desktop,  this is where you should store your data.  In the Regions, layers stored here will usually draw somewhat faster over Citrix, provided there are no other data source issues in your map document.  The T drive is for temporary storage and is purged every periodically.

 Citrix works by sending your mouse and keyboard actions to a server.  The actions are processed by ArcMap on the Server and an image is returned to your computer.  When you use data from IDFG Network Drives or from the LayerFetcher, the data and the processing  are in close proximity and the communication is fast.

Regional Network Drives (U, O, S)  - It possible to access a regional network drive through Citrix, but it is not advisable, because it will be very slow.  It will actually be pulling data back and forth over the network rather than just mouse clicks and images.  You should only store GIS data on a regional drive if you are only accessing it from a local desktop installation.  If you are at Boise HQ it is ok to use the O, S drives.

Your Local C: Drive – This will give you the fastest access to data if you are using a local desktop installation of ArcMap, however,  you will not be able to access it through Citrix, it is accessible only to you, and it is not automatically backed up.

 

Where am I?

In the office -  Here you have options.  You can use either or both depending the factors listed above.  If you have ever tried to use ArcMap, Excel, and Word at the same time, you have seen that everything slows down.  Using Citrix can be like having a second computer on your desk.

At another IDFG facility – As long as you are connected to the IDFG Network you will have access to the ArcGIS license server and the LayerFetcher data, so it possible to use the local ArcMap Installation on your laptop.  It is however, probably safer to put your map document and data on the Q drive and access through Citrix. If your project works on Citrix in your office it work anywhere you have a decent internet connection.

At home or anywhere not connected to the IDFG Network – Here the answer is easier. Citrix.  Your only other option is to purchase your own ArcGIS license, and you will not have access to any network data source.  You can access Citrix from anywhere with internet connectivity by going to https://linkup.idfg.idaho.gov/Citrix.

 

Other Considerations

Will you be sharing your map documents or data with others?  If the people you are sharing with are in your region and they are using a local install, then it’s ok to store data and map documents on your U drive.  Otherwise store on the Q drive and make sure your document works on Citrix.

 

How fast is your computer?  How full is your hard drive?  If your computer is old and slow you are better off using Citrix.  If your hard drive is near full you may not be able to install ArcMap.  You’ll need about 3 Gigs of space on your C drive to install.  Remember if you are doing any of the things on the “Not Appropriate for Citrix” list, you will need to install locally.

 

Problems with ArcMap on Citrix?

If you are unable to connect to ArcMap on Citrix or you get an error message when you try to open it, it may be a problem with the Network or Citrix itself.   Contact the Help Desk and let them know there’s a problem.  If it is running very slowly, it could be a network problem, but it is more likely to be a problem with a data source, data layer, or you map document.  The best way to diagnose is to start with a new blank map document and add your layers one by one.  If you are still having a problem, give me a call or send an email with the name and location of your map document and I will try to help to get to the bottom of the problem.

ArcGIS 10.2 Desktop Installation

Uninstall Previous Version

If you have any previous version of ArcGIS installed on your computer, you will need to uninstall before proceeding. [UPDATE 3/25/2015 - For 10.2.2 forward uninstall is not required]

-  From the start menu, select Control Panel.  Under Programs, select Uninstall program.

-  Select ArcGIS from the program list, then click Uninstall/Change.

-  Select Uninstall, then OK -> OK

 

                  Installing ArcGIS Desktop

Locate the the install program in

            U:\ESRI\ArcGIS10.2\Desktop.

1)  Open the Desktop folder and double click the setup application. 

2)  Accept the license agreement and click Next.

3)  Keep the default options and click Next until the installation begins.   The install  should take about 30 minutes.  Click Finish.

The ArcGIS Administrator Wizard window will come up automatically.

            -  Under  1. Select a Product, select Basic (ArcView) Concurrent Use.

 -  Under 2. Define a License Manager, select Define a License Manager Now.

             -  In the box below, type IFWISARC.

             -  Click OK -> OK.

You are done! For instrustructions on accessing the LayerFetcher Tools click here.  It takes two minutes.  This has not changed from 10.0.

Adding Excel .xlsx files to an ArcMap 10 document.

Don’t ask  me why but…  if you have office 2010 on your computer, you have to install the office 2007 system driver in order to see .xlsx files in ArcMap 10.

It’s a quick install.  You can find it here.

http://www.microsoft.com/download/en/details.aspx?displaylang=en&id=23734

Preventing Schema Locks in ArcSDE from ArcGIS Server 10.1.

By default, ArcGIS Server creates a schema lock on all layers and tables in a Map Service while the service is running.  This will prevent adding or deleting fields or records. 

At 10.1 this and a lot of other admin tasks can be done directly on the rest endpoint as described below.

By default, ArcGIS Server map services use schema locking to prevent other users from altering the geodatabase schema while the service is running. If the schema locks are impeding your workflow, you can disable them by using the ArcGIS Server Administrator Directory.

To disable schema locking for a service, perform the following steps:

  1. Navigate to http://<server name>:6080/arcgis/admin.
  2. Click services.
  3. Click the service for which you want to disable schema locking.
  4. Click edit.
  5. Add the "schemaLockingEnabled" = "false" property within the service JSON, as in the following example:

    …………………………………………………..

    "maxImageHeight": "2048",

    "schemaLockingEnabled": "false",

    "maxRecordCount": “1000",

    …………………………………………………..

  6. Click Save Edits.

This has been reblogged from

http://djmapper.tumblr.com/post/25068172061/prevent-arcgis-server-10-1-s...

 

NHD_LLID Tools for Creating Whole Stream LLID Routes Based on NHD

The NHD LLID Tools are an ArcMap 10 Add-in programmed in ArcObjects VB.net.  The purpose is to transfer the LLID whole stream route identifiers from our existing 1:100,000 whole stream route system to a whole stream route system based on the 1:24,000 NHD Hydrography, and to identify and resolve stream name conflicts between the two systems.  

The tool has three main components, organized into tabs on the ArcMap Add-in dockable window.  The first tab contains a tool to initiate an algorithm which assigns LLIDs to NHD ComID segments based on matching stream names.  The second tab provides a set of tools for assigning LLIDs to ComID segments for individual streams.  This tab also provides a tool for creating new LLID routes. The third tab provides a tool for identifying the location of stream name conflicts, as well as the nature of the conflict, how the stream was routed, why the stream was routed in this way, and what further action is required.  This portion of the application can be used without the LLID assignment functionality.

Click the following link to download the application, instructions,sample data, and sample map documents.  https://fishgame.idaho.gov/ifwis/download/NHD_LLID/NHD_LLID_Routing.zip

Unzip the LLID_RoutingTools file to C:\ .  This will create the directory C:\ NHD_LLID_Routing which includes the geodatabase NHD_LLID_Routing.mdb.  This file must be in this location for the program to function.  If you need to put the directory in another location contact tim.williams@idfg.idaho.gov

The NHD_LLID_Routing.zip contains the following:

- NHD LLID Routing Tools Instructions.doc. This document outlines how to use the tools to assign LLIDs to lines in the NHDFlowline layer and how to create a measured route system from these lines.

- The NHD LLID Routing Tools ArcMap 10 Add-in.

- A sample map document with no LLIDs assigned and the NHDFlowline layer unsymbolized. Start with this mxd if you would like to try out all the features of the NHD_LLID Tools. (NHDSampleUnrouted.mxd)

- A sample map document with LLIDs assigned. The State LLID route layer and the NHDFlowline layer are symbolized by LLID. The document also contains example IssuePoints highlighting discrepancies between the layers.  A completed NHD_LLID Route layer is also present. (NHDSampleRouted.mxd)

- A sample map document with the State LLID route layer and the NHDFlowline layer symbolized by Name and GNIS_Name. The document also contains example IssuePoints highlighting discrepancies between the layers.  (NHDSampleByName.mxd)

- The NHD_LLID_Routing.mdb geodatabase that supports the application.

- A sample NHD geodatabase where no LLIDs have been assigned (NHD17010304Unrouted).

- A sample NHD geodatabase where LLIDs have been assigned and a route system has been created (NHD17010304Routed).

ArcMap 10 Add-ins - Accessing the new LayerFetcher Tools

Add-ins are the new way to deploy and access ArcMap 10 custom applications and tools. There is a new and improved version of the LayerFetcher tools developed as an ArcMap 10 Add-in.   All you have to do to access the new LayerFetcher application is reference the network folder where it is located and turn on the toolbar.  To add the LayerFetcher , do the following:

1) Click Customize on the top menu bar in ArcMap 10 then click Add-In Manager.
2) Click click the Options tab then click the Add Folder button.
3) Navigate to Q:\IFWIS\AddIns and click OK. Then close Add-In Manager.
4) Click Customize then Tool Bars and check on the IDFG LayerFetcher Tools.

That’s it! No installation programs, no copying files.  And you only have to do the first three steps once.  Any other tools we place in this folder will automatically be available to you.  Just check on the tool bar. Any updates we do to existing Add-ins will be available without you doing anything.

If you find other Add-Ins that you would like to use, you can create your own AddIns folder in you folder on the Q:\ drive.  Just save the Add-in to that folder, and then follow the same steps listed above to reference it in the Add-ins Manager.
 

You can dock the LayerFetcher toolbar and LayerFetcher window anywhere on the main ArcMap window.