Raster Management & Optimization ¶

Author: Jeff McKenna
Contact: jmckenna at gatewaygeomatics.com
Original Author: HostGIS
Last Updated: 2022-01-25

Table of Contents

Raster Management & Optimization

Choose the right raster format for your needs ¶

The best/optimal data sources for MapServer in terms of speed-of-display is GeoTIFF for rasters.

Here are some other points to consider:

The GeoTIFF image format is the fastest to “decipher”, but once you get beyond a certain point, the disk reading may become slow enough to make it worthwhile to consider other image formats.
For GeoTIFFs larger than 1 GB, ECW images tend to render faster than GeoTIFFs, since decompressing the data (CPU and RAM) is faster than reading the uncompressed data (disk). The downside is that ECW is not open-source, and the licensing is often prohibitive. MrSID is another compressed image format that can be used.
JPEG2000 is a very slow image format, as is JPEG.

Spend time to review GDAL’s associated driver page for your chosen format ¶

This is a critical step, as MapServer relies on GDAL for raster data access. Each driver (GDAL format) has its own set of abilities and switches. Find your raster format and review its options here.

Connect to your data through GDAL ¶

For data management in MapServer, this should always be one of your first steps. Sometimes desktop GIS programs will display a format in a certain way but your data might not display in MapServer; checking how GDAL reads your raster file, will help you manage and understand the data. Here is an example using gdalinfo to read and return various important metadata regarding a specific raster, in this case a MrSID raster:

gdalinfo ortho_1-1_hc_s_mn163_2019_1.sid

      Driver: MrSID/Multi-resolution Seamless Image Database (MrSID)
      Files: ortho_1-1_hc_s_mn163_2019_1.sid
             ortho_1-1_hc_s_mn163_2019_1.sid.aux.xml
      Size is 50249, 116781
      Coordinate System is:
      PROJCS["NAD83 / UTM zone 15N",
          GEOGCS["NAD83",
              DATUM["North_American_Datum_1983",
                  SPHEROID["GRS 1980",6378137,298.257222101,
                      AUTHORITY["EPSG","7019"]],
                  TOWGS84[0,0,0,0,0,0,0],
                  AUTHORITY["EPSG","6269"]],
              PRIMEM["Greenwich",0,
                  AUTHORITY["EPSG","8901"]],
              UNIT["degree",0.0174532925199433,
                  AUTHORITY["EPSG","9122"]],
              AUTHORITY["EPSG","4269"]],
          PROJECTION["Transverse_Mercator"],
          PARAMETER["latitude_of_origin",0],
          PARAMETER["central_meridian",-93],
          PARAMETER["scale_factor",0.9996],
          PARAMETER["false_easting",500000],
          PARAMETER["false_northing",0],
          UNIT["metre",1,
              AUTHORITY["EPSG","9001"]],
          AXIS["Easting",EAST],
          AXIS["Northing",NORTH],
          AUTHORITY["EPSG","26915"]]
      Origin = (494792.400000000023283,5018068.799999999813735)
      Pixel Size = (0.600000000000000,-0.600000000000000)
      Metadata:
        GEOTIFF_CHAR__GeogAngularUnitsGeoKey=Angular_Degree
        GEOTIFF_CHAR__GTModelTypeGeoKey=ModelTypeProjected
        GEOTIFF_CHAR__GTRasterTypeGeoKey=RasterPixelIsArea
        GEOTIFF_CHAR__ProjectedCSTypeGeoKey=PCS_NAD83_UTM_zone_15N
        GEOTIFF_CHAR__ProjLinearUnitsGeoKey=Linear_Meter
        GEOTIFF_NUM__1024__GTModelTypeGeoKey=1
        GEOTIFF_NUM__1025__GTRasterTypeGeoKey=1
        GEOTIFF_NUM__1026__GTCitationGeoKey=NAD83 / UTM zone 15N
        GEOTIFF_NUM__2049__GeogCitationGeoKey=NAD83
        GEOTIFF_NUM__2054__GeogAngularUnitsGeoKey=9102
        GEOTIFF_NUM__2062__GeogTOWGS84GeoKey=0.000000,0.000000,0.000000
        GEOTIFF_NUM__3072__ProjectedCSTypeGeoKey=26915
        GEOTIFF_NUM__3076__ProjLinearUnitsGeoKey=9001
        IMAGE__ENCODING_APPLICATION=GeoExpress 10.0.1.5035
        IMAGE__FORMAT=MrSID/MG3
        IMAGE__INPUT_FILE_SIZE=30104685840.000000
        IMAGE__INPUT_FORMAT=GeoTIFF
        IMAGE__LTI_ESDK_VERSION=9.5.4.5035.Bob_5011_br win64-vc15/Release_md

        IMAGE__MODIFICATIONS=COMPRESSED CROPPED EMBEDDED REORDERED-BANDS MOSAICKED REPROJECTED
        IMAGE__PRINT_DENSITY_UNIT=in
        IMAGE__PRINT_X_DENSITY=200.000000
        IMAGE__PRINT_Y_DENSITY=200.000000
        VERSION=MG3
      Image Structure Metadata:
        INTERLEAVE=PIXEL
      Corner Coordinates:
      Upper Left  (  494792.400, 5018068.800) ( 93d 3'59.18"W, 45d18'57.98"N)
      Lower Left  (  494792.400, 4948000.200) ( 93d 3'56.57"W, 44d41' 7.27"N)
      Upper Right (  524941.800, 5018068.800) ( 92d40'54.44"W, 45d18'56.45"N)
      Lower Right (  524941.800, 4948000.200) ( 92d41' 6.94"W, 44d41' 5.78"N)
      Center      (  509867.100, 4983034.500) ( 92d52'29.30"W, 45d 0' 2.48"N)
      Band 1 Block=1024x128 Type=Byte, ColorInterp=Red
        Min=0.000 Max=228.000
        Minimum=0.000, Maximum=228.000, Mean=86.853, StdDev=53.485
        Overviews: 25125x58391, 12563x29196, 6282x14598, 3141x7299, 1571x3650, 786x1825, 393x913, 197x457, 99x229, 50x115, 25x58, 13x29, 7x15, 4x8
        Metadata:
          STATISTICS_APPROXIMATE=YES
          STATISTICS_MAXIMUM=228
          STATISTICS_MEAN=86.853391304348
          STATISTICS_MINIMUM=0
          STATISTICS_STDDEV=53.485489960597
          STATISTICS_VALID_PERCENT=100
      Band 2 Block=1024x128 Type=Byte, ColorInterp=Green
        Min=0.000 Max=224.000
        Minimum=0.000, Maximum=224.000, Mean=81.011, StdDev=45.552
        Overviews: 25125x58391, 12563x29196, 6282x14598, 3141x7299, 1571x3650, 786x1825, 393x913, 197x457, 99x229, 50x115, 25x58, 13x29, 7x15, 4x8
        Metadata:
          STATISTICS_APPROXIMATE=YES
          STATISTICS_MAXIMUM=224
          STATISTICS_MEAN=81.010782608696
          STATISTICS_MINIMUM=0
          STATISTICS_STDDEV=45.552372785005
          STATISTICS_VALID_PERCENT=100
      Band 3 Block=1024x128 Type=Byte, ColorInterp=Blue
        Min=0.000 Max=211.000
        Minimum=0.000, Maximum=211.000, Mean=90.109, StdDev=48.708
        Overviews: 25125x58391, 12563x29196, 6282x14598, 3141x7299, 1571x3650, 786x1825, 393x913, 197x457, 99x229, 50x115, 25x58, 13x29, 7x15, 4x8
        Metadata:
          STATISTICS_APPROXIMATE=YES
          STATISTICS_MAXIMUM=211
          STATISTICS_MEAN=90.109217391304
          STATISTICS_MINIMUM=0
          STATISTICS_STDDEV=48.70795205406
          STATISTICS_VALID_PERCENT=100

Note

You can use the extent values returned from gdalinfo (using Lower Left and Upper Right values) to paste into your mapfile’s EXTENT parameter. You can also notice in that summary the PROJCS/AUTHORITY line, which states that this data is currently in the EPSG:26915 projection.

Note

For Windows users, MS4W includes the gdalinfo utility, and all utilities mentioned here.

Overviews ¶

GeoTIFF supports the creation of “overviews” within the file, which is basically a downsampled version of the raster data suitable for use at lower resolutions. Use the gdaladdo program to add overviews to a GeoTIFF, and MapServer (via GDAL) will automagically choose which downsampled layer to use. Note that overviews significantly increase the disk space required by a GeoTIFF, and in some cases the extra disk reading may offset the performance gained by MapServer not having to resample the image. You’ll just have to try it for yourself and see how it works.

Tileindexes and Internal Tiling ¶

Tiling is mostly effective for cases where one commonly requests only a very small area of the image.

A tileindex is how one creates an on-the-fly mosaic from many rasters. This is described in the Tile Indexes MapServer document. That document describes common cases where a tileindex makes sense. In particular, if you have a very large raster and most requests are for a very small spatial area within it, you may want to consider slicing it and tileindexing it.

As an alternative to slicing and mosaicing, TIFFs do support a concept of internal tiling. Like a tileindex, this is mostly effective when the requests are for a small portion of the raster. Internal tiling is done by adding “-co TILED=YES” to gdal_translate, e.g.:

gdal_translate -co TILED=YES original.tif tiled.tif

Learn & Review the various GDAL utilities to manage your rasters ¶

GDAL raster commandline utilities are very powerful, and the more processing that you can do offline, the faster and easier it will be for MapServer to display your raster. For example, you might need a colorful ‘shaded relief’ from a raw DEM raster, in which case you could use the gdaldem utility, and then point your MapServer layer to that shaded relief raster. Or you might want to reproject all of your rasters to the desired output/display projection beforehand, by using the gdalwarp utility. Review all of the available GDAL raster utilities here.

Handling your raster LAYERS in the mapfile ¶

Please review the notes in the document Mapfile Tuning & Management.

Remote WMS ¶

Some remote WMS/OGC services (as WMS actually returns an image from a request sent to a remote server) can be slow or unreliable. Optionally you can use a monitoring service, such as GeoHealthCheck, to keep track of the service’s reliability for you.

You can also consider when the remote WMS layer or remote service should be used. For example, there may be a different WMS server (or a different set of imagery, or even vector outline maps) suitable for drawing the countries or states to orient the user. You could then have the WMS layer come on at a certain scale, or have the layer always available but turned off so the user can choose when to turn it on.

DOCS PREVIEW