go-iiif

What is this?

This is a fork of @greut's iiif package that moves all of the processing logic for the IIIF Image API in to discrete Go packages and defines source, derivative and graphics details in a JSON config file. There is an additional caching layer for both source images and derivatives.

I did this to better understand the architecture behind (and to address my own concerns about) version 2 of the IIIF Image API.

For the time being this package will probably not support the other IIIF Metadata or Publication APIs. Honestly, as of this writing it may still be lacking some parts of Image API but it's a start and it does all the basics.

And by "forked" I mean that @greut and I decided that it was best for this code and his code to wave at each other across the divide but not necessarily to hold hands.

Setup

Currently all the image processing is handled by the bimg Go package which requires the libvips C library be installed. There is a detailed setup script available for Ubuntu. Eventually there will be pure-Go alternatives for wrangling images. Otherwise all other depedencies are included with this repository in the vendor directory.

Once you have things likeGo and libvips installed just type:

$> make bin

Usage

go-iiif was designed to expose all of its functionality outside of the included tools although that hasn't been documented yet. The source code for the iiif-tile-seed or the iiif-transform tools is a good place to start poking around if you're curious.

Tools

iiif-server

$> bin/iiif-server -config config.json
2016/09/01 15:45:07 Serving 127.0.0.1:8080 with pid 12075

curl -s localhost:8080/184512_5f7f47e5b3c66207_x.jpg/full/full/0/default.jpg
curl -s localhost:8080/184512_5f7f47e5b3c66207_x.jpg/125,15,200,200/full/0/default.jpg
curl -s localhost:8080/184512_5f7f47e5b3c66207_x.jpg/pct:41.6,7.5,40,70/full/0/default.jpg
curl -s localhost:8080/184512_5f7f47e5b3c66207_x.jpg/full/full/270/default.png

iiif-server is a HTTP server that supports version 2.1 of the IIIF Image API.

Endpoints

Although the identifier parameter ({ID}) in the examples below suggests that is is only string characters up to and until a / character, it can in fact contain multiple / separated strings. For example, either of these two URLs is valid

http://localhost:8082/191733_5755a1309e4d66a7_k.jpg/info.json
http://localhost:8082/191/733/191733_5755a1309e4d66a7/info.json

Where the identified will be interpreted as 191733_5755a1309e4d66a7_k.jpg and 191/733/191733_5755a1309e4d66a7 respectively. Identifiers containing one or more ../ strings will be made to feel bad about themselves.

GET /{ID}/info.json

$> curl -s http://localhost:8082/184512_5f7f47e5b3c66207_x.jpg/info.json | python -mjson.tool
{
    "@context": "http://iiif.io/api/image/2/context.json",
    "@id": "http://localhost:8082/184512_5f7f47e5b3c66207_x.jpg",
    "@type": "iiif:Image",
    "height": 4096,
    "profile": [
        "http://iiif.io/api/image/2/level2.json",
        {
            "formats": [
                "tif",
                "webp",
                "jpg",
                "png"
            ],
            "qualities": [
                "default",
		"dither",
                "color"
            ],
            "supports": [
                "full",
                "regionByPx",
                "regionByPct",
                "sizeByWh",
                "full",
                "max",
                "sizeByW",
                "sizeByH",
                "sizeByPct",
                "sizeByConfinedWh",
                "none",
                "rotationBy90s",
                "mirroring",
                "baseUriRedirect",
                "cors",
                "jsonldMediaType"
            ]
        }
    ],
    "protocol": "http://iiif.io/api/image",
    "width": 3897
}

Return the profile description for an identifier.

GET /{ID}/{REGION}/{SIZE}/{ROTATION}/{QUALITY}.{FORMAT}

$> curl -s http://localhost:8082/184512_5f7f47e5b3c66207_x.jpg/pct:41,7,40,70/,250/0/default.jpg

Return an image derived from an identifier and one or more IIIF parameters. For example:

GET /debug/vars

$> curl -s 127.0.0.1:8080/debug/vars | python -mjson.tool | grep Cache
    "CacheHit": 4,
    "CacheMiss": 16,
    "CacheSet": 16,

$> curl -s 127.0.0.1:8080/debug/vars | python -mjson.tool | grep Transforms
    "TransformsAvgTimeMS": 1833.875,
    "TransformsCount": 16,

This exposes all the usual Go expvar debugging output along with the following additional properies:

CacheHit - the total number of (derivative) images successfully returned from cache
CacheMiss - the total number of (derivative) images not found in the cache
CacheSet - the total number of (derivative) images added to the cache
TransformsAvgTimeMS - the average amount of time in milliseconds to transforms a source image in to a derivative
TransformsCount - the total number of source images transformed in to a derivative

Note: This endpoint is only available from the machine the server is running on.

iiif-tile-seed

$> ./bin/iiif-tile-seed -options ID1 ID2 ID3...

Usage of ./bin/iiif-tile-seed:
  -config string
    	Path to a valid go-iiif config file
  -endpoint string
    	The endpoint (scheme, host and optionally port) that will serving these tiles, used for generating an 'info.json' for each source image (default "http://localhost:8080")
  -format string
    	A valid IIIF format parameter (default "jpg")
  -logfile string
    	Write logging information to this file
  -loglevel string
    	The amount of logging information to include, valid options are: debug, info, status, warning, error, fatal (default "info")
  -mode string
    	Whether to read input as a CSV file or from STDIN which can be represented as "-" (default "-")
-noextension
        Remove any extension from destination folder name.
  -processes int
    	The number of concurrent processes to use when tiling images (default 2)
  -quality string
    	A valid IIIF quality parameter - if "default" then the code will try to determine which format you've set as the default (default "default")
  -refresh
    	Refresh a tile even if already exists (default false)
  -scale-factors string
    	A comma-separated list of scale factors to seed tiles with (default "4")
  -verbose
    	Write logging to STDOUT in addition to any other log targets that may have been defined

Generate (seed) all the tiled derivatives for a source image for use with the Leaflet-IIIF plugin.

iiif-tile-seed and identifiers

Identifiers for source images can be passed to iiif-tiles-seed in of two way:

A space-separated list of identifiers
A space-separated list of comma-separated identifiers indicating the identifier for the source image followed by the identifier for the newly generated tiles

For example:

$> ./bin/iiif-tile-seed -options 191733_5755a1309e4d66a7_k.jpg

Or:

$> ./bin/iiif-tile-seed -options 191733_5755a1309e4d66a7_k.jpg,191/733/191733_5755a1309e4d66a7

In many cases the first option will suffice but sometimes you might need to create new identifiers or structure existing identifiers according to their output, for example avoiding the need to store lots of file in a single directory. It's up to you.

You can also run iiif-tile-seed pass a list of identifiers as a CSV file. To do so include the -mode csv argument, like this:

$> ./bin/iiif-tile-seed -options -mode csv CSVFILE

Your CSV file must contain a header specifying a source_id and alternate_id column, like this:

source_id,alternate_id
191733_5755a1309e4d66a7_k.jpg,191733_5755a1309e4d66a7

While all columns are required if alternate_id is empty the code will simply default to using source_id for all operations.

Important: The use of alternate IDs is not fully supported by iiif-server yet. Which is to say to the logic for how to convert a source identifier to an alternate identifier is still outside the scope of go-iiif so unless you have pre-rendered all of your tiles or other derivatives (in which case the check for cached derivatives at the top of the imgae handler will be triggered) then the server won't know where to write new alternate files.

Config files

There is a sample config file included with this repo. The easiest way to understand config files is that they consist of at least five top-level groupings, with nested section-specific details, followed by zero or more implementation specific configuration blocks. The five core blocks are:

level

	"level": {
		"compliance": "2"
	}

Indicates which level of IIIF Image API compliance the server (or associated tools) should support. Basically, there is no reason to ever change this right now.

graphics

	"graphics": {
		"source": { "name": "VIPS" }
	}

Details about how images should be processed. Because only libvips is supported for image processing right now there is no reason to change this. According to the bimg docs (which is the Go library wrapping libvips) the following formats can be read:

It can read JPEG, PNG, WEBP natively, and optionally TIFF, PDF, GIF and SVG formats if [email protected]+ is compiled with proper library bindings.

If you've installed libvips using the handy setup script then all the formats listed above, save PDF, should be supported.

Important: That's actually not true if you're reading this. It was true but then I tried running iiif-tile-seed on a large set of images and started triggering this error even though it's supposed to be fixed. If you're reading this it means at least one of three things: the bug still exists; I pulled source from gopkg.in rather than github.com despite the author's notes in the issue; changes haven't propogated to gopkg.in yet. Which is to say that the current version of bimg is pegged to the v1.0.1 release which doesn't know think it knows about the PDF, GIF or SVG formats yet. It's being worked on...

The VIPS graphics source has the following optional properties:

tmpdir Specify an alternate path where libvips should write temporary files while processing images. This may be necessary if you are a processing many large files simultaneously and your default "temporary" directory is very small.

features

	"features": {
		"enable": {},
		"disable": { "rotation": [ "rotationArbitrary"] },
		"append": {}
	}

The features block allows you to enable or disable specific IIIF features. Currently only image related features may be manipulated.

For example the level 2 spec does not say GIF outputs is required so the level 2 compliance definition in go-iiif disables it by default. If you are using a graphics engine (not libvips though) that can produce GIF files you would enable it here.

Likewise you may need to disable a feature that is supported by not required or features that are required but can't be used for one reason or another. For example libvips does not allow support for the following features: sizeByDistortedWh (size), rotationArbitrary (rotation), bitonal (quality).

Finally, maybe you've got an IIIF implementation that knows how to do things not defined in the spec. This is also where you would add them.

compliance

Here's how that dynamic plays out in reality. The table below lists all the IIIF parameters and their associate features. Each feature lists its syntax and whether or not it is required and supported according to the official spec but then also according to the example go-iiif config file, included with this repo.

This table was generated using the iiif-dump-config tool and if anyone can tell me how to make Markdown tables (in GitHub) render colours I would be grateful.

region

feature	syntax	required (spec)	supported (spec)	required (config)	supported (config)
full	full	true	true	true	true
regionByPct	pct:x,y,w,h	true	true	true	true
regionByPx	x,y,w,h	true	true	true	true
regionSquare	square	false	false	false	false

size

feature	syntax	required (spec)	supported (spec)	required (config)	supported (config)
full	full	true	true	true	true
max	max	false	true	false	true
sizeByConfinedWh	!w,h	true	true	true	true
sizeByDistortedWh	w,h	true	true	true	false
sizeByH	,h	true	true	true	true
sizeByPct	pct:n	true	true	true	true
sizeByW	w,	true	true	true	true
sizeByWh	w,h	true	true	true	true

rotation

feature	syntax	required (spec)	supported (spec)	required (config)	supported (config)
mirroring	!n	true	true	true	true
none	0	true	true	true	true
rotationArbitrary		false	true	false	false
rotationBy90s	90,180,270	true	true	true	true

quality

feature	syntax	required (spec)	supported (spec)	required (config)	supported (config)
bitonal	bitonal	true	true	true	false
color	color	false	true	false	true
default	default	true	true	true	true
dither	dither	false	false	false	true
gray	gray	false	false	false	false

Careful readers may notice the presence of an undefined (by the IIIF spec) feature named dither. This is a go-iiif -ism and discussed in detail below in the features.append and non-standard features sections.

format

feature	syntax	required (spec)	supported (spec)	required (config)	supported (config)
gif	gif	false	false	false	false
jp2	jp2	false	false	false	false
jpg	jpg	true	true	true	true
pdf	pdf	false	false	false	false
png	png	true	true	true	true
tif	tif	false	false	false	true
webp	webp	false	false	false	true

Support for GIF output is not enabled by default because it is not currently supported by bimg (the Go library on top of lipvips). There is however native support for converting final images to be GIFs but you will need to enable that by hand, below.

features.enable

	"features": {
		"enable": {
			"size": [ "max" ],
			"format": [ "webp", "tif" ]
		}
	}

Individual features for a given parameter are enabled by including the parameter name as a key to the features.enabled dictionary whose value is a list of specific feature names to enable.

features.disable

	"features": {
		"disable": {
			"size": [ "sizeByDistortedWh" ] ,
			"rotation": [ "rotationArbitrary" ],
			"quality": [ "bitonal" ]
		}
	}

Individual features for a given parameter are disabled by including the parameter name as a key to the features.disabled dictionary whose value is a list of specific feature names to disabled.

features.append

	"features": {
		"append": { "quality": {
			"dither": { "syntax": "dither", "required": false, "supported": true, "match": "^dither$" }
		}}
	}

New features are added by including their corresponding parameter name as a key to the features.append dictionary whose value is a model for that feature. The data model for new features to append looks like this:

	NAME (STRING): {
		"syntax": SYNTAX (STRING),
		"required": BOOLEAN,
		"supported": BOOLEAN,
		"match": REGULAR_EXPRESSION (STRING)
	}

All keys are required.

The supported key is used to determine whether a given feature is enabled or not. The match key is used to validate user input and should be a valid regular expression that will match that value. For example here is the compliance definition for images returned in the JPEG format:

		"format": {
	     	       "jpg": { "syntax": "jpg",  "required": true, "supported": true, "match": "^jpe?g$" }
		}

Important: It is left to you to actually implement support for new features in the code for whichever graphics engine you are using. If you don't then any new features will be ignored at best or cause fatal errors at worst.

images

	"images": {
		"source": { "name": "Disk", "path": "example/images" },
		"cache": { "name": "Memory", "ttl": 300, "limit": 100 }
	}

Details about source images.

images.source

Where to find source images.

Disk

	"images": {
		"source": { "name": "Disk", "path": "example/images" }
	}

Fetch source images from a locally available filesystem.

Flickr

	"images": {
		"source": { "name": "Flickr" },
		"cache": { "name": "Memory", "ttl": 60, "limit": 100 }
	},
	"flickr": {
		"apikey": "YOUR-FLICKR-API-KEY"
	}

Fetch source images from Flickr. You will need to provide a valid Flickr API key. A few caveats:

The code assumes the original Flickr (Auth) API and not the newer OAuth-flavoured API.
Signed API keys are not supported yet so you're limited to public photos.
The code calls the flickr.photos.getSizes API method and looks for the first of the following photo sizes in this order: Original, Large 2048, Large 1600, Large. If none are available then an error is triggered.
Photo size lookups are not cached yet.

Here's an example with this photo:

S3

	"images": {
		"source": { "name": "S3", "path": "your.S3.bucket", "region": "us-east-1", "credentials": "default" }
	}

Fetch source images from Amazon's S3 service. S3 caches assume that that the path key is the name of the S3 bucket you are reading from. S3 caches have three addition properties:

prefix is an optional path to a sub-path inside of your S3 bucket where images are stored.
region is the name of the AWS region where your S3 bucket lives. Sorry this is an AWS-ism
credentials is a string describing how your AWS credentials are defined. Valid options are:
env: - Signals that you you have defined valid AWS credentials as environment variables
shared:PATH_TO_SHARED_CREDENTIALS_FILE:SHARED_CREDENTIALS_PROFILE - Signals that your AWS credentials are in a shared credentials files and that go-iiif should use a specific profile
iam: - Signals that you are using go-iiif in an AWS environment with suitable roles and permissioning for working with S3. The details of how and where you configure IAM roles are outside the scope of this document.

For the sake of backwards compatibilty if the value of credentials is any other string then it will be assumed to be the name of the profile you wish to use for a valid credential files in the home directory of the current user. Likewise if the value of credentials is an empty string (or absent) it will be assumed that valid AWS access credentials have been defined as environment variables.

It is not possible to define your AWS credentials as properties in your go-iiif config file.

Important: If you are both reading source files and writing cached derivatives to S3 in the same bucket make sure they have different prefixes. If you don't then AWS will happily overwrite your original source files with the directory (which shares the same names as the original file) containing your derivatives. Good times.

URI

	"images": {
		"source": { "name": "URI", "path": "https://images.collection.cooperhewitt.org/{id}" }
	}

Fetch source images from a remote URI. The path parameter must be a valid (Level 4) URI Template with an {id} placeholder.

images.cache

Caching options for source images.

Disk

	"images": {
		"cache": { "name": "Disk", "path": "example/cache" }
	}

Cache images to a locally available filesystem.

Memory

	"images": {
		"cache": { "name": "Memory", "ttl": 300, "limit": 100 }
	}

Cache images in memory. Memory caches have two addition properties:

ttl is the maximum number of seconds an image should live in cache.
limit the maximum number of megabytes the cache should hold at any one time.

Null

	"images": {
		"cache": { "name": "Null" }
	}

Because you must define a caching layer this is here to satify the requirements without actually caching anything, anywhere.

derivatives

	"derivatives": {
		"cache": { "name": "Disk", "path": "example/cache" }
	}

Details about derivative images.

derivatives.cache

Caching options for derivative images.

Disk

	"derivatives": {
		"cache": { "name": "Disk", "path": "example/cache" }
	}

Cache images to a locally available filesystem.

Memory

	"derivatives": {
		"cache": { "name": "Memory", "ttl": 300, "limit": 100 }
	}

Cache images in memory. Memory caches have two addition properties:

ttl is the maximum number of seconds an image should live in cache.
limit the maximum number of megabytes the cache should hold at any one time.

Null

	"derivatives": {
		"cache": { "name": "Null" }
	}

Because you must define a caching layer this is here to satify the requirements without actually caching anything, anywhere.

S3

	"derivatives": {
		"cache": { "name": "S3", "path": "your.S3.bucket", "region": "us-east-1", "credentials": "default" }
	}

Cache images using Amazon's S3 service. S3 caches assume that that the path key is the name of the S3 bucket you are reading from. S3 caches have three addition properties:

prefix is an optional path to a sub-path inside of your S3 bucket where images are stored.
region is the name of the AWS region where your S3 bucket lives. Sorry this is an AWS-ism
credentials is a string describing how your AWS credentials are defined. Valid options are:
env: - Signals that you you have defined valid AWS credentials as environment variables
shared:PATH_TO_SHARED_CREDENTIALS_FILE:SHARED_CREDENTIALS_PROFILE - Signals that your AWS credentials are in a shared credentials files and that go-iiif should use a specific profile
iam: - Signals that you are using go-iiif in an AWS environment with suitable roles and permissioning for working with S3. The details of how and where you configure IAM roles are outside the scope of this document.

For the sake of backwards compatibilty if the value of credentials is any other string then it will be assumed to be the name of the profile you wish to use for a valid credential files in the home directory of the current user. Likewise if the value of credentials is an empty string (or absent) it will be assumed that valid AWS access credentials have been defined as environment variables.

It is not possible to define your AWS credentials as properties in your go-iiif config file.

Important: If you are both reading source files and writing cached derivatives to S3 in the same bucket make sure they have different prefixes. If you don't then AWS will happily overwrite your original source files with the directory (which shares the same names as the original file) containing your derivatives. Good times.

Non-standard features

go-iiif supports the following non-standard IIIF quality features:

Dithering

	"append": {
		"quality": {
			"dither": { "syntax": "dither", "required": false, "supported": true, "match": "^dither$" }
		}
	}

dither will create a black and white halftone derivative of an image using the Atkinson dithering algorithm. Dithering is enabled in the example config file and you can invoke it like this:

http://localhost:8082/184512_5f7f47e5b3c66207_x.jpg/pct:41,7,40,70/,5000/0/dither.png

And here's what you should see, keeping in mind that this screenshot shows only a section of the image at full size:

There are a few caveats about dithering images:

The first thing to know is that the dithering is a pure Go implementation so it's not handled by lipvips.
The second is that the dithering happens after the libvips processing.
This is relevant because there are some image formats where Go does not support native encoding. For example webp (which is weird since it's a Google thing...)
It is possible to track all of this stuff in code and juggle output formats and reprocessing (in libvips) but that code has not been written yet.
So you will need to track the sometimes still-rocky relationship between features and output formats yourself.

Primitive-ing

	"features": {
		"append": {
			"quality": {
				"primitive": { "syntax": "primitive:mode,iterations,alpha", "required": false, "supported": true, "match": "^primitive\\:[0-5]\\,\\d+\\,\\d+$" }
			}
		}
	},
	"primitive": { "max_iterations": 100 }

Note the way the primitive block is a top-level element in your config file.

primitive use @fogleman's primitive library to reproduce the final image using geometric primitives. Like this:

The syntax for invoking this feature is primitive:{MODE},{ITERATIONS},{ALPHA} where:

MODE is a number between 0-5 representing which of the primitive shapes to use. They are:
0: combo
1: triangle
2: rectangle
3: ellipse
4: circle
5: rotated rectangle
ITERATIONS is a number between 1 and infinity (a bad idea) or 1 and the number defined in the primitive.max_iterations section in your config file
ALPHA is a number between 0-255

For example:

http://localhost:8082/184512_5f7f47e5b3c66207_x.jpg/full/500,/0/primitive:5,200,255.jpg

Be aware that it's not exactly "fast". It's getting faster but it still takes a while. Also, this code should probably have a flag to downsize the input image for processing (and then resizing it back up to the requested size) but that doesn't happen yet. Basically you should not enable this feature as a public-facing web service because it will take seconds (not microseconds) or sometimes even minutes to render a single 256x256 tile. For example:

./bin/iiif-server -host 0.0.0.0 -config config.json
2016/09/21 15:43:08 Serving [::]:8080 with pid 5877
2016/09/21 15:43:13 starting model at 2016-09-21 15:43:13.626117993 +0000 UTC
2016/09/21 15:43:13 finished step 1 in 8.229683ms
2016/09/21 15:43:16 finished step 2 in 3.019413861s
…
2016/09/21 15:45:38 finished step 100 in 2m24.626232387s
2016/09/21 15:45:39 finished model in 2m25.611790848s

But it is pretty darn cool!

If you specify a gif format parameter then go-iiif will return an animated GIF for the requested image consisting of each intermediate stage that the primitive library generated the final image. For example:

http://localhost:8082/184512_5f7f47e5b3c66207_x.jpg/full/500,/0/primitive:5,100,255.gif

Which would produce this:

Here are examples where each of the tiles in an slippy image are animated GIFs:

Note: You will need to manually enable support for GIF images in your config file for animated GIFs to work.

Example

There is a live demo of the Leaflet-IIIF slippymap provider used in conjunction with a series of tiles images generated using the iiif-tile-seed utility available for viewing over here:

https://thisisaaronland.github.io/go-iiif/

The iiif-server tool also comes with a canned example (consisting of exactly one image) so you can see things in the context of a slippy map. Here's what you need to do to get it set up:

First, make sure have a valid go-iiif config file. If you don't then you can copy the example config included in this repo:

$> cp config.json.example config.json

Next, pre-seed some tiles for an image. You don't necessarily need to do this step but it's included to show you how it's done:

$> ./bin/iiif-tile-seed -config config.json -endpoint http://localhost:8082 -scale-factors 8,4,2,1 184512_5f7f47e5b3c66207_x.jpg

Note how we are specifying the endpoint where these tiles will be served from. That's necessary so that we can also pre-seed a profile description for each image as well as tiles.

Finally start up the iiff-server and be sure to pass the -example flag:

$> ./bin/iiif-server -config config.json -host localhost -port 8082 -example

Now if you visit http://localhost:8082/example/ in your browser you should see this:

Assuming you've pre-seed your tiles if you open up the network console in your browser then you should see something like this, namely that the individual tiles are returned speedy and fast:

Generating static images

The example included with go-iiif has an added super power which is the ability to create a static image of the current state of the map/image.

Just click the handy 📷 button to the bottom right of the image and you will be prompted for where you'd like to save your new image.

This is not a feature of go-iiif itself. It's entirely client-side magic in your browser but it's still pretty cool...

Performance and load testing

iiif-tile-seed

Processing individual or small batches of images go-iiif ranges from pretty fast to very fast. For example here is a picture of Spanking Cat width a maximum dimension of 4096 pixels:

$> ./bin/iiif-tile-seed -config config.json -refresh -scale-factors 8,4,2,1 184512_5f7f47e5b3c66207_x.jpg
[184512_5f7f47e5b3c66207_x.jpg] time to process 340 tiles: 27.537429902s

So any individual tile is pretty speedy but in the aggregate it starts to add up. I will need to do some continued digging to make sure that the source image isn't being processed unnecessarily for each tile. Here is the same image but with a maximum dimension of 2048 pixels:

$> ./bin/iiif-tile-seed -config config.json -refresh -scale-factors 4,2,1 184512_b812003c86c3525b_k.jpg
[184512_b812003c86c3525b_k.jpg] time to process 84 tiles: 1.894074539s

Note that we are only generating tiles for three scale factors instead of four. But that's not where things slow down as we can see seeding tiles for only three scale factors for the larger image:

$> ./bin/iiif-tile-seed -config config.json -refresh -scale-factors 4,2,1 184512_5f7f47e5b3c66207_x.jpg
[184512_5f7f47e5b3c66207_x.jpg] time to process 336 tiles: 26.925253066s

For processing large, or large volumes of, images the bottlenecks will be:

CPU usage crunching pixels
Disk I/O writing tiles to disk
Running out of inodes

That said on a machine with 8 CPUs and 32GB RAM I was able to run the machine hot with all the CPUs pegged at 100% usage and seed 100, 000 (2048x pixel) images yielding a little over 3 million, or approximately 70GB of, tiles in 24 hours. Some meaningful but not overwhelming amount of time was spent fetching source images across the network so presumably things would be faster reading from a local filesystem.

Memory usage across all the iiif-tile-seed processes never went above 5GB and, in the end, I ran out of inodes.

The current strategy for seeding tiles may also be directly responsible for some of the bottlenecks. Specifically, when processing large volumes of images (defined in a CSV file) the ifff-tile-seed will spawn and queue as many concurrent Go routines as there are CPUs. For each of those processes then another (n) CPUs * 2 subprocesses will be spawned to generate tiles. Maybe this is just too image concurrent image processing routines to have? I mean it works but still... Or maybe it's just that every one is waiting for bytes to be written to disk. Or all of the above. I'm not sure yet.

iiif-server

All of the notes so far have assumed that you are using iiif-tile-seed. If you are running iiif-server the principle concern will be getting overwhelmed by too many requests for too many different images, especially if they are large, and running out of memory. That is why you can define an in-memory cache for source images but that will only be of limited use if your problem is handling concurrent requests. It is probably worth adding checks and throttles around current memory usage to the various handlers...

Notes

The iiif-server does not support TLS yet.
There is no way to change the default quality parameter yet. It is color.

Bugs?

Probably. Please consult the currently known-known issues and if you don't see what ails you please feel free to add it.

Name		Name	Last commit message	Last commit date
Latest commit History 286 Commits
aws		aws
cache		cache
cmd		cmd
compliance		compliance
config		config
example		example
image		image
level		level
misc		misc
profile		profile
source		source
tile		tile
ubuntu		ubuntu
vendor/src		vendor/src
.gitignore		.gitignore
LICENSE		LICENSE
Makefile		Makefile
README.md		README.md
Vagrantfile		Vagrantfile
config.json.example		config.json.example
iiif.go		iiif.go

License

lelikg/go-iiif

Folders and files

Latest commit

History

Repository files navigation

go-iiif

What is this?

Setup

Usage

Tools

iiif-server

Endpoints

GET /{ID}/info.json

GET /{ID}/{REGION}/{SIZE}/{ROTATION}/{QUALITY}.{FORMAT}

GET /debug/vars

iiif-tile-seed

iiif-tile-seed and identifiers

Config files

level

graphics

features

compliance

region

size

rotation

quality

format

features.enable

features.disable

features.append

images

images.source

Disk

Flickr

S3

URI

images.cache

Disk

Memory

Null

derivatives

derivatives.cache

Disk

Memory

Null

S3

Non-standard features

Dithering

Primitive-ing

Example

Generating static images

Performance and load testing

iiif-tile-seed

iiif-server

Notes

Bugs?

See also

IIIF stuff

Go stuff

Slippy map stuff

Blog posts

Other stuff

About

Resources

License

Stars

Watchers

Forks

Releases

Packages 0

Languages

Packages