Hello! Today we’ll be learning about the MindsEye Artist’s Kit, a simple way for nearly anyone to produce their own deep learning artwork! Our goal here is to make this easy to use and simple to set up, using remote execution on Amazon EC2 to negate special hardware requirements. Only basic software skills, or a willingness to learn, are needed.

Although EC2 is a paid service, the prices are economical. We use P3.2XL instances, which cost about a nickel per minute. However, if you consider that this instance seems to perform almost 20x faster than my desktop, the $3 for an hour of compute on this machine is comparable to the $2–3 dollars of energy my home machine costs per day (A 1kW PC running at 100% for 24 hours at $0.12 per kW*hr — typically machines use power savings when idling or sleeping.) However, even though the price is reasonable and even though the software covered here does its best to clean up the instances after use, it is a good idea to double-check the AWS console to make sure you haven’t left any machines on when you are done!

Art Examples

What kind of images can we generate with this project? Here are a few examples. Keep in mind these are fairly raw first-generation examples. Your work could be way cooler, and I can’t wait to see it!

Background

At the fundamental level, we are using Neural Networks. Neural Networks are an artificial intelligence technique inspired by the biological brain found inside most humans. Each Neuron individually processes only a tiny amount of data, but through many layers of interconnections more advanced behaviors can emerge. The mechanics of a single layer of neurons work largely on the principles of Linear Algebra, and the network is generally trained using Back Propagation.

When processing visual data, however, we can take advantage of self-similarity within the data: A shape in the upper-left of the image is still the same shape even if it is in the lower-right in another image. Pixels which are next to each other tend to be of the same color. These spatial relationships allow us to impose this symmetry on the network itself, creating what is known as a convolutional network. Convolutional networks are great at processing image data and have been known to pick out edges, textures, and shapes at increasing levels of complexity.

One of our standard networks is VGG19, a pre-trained model built by some researchers at Oxford to process the ImageNet Challenge 2014 image recognition dataset. This “deep” convolutional network consists of 20–70 layers, depending on how you count them, and is nearly a gigabyte in size. This network was pre-trained for image classification, but we are going to re-use it for art generation by extracting the first few layers as “feature detection layers”. These first layers have implicitly been trained to respond to details that a relevant to object detection, with the idea that these details will also be the details of importance in the human visual processing system.

One of the first creative uses for these feature layers was the Deep Dream algorithm. This method modifies an image seeking to maximize the L2 norm of the activation of a given layer. This maximizes the signal of a given layer, thus whatever textures that layer was detecting should be visually enhanced. This can also use a weighted combination of the signal strengths of multiple layers, providing some control over the result.

A more flexible method uses an input “style” image to gather inspiration from. This style image is pre-processed by the feature layers, and then we collect metrics about these signals. Specifically, we track the mean value of each feature channel (similar to a color channel, but representing a larger “thing” such as a texture or a shape) and the mean of all cross-products, technically known as a Gram Matrix. By recording these metrics and then optimizing an image to have similar metrics, we can encourage any input image to have a similar distribution of shapes and textures; i.e. style transfer!

This is of course only a very, very brief theoretical overview. There are many other pre-trained networks out there, and there are many variations to these techniques. There is nearly an endless potential here for mathematically-based creativity!

Requirements

By using EC2 and Java, we can reduce the requirements for running this project down to commonly available and free components. These requirements are largely the same as for any Java software project:

Environment Setup

First, let’s assume you don’t have all the requirements installed. All you have is a computer, the internet, an email address, a web browser, and a credit card. No problem! You just need to setup 3 main things:

Amazon Web Services provides cloud computing facilities, including state of the art hardware for Deep Learning with CuDNN.

Git is a popular peer-to-peer version control system used to manage source code.

Java Development Tools are needed, including:

Project Setup

Pulling the project is as simple as any other Java project:

Execution

Execution of any script is easy:

First Time Setup

When run, the EC2Runner class looks for user-settings.json and ec2-settings.json, and if they are not found it starts a bootstrapping process to initialize itself. This is mostly automated, as long as the user has full permissions to AWS setup properly.

First, the user is prompted for email via standard input. There is no gui for this, so simply type your email into the console window and press enter. This email will be verified via AWS, sending an email to the provided address requesting confirmation to enable messages.

Next, using the aws user configured via the cli, the application sets up various AWS entities to support execution including:

Monitor Execution

Once the initialization logic completes, the remote process is launched. This is a complex, multi stage process involving launching a new EC2 instance, connecting to it, deploying code, and managing remote execution. This is largely transparent to the user, who will monitor the process over these stages:

At the end of execution, the instance should automatically shut down. It can also be manually terminated by using the AWS Console, linked to in the “start” email.

The MindsEye Artist’s Kit includes a variety of starter scripts, currently available in 3 families and 5 sub-types:

Script Families

Deep Dream — This process enhances latent patterns within an input image. This script uses the following parameters:

Style Transfer — This combines two images, using the content of one image and the style of another to render a unique combination of the two. This script uses the following parameters:

Texture Generation — Very similar to style transfer, this process lacks a content image and thus generates abstract imagery. This script uses the following parameters:

Script Sub-Types

Each family of script is provided in several forms, each supporting a slightly different use-case:

Script Output Examples

For illustration, here are some prior runs of each script. Keep in mind these are works in progress. Feel free to submit your results and improvements!

Deep Dream

Style Transfer

Texture Generation

Further Reading

Enjoy!

Big Data Engineer and Artificial Intelligence Researcher