“Outpainting” is using AI to expand any image in any direction

DALL-E 2 for “computer science, digital art“

DALL-E is a neural network that turns images into paintings. It was created by OpenAI, an artificial intelligence research lab. The name DALL-E comes from the Disney film Wall-E and the name of the famous artist Salvador Dali. DALL-E is trained on a massive dataset of images from the internet. The goal of DALL-E is to generate images from textual descriptions, such as "an orange with a blue stripe." To do this, DALL-E uses a Recurrent Neural Network (RNN). RNNs are a type of neural network that can process sequences of data, such as text. DALL-E is not the first neural network to generate images from text descriptions, but it is the first one to do so in such a high resolution. For example, the image to the right was generated by DALL-E from the description "computer science, digital art." As you can see, DALL-E was able to create a beautiful high-resolution image. This shows that DALL-E has a deep understanding of the content of images.

We used the outpainting prompt “fill with flowers“ to achieve this incredible result.

So how does “outpainting” work?

The "outpainting" feature of DALL-E allows users to expand an image in any direction. For example, if you have an image that is too small to fill the desired frame, you can use outpainting to expand it toward the left and right sides of the image. This is a great way to add more detail to an image without having to start from scratch. To use outpainting, simply select the image editing tool from the menu and then click and drag a box in the direction you want to expand the image. The further you drag, the more the image will be expanded. Outpainting is a great way to add more interest to an otherwise plain image.

And technically speaking (but still for AI noobs)?

Outpainting analyses the original image and labels it (e.g. chair, sunset, computer, digital art, purple floor, wall on the left of the window, and very many more). Then new images are generated, let’s say in a random fashion for now - this varies by model and is rather exhaustive as a topic, and these generated images are labeled too and matched to the millions of original labels to see how well they match. The user is presented with the best variants (and can opt to re-generate more suggestions) and once confirmed the picture is extended and the process can start over. Outpainting is actually quite fun to play with using DALL-E, which is now available at a very low cost with a few free samples to get started.

Interest in outpainting over time

As you can see outpainting is a brand new topic and interest is rapidly rising, especially in China.

References

1. OpenAI. (2022). DALL-E 2: Creating Images from Text. OpenAI. https://openai.com/research/dall-e-2/
2. Radford, A., Kim, K., Hallacy, C., et al. (2021). Learning to Generate Images from Text. arXiv preprint arXiv:2102.12092.
3. Brown, T. B., Mann, B., Ryder, N., et al. (2020). Language Models are Few-Shot Learners. arXiv preprint arXiv:2005.14165.
4. OpenAI. (2022). DALL-E 2. OpenAI. https://openai.com/research/dall-e-2/
5. Sitzmann, V., Chen, R., Russell, B., et al. (2020). SIREN: Implicit Neural Representations with Periodic Activation Functions. arXiv preprint arXiv:2006.09661.