Max The Rocket

What is OpenGL

OpenGL is a low-level, 3D graphics and modeling library that is highly portable and very fast. It is used for rendering real-time 3D and 2D graphics such as games, music visualizations, screen savers and data visualizations on many platforms and systems like Windows, MacOS, Linux as well as mobile devices and gaming consoles. Essentially, OpenGL is an API that allows software programs to send drawing functions directly to the GPU (Graphics Processing Unit), located on a computer’s graphic card. Since graphic cards are designed and optimized for handling graphics, all the drawing and 3D rendering calculations are processed much faster on the GPU than on the computer’s CPU. OpenGL is an open standard API, developed and maintained by non-for-profit Khronos Group, which means that any graphic card manufacturer, such as Nvidea and ATI, are able to implement OpenGL in their hardware without any charges or royalties.

What OpenGL is NOT

• OpenGL is not a programming language
• It is not a game engine
• It is not a framework like Processing, OpenFrameworks (although these frameworks do use OpenGL)
• It is not able to create operating system windows or user interface objects
• It cannot read and write files

Without a programming language, OpenGL is useless. OpenGL requires a programming language to give it a window (called a context) to draw in, to allow user interaction, to create animations and provide it with data like 3D objects, textures and colours.

Virtually any programming language can execute OpenGL functions (C++, Java, Python, C# just to name a few), and it doesn’t matter which language is used, OpenGL will render the graphics the same way.

OpenGL is a procedural graphics API as opposed to descriptive. Instead of describing the 3D scene and how it should appear, the program needs to call OpenGL functions in a specific sequence in order to achieve a certain effect. These functions are used to draw graphics primitives such as points, lines, and triangles in a three-dimensional space. OpenGL also supports texture mapping, blending, transparency, animation, and many other special effects and capabilities.

My Research Approach

With the help of various tutorials and books, a list of which I provided in the end of the article, I created a web application that showcases some capabilities of OpenGL. I utilized a new technology called WebGL, which allows to execute OpenGL calls using Javascript withing a web browser. Keep in mind that WebGL is currently only available on the latest version of Google Chrome and Firefox (Safari and Opera should support it as well, although I haven’t tested it). Since the entire application is written in Javascript, and some HTML, the source code is available for anyone to view.

You may view the full application here: http://maxtherocket.com/projects/experimental/webgl/
Click left and right arrows to move through different scenes.

Process

I will outline the process of creating my application, but I will not go into too much technical detail. The description will cover the steps I had to take in order to render various 2D and 3D graphics, using OpenGL function calls. I will also display parts of the code that I used, if you are interested, full source code is available if you view the source on the page of my application.

Step 1 [Creating an OpenGL window]

WebGL lets you create an OpenGL window, it is actually called a context, inside an HTML5 element called CANVAS. Here’s the Javascript code that creates an OpenGl context, it executes as soon as the page loads:

canvas = document.getElementById("glcanvas");
initWebGL(canvas);      // Initialize the GL context
// Here's the initWebGL() function
function initWebGL() {
  gl = null;

  try {
    gl = canvas.getContext("experimental-webgl");
  }
  catch(e) {
  }

  // If we don't have a GL context, alert the user
  if (!gl) {
    alert("Unable to initialize WebGL. Your browser may not support it.");
  }
}

After the gl object has been initialized, it will contain all the OpenGL functions.

And HTML code for the canvas element, which sets up the dimensions of the OpenGL window:

<canvas id="glcanvas" width="640" height="480"></canvas>

In a language like C++ creating the window for OpenGL would be different since you would have to create an operating system window.

Step 2 [Set a couple settings]

We need to tell OpenGL how to render the geometry that we’ll be creating in the 3D scene.

gl.enable(gl.DEPTH_TEST); // Enable depth testing, so 3D objects don't overlap each other
gl.depthFunc(gl.LEQUAL);  // Near things will obscure far things, this can be set the other way around, which would be weird

Step 3 [Setup Shaders] (This one is important!)

Shaders in OpenGL play a very important role. Shader code tells the hardware, the GPU, how to render 3D shaped in perspective and how to colour them. Without shaders, OpenGL cannot render anything. Shaders are written in a language very similar to C, called GLSL (GL Shading Language). Shaders come in 2 parts, a vertex shader and a fragment shader.

• The vertex shader manipulates each 3D point that we load into the OpenGL scene
• The fragment shader is responsible for colouring the objects and it manipulates each pixel of an object

Here’s is the source code for the basic shaders that I used in my application:

Vertex Shader

// This is the Vertex shader
// This little program loops through each 3D point (a vertex) that the OpenGL has in the scene
attribute vec3 aVertexPosition; // This variable contains a single 3D point, each point gets passed into this variable as the program loops through vertices

uniform mat4 uMVMatrix; // A translation matrix, used for moving/rotating shapes in 3D space
uniform mat4 uPMatrix; // Perspective matrix, creates the 3D look, pretty much a camera

void main(void) {
   gl_Position = uPMatrix * uMVMatrix * vec4(aVertexPosition, 1.0); // This is where the point gets translated from 3D space to 2D space
   vColor = aVertexColor; // Here we're just passing the color of a vertex along to the fragment shader
	// there's no need to have this last line if we don't use the colours from vertices
}

Fragment Shader (Pixel Shader)

// This is the fragment shader
  void main(void) {
   // gl_FragColor is an OpenGL variable, it will draw the color that we pass it
   // with the variable set to vec4(1.0, 1.0, 1.0, 1.0) OpenGL will color all objects plain white
   // vec4 is a data type, vec4 stands for: vector with 4 values
   // each of the values represent a color vec4(red, green, blue, opacity)
   // By setting all the values to 1, we get white
    gl_FragColor = vec4(1.0, 1.0, 1.0, 1.0);
  }

Shaders are very powerful, since they loop through every 3D point and every pixel for each frame of rendering (and it’s all done really fast). The colour of each pixel can be changed based on its relation to other pixels, 3D points and lights in the scene. 3D points can be modified as well through a vertex shader as well.

This is how I initialized the shaders in WebGL:

// Create a place for a shader in OpenGL memory
shader = gl.createShader(gl.FRAGMENT_SHADER);
// Load the shader source code
gl.shaderSource(shader, theSource);
// Compile the shader
gl.compileShader(shader);

Step 4 [Adding Geometry]

Since the code for geometry gets fairly lengthy, I will only explain here the steps required to create a square shape in OpenGL.

In OpenGL, all geometry is created with points. There is a way to tell OpenGL how to connect and fill the points that you create. OpenGL offers 10 different drawing modes, each will render the points differently. Here’s a good graphic that showcases all the modes:

I passed 4 points into OpenGL geometry buffer and then used a mode called TRIANGLE_STRIP to draw the square shape.

  // this creates a place in GPU memory for our array of points
  squareVerticesBuffer = gl.createBuffer();
  // then we 'bind' the buffer, get the memory ready to receive the points
  gl.bindBuffer(gl.ARRAY_BUFFER, squareVerticesBuffer);
// Here's our list of 4 points
// each line represents a point in 3D space
// the order is x, y, z
// we leave the z as 0 since we are drawing a 2D shape
// The center of the square will be at coordinate 0,0,0
  var vertices = [
    1.0,  1.0,  0.0, // top right
    -1.0, 1.0,  0.0, // top left
    1.0,  -1.0, 0.0, // botom right
    -1.0, -1.0, 0.0 // bottom left
  ];
  // Now pass the list of vertices (points) into WebGL to store the shape
  gl.bufferData(gl.ARRAY_BUFFER, new Float32Array(vertices), gl.STATIC_DRAW);

All objects created in OpenGL are automatically positioned at point 0x,0y,0z, and that point, in the cartesian coordinate system, is usually placed at the center point of the OpenGL window. When creating vertices for an object we have to keep in mind that all the points extend from the center of the object, so a vertex at position (1.0x ,1.0y, 0z) means that it’s positioned 1 unit of x and 1 unit of y away from the center of the screen.

Now that we stored the points in OpenGL memory we have to pass the vertices to the vertex shader and tell OpenGL to draw this shape.

Step 4 [Drawing]

Before we draw anything to the screen we need to clear the canvas. If we are animating something, we’ll need to clear the canvas on each frame and then re-draw everything again. This might seem counter-intuitive but, at lower level, that is how real-time computer animations works. So each frame we have to keep track of the position and states of each object.

Here’s how to clear the screen:

// Clear the canvas before we start drawing on it.
gl.clear(gl.COLOR_BUFFER_BIT | gl.DEPTH_BUFFER_BIT);

Then we need to create a perspective matrix, this matrix will tell the vertex shader how to render the 3D scene to a 2D image. This matrix acts as a camera that shows us the 3D scene.

I used special helper functions to create this camera matrix:

 // Establish the perspective with which we want to view the
  // scene. Our field of view is 45 degrees, with a width/height
  // ratio of 640:480, and we only want to see objects between 0.1 units
  // and 100 units away from the camera.
  perspectiveMatrix = makePerspective(45, 640.0/480.0, 0.1, 100.0);
// THis matrix gets passed to the vertex shader later on

If you are interested, you can view the full function in the glUtils.js file, it contains various 3D matrix helper functions.

// Get the points from memory
gl.bindBuffer(gl.ARRAY_BUFFER, squareVerticesBuffer);
// Pass the points array to the vertex shader (I initialized vertexPositionAttribute earlier in the program)
gl.vertexAttribPointer(vertexPositionAttribute, 3, gl.FLOAT, false, 0, 0);
// Tell OpenGL to draw
gl.drawArrays(gl.TRIANGLE_STRIP, 0, 4);

The drawArrays function simply tells OpenGL to execute the shaders, passing the number of points to take from the array and the mode to draw them with, which will then draw inside the window.

The steps I outlined above explain how to draw a simple 2D shape in OpenGL using Javascript. I also explained a fair bit about how OpenGL works. In my demo application, I have examples of animation, use of colour and texturing a 3D object. Below, I will briefly explain how these features we implemented.

Using Colour

Instead of colouring the entire shape with one colour, we can give each vertice it’s own colour. The cool thing about OpenGL is that the shaders will automatically fill the space between the vertices with a gradient, if the colours of each vertex is different. Just look at the image on the left. I gave each corner vertex a different colour and OpenGL figured out the rest. If I gave each vertex the same colour, the square would have been a solid colour.

Here’s how I set the colours using Javascript:

var colors = [
		    1.0,  1.0,  1.0,  1.0,    // white
		    1.0,  0.0,  0.0,  1.0,    // red
		    0.0,  1.0,  0.0,  1.0,    // green
		    0.0,  0.0,  1.0,  1.0     // blue
		  ];
		  squareVerticesColorBuffer = gl.createBuffer();
		  gl.bindBuffer(gl.ARRAY_BUFFER, squareVerticesColorBuffer);
		  gl.bufferData(gl.ARRAY_BUFFER, new Float32Array(colors), gl.STATIC_DRAW);

As you can see, this is very similar to how I created the points for the square. The number of colours has to be the same as the number of points in an object.

3D Objects

Theres are no 3D primitives in OpenGL, no cubes, no spheres, no cylinders, every 3D object has to be created by using vertices. So to create a 3D cube I had to create vertices for 6 different squares and place them accordingly within the 3D space. In OpenGL all 3D objects have to be constructed from 2D planes, which are placed together tightly, to make the objects look solid. There’s no magic for creating 3D objects, just lots of code.

Animation

Im the demo application I am rotating a 3D cube. In order to rotate the object OpenGL I had to apply another matrix, on top of the perspective matrix, to the object’s vertices inside the vertex shader, this matrix is called a transformation matrix. The transformation matrix modifies the position, rotation and scale of the object’s coordinate system and, as a result, transforms the object.

This is the code I am using to set the rotation matrix of the cube:

mvRotate(rotation, [1, 0, 1]);
function mvRotate(angle, v) {
  var inRadians = angle * Math.PI / 180.0;

  var m = Matrix.Rotation(inRadians, $V([v[0], v[1], v[2]])).ensure4x4();
  multMatrix(m);
}

The mvRotate function accepts an angle of rotation and a vector representing the axis of rotation. It then uses some helper math function from a math library called Sylvester to generate that matrix. I increment the rotation variable on each frame so that the angle of rotation increases, causing the cube to spin. The translation matrix could affect many objects at once or just one object, depending on how many objects are rendered using the matrix.

This is the part of the shader code that applies the transformation matrix (it’s called uMVMatrix)

gl_Position = uPMatrix * uMVMatrix * vec4(aVertexPosition, 1.0);

Image Texture

Applying an image texture onto a 3D object requires a few extra steps. We need to first load the image into the OpenGL buffer and tell it how to treat it (wich scaling functions to use when scaling the image). Then we need to tell OpenGL how to place the image onto an object through an array of texture coordinates. We then need to use modify the shader to grab the colour for each pixel on the object from the image.

This is the code for loading the image from a url to the OpenGL buffer:

// This function loads the image from a url, when the image finishes loading an OpenGL texture gets created
function initTextures() {
  cubeTexture = gl.createTexture();
  cubeImage = new Image();
  cubeImage.onload = function() { handleTextureLoaded(cubeImage, cubeTexture); }
  cubeImage.src = "images/fridge.jpg";
}

function handleTextureLoaded(image, texture) {
  gl.bindTexture(gl.TEXTURE_2D, texture); // Bind texture
  gl.texImage2D(gl.TEXTURE_2D, 0, gl.RGBA, gl.RGBA, gl.UNSIGNED_BYTE, image); // Create texture from image
  gl.texParameteri(gl.TEXTURE_2D, gl.TEXTURE_MAG_FILTER, gl.LINEAR); // Scaling functions for enlarging
  gl.texParameteri(gl.TEXTURE_2D, gl.TEXTURE_MIN_FILTER, gl.LINEAR_MIPMAP_NEAREST); // Scaling functions for sizing down
  gl.generateMipmap(gl.TEXTURE_2D);
  gl.bindTexture(gl.TEXTURE_2D, null); // Complete
	resetBuffers();
}

This is the code for specifying texture co-ordinates. Pretty much we’re just telling OpenGL how to place the image on the object. These points mean that the corners of the image should be placed on the corners of the square.

var textureCoordinates = [
// Front
0.0,  0.0, // Top left
1.0,  0.0, // Top right
1.0,  1.0, // Bottom right
0.0,  1.0, // Bottom left
// Back
0.0,  0.0,
1.0,  0.0,
....

This is a part of the new shader code that reads the colour of each pixel in the image and applies it onto the object:

gl_FragColor = texture2D(uSampler, vec2(vTextureCoord.s, vTextureCoord.t));

Conclusion

WebGL provides a direct access to OpenGL, without much abstraction (I refer to abstraction as helper classes and functions that help create primitive 3D shapes, lights and provide ways to easily add colour and texture to objects). This low-level access to OpenGL functions, through WebGL, helped me to get a better understanding of how OpenGL works.

While OpenGL is very powerful, it also requires a lot of code, especially the process of creating objects and adding colours and textures to those objects. I think it is more productive if those tasks are done through abstracted classes and helper functions. Game engines and frameworks like Processing and OpenFrameworks already do these abstractions, and do them well. The main benefit of digging deeper into OpenGL comes from mastering the shading language. Since shaders offer a powerful, hardware optimized, way to manipulate how objects get drawn to the screen, learning the OpenGL shading language is a critical skill to have when working with real-time 3D graphics.

Unfortunately, during my research I barely had a chance to learn how to create custom shaders, since all my time went into learning how to create objects and setting up OpenGL for rendering. My next step is to get a better understanding of shaders and attempt to create my own. I will use a framework for dealing with object creation, such as OpenFrameworks, so that I can focus just on OpenGL shader development.

References

• WebGL Tutorial by Mozilla Developer Network
• WebGL Tutorial from learingwebgl.com
• Official Web GL Specification
• OpenGL SuperBible: Comprehensive Tutorial and Reference (5th Edition)

Motorized from Max Rusan on Vimeo.

Storyboards

Motorized is a one-minute, video installation. The video demonstrates the role of digital displays within a typical urban city in North America. Motorized focuses on the fast-paced human traffic in the core of the city. The video shows the city’s dependence on electronic displays to keep the constant flow of people under control. Through the juxtaposition of video clips showing both, human traffic and electronic displays used in downtown Toronto, as well as vehicle traffic, which is a more familiar system of flow control, the video demonstrated the relation of electronics and the control of human traffic.

The first part of the video, which is before 3 white arrows fill the screen, shows a car moving from a suburban area into an urban area of the city. This is done through the use of video clips that show a view from within a moving car. The side view from a window, which shows an area filled with a lot of trees and lake Ontario in the background, along with the view from the front window, which shows the car driving through the same area, clearly demonstrate that the car starts off driving in a suburban area. Consecutive clips, like the view of underneath the bridge and the view from the front window showing a saturation of buildings on both sides of the road indicate that the car is going into an urban area. The contrast between the suburban area and an urban area exaggerates the abundance of digital displays within the core of the city, since the occurrence of digital displays in a suburban area is minimal. The 3 digital arrows mark a transition from suburban area to a busy urban area. Right after the 3 white arrows fill the screen, an establishing shot of Yonge and Dundas square is shown and the viewer is places in the core of Toronto. To make the digital presence within the busy downtown core more obvious, the screen is filled with digital scrolling text. A flow of human traffic is shown after another digital display is shown, a digital clock. The flow of human traffic is sped up to focus the viewer’s attention on the actual flow, and not on individual people. This juxtaposition of human traffic and electronic displays show the role of electronics in controlling the flow of people within a busy city environment. The crosswalk countdown in the end of the video, further illustrates how people are controlled by digital displays just like the end of the video is controlled by the same countdown.

Through Motorized I want to remind the viewers of how important digital displays are in a busy city like Toronto. The displays control the flow of the city as well keep the people safe. Do demonstrate my concept I chose to use time-based media since it was the best media to show the flow of human and vehicle traffic and to convey the dazzle of digital displays. It is important to notice how these electronic devices became such a significant part of North American urban society.

Pamela vs. Courbet, L’Origin Du Monde by maxtherocket

A podcast review of a photo mosaic work by Alex Guofeng Cao entitled Pamela vs. Courbet, L’Origin Du Monde.

Dr. Strangelove Dr. Strangelove by maxtherocket

A podcast reviewing a work by a Toronto artist Kristan Horton entitled Dr. Strangelove Dr. Strangelove

Pong Prom by maxtherocket

In this podcast I review an interactive performance called Pong Prom by a Toronto New Media artist, Edward Keeble.

Check out Edward’s website: http://edwardkeeble.com/projects/pong-prom/

Ghostbusters by maxtherocket

Ghost Beat re-creates the Ghostbusters theme song by Ray Parker Jr. using sounds of whistles, hums, knocks and kicks that I recorded myself. I used the actual notes of the song so the tempo and instrument notes match the original song, but the sound of the instruments sounds completely different. With so many choices of tools for creating music the only limitation is creativity of the creator. Through this work I am hoping to demonstrate that anything can be used as an instrument when creating music, all the sounds you need to create music can be found within your room or office and the ability to create sounds using you mouth should not be overlooked. By using a song that most people are familiar with and by avoiding the instruments used in the theme song — by avoiding instruments altogether — I am hoping to make it perceptible that it is possible to make music by using only recorded sounds.

It is difficult for the listener to recognize the Ghostbusters theme melody from the first 10 seconds the Ghost Beat. I did this on purpose so the listener is forced to concentrate on the actual sounds used, instead of the melody. The sounds used in the beginning of the song consist of hums and a recording of a homeless person shouting. I made the sound for the drum beat of the song by taping on my desk with my finger. For the melody of the song I used a recording of me whistling. The rest of the underlying sounds are a combination of hums and whistles.

Ghost Beat uses midi data from the Ghostbusters theme song to trigger recorded sound samples to play, demonstrating, to the listener, the creative possibilities of using software to create music. After listening to Ghost Beat the listener should be encouraged to explore the idea of making music with sounds that are present in everyday life. It is important to point out to the public that making music is simple and there is no need for a musical instrument, Ghost Beat underlines that creating music is accessible to anyone.

View the site

This is the website for an annual 4D Events party at the Guvernment. This year I wanted to give the site a look and feel of love, happiness, and a little hippie. I used a lot of circular shapes, smooth curves and an explosion of vibrant colors. Doesn’t this website make you happy when you look at it? Doesn’t it scream Unity? Well it should.

I used Papervision 3D for the Unity logo at the top, it moves with your mouse . I thought the video on the background is a good way to show what the party is all about, which worked out well. Since the site if full page flash it was a quite challenge to figure out how to move all the elements on stage when the window gets resized, not to mention resizing the video. But I love challenges.

View the Project

This is application that displays random, most interesting photos from Flickr. For each photo the application searches for a YouTube video that matches the photo’s tags and plays the sound from the YouTube video. Basically the application created a soundtrack for a photo being viewed.

I had to create a database of suitable photo tags for searching videos on YouTube. Most tags in Flickr photos are specific to photographs only like color, camera models and photography techniques. I had to hand-pick the tags that would be appropriate for YouTube video tags.

I created this app as a project for my application to Ryerson University. The interface is kinda weak, I was tight on time when I made it. I am planning on creating a cooler interface for this. Maybe something with 3d, a sound spectrum and an option to view the actual video that the sound is taken from.

This is an interactive TSN Broadcast Museum exhibit for the Hockey Hall of Fame. The exhibit is on the history of TSN’s hockey broadcasting. The project combines video, audio and images in a 3D interface. I used Papervision3D for the 3D interface and coded everything in Actionscript 3. The exhibit is controller by gesture recondition. I didn’t have to implement the gesture recondition technology myself, it was already in place. i just had to work with basic mouse events. But the gesture recodnition system is just awesome.

View The Project

Here’s a pretty good description of CASA Lib from its website:

CASA Lib is a flexible ActionScript library designed to streamline common chores and act as a solid, reliable foundation for your projects. It provides a core set of classes, interfaces, and utilities to get you coding faster and more reliably without getting in the way.

If you work with actionscript YO MUST GET THIS LIBRARY! I ran into CASA Lib on he web a while ago and I can’t believe I used to work with actionscript without this library. CASA Lib has saved me a lot of time, it takes care of the code that you have to type over and over again like a zombie. Save yourself time and use CASA Lib!

I’m sure there are other actionscript libraries out there, if you know of any good ones like CASA Lib post them in the comments.