If you happen to have one of these 720KB floppy disks lurking somewhere in your Sierra adventure game collection, then chances are you are not alone. Versions 2.0D and 2.0F of Space Quest II were not exactly uncommon.

A simple directory listing

Listing the files on the disk doesn’t reveal anything unusual:

The listing shown is from version 2.0D. There are no mysterious additional files, in fact it looks like any other Sierra game disk. Nothing is out of place.

The timestamps show that the game’s main data files, i.e. PICDIR, LOGDIR, VIEWDIR, SNDDIR, VOL.0 and VOL.1, were built on the 14th March 1988. The .OVL files are dated the 15th March 1988, and the AGI interpreter code itself is dated the 18th March 1988. These file timestamps capture a week of activity during which someone in the Sierra office was focused on preparing the 2.0D version of Space Quest II.

One thing that is a little curious about the directory listing above is that the “free” space on the disk is greater than the used space. 302,918 bytes are used and 402,432 bytes are supposedly free.

Using a hex editor

To take a closer look at what is on the disk, and to peek into that supposedly unused part, we need a tool called a hex editor. Back in the 1980s, a commonly used tool for this came with Norton Utilities. These days a great modern equivalent is the excellent HxD Hex Editor written by Maël Hörz.

For a freshly formatted DOS floppy disk, we’d expect all of the unused sectors to be filled with the 0xF6 byte value (i.e. DOS’s default format ‘filler’ byte), which is indeed the case for Disk 2 of Space Quest II version 2.0D, as shown below:

There isn’t, however, a single unused sector on Disk 1 that is filled with the 0xF6 byte, in fact the longest string of consecutive 0xF6 bytes on Disk 1 is only two. Given that over half the disk is “free” space, then clearly the disk had not been formatted prior to being prepared as a SQ2 master disk. The following shows an example of a section of the SQ2 Disk 1 disk that is marked as unused:

Rather than being filled with the 0xF6 byte, it is instead filled with something that looks like C source code. This strongly suggests that the master disk was used for some other purpose prior to being the SQ2 Disk 1 master disk. The files were then deleted, but the disk was not properly formatted afterwards. Due to the way the DOS FAT file system works, deleting a file does not actually remove the data. Instead it simply marks the sectors as no longer being in use, so that in the future they could be used for a new file. If those sectors are never used again for a new file, then they will keep what they previously had stored on them, and so it was important that people fully formatted a floppy disk after storing sensitive files on them.

AGI interpreter source code

It is difficult to see exactly what the above data looks like when viewed in a hex editor, but it clearly looks like text, so let’s copy and paste the ASCII text on the right hand side into a text editor to see what we have:

As we can see, it certainly does look like C source code. There are two functions defined, one called DisplayStatusLine and another called StatusLineOn. In the case of DisplayStatusLine, it appears to be displaying a line of text that includes the current score and whether the sound is on or off. Does this look familiar? You bet it does. Take a look at the top of the Space Quest II screen shot below:

That particular piece of C source code is responsible for displaying the white status bar at the top of the game screen, with the black text that shows the user what their current score is. This source code is a part of the AGI interpreter itself!

This is just one small example. Scrolling through more of the unused sectors in the hex editor reveals that there is a large amount of such source code. Not only that but the code is stored over consecutive sectors, i.e. it is not fragmented as you might expect, so it is relatively easy to extract this data from the disk and then split it up into separate C source files. The split points are easy to determine, since each file has a comment at the top saying what the name of the source file is. Splitting into the separate files gives us 93 files in total, made up of 75 C source files, 16 assembly language source files, and 2 DOS BAT files:

In total, there are over 15000 lines of code and most of these files are complete.

It turns out that this Space Quest 2 game disk contains about 70% of the source code of Sierra On-Line’s AGI interpreter, complete with comments and change history. We will look at how that 70% figure was calculated later in the article.

When we say “the AGI interpreter”, what we are referring to is the file named “AGI” in the directory listing that was shown earlier, along with the the .OVL files. The “AGI” file is the executable that performs the actual interpretation of the game data contained in the LOGDIR, PICDIR, VIEWDIR, SNDDIR, OBJECT, WORDS.TOK and VOL files.

Source file change history

Some of the source files contain a change history within the header comment at the top. The following is an example from the ANIMATE.C source file:

The header comment starts by stating the name of the source file. It then gives a short description of what it does, in this case to “Handle one cycle of animation in an adventure game”. Next we have a line that says “compile: MWC”. This appears to state the name of the C compiler that was used to compile the C source. MWC was a C compiler from the Mark Williams company that was very popular in those days. After that, the header has a “Change History” section and this makes for very interesting reading. It mentions the date, time, initials of the person making the change, and a description of the change.

The programmers have identied themselves using their initials, including their middle name initial. JAS is none other than Jeff Stephenson, the main programmer who was working on the AGI interpreter code, and DCI is Chris Iden. If you look at the game credits for the AGI games from 1985-89, you will see that Jeff and Chris were always credited with working on the game development system.

Robert Heitman is also listed, but his focus was mainly on the graphics tools (i.e. the Picture Editor (PE) and View Editor (VE)), whereas Jeff and Chris worked primarily on the interpreter code. So it isn’t surprising to see their initials in the change history for the C source code of the AGI interpreter.

AGI.EXE memory map

In addition to the 93 AGI interpreter source code files, the SQ2 2.0D 720KB Disk 1 disk also contains over 2000 lines of a Memory Map of the AGI.EXE executable. For production releases of AGI games, the interpreter executable is called simply AGI, and is not directly executable, but during the development of the games, Sierra On-Line used a version of the interpreter that had the .EXE extension and could be run directly. Someone at Sierra generated a memory map of the AGI.EXE executable, i.e. the AGI interpreter, on the 7th October 1987. A small section of it is shown below:

This gives a potential date to the collection of source code files. It ties in with the most recent change history comment in the code, which is from September 1987.

The memory map also gives us a fairly complete list of the modules and source files that make up the AGI interpreter. Each module specifies the file that it is defined in. If we count the number of distinct source files mentioned, it comes to 98. The number of those source files that are present in full on the SQ2 disk is 71. This is where we get the figure of roughly 70% for the amount of AGI interpreter source code that is on the SQ2 disk. For some of the modules, only the C header file is included, and so those modules are not included in this calculation.

Sierra’s intellectual property

In 1984, Sierra On-Line was lucky to survive as a business. Some time around the release of King’s Quest, in May/June of that year, Ken Williams had to lay off around 100 of his employees, reducing the head count from approximately 130 down to about 30. Although it was a struggle, part of what turned their fortunes around was the success of the AGI adventure game system and the games that were built using it. By the end of 1984, King’s Quest had entered the top 20 on the software sales charts for computer games. It would stay there for the next half year, right up until the release of King’s Quest II, and it wasn’t long until King’s Quest II was also in the charts. This was helped a lot by the deal that Sierra made with Tandy Radio Shack to sell Tandy versions of the games in the Radio Shack stores.

The AGI games continued to be best sellers from 1985 to 1988. Over those years, the AGI interpreter was Sierra On-Line’s primary means of making money and therefore a core part of their intellectual property. They not only invented the 3D animated graphic adventure game genre but also had a head start on their competitors that lasted several years. It is safe to say that the source code for the AGI interpreter is something that Sierra would have preferred didn’t fall into the hands of their competitors.

For 70% of the source code to end up being copied en masse and sent out to tens, if not hundreds of thousands of their customers, was a big blunder.

How did this happen?

When a new release version of a game was prepared by Sierra, it involved creating a “production copy” master disk to be used by the FormMaster disk duplication machine. This machine did not simply copy files from the the master disk to each copy but instead copied every byte of every sector of the disk, regardless of whether the sectors were currently being used or not. What this meant in the case of the version 2.0D and 2.0F SQ2 Disk 1 disks is that it also copied the supposedly unused 402,432 bytes, despite there being no reason to. This is why the preparation of a master disk involved ensuring that it started out fully formatted before the game was copied on to it, and for the most part, Sierra On-Line got this step right. Most original Sierra game disks were formatted before use. It would seem that someone forgot to do this for the Space Quest II version 2.0D Disk 1 disk, and the same disk was then also used for version 2.0F of SQ2.

This meant that potentially hundreds of thousands of SQ2 disks sent to customers and retail stores had a copy of 70% of the AGI interpreter source code hidden on them.

Dodged a bullet

It was almost certainly an unintentional mistake. Surprisingly, no one appeared to have noticed that this happened, not Sierra, not their competitors or their customers, and it was only discovered decades later, the first known discovery of it by online user NewRisingSun in October 2016.

It is also fortuitous that this happened at the end of the AGI era. In March 1988, Sierra had already developed their SCI adventure game system and were about to release the first game using it, which would be King’s Quest IV. So accidentally leaking the AGI interpreter source code into the public, should someone have noticed it, may not have been the issue it could have been only a year or two before that.

It makes for an interesting digital archeaology story though, and it allows us, 36 years later, to see how our heros at Sierra On-Line wrote their AGI interpreter.

For those who are interested, I have uploaded the extracted AGI interpreter source code into a github repo: https://github.com/lanceewing/agi

Check out AGILE, the web based AGI interpreter, whose implementation was guided in part by the original AGI source code.

🎂 Happy 40th Anniversary to King’s Quest!! 🥂

Give it a try!

Note that this isn’t a conversion of the original games into another form, so is quite different to sarien.net in that regard. AGILE is instead a nearly 100% accurate implementation of the AGI interpreter that was used by Sierra to run those games, and being so, it will run the games from the original game files that you may still have lying around somewhere from when you first played them back in the 80s!

Most of the original Sierra On-Line games are still available for purchase online, so for legal reasons, AGILE does not come prepackaged with those games. You must have your own copy and use the import feature to load the game into AGILE. It supports importing from both a folder containing the game, or a ZIP file containing the game. After it has been imported, it will remain in your browser’s storage so that you won’t have to import it again. To purchase the original games online (should you no longer have them on your bookshelf), check out gog.com and Steam.

How to run the games

Start by going to https://agi.sierra.games

AGILE should run on most modern web browsers. It does, however, rely on some browser APIs that are relatively recent, such as the SharedArrayBuffer and Origin Private File System. It has been tested on Chrome, Edge, Firefox and Safari, both the desktop and mobile versions. If it doesn’t work for you, then check to make sure that you have updated your browser to the latest version.

It starts by showing the title page, as shown below:

There is a small question mark icon in the top right that pops up a dialog with a brief description. And take note of the pagination indicator dots at the bottom of the screen. The screen works in a very similar way to the user interface of a mobile device. If you are accessing the website on a touch screen device, then you can swipe to the right to get to the next page. If you are on desktop, you can use the right arrow key, or drag/fling with your mouse, or click on the small right arrow at the bottom of the screen.

The first page to the right has thumbnails for the original Sierra On-Line games. Notice that they are all faded out:

This is to indicate that they are not imported yet. As mentioned above, this is for legal reasons. If you click on one of these, it will open a dialog telling you that you must import your own copy of the game for legal reasons. It also asks you if you would like to import from a ZIP file or a folder. After completing the import process, the thumbnail for the game will no longer be faded out. The game is now imported into the OPFS storage in your browser. If you click on the thumbnail again, it will run the game!

In addition to supporting the import of original Sierra games, AGILE comes pre-packaged with over 100 fan-made AGI games and demos; simply swipe again to the right to see several pages of them:

These are games that fans of Sierra On-Line built themselves to run on the same AGI interpreter system and so they will therefore also run on AGILE. I have included them so that they and their authors can share in the 40th anniversary celebration. For those games that do not have in-game credits, please refer to the following web page to see who the authors were:

Fan AGI Release List

Be sure to check out Let Them Eat Cake, a new fan-made game written by Russ Danner to commemorate the 40th anniversary of King’s Quest! It’s well worth a play, and is a great tribute to the original game.

How does it work?

The story of AGILE begins several years ago when I first wrote a version of the interpreter in C#. That version runs as a desktop application. Development on the new web version of AGILE began in November 2023, in hopes that it would be fully complete by the 10th May 2024, to be a tribute to, and to coincide with, the 40th anniversary of the release of King’s Quest.

It started out as a port of the C# AGILE Sierra AGI interpreter project to Java, using the libGDX cross-platform development framework, targeting Desktop, Android and HTML5/GWT. The AGILE interpreter code itself is a straight conversion of the C# code to Java, but for the “AGI Library” part, it is instead using a stripped down version of the JAGI project, which was already written in Java and already had code for loading AGI resources. JAGI was originally written by a mysterious author known as Dr Zoltan and was further extended by myself and Mark Yu.

The reason why the libgdx framework was chosen (and why the JAGI bit is stripped down to the bare minimum), was to get it working as a web app, by primarily targeting the GWT/HTML libgdx platform. GWT (i.e. the Google Web Toolkit) is used by libgdx to transpile the Java code into JavaScript, thus the reason why it is written mostly in Java but is able to run on the web. Some parts had to be written in native JavaScript, where libgdx and GWT didn’t provide an out of the box way of doing something.

JavaScript is by default single threaded, which isn’t compatible with how AGI blocks waiting for input in some scenarios. The browser tab would simply hang. To address that issue, a web worker was needed, as it allows code to be run outside of the browser’s main UI thread. Unfortunately, libgdx and GWT did not provide direct access to that, but by using a project called gwt-webworker written by Manfred Trammel, I was able to get libgdx to support running the code in a web worker. I then used the SharedArrayBuffer as the primary means of communication between the browser UI thread and the web worker, by both sides having shared simultaneous access to certain parts of the AGI interpreter state.

For those who are interested, the source code can be seen here:

https://github.com/lanceewing/agile-gdx

I hope you all have fun trying it out and reminiscing about King’s Quest and the other AGI adventure games!

And finally, a big thank you to our heroes who 40 years ago created King’s Quest. To Ken & Roberta Williams, Arthur Abraham, Charles Tingley, Greg Rowland, Ken MacNeill and Doug MacNeill: Thank you for bringing to life such an amazing and enduring concept, and a game and franchise that we will continue to remember for decades to come.

My goal for the 2020 js13kgames contest this year was to use them for as much as I could.

Scene Composition

Composing a scene solely with emojis isn’t something that is 100% achievable. We can use things like buildings, trees, and vehicles to build up a scene, but there are no emojis that we could use for the background; well, not ones that are consistent across all platforms. Perhaps the cityscape emojis could be useful as backgrounds in some situations, but what we really need is something more generic.

We have to start with some kind of default background. Since most of my locations were outside, and with buildings, I decided a red brick wall was something that most would have in common. I also added a blue sky, with a gradient, some green grass, a footpath and a road.

The above is all created with CSS, most of it as a single linear gradient. On top of this, I added the emojis that made up the current scene:

All of these larger sized emojis, that are used to compose the background scene itself, are drawn on individual canvas elements using the renderEmoji method shown in the first part in this series. The choice for drawing them on a canvas, rather than simply as text, was originally to support Windows 10 emojis, as discussed in that post.

As the game progressed, I discovered that there were other benefits to having them drawn on a canvas, which we will cover in a later section of this article entitled “Determing the thing”.

Different Shades

Sometimes the default appearance of an emoji, such as a tree, gets a bit monotonous. In “The King’s Missing Page”, I use a row of trees in the background for most of the scenes. This looked a bit plain when there wasn’t some kind of variation:

Applying a CSS filter to darken one of the rows of trees makes it look a lot more interesting:

.dark {
  filter: brightness(0.4) saturate(3);
}

The same technique could be used to apply some kind of variation to any of the emojis.

The unicode standard itself supports variations to certain types of emoji, such as those showing skin colour, using something called “modifiers”, but for a tree, a building or a vehicle, there isn’t such a mechanism at the current time, which is why a CSS filter is useful to introduce a bit of variation.

Point and Click Interface

“The King’s Missing Page” uses an old school graphic adventure point-and-click user interface, similar to those seen in some of the Sierra On-line games, such as King’s Quest 5 and 6. It is very easy to create such an interface using emojis, due to there being emoji characters that look virtually the same as the icons that were used in those old school games:

A walking person, an eye, a hand, and a speech bubble. These are all familiar to those who have played such classic games:

I designed my game so that only five action verbs were required: “Walk to”, “Look at”, “Pick up”, “Talk to” and “Use”. Four of these are represented by the four icons on the left of the image above.

The rest of the “action” bar is taken up by the inventory, a scrollable area inside which emojis are also used. The unicode standard includes emojis that cover a wide range of different inventory items that could be found in an adventure game, and they all have the look that you’d expect from such a game. It’s a perfect source for item images.

For all of the emojis on the “action” bar, whether they be verb icons, or item icons, they are rendered simply as emoji text.

<div id="commands">
    <span data-name="Walk to">🚶</span>
    <span data-name="Look at">👁️</span>
    <span data-name="Pick up">🤚🏼</span>
    <span data-name="Talk to">💬</span>
</div>

<div id="itemlist">
    <span data-name="map">🗺️</span>
    <span data-name="rose">🌹</span>
    <span data-name="tulip">🌷</span>
    <span data-name="briefcase">💼</span>
    <span data-name="bank card">💳</span>
    <span data-name="water pistol">🔫</span>
    <span data-name="cash">💵</span>
</div>

There is no need to render these ones on a canvas as was done for the scene composition. These action bar emojis are drawn a lot smaller, making the black outline seen on Windows emojis far less noticeable.

Mouse Cursors

Whenever a verb icon is clicked, the mouse cursor changes to match that verb icon. Likewise, when an inventory item is clicked first (i.e. not after clicking on a verb icon), then the “Use” verb is implied, and the mouse cursor changes to match the inventory item.

All of these mouse cursors can be rendered using emojis. When the game first starts up, it renders the various mouse cursors up front, using the same rendering code used for rendering the sprites on the screen (the renderEmoji code can be seen in part 1 of this series). A mouse cursor uses a much smaller emoji size though.

The following code shows how the mouse cursors are rendered:

// Initalise the mouse cursors.
// Note: Firefox ignores custom cursors bigger than 32x32 when near the Window edge.
let cursorSize = navigator.userAgent.match(/Firefox/)? 32 : 50;
this.cursors = {};

['🚶','🤚🏼','⬆️','💬','⬇️','⏳','↖️','👁️','⬅️','➕','↗️','🤏🏼','➡️','❔','↙️','🔍','↘️']
).forEach((c,i) => {
    // Calculate hotspot Y position for cursor.
    let hotspotY = [cursorSize-1, cursorSize/2][i % 2];
    // Render emoji at cursorSize and build CSS custom cursor definition.
    this.cursors[c] = `url(${Util.renderEmoji(c, cursorSize, cursorSize)[0].toDataURL()}) ${cursorSize/2} ${hotspotY}, auto`;
    // Add custom property to body tag with name matching emoji char and value the CSS cursor value.
    document.body.style.setProperty(`--${c}`, this.cursors[c]);
});

The rendered cursors are stored in two different places: One place is in an instance variable within the game, and the other place is as a CSS custom property on the document body tag. The latter case is so that CSS declarations can refer to these cursors.

Cursor Size Restrictions

As mentioned in the comments included in the above code snippet, Firefox behaves strangely when a mouse cursor is greater than 32x32. This is for security reasons and is discussed in this Mozilla ticket. The security concern is that large mouse cursors can be used to deliberately trick users when drawn over the native UI of the browser, something that tech support scammers have taken advantage of.

In Firefox, if the cursor is bigger than 32x32, and it moves close to the edge of the window, then the custom cursor disappears. When I say “close to”, it appears to happen when any part of the cursor starts moving outside of the window, rather than when the hotspot moves outside of the window. This is really quite noticeable. For a 50x50 custom mouse cursor, it switches to the default pointer cursor and its quite obvious that the hotspot of the pointer is still 25 pixels from the edge.

This is unfortunate, since rendering color mouse cursors at a size of 50x50 looks much nicer than 32x32.

Chrome also has this ticket related to the same issue, but they appear to have handled it in a different way. For cursors larger than 32x32 in size, the cursor doesn’t disappear until the hotspot moves outside of the window. This is more in line with what the user would expect. By playing the game, the user becomes familiar with where the cursor hotspot is, which is why the behaviour in Firefox is quite jarring.

Due to the Firefox behaviour, “The King’s Missing Page” deliberately renders mouse cursors in Firefox at a size of 32x32 to avoid the issue, whereas for all other browsers, it uses 50x50.

Cursor Changes

The default verb action in the game is “Walk to”. Whenever a sentence is composed and execued, e.g. “Look at chipmunk” or “Pick up tulip”, the verb switches back to “Walk to”. This verb is a special verb, in that it behaves both as the “Walk to” verb, but also in some cases as if no action verb is selected, such as when the mouse is over the inventory items. The player obviously can’t “Walk to” an inventory item, so in this scenario if an inventory item is clicked on, the “Use” verb is assumed.

When the currently active verb is “Walk to”, then the cursor also changes to various arrows when the mouse passes over certain things on the screen, for example, changing to an upwards arrow when over a building, or downwards arrow when over the road. Certain parts of the footpath should change it to diagonal arrows or left/right arrows.

The mouse cursor also changes to an hour glass (⏳) when user input is disabled.

// Update cursor. If input disabled, shown hour glass; otherwise verb icon.
let newCursor = this.cursors[this.inputEnabled? this.verbIcon : '⏳'];
if (newCursor != this.currentCursor) {
    // If the cursor has changed, update cursor on wrap element.
    this.wrap.style.cursor = newCursor;
}
this.currentCursor = newCursor;

The above code shows the case where we simply update the cursor on the main wrapper element whenever it needs to change. It doesn’t show how the unique behaviour of the “Walk to” verb is handled.

Custom CSS Property Fallback

I struggled for a while to get that particular part of the mouse cursor behaviour to work as I wanted it to. The problem was that I wanted every other verb icon (i.e. the eye, hand and speech bubble) to always be the mouse cursor regardless of what element on the screen the mouse was moving over. Those particular cursors had higher precedence. But for the walking icon, it had the lowest priority and the mouse cursor needed to instead respond to the properties defined in the CSS.

I knew that putting “cursor” CSS rules for the various elements I wanted the cursor to change on was a key part of the solution. But it needed to be conditional on what verb was currently active. I started out by thinking that I could have a single custom CSS property called “–c” that the Javascript could update based on the requirements I’ve described. There just didn’t seem to be a way to make that work though.

I then remembered that the CSS var function supported a fallback value. This got me thinking: If the “–c” custom CSS property was set only when the active verb icon was not “Walk to”, then I could make use of the CSS variable fallback mechanism.

if (this.verbIcon != '🚶') {
    this.wrap.style.setProperty('--c', newCursor);
} else {
    this.wrap.style.removeProperty('--c');
}

With the above bit of Javascript in place, the CSS could then look like the following to achieve what I wanted. Each area of the screen that required this behaviour could define a “cursor” property value with “–c” as the overriding CSS property value, and a fallback to a specific cursor relevant only to that element:

.building {
  cursor: var(--c, var(--⬆️));
}
.road {
  cursor: var(--c, var(--⬇️));
}
.left_path {
  cursor: var(--c, var(--↗️));
}
.right_path {
  cursor: var(--c, var(--↖️));
}
.left_path.down {
  cursor: var(--c, var(--↙️));
}
.right_path.down {
  cursor: var(--c, var(--↘️));
}

Determining the ‘thing’

“The King’s Missing Page” has a section of the screen that shows the sentence that is being constructed to tell “Detective Pip” what to do. The sentence is formed from the verb, e.g. “Look at”, and a thing, e.g. “tree”.

The “thing” part of that sentence is usually a sprite within the current scene. These sprites are instances of custom x-sprite elements and contain a canvas element. Both of these elements are rectangular in shape, which is true of most HTML elements.

A mouse move event handler is registered with each of the sprites. This event is fired whenever the mouse moves over the box within which the element exists. The idea of the event handler is to update the “thing” part of the sentence, e.g. “Look at” “tree” or “Walk to” “house”.

The problem with this is that the visual part of the sprite is never rectangular in shape. For something like a palm tree, quite a lot of that bounding box is transparent. It wouldn’t be correct to update the “sentence” to include the sprite until the mouse cursor is actually over the visual part of the sprite.

In some cases, there might be another sprite behind a sprite, e.g. a building or wall behind a tree, so rather than the sentence being updated to include “tree”, if the cursor is over the transparent part of the sprite, then it should instead fallback on the sprite behind it.

The following code shows how this is done:

/**
 * Handles mouse move events, primarily so that the 'thing' property is updated 
 * when the mouse moves over objects on the screen. If the object has a canvas,
 * then it uses the image data to determine when exactly the mouse is over the
 * object. If the pixel is transparent at that position, then it falls back on
 * whatever is underneath the object.
 * 
 * @param {MouseEvent} e The MouseEvent for the mouse move event.
 */
objMouseMove(e) {
    // Get's the element that the mouse is moving over.
    let target = e.currentTarget;

    // Only certain elements in the game have a canvas, i.e. the game sprites.
    if (target.canvas) {
        // Map the mouse position to an X/Y position within sprite canvas.
        let rect = target.getBoundingClientRect(); 
        let x = ~~((e.clientX - rect.left) / this.scaleX);
        let y = ~~((e.clientY - rect.top) / this.scaleY);

        // Get the pixel data from the canvas.
        let { width, height } = target;
        let ctx = target.canvas.getContext('2d');
        let imgData = ctx.getImageData(0, 0, width, height);

        // If the pixel at the X/Y position is transparent, then get sprite underneath.
        if (!imgData.data[(y * (width << 2)) + (x << 2) + 3]) {
            // Get all elements that have a canvas (i.e. sprites) in current mouse position.
            let elements = document.elementsFromPoint(e.clientX, e.clientY).filter(s => s instanceof Sprite);
            // Index 0 is the currentTarget, so index 1 is the one immediately below.
            target = elements[1]? elements[1] : null;
        }
    }

    this.thing = target? target.dataset.name : '';
}

Elements From Point

The mouse position is mapped to a pixel position within the canvas. If that pixel is not transparent, then the “thing” will be the sprite that has received the event. If, however, the pixel is transparent, then the document.elementsFromPoint function is invoked to get all elements at the current mouse position.

The elements are in Z order, so the first one will be the sprite that received the mouse move event. The next sprite element in the list will be the one immediately behind it, from which the name of the “thing” will be taken. The code above is looking only at the element immediately behind the current target, but it could be extended to continue looking back through the element list, so as to support more than two sprites overlapping.

People

One of the main limitations of using purely emojis to create an animated point-and-click adventure game is that all the emojis relating to people are of static poses. It isn’t really possible then to use the walking person emoji for your main character, since the movement wouldn’t have any kind of cycling animation, such as moving legs and arms.

I tried to avoid drawing attention to this by using the main character that I had used in my previous games, i.e. “Pip”, which isn’t an emoji. But it left me with a problem as to how to handle the other people in the game. There are plenty of different people emojis, but almost all of them are of the top part of the body.

This gave me an idea that I could use them in a “behind the counter” scene, where only that portrait view of the person is visible. The following is an example from my game of the bride in the church:

And this one is of the hotel clerk:

This is how I handled inside locations, and I think it worked out quite well.

Favicon

As a final point, and although I didn’t do this for “The King’s Missing Page”, it would be trivial to use the same renderEmoji method shown in part one of this series to render and then set the favicon for the page. The following code is an example of how this could be done:

let favicon = document.getElementById('favicon');
favicon.href = Util.renderEmoji('👁️', 32, 32).toDataURL();

I also learnt that the frequency at which different operating systems updated their emoji fonts to support new Unicode versions varied, so that some platforms are more likely to be 3 or 4 years behind the standard, whereas others are completely up to date.

The js13kgames emoji rule

As this year’s contest was starting, I became aware of a js13kgames rule that I hadn’t previously realised was there. It reads as follows:

“you are allowed to ask users to live-load a web font to support some characters or emoji on devices that can’t display them properly, but you have to make sure your game will work without them”

This intrigued me a bit. The Twemoji font was one such font that could be used. Others include Noto Color Emoji, OpenMoji, and perhaps JoyPixels (formerly EmojiOne). The above rule was a bit vague on what was allowed though. Could I always load and use the font? That would make the experience consistent across platforms, but it felt a little bit like cheating. The rule states that “you have to make sure your game will work without them”.

I think it is clear from the wording of the rule that you need to make an effort first to use the emoji font that is available to the platform that the game is running on. One solution would be to pop up a message asking the user to choose between the platform specific font or the live loaded font. I couldn’t help feeling like this would distract from the game itself and draw attention to the fact that an external resource was being used.

Detecting the supported emoji unicode version

I decided instead to implement an automatic solution, one that would detect whether the platform supported the necessary emoji characters, and only if it didn’t would it “live load” the chosen fallback font. This felt like it was in the spirit of the rules.

The plan I had for my game initially depended on emojis up to and including unicode 12 being available, so I needed a way of detecting whether the platform it was running on supported that version. This seemed a bit tricky at first, but then I discovered that there was a unicode character (\uffff) that would always render the same as an unsupported character (￿). Different platforms render an unsupported emoji character in different ways, but a given platform will render all unsupported emoji characters in the same way.

I was therefore able to render the known unsupported character first and then compare that to the result of attempting to render the Unicode 12 character. If they matched, then Unicode 12 was not supported and it would automatically enable the fallback font.

/**
 * Detects what Emoji Unicode version is available by default.
 */
static detectEmojiVersion() {
    // The game requires at least Unicode 12 to work without Twemoji. We 
    // work out whether it supports Unicode 12 by rendering a known 
    // unsupported char and then compare it to what is rendered for a 
    // Drop of Blood, which is part of Unicode 12.
    let NO_EMOJI = Util.renderEmoji('\uffff', 100, 100).toDataURL();
    if (Util.renderEmoji('🩸', 100, 100).toDataURL() == NO_EMOJI)) {
        Util.twemoji = true;
        document.body.classList.add('twemoji');
    }
}

With that class added to the body, the game can then dynamically load the chosen fallback font. If there is a @font-face declaration block for the font, the browser will not automatically load it. But if one of the CSS selector blocks becomes active, perhaps in response to the “twemoji” class being added to the DOM in the example above, then the browser will download the font.

@font-face { 
  font-family: "twemoji";
  src: url('https://lanceewing.github.io/twemoji.ttf');
}
/* The browser will only download the twemoji font if the twemoji class is present */
.twemoji #controls {
  font-family: "twemoji";
}

By using a technique like this, the font is only downloaded when the platform the game is running on doesn’t support the necessary emoji version.

In the example above, we used Twemoji as the fallback font. As mentioned earlier, there are several other options we could have gone with and perhaps Twemoji wasn’t necessarily the best one. In order to make the best choice, I had to learn more about the various available fonts and formats, and their pros and cons.

Emoji fonts and formats

The automatic fallback mechanism described above is just that: a fallback mechanism. The normal case would hopefully be when the platform already supports the required unicode version. So let’s have a look at those “standard/default” fonts first, alongside some of the other alternatives.

The following image shows how a single emoji character appears differently depending on what font is used (images sourced from Emojipedia):

The examples on the left are the default emoji fonts on particular operating systems, whereas those on the right are not associated with any particular OS. These eight are not the only Emoji fonts in use, but at the time of writing, they are probably the main ones that are still under active development. Some are available for free use, some for a fee, and others are fully proprietary and restricted solely to certain operating systems, apps or devices.

Games submitted for the js13kgames competition are likely to encounter only the following emoji fonts: Segoe UI Emoji (Windows), Apple Color Emoji (Mac/iPhone/iPad), Noto Color Emoji (Android/Linux), and Samsung.

As we can see in the image above, the same emoji looks quite different across those four emoji fonts. The Apple, Noto, and Samsung fonts are all bitmap fonts that look like hand painted pictures. The Windows font, on the other hand, is a vector font with layers of flat colours, and has the thick black outline that was discussed at length in part 1 of this series of articles.

These are not the only differences. Each vendor has drawn the house image itself in a different way. A different number of windows. Some with a chimney, some without. Even when they have the same features, those features don’t always align in the same places. These differences mean that a game can’t rely too much on what the emoji looks like. They have to be treated in a more generic way.

Multi-color Glyphs

In addition to a different appearance, there are several standards and formats for how Color Emojis are stored in a font file. Microsoft, Google, Apple, and Mozilla/Adobe all came up with different formats, and unfortunately they’re all being used.

They all do essentially the same thing, which is to extend the standard font file formats to include color image data, but they do this in different ways.

I realised after learning about these different formats that whichever “fallback” font I used, it would need to be in a format that worked on as many platforms and browsers as possible. All four formats are now incorporated into the OpenType spec, but support varies across the different browsers and platforms.

SBIX was ruled out early on, since only Apple was using it, and their font was not freely available. SBIX is supported by Safari, Chrome and Edge, but not Firefox. Even though the format is supported in several browsers, it is only useful on Apple devices.

OpenType-SVG

OpenType-SVG was ruled out next, due to Chrome’s refusal to support it, mainly due to performance concerns. Now that Edge is using Chromium, Edge also no longer supports OpenType-SVG.

If you read up on OpenType-SVG, you’ll see that it was at one time mentioned as being well positioned to become the golden standard. It doesn’t seem like this is how things have panned out, in fact with Chromium having no plans to implement it, and Edge now using Chrominum, the browser support has actually regressed in recent times.

The following website is a good one for testing OpenType-SVG support on your browser:

http://xerographer.github.io/reinebow/

I tried the above web page while writing this article and can confirm that it was only in Firefox that the color fonts were displayed.

The following table shows what I believe the current state to be. It is a modified version of the table that appears in Ollie Williams’ article “It All Started With Emoji: Color Typography on the Web” on CSS-Tricks. All I’ve changed is the OpenType-SVG in Edge, which is no longer available:

CBDT/CBLC

CBDT/CBLC didn’t have consistent support across all browsers either. I couldn’t make Noto Color Emoji load using @font-face in Firefox no matter what I tried. By default, the sanitizer in Firefox rejects CBDT/CBLC, but there is a preference (gfx.downloadable_fonts.keep_color_bitmaps) to turn on support. Unfortunately Noto Color Emoji characters still do not show even after enabling this preference.

I was also a bit skeptical about how good a bitmap format emoji font would look like when drawn at large pixel sizes, such as 400px. Would it look too blurry? The following is an example of using Noto Color Emoji:

The castle in the above image appears quite blurry. Although not so noticeable on a mobile device, it becomes quite obvious on a desktop monitor. The bitmap image for the emoji has been stretched bigger than it is designed to be.

A vector format wouldn’t suffer from this.

COLR/CPAL

What did that leave me with? COLR/CPAL. It appeared to be supported by all browsers, and had been for some time. Microsoft had been quite cleaver in creating a format that utilised the glyph format already used in font files. It wasn’t as flexible as SVG, but it was a vector format and so did scale well. The only problem was that the Windows “Segoe UI Emoji” format was the only COLR/CPAL font that I’d heard of so far.

I couldn’t use the Windows emoji font as the fallback font. Technically though, I probably could, and I did think about it for a bit, but it wasn’t the right approach, because Microsoft didn’t allow this, so would violate the licensing section of the js13kgames rules.

Most of the free to use open source emoji fonts were not in the COLR/CPAL format. But after a bit of searching, I discovered that Mozilla had created COLR/CPAL versions of a couple of them.

EmojiOne Mozlla Font

The first font they did this with was the EmojiOne font. Firefox used to ship with a COLR/CPAL version of EmojiOne, but Mozilla replaced this with with a COLR/CPAL version of the Twemoji font back in April 2018, in response to EmojiOne no longer being open source.

For the same reason, I couldn’t really use the latest EmojiOne font (i.e. JoyPixels) either. Perhaps I could use the COLR/CPAL version that Mozilla produced before the EmojiOne licensing changed, since EmojiOne v2 was bound by an opensource license and therefore Mozilla’s conversion of v2 would also be free to use. The drawback with that though was that it would limit the game to Unicode 9, which would miss out on the past few years of new emoji characters.

Twemoji COLR font

The Twemoji COLR/CPAL font, on the other hand, appeared to be perfect for my needs. It ticked all the boxes and it looked good as well:

So this is what I choose to use as my fallback font for most of the time I was working on my 2020 js13kgames entry.

Cross Platform Testing

I did some testing on Windows, Macbook, Android and a Samsung phone. Windows 10 was using Unicode 12. After a little research, it appeared that Windows kept its emoji font up to date through regular Windows updates.

The Android and Samsung phones, assuming they were on the latest Android release, also were on Unicode 12. The following video is a full playthrough of “The King’s Missing Page” on an Android phone:

What surprised me is that the Apple Macbook Pro I tested on was only at Unicode 9. It seemed that the MacOS was not updating its emojis as regularly as other operating systems were. This was unfortunate, because the Apple emojis were argubably the most visually appealing, despite being a bitmap format and therefore prone to the blurring issue.

It was possible that there were a lot of Macbooks out there that were also several Unicode versions behind, which would mean that my game would fall back on the Twemoji font rather than showing the native Apple Color Emoji font.

I worried over this for a while, because it felt to me that potentially a rather high proportion of players out there would end up seeing the fallback Twemoji font.

I felt strongly enough about this that I decided to do a stock take of all of the emojis I was using from Unicode versions 10, 11 and 12. Perhaps I could replace those emojis with something similar from earlier Unicode versions, which is exactly what I ended up doing. Before long, the requirement to have Unicode 12 available had been reduced to Unicode 9.

EmojiOne V2 Revisited

Out of curiosity more than anything, having reduced the required Unicode verson back to verson 9, I wondered what the game would look like with the EmojiOne COLR font.

It didn’t look too bad, but it did show how emojs from different fonts can be drawn from different angles, which makes placing them on the screen, in a way that works across all possible fonts, a bit tricky. In the following example, the lorry on the left is not meant to be parked on the grass facing out. In all other emoji fonts that I’d seen, this emoji is drawn side on, similar to the vehicle on the right.

A Reversal

Having been through the process of changing the game to use only Unicode 9 or below, in the end I felt that perhaps the fallback mechanism would no longer be needed, and I could save some precious bytes that could be used elsewhere.

So despite implementing the fallback mechanism described earlier in this article, and choosing a suitable font to use as that fallback, I ended up removing it from the final version of the game. To be honest, there was still a part of me that didn’t feel like it was in the spirit of the js13kgame contest to rely on a live loaded font file, so I felt happier that I didn’t need it afterall.

After the contest had ended, I received feedback from someone who played my game who said that certain emojis were not showing on his machine. So perhaps even Unicode 9 wasn’t old enough to justify removing the fallback mechanism. I had to draw the line somewhere though.

Takeaways

The following are the key points that I took away from my time researching the various emoji fonts:

• COLR/CPAL currently has the widest support across browsers and platforms.
• The Mozilla Twemoji COLR font is therefore a good fallback font, and is up to date and free to use.
• OpenType-SVG surprisingly has the worst support across the various browsers.
• Apple (SBIX) and Google (CBDT/CBLC) fonts do not scale particularly well at the larger sizes needed to render things like full size buildings in a game.
• If you’re not using a “live loaded” fallback font, then using newer emoji characters risks them not being rendered on some people’s machines.

Part 3…

That is all for this part.

In the next part, I’ll be looking much closer at how I have used emojis in “The King’s Missing Page”, focusing primarily on the mouse interaction with the emojis.

For those who have seen my previous js13k entries, you will have seen that two of them are point and click graphic adventures. I also entered the inaugural gamedev.js jam earlier this year, with what was once again a point and click adventure.

Since I first heard of the js13kgames game jam back in 2013, I have wondered whether it would be possible to create a point and click adventure game that felt like a full and challenging game. Was 13k enough space to create something like the old Lucasfilm and Sierra On-line classics of the 80s and 90s?

My first two attempts (shown above) fell a little short of my vision, mainly because I ran out of time before I ran out of space. One thing became obvious through those attempts though, that if I wanted to add lots of rooms, they’d all end up looking nearly the same. Creating distinct looking scenes took up valuable bytes.

Emojis as an art asset

Earlier this year a solution to creating more distinct looking scenes occurred to me. I had seen games in previous years of the js13kgames contest that had used emoji characters. Looking through the full set of emojis, I could see lots of different buildings, a few different trees, people, animals, and items. Could this be the way to create a more interesting graphic adventure game?

The main problem appeared to be how they are rendered on Windows 10. As shown in the image above, emojis on Windows are rendered with a relatively thick black outline. The bigger you draw the emoji, the thicker the black outline becomes. Buildings, vehicles and trees all have ugly outlines that look out of place.

Removing the black outline

Emojis are unicode characters rendered using a particular font. The Windows emoji font is called “Segoe UI Emoji”. It isn’t possible to style that font to remove the thick black outline, because its part of the graphics that make up the colour glyph of the character.

I spent some time playing around on codepen.io to see if could draw an emoji to an HTML canvas and then manipulate the pixels to remove the black outline. The idea I came up with was to apply a pixel fill algorithm starting at an area of the canvas outside of where the emoji is and then customising the fill algorithm to detect pixels that look like they are part of the black outline and to then remove them.

The following Pen shows the final code that I arrived at. You’ll need to view this on a Windows machine to appreciate what it is doing:

See the Pen Emoji on Canvas by Lance Ewing (@lance_ewing) on CodePen.

Why use a fill algorithm? This is mainly so that areas of black inside the emoji are not also removed. It is only the black around the edge of the emoji that we’re interested in removing. So the fill floods in towards the edges of the emoji and then stops when it gets to the other side of the black outline.

Detecting the black outline

There is a bit more to it than what I described above. We might naively think that the whole of the black outline is pure black, by which I mean an RGB value of 0,0,0. Due to antialiasing with both the area outside the emoji, and the area within the emoji, the outline takes on various subtle shades and tints. This is what makes the outline look smooth when rendered, but it is a pain to try to remove.

The following code does a reasonable job:

// Get the RGBA values for the pixel at the current position.
let [red, green, blue, alpha] = imgData.data.slice(pos, pos + 4);

// Calculate the perceived brightness of the pixel.
let brightness = Math.round(Math.sqrt(
    (red * red * 0.299) + 
    (green * green * 0.587) + 
    (blue * blue * 0.114)
));

let spread = false;
if (alpha == 0) {
    // If the pixel is fully transparent, then let flood fill spread.
    spread = true;
}
else if ((red == 0) && (green == 0) && (blue == 0)) {
    // If the pixel is pure black, then let flood fill spread, and 
    // make pixel fully transparent.
    imgData.data[pos + 3] = 0;
    spread = true;
}
else if (brightness < 70) {
    // If the RGB value of the pixel is below a certain perceived 
    // brightness level, then adjust the alpha channel to match 
    // and let the flood fill spread.
    imgData.data[pos + 3] = Math.round(brightness * (255 / 70));
    spread = true;
}

The first case, where the alpha channel is zero, is simply the fill flooding into the transparent background parts of the canvas. The second case, where RGB are all zero, is where the pixel is pure black and therefore deemed to be part of the black outline, and so the fill floods into that pixel and it is set to transparent.

If we stopped there, and didn’t have the third case, then the resulting image would have a jagged edge (see tree on the left in the image below):

To avoid this, the third case takes advantage of the existing antialiasing between the inside edge of the black outline and the colourful interior of the emoji. It detects this by calculating the perceived brightness of the pixel, and then if it is below a certain threshold, it adjusts the alpha channel of the pixel to match the brightness. This has the effect of making the edge of the emoji look smooth after removing the black outline.

The threshold of 70 was determined by trial and error after testing the code with various Windows emojis.

Resizing after removal

If an emoji is drawn at a large font size, let’s say 300px, then removing the thick black outline results in an image noticeably smaller than the originally desired size.

Consider also that in order to allow the flood fill to spread the whole way around the emoji, the initial canvas size is deliberately made higher and wider than the desired height and width. Not too much though, otherwise time would be wasted filling into areas that are not needed.

As a consequence of the two factors mentioned above, after the black outline is removed there is quite a noticeable empty area beyond the four edges of the emoji that needs to be trimmed off. The canvas needs to be cropped so that its width and height exactly fits the emoji.

With this in mind, the fill algorithm also keeps track of where the four visible edges of the emoji are as it is spreading around the emoji:

if (imgData.data[pos + 3]) {
    if (x < minX) minX = x;
    if (x > maxX) maxX = x;
    if (y < minY) minY = y;
    if (y > maxY) maxY = y;
}

In the outline detection code shown earlier in the article, imgData.data[pos + 3] represented the alpha channel of the current pixel. Setting it to 0 makes it fully transparent. If after processing the current pixel it is determined to not be fully transparent, then it makes up a visible part of the emoji and therefore the current max and min X and Y values are updated. After processing the whole image, the minX, minY, maxX and maxY values give the left, top, right and bottom visible edges.

Simply cropping the canvas to this size is not enough though. The game requires an emoji to be displayed at a given width and height. So in addition to cropping, the canvas is resized in both directions:

// Redraw the canvas, so that we can remove white space and add a shadow.
let emojiCanvas = document.createElement('canvas');
let newWidth = ((maxX - minX) + 3);
let newHeight = ((maxY - minY) + 3);
emojiCanvas.width = width;
emojiCanvas.height = height;
let emojiCtx = emojiCanvas.getContext('2d');
if (newWidth > 0 && newHeight > 0) {
    emojiCtx.shadowColor = "black";
    emojiCtx.shadowBlur = 3;
    if (flip) {
        emojiCtx.translate(width, 0);
        emojiCtx.scale(-1, 1);
    }
    emojiCtx.drawImage(
        canvas,
        minX-1, minY-1, newWidth, newHeight,
        0, 0, width, height,
    );
}

In the code above, the canvas variable is the original canvas that was manipulated to remove the black outline. The emojiCanvas is the new canvas that is set to the desired size of the emoji. The detected visible edges of the emoji on the original canvas are then used to specific what part to draw to the new emojiCanvas, which is resized in each direction to the desired size, and we’ve thrown in a flip variable in case we want the emoji to face the other direction.

Note that we also have a small shadow blur. This isn’t required, but it does give it a slightly smoother and more defined edge.

Performance

There is certainly an overhead in performing this kind of image manipulation, so we need to keep it to a minimum. Sometimes the same emoji needs to be drawn many times in the same scene, such as a background of many trees of the same type and size. To address this, the game caches the image data for a given emoji and size, so that if the same combination is encountered again, in any scene, it uses the cached image data instead of performing the same image manipulation.

The end result

Removing the black outline from the Windows emojis in this way allows more attractive scenes to be created. If we render the same scene shown earlier in this post using the described approach, then the end result is as follows:

That’s much nicer!

Obviously this rendering technique is only of concern on a Windows machine, but it’s likely that many of the participants in the js13kgames contest are using Windows and will see the game like this, and since the participants are the voters, it is an important consideration.

Windows isn’t the only operating system that I had to cater for in choosing to use emojis. The topic of the next post in this series looks at some of the things I discovered about emoji fonts in relation to cross platform and browser compatibility.