In this series of blog posts, I’m doing to teach you how I did it. I assume you know a bit about regular expressions, C++ and Qt. I will try to simplify the theory behind the best I can.

Bear in mind that my method is not the most efficient way to implement a scripting language. My goal with this project was to learn and have fun, not to worry about performance too much.

Basics of compilation

Overview of the process

In its simplest form, the compilation process is the act of translating an input into another format. Your C++ compiler takes a text file with instructions in it and translate it into machine code.

A compiler needs several building blocks to understand a language and create another output. There are the lexical analysis (Lexer), the grammar analysis (Parser),  an abstract syntax tree (AST) and an Interpreter or a CodeGenerator in a compiled language. I left out the semantic analysis because in our case, it will be done in the interpreter.

What’s lexical analysis and grammar analysis ? We going to find out in the following sections.

Lexical analysis

First let’s take a English sentence and identify the various lexical class.

My father gave me a nice gift.

If you were paying any attention in our English or French class. You will identify these as

Identifying the various components of a sentence helps us in the grammar analysis by giving us a simplified representation of the sentence. If we put the sentence like this:

article noun verb pronoun article adjective noun

It is now easier to tell that our sentence is grammatically correct. If we shuffle the words around a bit like this:

father my me gave nice a gift ->
noun article pronoun verb adjective article noun

We’re still able to identify the various components but the sentence make no sense from a grammar point of view. A noun is not supposed to be followed by a article for instance.

In a programming language, we can apply the same process

void PrintDebugInformation(const char *format, int arg1)
{
    printf(format,arg1);
}

Here’s the various components

Thus the job of a Lexer is to identify lexical classes, called tokens, from the input text. Like the English grammar analysis, it will simplify the grammar analysis. Note that I named the lexical class all in caps. It helps to separate them from a grammar rule.

Grammar analysis

A grammar is a set of rules to describe a syntax, A syntax is how the lexical classes are structured together to create a cohesive and logical form.  Let’s take a C function call as an example.

printf(format,arg1);

The lexical analysis give us this:

IDENTIFIER LEFT_PARENTHESIS IDENTIFIER COMMA IDENTIFIER RIGHT_PARENTHESIS STATEMENT_END

We could formulate a rule to identify a function call like:

functionCall ::= IDENTIFIER LEFT_PARENTHESIS IDENTIFIER COMMA IDENTIFIER RIGHT_PARENTHESIS STATEMENT_END

But it assume we pass only 2 arguments to our functionCall. What if we want to support 0, 1 or more arguments ? What if we could use the zero-or-more operator(*) from regular expression ? Using the * operator, our functionCall rule looks like this

functionCall ::= IDENTIFIER LEFT_PARENTHESIS IDENTIFIER (COMMA IDENTIFIER)* RIGHT_PARENTHESIS STATEMENT_END

Okay, now we can express various function arguments, but what if our functionCall does not have arguments ? We can use another regular expression operator, the zero-or-one operator(?) on the first IDENTIFIER after the parenthesis to parse the functionCall correctly.

functionCall ::= IDENTIFIER LEFT_PARENTHESIS IDENTIFIER? (COMMA IDENTIFIER)* RIGHT_PARENTHESIS STATEMENT_END

We should be fine with our functionCall rule right ?  What if we want to pass a number to a function ? We could use the alternative operator(|) right ?

functionCall ::= IDENTIFIER LEFT_PARENTHESIS (IDENTIFIER|NUMBER)? (COMMA (IDENTIFIER|NUMBER))* RIGHT_PARENTHESIS STATEMENT_END

While it could work if we only want to support variables and numbers as function arguments, it is not flexible enough if we want to support different types of arguments like a math expression, a function call, etc. We will use another grammar rule to express the term of a functionCall

term ::= IDENTIFIER | NUMBER
functionCall ::= IDENTIFIER LEFT_PARENTHESIS term? (COMMA term)* RIGHT_PARENTHESIS STATEMENT_END

During the parsing process, the term rule in the functionCall rule will be expanded into the right lexical class. Here’s an example of a concrete parse tree of the function call defined above. As you can see, our lexical classes are all leaves in the tree. The goal of a rule is to create leaf nodes in a parse tree (conceptually of course).

The Parser job is to take a text input and validate the syntax of a language. It uses the grammar rules to define the order of parsing. But it can do more than just validate the syntax of a language. For each rule, we can add code to be executed each time the parser encounter that rule. It can be used to create the nodes in the Abtract Syntax Tree for instance.

In a next part, we’ll go deeper into defining grammar rules and how it translate to C++ code.

Abstract Syntax Tree

An abstract syntax tree is a tree thats represent operations in your language. It is called abstract because it doesn’t store the whole source code, it only store the data needed for each operation. Why we don’t store the whole information ? Because most information is redundant or only used to identify a language construct. Let’s take a variable declaration statement.

int myVariable = 3*5*x;

It generates this Abtract Syntax Tree:

As you can see, the int type, equal sign and the semicolon are not expressed in the tree. We just don’t need them. I also put the data stored in each node to show that an abstract syntax is not a dumb representation of the source code syntax, it is used for interpretation or code generation.

Interpreter

An Interpreter takes an AST, traverse it and execute an operation on each node. In a nutshell, it’s the Interpreter pattern in the GoF Design Patterns book. We could also use the Visitor pattern to visit each node and make the AST independent of the code execution mechanism (ex: interpreted, JIT, compiled)

Conclusion

In this part, I introduced you to various concepts related to compilation: lexing, parsing, abstract tree and interpretation. In the next part, we will start coding the scripting language by implementing mathematic expression in our scripting language. Don’t worry, I won’t fall in the trap of others compilations tutorial, I will go further than math expressions

What is Mayanezz ?

Mayaznezz is 2d game top-view action adventure game (think Zelda) set in a fictional Maya universe. We play the role of a young Mayan warrior pursuing the power of 6 Mayans Gods. The story outline is still to be determined. Although we know we’ll mix some elements from the Mysterious Cities of Gold in it.

Each main dungeon will be available from the beginning of the game. Dungeons will not require special items to be completed, although the items will make the dungeons much easier.

The items will be scattered across the land, either given by key people or found in secret places.

Note that Mayanezz is code name, it is not the final name of the game

The Team

Meet our team ! Each member is involved in every part of the process, except for the graphic side.

• Fred Imbeault: Programmer, ScrumMaster, Game Design, Level Design
• Jimmy Provencher: Programmer, Game Design, Level Design, Story Planner
• Bryan Bergeron: Integration programmer, Artist, Game Design, Level Design
• and me: Programmer, Level Design, Story Planner

First meetings

Each game development team (3 for this year) is assigned to a local where only the member of the teams has access. It took about 3 weeks to get our local. Thus, our first meeting was delayed a lot. During our first meeting, we got a crash course of SCRUM from Fred. For those who doesn’t know SCRUM, it’s a agile project management methodology used by many game and software company. SCRUM is focused on user stories, which are requirement based on the point of view of the user. For example: “As a player, I want to attack enemies”.

After the crash course on SCRUM, we started the product backlog. A Product Backlog is a collection of user stories that we want to get into the game. It is saved during the whole project and serve as basis for the sprint.

A Sprint is a iteration in the process. It’s usually last from 2 to 4 weeks (team choice). We chose user stories from the backlog based on our velocity. A velocity is the amount of user points we can do in a sprint.

In our cases, the Sprint 1 looks like this:

• Allow the character to navigate on a map (screen size)
• Listen to background music
• Showing a enemy with basic movements

Our sprint time is set to 4 weeks. Our velocity is quite low because we don’t work full time on the project.

Setuping the environment

A great deal of our time was to setup our environment. Each PC is installed with Windows 7 Professional, yes the final version (gotta love MSDN-AA) with Visual Studio 2008 SP1. We’ve setup our own server, which is running on Windows Server 2008. The server is running VisualSVN server for source code management, Trac+Agilo for SCRUM, ticket management and wiki, CruiseControl.NET for continous integration. As for myself, I am using DevExpress Refactor for C++ and AnkhSVN Visual Studio plugin.

Technologies used

We are coding our own engine from scratch in C++.  Most of the backend will use SDL and related libraries. We are using UnitTest++ and Google Mock for unit testing. We are planning to use Eigen for linear algebra related code. For map editing, we will modify Tiled to suit our needs. For scripting, we are going to use Lua.

As for today

The code is started ! I started working on the user story “Listen to background music”. It should be done soon enough and move to another task.

See you next time !

In order to get a feel of the code, I’m going to do some small mechanical refactorings. I will do only safe renames and extract method in the class related to Block and Piece. I will not try to alter behavior before I write my characterization tests (will explain in a later article). I am using Visual Studio 2005 and Code Rush(which include Refactor! C++) free version. Visual Assist X will be fine too but it costs money.

Let’s go deep into the code !

The first thing I noticed when reading the code related to Piece and Block was this:

void Piece::moveBlocks(int deltaX, int deltaY)
{
 for(int i=0; i<TetraSize; i++)
 {
  m_tetra[i].move( m_tetra[i].x() + deltaX, m_tetra[i].y() + deltaY );
 }
}

Why do I need to query the X and Y coordinates of each block forming the tetra in order to move them ? This is a violation of the principle “Tell, don’t ask“. I’m going to refactor in small steps (like strongly suggested in Martin Folwer’s Refactoring).

First I renamed the move method of Block class to moveAbsolute since it’s clearly what this method is doing. After, I created a new method called Block::move which really moved according to deltaX and deltaY. Then, I looked up all the calls of moveAbsolute and make then use the new move() if they required this new behavior.

I found a interesting thing. All calls to moveAbsolute are in my poor handling of the Piece. Thus in the future, we will be able to remove this method after several refactorings. This single rename make my bad code stood out of the rest.

After, my random look into the code, I saw this big function:

SDL_Surface *BlockSurfaceManager::get(BlockColor color)
{
 if( d_ptr->surfaceCache.count(color) == 0 )
 {
  SDL_Surface *newSurface = SDL_CreateRGBSurface(SDL_HWSURFACE, BlockWidth, BlockHeight, ColorDepth, 0, 0, 0, 0);

  SDL_Rect blockRect;
  blockRect.x=0;
  blockRect.y=0;
  blockRect.w=BlockWidth-1;
  blockRect.h=BlockHeight-1;

  Uint8 r=0,g=0,b=0;
  switch(color)
  {
   case BlockCyan:
    r=0; g=240; b=240;
    break;
   case BlockBlue:
    r=0; g=0; b=240;
    break;
   case BlockOrange:
    r=240; g=160; b=0;
    break;
   case BlockYellow:
    r=240; g=240; b=0;
    break;
   case BlockGreen:
    r=0; g=240; b=0;
    break;
   case BlockPurple:
    r=160; g=0; b=240;
    break;
   case BlockRed:
    r=240; g=0; b=0;
    break;
  }

  SDL_FillRect(newSurface, &blockRect, SDL_MapRGB(newSurface->format, r,g,b));

  d_ptr->surfaceCache[color] = newSurface;
 }

 return d_ptr->surfaceCache[color];
}

As you can see, the if block is creating a new surface if the color has not been asked yet. The problem is this block is in the way of the main logic of my method, which is to retrieve a cached surface for the given color. Using “Extract Method”, I can make my intent more clear.

SDL_Surface *BlockSurfaceManager::get(BlockColor color)
{
 if( d_ptr->surfaceCache.count(color) == 0 )
 {
  createCachedSurfaceFor(color);
 }

 return d_ptr->surfaceCache[color];
}

I could go very zealot while refactoring this class. For the intend of this article, I’m going to ‘give my life for Aiur’ errrr make my intention even more clear

I will extract the if condition into its own method:

SDL_Surface *BlockSurfaceManager::get(BlockColor color)
{
 if( hasNoCachedSurfaceFor(color) )
 {
  createCachedSurfaceFor(color);
 }

 return d_ptr->surfaceCache[color];
}

bool BlockSurfaceManager::hasNoCachedSurfaceFor(BlockColor &color)
{
 return d_ptr->surfaceCache.count(color) == 0;
}

As you can see, it makes my intent more clear. Also, when you extract method with CodeRush, please double check the header if your method has the correct visibility.

For the beverity of the article, I will not describe in details the mechanics of the refactoring of createCachedSurfaceFor() but here’s for your viewing pleasure:

void BlockSurfaceManager::createCachedSurfaceFor(BlockColor &color)
{
 d_ptr->surfaceCache[color] = createBlockSurface(color);
}

SDL_Surface* BlockSurfaceManager::createBlockSurface(BlockColor &color)
{
 SDL_Surface *newSurface = SDL_CreateRGBSurface(SDL_HWSURFACE, BlockWidth, BlockHeight, ColorDepth, 0, 0, 0, 0);

 Uint8 r=0,g=0,b=0;

 getBlockColorRgb(color, r, g, b);

 drawBlockSurface(newSurface, r, g, b);

 return newSurface;
}

void BlockSurfaceManager::drawBlockSurface(SDL_Surface *newSurface, Uint8 &r, Uint8 &g, Uint8 &b)
{
 SDL_Rect blockRect = createBlockRect();
 SDL_FillRect(newSurface, &blockRect, SDL_MapRGB(newSurface->format, r,g,b));
}

SDL_Rect BlockSurfaceManager::createBlockRect()
{
 SDL_Rect blockRect;
 blockRect.x=0;
 blockRect.y=0;
 blockRect.w=BlockWidth-1;
 blockRect.h=BlockHeight-1;
 return blockRect;
}

void BlockSurfaceManager::getBlockColorRgb(BlockColor &color, Uint8 &r, Uint8 &g, Uint8 &b)
{
 switch(color)
 {
 case BlockCyan:
  r=0; g=240; b=240;
  break;
 case BlockBlue:
  r=0; g=0; b=240;
  break;
 case BlockOrange:
  r=240; g=160; b=0;
  break;
 case BlockYellow:
  r=240; g=240; b=0;
  break;
 case BlockGreen:
  r=0; g=240; b=0;
  break;
 case BlockPurple:
  r=160; g=0; b=240;
  break;
 case BlockRed:
  r=240; g=0; b=0;
  break;
 }
}

The code has more lines, but the intend of the code is more clear. I stop here for the moment but I should really factor the surface creator into its own class. BlockSurfaceManager has two responsibilities: Managing the surface cache AND creating the surface. This does not follow the Single Responsibility Principle where each class should have one reason to change.

I will commit the changes I’ve made. Let’s return back to Piece. In Piece::rotate() I have this snipped of code

if( type() == Piece_Square )
 {
  m_direction = newDirection;
  return;
 }

 Block *rotatedBlocks = getBlocks(newDirection);
 if( tryMove(0, 0, rotatedBlocks) )
 {
  delete[] m_tetra;
  m_tetra = rotatedBlocks;
  m_direction = newDirection;
 }
 else
 {
  delete[] rotatedBlocks;
 }

I smell something bad in here. The base class make assumption about its derived classes. I will leave it here for now because I haven’t written my test and I could alter behavior by removing the faulty block.

Here’s another obscure method:

bool Piece::tryMove(int deltaX, int deltaY, Block *block)
{
 bool canMove=true;

 for(int i=0; i<TetraSize; i++)
 {
  if( (block[i].x() + deltaX) >= BoardEndX ||
   (block[i].x() + deltaX) < BoardX ||
   m_board->blockAt(block[i].x() + deltaX, block[i].y() + deltaY) ||
   (block[i].y() + deltaY) < BoardY ||
   (block[i].y() + deltaY) >= BoardEndY
   )
  {
   canMove=false;
   break;
  }
 }

 return canMove;
}

The if condition is confusing as hell. With simple reordering and extract method, it becomes this:

if( isOutsideBoard(block[i], deltaX, deltaY) ||
   hasBlockingBlock(block[i], deltaX, deltaY) )
  {
   canMove=false;
   break;
  }

bool Piece::isOutsideBoard(const Block &block, int deltaX, int deltaY) const
{
 return (block.x() + deltaX) >= BoardEndX ||
   (block.x() + deltaX) < BoardX ||
   (block.y() + deltaY) < BoardY ||
   (block.y() + deltaY) >= BoardEndY;
}

bool Piece::hasBlockingBlock(const Block &block, int deltaX, int deltaY)
{
 return m_board->blockAt(block.x() + deltaX, block.y() + deltaY);
}

Make much more sense. Let’s commit this. As for the derived classes of Piece, I won’t ever dare to touch them before I write some tests.

Let’s call done for the day. In the next episode, I will show you how to make the code into test harness.

The trick is to add this little line into your assembly properties (AssemblyInfo.cs)

[assembly: InternalsVisibleTo("NameOfYourTestAssembly")]

Now all class with the internal visibility will be accessible to your unit tests. For private visibility, one solution is to use reflection or just look on Google

Hello, welcome to my series of article about refactoring game legacy code. Game legacy code is no different than other types of code. What’s makes game different than others type of codes ? With a game, you are dealing with a event driven, simulated real-time architecture. They have different needs and requirements than a business, Web or rich client application. Games need to react fast to events such as ‘gun was triggered’, you have parts of your code that are called 30 or 60 times per second, loading times need to be short, etc.

There is a lot of things going on in a game, where in a typical business application you just need to collect some input, process it and persist into a database (good ‘ol CRUD).

The subject

The subject is my first complete game ever written. If you look at the projects sections of this site, you’ll see some failed attempts. The game is a Tetris clone called “That Game From Russia (you know it)”. It is written in C++ using SDL library. It was done in two weeks back in January 2009. If you have read my article Paradigm Shift, you know this was before I read many books and articles that changed my way of thinking about code, design and architecture.

The code can be found at http://code.google.com/p/gamefromrussia/

We start with SVN revision 5.

Current state of the architecture

The architecture is based on a basic state machine. There is a bit of duplicated code scattered around the code, like the input handling. The code is pretty tightly coupled. They are little to no abstraction in the code. SDL types are found in public API. When I created the code, I intended to do my Tetris in one day. Yeah silly I know. I didn’t intend to create a flexible architecture. I wanted to create a Tetris for fun thus I didn’t try to care that much about the craft of my code.

Anyway so you have the states (DropPiece, InitGame, Setup, GameOver, RemoveLines, SetNextPiece, PauseState, LinesRemoveAnimation) that define the core behavior of the game. Each state has a enter(), execute() and exit() methods. enter() initialize the state, execute() is called on every iteration of the game loop and exit() is called before changing to another state.

The “domain” or the core of the game are in the classes Block, BlockLine, Board, Piece. Block is a single block. BlockLine is a line of blocks in the Board. Board contains a list of block lines. Piece is a collection of blocks that represent the shapes in a Tetris game.

The other classes are support classes. SoundPlayer manages sound effects and music. TextPrinter manage drawing the current level, the score and the number of lines.

What we are trying to refactor ?

What I want to refactor is all the code related to the Piece handling. I want to make the rotation code easier to understand. Currently, for each type of piece, you have obscure code that move the blocs depending of the direction. It assume that the x, y is for the center block and the other blocks are moving according to this block. While developing this code, I had sketches on paper to help me map the blocks for each piece. I shouldn’t needed a piece of paper to manage the complexity of this code.

The right way to rotate a piece is to use a rotation matrix and apply it to each of the blocks coordinate. I found this when looking at other Tetris clone code. You don’t need to use a matrix explicitly thought. I will explain the solution a little deeper in a following article.

When you refactor code, you should keep in mind your main goal. When dealing with legacy code, you can be tempted to do large scale refactoring of your ugly, stupid code. Having a goal help you focus of refactoring code that matters for what you are trying to accomplish.

Why I am doing this ?

In this case, I want to train my refactoring, TDD, SOLID skills in a C++ code base in preparation for a much larger game project this fall. Also, I am to experiment some techniques illustrated in Michael C. Feathers book Working Effectively with Legacy Code.

Refactoring legacy code is hard

I don’t claim to be an expert in refactoring legacy code. In fact, it’s quite the opposite. I am still a beginner in refactoring legacy code. Up to this point, I only refactored one legacy code base I written in C# back in 2007. It is quite a hard journey depending of the quality of the code base. My current code base has a lot of inner, subtle dependencies that we will need to break over time.

Next episode

In next episode, we’ll start doing some code cleanups to get a feel of the code base before setting up the tests in preparation for larger refactorings.

First of all, I used to be a Linux user AND developer (for KDE) all the way. I used Linux primely on my desktop and my laptop. I used to be a KDE contributor until University took all my time. Up to that point, I managed to do all my stuff in Linux except for certain things that really required Visual Studio or a Windows-only program. Now I need Visual Studio more than ever, and since I was tired to swag my girlfriend’s laptop every time I needed Windows, I wiped out my Mac partition and installed Windows Vista. Oh the horror some might say, but nowadays I’m been using Windows 7 RC and there were much rejoicing. I haven’t felt the need to restore GRUB on my laptop and suspend-to-RAM is painless.

The seed that started this whole shift is when I was working on my summer work and I grew tired of writing the same CRUD database stuff and business rules. From this day, I decided that I needed something more challenging. Fortunately I was in the good curriculum, only my future goal changed.

Then I (re)found gaming. I thought games would be the challenge I was looking for.

When I started university, I have no intention to become a game developer, I didn’t play game that much for several years where in my young years I used to be a Nintendo freak. My goal when leaving University was to join a open source company to work on free software or find something that wasn’t related to IT(not custom apps for biz) or gaming. Now, I want to join a game company or maybe form my own with friends, time will tell.

Back to open source. Like I said, I was a big Linux user. I still think they are cool stuff in Linux and open source is nice. I still use a lot of free software. But the reality as a developer is slapping me back to the face. The fact is Windows is the dominant platform and if you want to get the most users, either you go the Web approach (which I kinda dislike) or you develop for Windows. Also, I want to be a game developer and Windows is THE platform for developing games no matter the target platform.

Beside my switch in careers goals, I also change a lot my mentality. Since the last months I’ve been reading a lot of books about software practices such a Clean Code, Agile Patterns Principles and Practices, Refactoring, Code Complete. I also read a lot of articles from Joel on Software. I might say that I got influenced a lot by those articles and that changed my way of thinkings about software.

Lots of changes. Lots of contracting ideas that I need to sort out and came out with the mixture that will work for me.

This new API use a philosophy called “Write once, read multiple of times.”, I use the same philosophy for some time while doing libpapillon. I will give you a case, when you read a constructor like this new KDialog(i18n("Caption"),parent, name, User1 | User2, User1, true, ...). This make sense when you first write it because you have the API documentation open in another window. Now, re-read this code 1 month later, the code will make less much sense and you will need to remember and try to guess what each parameter do. By simplicifing your API, you make your code more readable and more maintainable, not just for you but for yours followers too because we are in open source world and other people read our code.
Updated code:

KDialog *newDialog =new KDialog(parent);
newDialog->setCaption( i18n("Caption") );
newDialog->setButtons( KDialog::User1 | KDialog::User2 );
newDialog->setDefaultButton( KDialog::User1 );

Easier to read isn’t ?

Another philosophy should be used while designing API is make your interface ease to use and harder to misuse. In fact, that’s one item from book “Effective C++ 3rd Edition.”. Every serious C++ Programmer should have a copy of this book IMO. The goal is to reduce the complexity of your API to avoid confusion when using it, thus avoiding bugs and unexpected behavior. Also to report error at compile-time and not at runtime. I will reuse the example from the book (that I paste from my memory ).

Here a simple encapsulation of a Date.

class Date
{
public:
Date(int year, int month, int day);
};


What happen if someone call Date(21, 02, 1986) or Date(02, 21, 1986) ? This can lead to an unexpected behavior.

Here a good way to encapsulate Date:

struct Month
{
explicit Month(int);
};
struct Day
{
explicit Day(int);
};
struct Year
{
explicit Year(int);
};
class Date
{
public:
explicit Date(Day day, Month month, Year year);
};

By making use of keyword explicit, you avoid implicit conversion to int and compiler will generate an error if you try to use the wrong date component.