I have made some progress in managing resources. I'm almost up to the point where I need to start thinking about incorporating drawing code to see the progress, but I'm resisting the temptation. Once I have that up, it's going to take too much attention away from drawing the basic infrastructure.
Instead, I'm looking into how to manage my resources. There are three basic things you need to do with resources (be they models, entities, sounds or whatever). They need to be loaded from files (or created procedurally), they need to be called during play, and they have to be unloaded when not needed any more. My current problem is with all three.
I'm trying to determine what the best way to handle them in each situation is. Before calling up a specific file to load, I need to know what specific resource I need, and check if it is already loaded. If it isn't, then I look up the file and load it. Consider the case of models, for example. Several entities might use the same model. There's no point in storing in the entity the filename of the model, since then I'd be loading the model once per entity. I'd need a table of some kind that looks up a specific filename and checks to see if it is loaded or not. If it is loaded, it would provide the 'model id' that the system assigned it when loading it, and that can be used to refer to it. Otherwise it loads the file, assigns it a model id, and then returns it.
Hmm, that could work, I guess.
I already implemented a way to ensure that the vector holding the pointers to the resources doesn't fill up with holes at least. I'll keep a count of how many pointers are stored in it, which increases when added to and decreses when one is freed. Then I can know if any holes develop by comparing the size of the vector to the number of resources loaded. If when loading a new resource I find that there's a hole, I'll search the vector for the hole and use the first I come upon.
So coming up, more design work. I'm kinda sorta stuck in that I don't know where exactly to move forward. All of the code needs to be worked on. I'll spend the next week looking at the event handling, that should provide some interesting avenues to make inroads.
Sunday, June 26, 2011
Friday, June 24, 2011
Main loop
I've been working on the infrastructure for the program. Matters like resource handling, memory structure and access to functionality. These are important, but don't describe the program itself. I can have test runs with programs that try the functionality of different aspects of the program, but they're not what I'm after.
Games have a pretty standard structure. They're basically a loop, where you poll for input, update the state, and render to the screen. This loop runs dozens of times per second in a normal game. A game running at thirty fps is running through this loop thirty times per second. Each run is generally referred to as a frame, since you can consider the discrete changes in state between each run a snapshot, and the sequence of snapshots a movie.
So, after I got entities and models being loaded into my resource managers, I figured I ought to do something with them. I'm going to replicate the loop in the 'game', though for the moment there's not going to be any even handling or much of a game state. Or actual rendering.
This structure fits within the 'Execution' method. After setting things up, within a while loop (with a suitable 'running' variable) I call a 'Render' method. The method should start with a ModelView matrix stack set to the Identity matrix. The projection matrix is set once and should not change unless something dramatic happens, like resizing the screen or changing the FOV, so it's not handled in here.
Then, we multiply the ModelView matrix by the View matrix, which represents the camera position. Then, we start calling the list of entities. For each entity, we add the model matrix to the stack, draw the entity, and pop the ModelView stack. This is it as far as rendering is concerned.
I don't really need to be concerned that I'm not working with OpenGL yet. Just handling the information back and forth should be enlightening. For example, I'm thinking I really need to keep everything together, no 'layers' really. They can be derived from different classes, but I can't isolate them based on it.
Games have a pretty standard structure. They're basically a loop, where you poll for input, update the state, and render to the screen. This loop runs dozens of times per second in a normal game. A game running at thirty fps is running through this loop thirty times per second. Each run is generally referred to as a frame, since you can consider the discrete changes in state between each run a snapshot, and the sequence of snapshots a movie.
So, after I got entities and models being loaded into my resource managers, I figured I ought to do something with them. I'm going to replicate the loop in the 'game', though for the moment there's not going to be any even handling or much of a game state. Or actual rendering.
This structure fits within the 'Execution' method. After setting things up, within a while loop (with a suitable 'running' variable) I call a 'Render' method. The method should start with a ModelView matrix stack set to the Identity matrix. The projection matrix is set once and should not change unless something dramatic happens, like resizing the screen or changing the FOV, so it's not handled in here.
Then, we multiply the ModelView matrix by the View matrix, which represents the camera position. Then, we start calling the list of entities. For each entity, we add the model matrix to the stack, draw the entity, and pop the ModelView stack. This is it as far as rendering is concerned.
I don't really need to be concerned that I'm not working with OpenGL yet. Just handling the information back and forth should be enlightening. For example, I'm thinking I really need to keep everything together, no 'layers' really. They can be derived from different classes, but I can't isolate them based on it.
Thursday, June 23, 2011
Back to the drawing board
Ugh, so annoying. I had the right idea in the previous posts with the design of the program, but I went a little too overboard with separating the functions of the different parts of the program. Too much insulation between the parts that manipulate my resources and the parts that read files. The problem being, there's too much interconnectedness in those two fields to have separated them so much.
It appears I'm going to have to fold the application layer into the resource layer. Otherwise, in order to load a model file, I have to break away from resource handling. Which is just dumb. Yeah, a link between layers would solve it, but it's poor form. If I need the connection to be porous between the two, I shouldn't be insulating them from each other so much.
Once I have things working together, I can reevaluate if I need to split them apart to clear up things. Overcomplicating things at the outset is never a good idea.
It appears I'm going to have to fold the application layer into the resource layer. Otherwise, in order to load a model file, I have to break away from resource handling. Which is just dumb. Yeah, a link between layers would solve it, but it's poor form. If I need the connection to be porous between the two, I shouldn't be insulating them from each other so much.
Once I have things working together, I can reevaluate if I need to split them apart to clear up things. Overcomplicating things at the outset is never a good idea.
Wednesday, June 22, 2011
Framework testing, reconsidered
It's usually easy to recognize bad code. It's big, clunky, ugly, and you hate writing it. It can get the job done, for certain values of 'the job' and 'done', but it isn't pleasant for anyone.
I had been struggling with part of my design, no matter how I looked at it I couldn't get it to do what I wanted it to. Not without creating a mess of code that was painful to look at. I've finally figured out that the problem was my design.
I had started well enough, separating tasks and delegating responsability. However, I went a bit too far down that path, to the point that I was hoping for classes at the bottom of the ladder to do too much. The class that handles the Entities, for example, should not be in charge of getting the entities' models loaded. In short, I was trying to make my classes too smart, and in trying to give them the freedom to act they had to learn about parts of the program that were beyond their scope.
So going forward, I'm going to be taking another look at how I am trying to handle this. The only 'upwards' communication the classes will have is answering queries by the holding object. No trying to start other jobs, just getting the specific function requested done, possibly returning a value as appropriate.
More tomorrow, when I start rambling on about what I'm trying to do again.
I had been struggling with part of my design, no matter how I looked at it I couldn't get it to do what I wanted it to. Not without creating a mess of code that was painful to look at. I've finally figured out that the problem was my design.
I had started well enough, separating tasks and delegating responsability. However, I went a bit too far down that path, to the point that I was hoping for classes at the bottom of the ladder to do too much. The class that handles the Entities, for example, should not be in charge of getting the entities' models loaded. In short, I was trying to make my classes too smart, and in trying to give them the freedom to act they had to learn about parts of the program that were beyond their scope.
So going forward, I'm going to be taking another look at how I am trying to handle this. The only 'upwards' communication the classes will have is answering queries by the holding object. No trying to start other jobs, just getting the specific function requested done, possibly returning a value as appropriate.
More tomorrow, when I start rambling on about what I'm trying to do again.
Project Demo, Reading a file
I'll scrap the interface bits out for now. The class holding the layers already holds the information I was trying to model with them, so it may have been redundant to try and store it as well inside each object. Instead I'll see what I need to get a file on request.
For starters, I need to configure the resource directories. For now, I'll store the "resource/model" directory in a std::string. I'll do the assignment in an initialization method. Then when asking for a given model, I only need to ask for the file name and it'll know where to look for it.
Reading a file is going to generally return a 'resource' of some kind. In this case a list of vertices that makes up the model. I'll store them in a vector. So for now, the ReadModel method will read a file and return a pointer to a vertices resource, which holds a vector of Vertex structs. Vertex contains 3 floats for coordinates, 3 for normal direction, and 2 for texture coordinates.
I'll use the c++ fstream libraries to handle the files, rather than Cs FILE. In the 'read' method, I'll create a ifstream object, and a string to build the full path. I'll then open the file (in read-only mode) and read the test 'triangle.tvl' file.
We now have a method for reading TVL files. Normally we'd need the model information when we created an Entity. The entity manager should know what kind of model each entity uses. When loading the entity, it should ask the model manager if the model is loaded.
This raises the question of how to refer to each model. This is something I'm not going to be happy with for a long time, so for now I'll use a number to refer to them. So our triangle will have the model id 0. Of course, now we need some way to store our entities as well. Which means another custom file format. And so is the TE (Text Entity) format born. For now, all it will have is the model id.
We quickly add another method in the application layer to read TE files and return Entities. We then add an int field to the Entity class and call it idModel, which is the single value to be read from the file. We also add a method to load the value into the field.
Ok, so now we can read the files, and the application layer returns pointers to the created objects. We need to add code to get the program to call those methods, though. We're trying to avoid building a whole program though. For the purposes of this demo, for example, the engine layer might as well not be there. We start with what would be the normal entry point of the program, the main function. There, we create a Game object. Once that is done, we should get the program going. We can call an 'Execute' method in game, which get the game going.
As soon as the game starts (within this execution method) we have to set the different layers up so they can be ready to work. For example, we have to set the resource paths we use in the Application Layer to read files. Eventually we'll have a configurable 'config.ini' type file that contains much of that information, but for now we'll just hard code the values in an initialization method for each layer.
So main calls game.execute, execute calls the application layer's initialization (the only one that needs to e initialized so far) and then.... Well, we could do a quick loop to simulate the normal running of the game. We load the first entity with a load entity function in the entity manager.... No. We try to get an entity. That would emulate what a method in the engine layer might try to do. So we ask for an entity... hmm. How do I ask for an entity? An entity id? But entities come and go. Normally the rendering engine just ask for a list of all entities that ned to be drawn. Well, entities do need to be 'created'. When the game starts it would create at the very least an entity to represent the player character. So we start with a 'create entity' method.
Hmm. Ok, so there could be a 'load resources' method in the resource layer as a way to initialize the bare necessities. This function would call a load function for each manager, or in the case of the entity, a create method. Ah, here we run into the problem with linking things together I scrapped before. The entity manager has a filename it wants to load, but the logic is over in the application layer.
Ok, at least I have a different perspective to assail this problem now. I want to have a good solution for this, since so much of the design depends on it.
And now I'm stuck again, thanks to the joys of the proper order for including files. I may have found a way around this, though.
For starters, I need to configure the resource directories. For now, I'll store the "resource/model" directory in a std::string. I'll do the assignment in an initialization method. Then when asking for a given model, I only need to ask for the file name and it'll know where to look for it.
Reading a file is going to generally return a 'resource' of some kind. In this case a list of vertices that makes up the model. I'll store them in a vector. So for now, the ReadModel method will read a file and return a pointer to a vertices resource, which holds a vector of Vertex structs. Vertex contains 3 floats for coordinates, 3 for normal direction, and 2 for texture coordinates.
I'll use the c++ fstream libraries to handle the files, rather than Cs FILE. In the 'read' method, I'll create a ifstream object, and a string to build the full path. I'll then open the file (in read-only mode) and read the test 'triangle.tvl' file.
We now have a method for reading TVL files. Normally we'd need the model information when we created an Entity. The entity manager should know what kind of model each entity uses. When loading the entity, it should ask the model manager if the model is loaded.
This raises the question of how to refer to each model. This is something I'm not going to be happy with for a long time, so for now I'll use a number to refer to them. So our triangle will have the model id 0. Of course, now we need some way to store our entities as well. Which means another custom file format. And so is the TE (Text Entity) format born. For now, all it will have is the model id.
We quickly add another method in the application layer to read TE files and return Entities. We then add an int field to the Entity class and call it idModel, which is the single value to be read from the file. We also add a method to load the value into the field.
Ok, so now we can read the files, and the application layer returns pointers to the created objects. We need to add code to get the program to call those methods, though. We're trying to avoid building a whole program though. For the purposes of this demo, for example, the engine layer might as well not be there. We start with what would be the normal entry point of the program, the main function. There, we create a Game object. Once that is done, we should get the program going. We can call an 'Execute' method in game, which get the game going.
As soon as the game starts (within this execution method) we have to set the different layers up so they can be ready to work. For example, we have to set the resource paths we use in the Application Layer to read files. Eventually we'll have a configurable 'config.ini' type file that contains much of that information, but for now we'll just hard code the values in an initialization method for each layer.
So main calls game.execute, execute calls the application layer's initialization (the only one that needs to e initialized so far) and then.... Well, we could do a quick loop to simulate the normal running of the game. We load the first entity with a load entity function in the entity manager.... No. We try to get an entity. That would emulate what a method in the engine layer might try to do. So we ask for an entity... hmm. How do I ask for an entity? An entity id? But entities come and go. Normally the rendering engine just ask for a list of all entities that ned to be drawn. Well, entities do need to be 'created'. When the game starts it would create at the very least an entity to represent the player character. So we start with a 'create entity' method.
Hmm. Ok, so there could be a 'load resources' method in the resource layer as a way to initialize the bare necessities. This function would call a load function for each manager, or in the case of the entity, a create method. Ah, here we run into the problem with linking things together I scrapped before. The entity manager has a filename it wants to load, but the logic is over in the application layer.
Ok, at least I have a different perspective to assail this problem now. I want to have a good solution for this, since so much of the design depends on it.
And now I'm stuck again, thanks to the joys of the proper order for including files. I may have found a way around this, though.
Tuesday, June 21, 2011
Project Demo, Part 2
As I said before, I wanted to do a quick implementation of the base program design. Of course, since we started from a fairly high level of abstraction, getting it down to code requires making several decisions, and it's always best to document these somewhere other than the code base.
For starters, the program is contained in a Game class. I created a Layer base class, and Layer classes for each layer. The Game class holds one of each layer; Application, Resource and Engine. I also created an Interface base class, from which all the interfaces inherit. Something I'm not happy with is the names I gave them. The Resource Interface in the Application Layer is not the same as the Resource Interface in the Engine layer. This means that for N number of layers, I could have up to N*N-1 kinds of interfaces. This may be excessive, since not all Layers are connected to all others (Engine is only connected to Resource, for example). However, it still requires a multitude of classes. I may eventually figure a solution to this, but for now it should be enough for my purposes.
I spoke a great deal about 'connecting' the different layers, and communications between them. This, unfortunately, cannot be done from within the class. I can't have an internal 'link to layer' method in the Interface class, since it doesn't know about anything outside itself. Instead, to link two layers together, the containing structure (Game in this case) has to pass each a pointer to the other. I could have a method in the Layer itself that links the contained interface to a pointer given to it, though.
To support this, I've given the Interface base class an Interface pointer called link. I also gave Interface a method, 'LinkTo(Interface*)' that takes a pointer to an interface and links the interface to it. On the Layer base class, I created another method that takes an Interface and a pointer to an interface, and calls the LinkTo method in the Interface with the pointer as a parameter.
Now I need a method to ask each Layer the address of their interfaces, so that other layers can link to them. I want to keep this general, so I don't have to write one method for each Layer, but in order to make it part of the Layer class Layer must know what interfaces the layer will ultimately have, which is something known by the derived classes. The problem is the number of interfaces is variable. Let such a method be called "GetInterface". It would have to return an array of some kind with all the Interfaces the Layer has. Then the calling code would have to determine which interface should connect where.
I could hack some brute force, "this goes here" sort of code, where I manually link each pair of interfaces, but that isn't elegant. Part of the problem goes back to what I didn't like about my interface code before; I'm making too many derived classes. This will require some more thinking.
In the meantime, I'll move over to other issues I have dealt with.
I included a Resource* vector in the Manager base class. If any of the managers ever need a different kind of storage, I'll think of something. Something else I decided on was to create a simple 'vertex list' file format to store simple models in. This is going to help avoid cluttering the code with hard coded models, and the infrastructure required to read one kind of file type can easily be extended to read others.
The standard for the file will be simple. The file is going to be text based. The file header will contain, first, the file version (starting at 1.0). This is so that if I change the format in the future, I don't have to scrap the files I may already be using. It's also just good form to version file formats. In the next line, the number of vertices contained in the file. In the next line, and for every line afterwards, the x, y and z values of the vertex, separated by white space, followed by the x, y and z values of the normal at each point, followed by the u, v values of the texture coordinates. The format will permit comments in lines that start with '#'. All values are mandatory, and upon finding a malformed line the file will stop being read.
This should help with testing, once everything else is working.
For starters, the program is contained in a Game class. I created a Layer base class, and Layer classes for each layer. The Game class holds one of each layer; Application, Resource and Engine. I also created an Interface base class, from which all the interfaces inherit. Something I'm not happy with is the names I gave them. The Resource Interface in the Application Layer is not the same as the Resource Interface in the Engine layer. This means that for N number of layers, I could have up to N*N-1 kinds of interfaces. This may be excessive, since not all Layers are connected to all others (Engine is only connected to Resource, for example). However, it still requires a multitude of classes. I may eventually figure a solution to this, but for now it should be enough for my purposes.
Design grows more detailed as I try to implement it
I spoke a great deal about 'connecting' the different layers, and communications between them. This, unfortunately, cannot be done from within the class. I can't have an internal 'link to layer' method in the Interface class, since it doesn't know about anything outside itself. Instead, to link two layers together, the containing structure (Game in this case) has to pass each a pointer to the other. I could have a method in the Layer itself that links the contained interface to a pointer given to it, though.
To support this, I've given the Interface base class an Interface pointer called link. I also gave Interface a method, 'LinkTo(Interface*)' that takes a pointer to an interface and links the interface to it. On the Layer base class, I created another method that takes an Interface and a pointer to an interface, and calls the LinkTo method in the Interface with the pointer as a parameter.
Now I need a method to ask each Layer the address of their interfaces, so that other layers can link to them. I want to keep this general, so I don't have to write one method for each Layer, but in order to make it part of the Layer class Layer must know what interfaces the layer will ultimately have, which is something known by the derived classes. The problem is the number of interfaces is variable. Let such a method be called "GetInterface". It would have to return an array of some kind with all the Interfaces the Layer has. Then the calling code would have to determine which interface should connect where.
I could hack some brute force, "this goes here" sort of code, where I manually link each pair of interfaces, but that isn't elegant. Part of the problem goes back to what I didn't like about my interface code before; I'm making too many derived classes. This will require some more thinking.
In the meantime, I'll move over to other issues I have dealt with.
I included a Resource* vector in the Manager base class. If any of the managers ever need a different kind of storage, I'll think of something. Something else I decided on was to create a simple 'vertex list' file format to store simple models in. This is going to help avoid cluttering the code with hard coded models, and the infrastructure required to read one kind of file type can easily be extended to read others.
Sample tvl (Text Vertex List) file
The standard for the file will be simple. The file is going to be text based. The file header will contain, first, the file version (starting at 1.0). This is so that if I change the format in the future, I don't have to scrap the files I may already be using. It's also just good form to version file formats. In the next line, the number of vertices contained in the file. In the next line, and for every line afterwards, the x, y and z values of the vertex, separated by white space, followed by the x, y and z values of the normal at each point, followed by the u, v values of the texture coordinates. The format will permit comments in lines that start with '#'. All values are mandatory, and upon finding a malformed line the file will stop being read.
This should help with testing, once everything else is working.
Program Design, Demo
There's an annoying issue when designing a program; you can't see the result of the desig until you are done. But sometimes, in order to make a design decision, you might want to see what you have working, get a feel for it. It's an iterative process. There's a reason so many programs minimize or ignore the need for design: designing is boring, compared to actually coding and creating something that works.
There's always the temptation to design as you code and fix things later. Then when the code is crisscrossed with hacks made to get it working, you feel like scrapping it and starting over. I'm going to try to fight that urge and do things right. That means doing a few tests as I design, settling on methods and structure, and leaving the actual program for the future. That's where Demos come in.
I have designed a high level overview of the program. A bit of the class hierarchy, penciled in the data structures I plan to use and the like. I'm close to having the parts in place for the Rendering Engine to actually have something to render. What I'm going to do is make a quick mock-up of the program, with a sparse implementation of everything I talked about until now, and try to get it all ready for prime-time.
I'm not going to actually render anything. No OpenGL calls or anything like that. This is meant to make sure that what I have works, and identify any needs I forgot about. It's going to be a console only application. So let's have a quick rundown of how the program should work:
Once we decide on the scope of the demo, we program it. Gotta jump in Code::Blocks for this....
There's always the temptation to design as you code and fix things later. Then when the code is crisscrossed with hacks made to get it working, you feel like scrapping it and starting over. I'm going to try to fight that urge and do things right. That means doing a few tests as I design, settling on methods and structure, and leaving the actual program for the future. That's where Demos come in.
I have designed a high level overview of the program. A bit of the class hierarchy, penciled in the data structures I plan to use and the like. I'm close to having the parts in place for the Rendering Engine to actually have something to render. What I'm going to do is make a quick mock-up of the program, with a sparse implementation of everything I talked about until now, and try to get it all ready for prime-time.
I'm not going to actually render anything. No OpenGL calls or anything like that. This is meant to make sure that what I have works, and identify any needs I forgot about. It's going to be a console only application. So let's have a quick rundown of how the program should work:
This is what we're going to test
Once we decide on the scope of the demo, we program it. Gotta jump in Code::Blocks for this....
Monday, June 20, 2011
Program Design, Application Layer
So I talked about how to store the resources, and how to hand them to the Engine Layer, but there's still the matter of generating them.
Before they can be stored, shaders have to be read from files, sent to the GPU, compiled, and linked. Only then can they be used, and there is no point storing a broken shader. Textures also need to be read and loaded. As well as models. Configuration options can also be read from files.
All these parts of the program doing similar actions indicate a need to separate the file reading from the parsing. How to do it, though, and what to pass along?
Let's start with the shaders. The source for each kind of shader is found in .vp and .fp (and .gp) files. These stand for Vertex, Fragment and Geometry programs. They are regular text files, nothing fancy. The way I'm doing it right now is by calling a function in the GLTools library, which takes the name of the files and loads the shaders. It works, but it violates certain rules I want to enforce, such as the resource layer knowing about resources, not about the file system.
Still, we can scavenge the function (the license on it is very permisive). Interestingly, the function already separates the code that reads the file from the code that loads the function, which is exactly what we want to do.
glShaderSource is the OpenGL API call we want to use to load the shader to the GPU. The way it wants us to send it the shaders is as a pointer to a null terminated string of GLchars, which is essentially an array ending in '\0'. This means that, in this case, we'd want the file reading part of the program to load the file into an array in memory, and send a pointer to the array to the ShaderManager, which would then be in charge of loading it in the GPU and storing the result of the operation.
This works for any text files, but textures are genrally stored as image files. These are genrally binary encoded and compressed. So how to handle them? One alternative is to read the file, and pass the content in the state that it was received to the resource layer, and let the Resource Layer handle compression and the like. Another is to let the Application Layer deal with the matter of filetypes and enconding and just hand the resulting data to the Resource Layer.
I feel more comfortable with this second approach, myself. For one, it is not necessarily true that only the Texture Manager will want to decode image files. Heightmaps, for example, are one way of storing terrain topography. If I ended up using a picture format for terrain and the same format for a texture, I'd have to repeat the decoding process in each manager. This is what I want to avoid.
As a result, I'll keep functions to decode each filetype in the application layer, and will simply pass along pointers to arrays of bytes to the rest of the program when required. These can then be interpreted as appropriate by each part of the program.
Passing along the full file can be problematic however. Specially in binary files, which can contain structures. There would be little point in decoding unless we were to turn the file into something usable. Now, in the case of shadrs, just passing along a null terminated string is fine, that is what we want. But in the case of images, for example, we could pass along a more elaborate structure, including the height and width of the image, color depth, etc. This is the information the Resource Layer wants.
We can repeat the structure we came up with before for handing resources to the Engine Layer in the communication between the Application and Resource Layers. For starters, we would need a base class for the different structures we might need to pass along. We could re-use 'Resource'. Then we'd need types for the different sort of information that is being entered into the system. "Text" for char arrays (there's a multitude of string classes that could be used), "Image" for textures or heightmaps, and the name of whatever appropriate structure is being sent for other, more complex files.
We can think of these as 'un-processed' resources, which are processed by the Resource Layer into the game resources our engines can use.
As before, we don't want each of our managers dealing with the functions in the Application Layer personally. Instead, when a request for a certain resource is sent their way, they should check if the resource is already stored and return it or, if not, forward the request to load it from file, store it, and return it to whoever asked for it. This forwarding should be done through an interface into the Application Layer, the same way the Engine Layer connects to the Resource Layer.
This takes care of the communication aspect, but we still need to determine how to model the application layer, which for now would consist simply of a smattering of functions. Such as loading files and converting them to and from different file types.
Before they can be stored, shaders have to be read from files, sent to the GPU, compiled, and linked. Only then can they be used, and there is no point storing a broken shader. Textures also need to be read and loaded. As well as models. Configuration options can also be read from files.
All these parts of the program doing similar actions indicate a need to separate the file reading from the parsing. How to do it, though, and what to pass along?
Factoring the 'ReadFile' code away
Let's start with the shaders. The source for each kind of shader is found in .vp and .fp (and .gp) files. These stand for Vertex, Fragment and Geometry programs. They are regular text files, nothing fancy. The way I'm doing it right now is by calling a function in the GLTools library, which takes the name of the files and loads the shaders. It works, but it violates certain rules I want to enforce, such as the resource layer knowing about resources, not about the file system.
Still, we can scavenge the function (the license on it is very permisive). Interestingly, the function already separates the code that reads the file from the code that loads the function, which is exactly what we want to do.
glShaderSource is the OpenGL API call we want to use to load the shader to the GPU. The way it wants us to send it the shaders is as a pointer to a null terminated string of GLchars, which is essentially an array ending in '\0'. This means that, in this case, we'd want the file reading part of the program to load the file into an array in memory, and send a pointer to the array to the ShaderManager, which would then be in charge of loading it in the GPU and storing the result of the operation.
This works for any text files, but textures are genrally stored as image files. These are genrally binary encoded and compressed. So how to handle them? One alternative is to read the file, and pass the content in the state that it was received to the resource layer, and let the Resource Layer handle compression and the like. Another is to let the Application Layer deal with the matter of filetypes and enconding and just hand the resulting data to the Resource Layer.
Decoding in the Resource Layer vs. decoding in the Application Layer
I feel more comfortable with this second approach, myself. For one, it is not necessarily true that only the Texture Manager will want to decode image files. Heightmaps, for example, are one way of storing terrain topography. If I ended up using a picture format for terrain and the same format for a texture, I'd have to repeat the decoding process in each manager. This is what I want to avoid.
As a result, I'll keep functions to decode each filetype in the application layer, and will simply pass along pointers to arrays of bytes to the rest of the program when required. These can then be interpreted as appropriate by each part of the program.
Passing along the full file can be problematic however. Specially in binary files, which can contain structures. There would be little point in decoding unless we were to turn the file into something usable. Now, in the case of shadrs, just passing along a null terminated string is fine, that is what we want. But in the case of images, for example, we could pass along a more elaborate structure, including the height and width of the image, color depth, etc. This is the information the Resource Layer wants.
We can repeat the structure we came up with before for handing resources to the Engine Layer in the communication between the Application and Resource Layers. For starters, we would need a base class for the different structures we might need to pass along. We could re-use 'Resource'. Then we'd need types for the different sort of information that is being entered into the system. "Text" for char arrays (there's a multitude of string classes that could be used), "Image" for textures or heightmaps, and the name of whatever appropriate structure is being sent for other, more complex files.
We can think of these as 'un-processed' resources, which are processed by the Resource Layer into the game resources our engines can use.
As before, we don't want each of our managers dealing with the functions in the Application Layer personally. Instead, when a request for a certain resource is sent their way, they should check if the resource is already stored and return it or, if not, forward the request to load it from file, store it, and return it to whoever asked for it. This forwarding should be done through an interface into the Application Layer, the same way the Engine Layer connects to the Resource Layer.
Interfaces connecting the different layers
This takes care of the communication aspect, but we still need to determine how to model the application layer, which for now would consist simply of a smattering of functions. Such as loading files and converting them to and from different file types.
Program Design, Managing Resources
So I started describing how I wanted to hold my information. My first idea was vectors of each type:
Then, I can create vectors of pointers to Resource types. Each caller would then be responsible for asking the right manager for the information, and handling it correctly, but the 'pipes' moving the information across can all be the same.
Vectors to hold shaders, entities and models
A vector, for those not familiar with the C++ standard library, is a form of storage consisting of as many adjacent 'boxes' as you need. Each of the boxes in a given vector can contain one type of data.
You can access any of the boxes directly by their index. In the example above, asking for shader[0] gives you one value of type GLuint, while asking for entity[0] gives you an Entity object and asking for model[0] gives you a GLTriangleBatch object.
This presents a few problems. For example, I have to keep one kind of calling code for each, and ready a different kind of object to hold the information. This kind of duplication will make things difficult in the long run. As I was saying last time, a solution for this was to create a Resource class, from which every other kind of resource would inherit.
While I can make Entity inherit from Resource easily, GLuint is a primitive type defined in the OpenGL standard, and GLTriangleBatch belongs to the GLTools library in the OpenGL SuperBible. They can't inherit from Resource. Instead, I can create wrapper classes, Shader and Model, which simply contain elements of their types, and inherit from Resource.
Resource inheritance graph
Then, I can create vectors of pointers to Resource types. Each caller would then be responsible for asking the right manager for the information, and handling it correctly, but the 'pipes' moving the information across can all be the same.
Resource pointer vectors for shaders, entities and models
Giving the engine the resource it needs, then, would mean giving it a pointer to where in memory the resource is stored. This way the engine can know everything it needs to know about the resource, without worrying about its storage or lifetime. Looking back at the interface graph, however, we notice that our interface needs to know about every manager. Managers belong in the Resource layer, and the interface belongs in the Engine layer. Giving one an insight into the inner workings of the other is troublesome.
We can add an additional layer of insulation, and create an Engine interface in the resource layer. The Resource Interface would just need to know how to call the Engine interface, then, and it would be this component that knows about all the managers, and who holds what bit of information. The Resource interface would then just ask for 'model #0', and the engine interface would provide the Resource pointer to 'model #0', without the engine interface needing to know anything about managers or the resource managers knowing what engine asked for it.
Interfaces insulating the Resource and Engine Layers
So we have now fleshed out a bit the design for the Engine and Interface layers, and worked out how to keep the details of each local to themselves. We could create more engines and resource types, and would not need to change much to accommodate them.
Program Design, Class Structure
Last post, I had decided on a basic structure for the program. Something that looked like this:
With that done, we can start thinking of what actually needs to be done, and create a hierarchy of classes that will get everything working.
In the Engine Layer, for example, we'll be having a Rendering Engine, a Physics Engine, a Gameplay Engine, AI Engine, etc. Each of them will drive one of the game's subsystems, called in turn by the Game. We want these to have a common interface where it comes to communicating with the Resource Layer. They'll generally want to ask for the same information, and trying to standardize how they do it can save us work in the long term.
For now I'll concentrate on getting the Rendering Engine working, which is the shiniest part of the project. Getting it working can help us see what is going on, and it'll make it obvious later on if one of the other parts of the game is acting oddly.
So, we start with a RenderEngine class. This class is going to inherit from an Abstract Base Class (ABC) called Engine. Doing this will allow us to refer to it as "a kind of" Engine, same as any other Classes we derive from it.
In inheritance graphs, arrows point from the child to the parent. What inheritance means, in essence, is that the child has all the same methods and data members as the father, as well as any defined in itself. The child can be handled by pointers of the parent's type, and any methods or data members in the parent can be safely accessed through it. In short, you could treat an object of the child's type through a pointer of the parent's type as if it were an object of the parent's type.
So, what does the rendering engine need? It needs access to the list of entities it must draw (ideally sorted or culled in some way, though we'll start with just getting it working with all of them), it needs access to the models that represents each entity, it needs the ModelViewProjection matrix and it needs the shaders that will be in charge rendering to the screen. It would also need textures, but not just yet.
These are a lot of different resources that must be handled. So how do we get a hold of them?
Well, we need a way of storing them in the Resource Layer. We can have a vector for each different kind of resource, for example. We can then use it's position in the vector as a unique identifier for each.
However, each kind of resource is fundamentally different. The way to prepare a shader for use is very different from the way a model or a texture are prepared, for example. We need specialized functions for how to store each. What we can do is create several manager classes, each specialized in handling a specific kind of resource, but with a common interface for delivering them in a predictable manner.
Certain functions will be common to all the managers, even if their implementation is different in each. This again is a reasonable situation to use inheritance. We can have a Manager ABC, from which several different managers will inherit (ShaderManager, ModelManager, EntityManager, etc).
Then each manager would have a structure to hold the associated resource. It can be a vector in some cases, but for others it may eventually be best to use some other kind of structure.
So the rendering engine would then need only call on the shader manager to provide a given shader when needed, and the model manager for models, etc. However, we can see two problems with this. One, our engine would need to know how to talk to each manager. And since the managers are external to the engine classes, each engine will need to store the address of each manager.
This is a lot of duplicated information, not all of which would be of use. Instead, we could have a single system in charge of interfacing with the resource layer, and all the different engines would instead talk to it. Then, we'd only need to store it's location in each, and each engine could just focus on asking this interface for a kind of resource, instead of knowing how to talk to each manager.
This is starting to look decent. However, there's another issue to deal with. Each of these resources is of a different type, and both the handling and storage of them breaks the uniformity we'd rather have. Models, for example, are being stored as GLTriangleBatch objects, whereas Shaders are stored as GLuint types and Entities as Entity objects. When trying to draw the models, the Rendering engine has to call a method of the GLTriangleBatch class, and to use the Shader it has to use an OpenGL API call with a GLuint parameter.
This means that while we went some ways into simplifying the connection to the Resource Layer, we have to have several different types of call methods in our interface. This is a heavy burden, and if we change the way we store any of them, it means changing the interface. The interface shouldn't concern itself with the implementation of the resources, it should just be getting them for the Engines to use.
We can solve this by standardizing the way we refer to our resources. We can, in effect, create a Resource ABC all other resource types inherit from. We can then store and pass along Resource type objects, and it will work equally well for shaders or models or entities. Only we can't pass along the Resource object (which, since it's an abstract class, can't be instantiated), but instead we'll be storing and passing along pointers to them.
This is so because while we can point to something that inherits from Resource with a Resource pointer, if we try to set up storage for a Resource object, objects derived from it won't fit in it. So we must store pointers to them instead.
This means our storage structures will no longer be holding the information. Instead, they'll be holding pointers to where the information we want is. It'll be up to the person asking for it to know what kind of information they asked for, however, to avoid calling members or methods from another type, which could kill the program.
Next post I'll look into this in more detail.
Basic program structure
In the Engine Layer, for example, we'll be having a Rendering Engine, a Physics Engine, a Gameplay Engine, AI Engine, etc. Each of them will drive one of the game's subsystems, called in turn by the Game. We want these to have a common interface where it comes to communicating with the Resource Layer. They'll generally want to ask for the same information, and trying to standardize how they do it can save us work in the long term.
For now I'll concentrate on getting the Rendering Engine working, which is the shiniest part of the project. Getting it working can help us see what is going on, and it'll make it obvious later on if one of the other parts of the game is acting oddly.
So, we start with a RenderEngine class. This class is going to inherit from an Abstract Base Class (ABC) called Engine. Doing this will allow us to refer to it as "a kind of" Engine, same as any other Classes we derive from it.
Engine inheritance graph
So, what does the rendering engine need? It needs access to the list of entities it must draw (ideally sorted or culled in some way, though we'll start with just getting it working with all of them), it needs access to the models that represents each entity, it needs the ModelViewProjection matrix and it needs the shaders that will be in charge rendering to the screen. It would also need textures, but not just yet.
These are a lot of different resources that must be handled. So how do we get a hold of them?
Well, we need a way of storing them in the Resource Layer. We can have a vector for each different kind of resource, for example. We can then use it's position in the vector as a unique identifier for each.
However, each kind of resource is fundamentally different. The way to prepare a shader for use is very different from the way a model or a texture are prepared, for example. We need specialized functions for how to store each. What we can do is create several manager classes, each specialized in handling a specific kind of resource, but with a common interface for delivering them in a predictable manner.
Certain functions will be common to all the managers, even if their implementation is different in each. This again is a reasonable situation to use inheritance. We can have a Manager ABC, from which several different managers will inherit (ShaderManager, ModelManager, EntityManager, etc).
Manager inheritance graph
So the rendering engine would then need only call on the shader manager to provide a given shader when needed, and the model manager for models, etc. However, we can see two problems with this. One, our engine would need to know how to talk to each manager. And since the managers are external to the engine classes, each engine will need to store the address of each manager.
This is a lot of duplicated information, not all of which would be of use. Instead, we could have a single system in charge of interfacing with the resource layer, and all the different engines would instead talk to it. Then, we'd only need to store it's location in each, and each engine could just focus on asking this interface for a kind of resource, instead of knowing how to talk to each manager.
Link between the Engine and Resource Layers
This is starting to look decent. However, there's another issue to deal with. Each of these resources is of a different type, and both the handling and storage of them breaks the uniformity we'd rather have. Models, for example, are being stored as GLTriangleBatch objects, whereas Shaders are stored as GLuint types and Entities as Entity objects. When trying to draw the models, the Rendering engine has to call a method of the GLTriangleBatch class, and to use the Shader it has to use an OpenGL API call with a GLuint parameter.
This means that while we went some ways into simplifying the connection to the Resource Layer, we have to have several different types of call methods in our interface. This is a heavy burden, and if we change the way we store any of them, it means changing the interface. The interface shouldn't concern itself with the implementation of the resources, it should just be getting them for the Engines to use.
We can solve this by standardizing the way we refer to our resources. We can, in effect, create a Resource ABC all other resource types inherit from. We can then store and pass along Resource type objects, and it will work equally well for shaders or models or entities. Only we can't pass along the Resource object (which, since it's an abstract class, can't be instantiated), but instead we'll be storing and passing along pointers to them.
This is so because while we can point to something that inherits from Resource with a Resource pointer, if we try to set up storage for a Resource object, objects derived from it won't fit in it. So we must store pointers to them instead.
This means our storage structures will no longer be holding the information. Instead, they'll be holding pointers to where the information we want is. It'll be up to the person asking for it to know what kind of information they asked for, however, to avoid calling members or methods from another type, which could kill the program.
Next post I'll look into this in more detail.
Sunday, June 19, 2011
Basic Design
The first step for building successful software is Design. A good design doesn't need to be the ultimate expression of the program, and it can always be redesigned in the future, but without an overarching structure it is difficult to know in which direction to start off.
For this project, I started with the basic loop structure every game has. This refers to the fact that in games, you initialize the game state (load resources, etc), then until the player closes the game you check for input, work on the input, and render to the screen. Then when the game is over you clean up and exit.
But that refers to the superstructure. The interesting stuff happens in between the lines. What I have been working on is the infrastructure that allows all that to happen.
After some thinking, I came to the conclusion that there are two main problems a game has to deal with. One is managing the game's resources; keeping track of entities, loading and removing models, textures, shaders, etc. The other issue is running the engines that use those resources to display the game, run collision detection, manage the game logic, etc.
Something like this:
From this initial design, some general needs become evident. We can't have all our resources jumbled up in a big pile. We need to organize them, and be able to provide them when required. Likewise, we must organize the different engines in some way as to be able to call each when it's their turn to act.
Additionally, there is another level we must consider, an Application Layer, that would be in charge of managing the interaction between the Game and the host machine. Fetching files and loading them in memory and the like. This could be a sort of resource, but we want to avoid giving any part of the program too broad a definition.
There is one further element of the program I haven't yet fit into my design, however, referring to the creation of resources. The backbone of my program is to eventually procedurally generate many of the resources. This wouldn't belong in any of the previous three layers, implying the need for another, a 'procedural layer' of some kind.
The Engine Layer should be completely decoupled from either the procedural or application layers, however the resources should be generated by either one of them. I'll have a look at each layer, as well as the communication between them, in the coming posts.
For this project, I started with the basic loop structure every game has. This refers to the fact that in games, you initialize the game state (load resources, etc), then until the player closes the game you check for input, work on the input, and render to the screen. Then when the game is over you clean up and exit.
But that refers to the superstructure. The interesting stuff happens in between the lines. What I have been working on is the infrastructure that allows all that to happen.
After some thinking, I came to the conclusion that there are two main problems a game has to deal with. One is managing the game's resources; keeping track of entities, loading and removing models, textures, shaders, etc. The other issue is running the engines that use those resources to display the game, run collision detection, manage the game logic, etc.
Something like this:
From this initial design, some general needs become evident. We can't have all our resources jumbled up in a big pile. We need to organize them, and be able to provide them when required. Likewise, we must organize the different engines in some way as to be able to call each when it's their turn to act.
Additionally, there is another level we must consider, an Application Layer, that would be in charge of managing the interaction between the Game and the host machine. Fetching files and loading them in memory and the like. This could be a sort of resource, but we want to avoid giving any part of the program too broad a definition.
There is one further element of the program I haven't yet fit into my design, however, referring to the creation of resources. The backbone of my program is to eventually procedurally generate many of the resources. This wouldn't belong in any of the previous three layers, implying the need for another, a 'procedural layer' of some kind.
The Engine Layer should be completely decoupled from either the procedural or application layers, however the resources should be generated by either one of them. I'll have a look at each layer, as well as the communication between them, in the coming posts.
Saturday, June 18, 2011
At last, progress!
Well, there's quite a bit that has happened since my last post here. As I was saying, I had decided to settle on SFML to handle context creation and interaction with the OS. After deciding on it, I thought long and hard about the program's structure, the basic design of the classes and the like.
I want to think that I came up with a solid design. It's helped me organice my thoughts and given me clear targets for the programming aspect. I also have to thank the OpenGL SuperBible, 5th Edition, for providing a solid grounding on the basics of graphics programming, and a set of functions that came in handy to deal with some common operations I'll be performing.
Now, what the program actually is doing is not very exciting. Here's a sample of the output:
What is interesting, instead, is how it's doing it.
I won't be describing the design of the program today, but let's just say there's enough material for several posts. It's still subject to change, and I've already identified some aspects of the program that don't entirely fit in the framework, but it's cool stuff.
Oh, I'm also working for real with source control, Git and Mercurial actually. I, err, couldn't make up my mind, so I'm giving both a spin. Also testing their associated sites, GitHub and BitBucket. They have some cool additions besides serving as storage for repositories, like wikis and issue tracking, but I haven't tried those out yet.
I'm very excited about all this.
I want to think that I came up with a solid design. It's helped me organice my thoughts and given me clear targets for the programming aspect. I also have to thank the OpenGL SuperBible, 5th Edition, for providing a solid grounding on the basics of graphics programming, and a set of functions that came in handy to deal with some common operations I'll be performing.
Now, what the program actually is doing is not very exciting. Here's a sample of the output:
What is interesting, instead, is how it's doing it.
I won't be describing the design of the program today, but let's just say there's enough material for several posts. It's still subject to change, and I've already identified some aspects of the program that don't entirely fit in the framework, but it's cool stuff.
Oh, I'm also working for real with source control, Git and Mercurial actually. I, err, couldn't make up my mind, so I'm giving both a spin. Also testing their associated sites, GitHub and BitBucket. They have some cool additions besides serving as storage for repositories, like wikis and issue tracking, but I haven't tried those out yet.
I'm very excited about all this.
Monday, June 13, 2011
Deciding on a new framework
After comparing, I've decided to switch to SFML to handle the OS specific aspects of the program. I didn't care for freeGLUT taking over the program loop, and SDL is going to be removing support for threading in upcoming versions. SFML mostly stays out of the way, and supports threading, which are two things I care about.
I'll have something to talk about soon,
I'll have something to talk about soon,
Monday, June 6, 2011
Back to basics
It might have looked like I left this again, but no, I've been busy working. I finally got my hands on a decent resource (OpenGL SuperBible, 5th Edition) and I'm looking to move everything to OpenGL 3.3 and start using shaders.
The good news is that this means I'm going to have far more powerful tools at my disposal. The downside is that I have to learn a lot, and I've spent all this time reworking everything from the ground up. I'll soon have a triangle up and running.
The upcoming examples may use GLUT instead of SDL, since that's what the Bible uses, but I've not decided that that's what I'll end up using.
The good news is that this means I'm going to have far more powerful tools at my disposal. The downside is that I have to learn a lot, and I've spent all this time reworking everything from the ground up. I'll soon have a triangle up and running.
The upcoming examples may use GLUT instead of SDL, since that's what the Bible uses, but I've not decided that that's what I'll end up using.
Subscribe to:
Posts (Atom)