eLatin eGreek eLearn

More wired than a Roman Internet café

Rubric for assessing historical scenario-building for Civilization

One of the things that always amazes me about playing Civilization IV (or indeed, just about any game you’d care to name) is what might be called the ‘metagame’ - the discussions on the forums, the fansites, the user-created mods. It seems to me that this is one of the most important aspects of the educational use of commercial games. On Civfanatics, there is a discussion entitled ‘the Rise and Fall of Rome‘ which I find absolutely fascinating. These folks are not historians, they are not classics students, but in the course of trying to make an historically ‘authentic’ simulation of Roman culture they embrace such difficult concepts as the conditions behind the emergence of the Social War - and then they devise a way to allow for the possibility of a Social War emerging in the game play! (other historical scenarios in Civ IV available here)

That is the kind of discussion I would want to emerge in my classroom, were I to formally assign the creation of a Civ mod or scenario as part of the assessment of the course. The problem that I’m addressing in this post though is how would I assess the scenario, and the metagame? I’ve addressed the problem of assessment when students play a scenario (in my ‘Year of the Four Emperors’ scenario for Civ IV I assigned a ‘game diary’ that asked pointed questions of the students at particular points in the game) but I’ve only started to grapple with the problem of assessing construction recently. How can you be fair and assess two individual students, one who has a good technical grasp of python, xml, and scenario building but is hazy on the history, and one who knows the history but freezes at the sight of the worldbuilder? How do you mark the mass of material that will be produced as a byproduct? How do you manage the paper trail?

I had a similar problem during my dark old days as a high school teacher of technical drawing. The solution there was a rubric, and I think the solution here might also be a rubric. Rubrics have the advantage of boiling everything down to a checklist of various criteria. Your students can see at a glance what you are looking for, and they can see what they have to do to achieve a good grade. As the prof, you save yourself time, energy, and headaches. Below is my proposed rubric for marking the creation of a scenario for Civilization IV:

Rubric for assessing historical scenario-building for Civilization

The first criterion addresses the question: has the student selected a good problem to try to render in a scenario? Civilization has built in assumptions about how history unfolds. Does the proposed scenario play to those assumptions, or does it challenge them?

The second criterion assesses whether the student has assembled the appropriate secondary or primary literature to ensure the ‘authenticity’ of the scenario (and a very good student will explore just what makes for an authentic scenario).

The next two criteria are asking the student to plan out the scenario on paper first. Where will the issues be? What kind of a map? What scale is appropriate both geographically and chronologically? Clear writing = clear thinking = an easier time of building the scenario. My own scenarios at first suffered from woolly design…

The ‘demonstrates understanding’ criterion might be the place to assess whether the student realizes the problems of simulating history…?

The ‘uses forum/wiki’ criterion - I envision having a group forum or wiki for students to talk out their design problems, and to offer help, hints, and suggestions to each other as they design their scenarios. I’m envisioning each student designs their own scenario, but I want the experience to be a social one. This is especially important for my distance education students…

‘Identify design issues’ - I’m not sure whether to keep this or to discard it. It really should be moved up to the ‘design’ part of this rubric. I do want the students to be demonstrate that they are aware of the constraints the Civilization environment imposes.

The last two are performance related. A student who is otherwise a poor historian (and would get low grades in an essay-based course) would here have a chance to pick up some points - and demonstrate their historical knowledge through making.

So, that’s all off the top of my head this morning. I would be interested to know how others have approached (or if they’ve approached) the problem of assessing the use of games in an educational context in this manner. Should the rubric be expanded? Contracted? Is it hitting the right targets?

Views: 261

Comment by Laura Gibbs on November 15, 2007 at 7:11pm
Hi Shawn, this is a fascinating topic and one that I have wrestled with very much over the past years of teaching online where I have tried to change my role as teacher to really being more of a "coach" - putting more and more responsibility on the students wherever possible, giving them tasks to do which they can do, while relieving me to do tasks that only I am able to carry out in the class. I use rubrics in a more basic way than what you have outlined here, but it is critical to my class, so I thought I would show you how I use rubrics, and how it has helped me focus my efforts in the most valuable way. Teaching is a TIME SINK and one of my main goals is to make sure my teaching job takes exactly 40 hours per week or my time: no more, no less - it is a challenge! But after years of working on the method, I think I've found a solution for me.

Like you, my goal is to have the students produce a complex, multidimensional project that embodies technical ability, research, writing, personal creativity, elaboration of structure, etc.

I have not tried to find a rubric that encompasses that whole range of achievement, however - instead, I focus on the BASIC minimum requirements only when it comes to the rubric, and let students learn from looking at other students' work what the higher levels of achievement involve. At the beginning of the semester, they learn this by looking at past student projects (I have a huge archive online now), and then as the semester progresses, they are inspired to higher achievement by looking at the work of fellow students. That becomes what we could call the "implied" rubric for the class.

First, a caveat about my grading process which may make my approach less useful for you. Your rubric makes some basic assumptions about objective achievement and objective competencies that I use only at a much lower level. I use the rubric only for BASIC competency issues, and by basic, I mean really really really basic. There is an enormous range of ability in my students - in any of my classes, even though they are college classes, I have students whose reading and writing abilities range from barely 6th grade to post-graduate level. So, I do not feel comfortable with an achievement-based grading system (objective grading, call it what you will). That is the main reason why my rubrics define a minimum level of achievement, but they do not define a top level.

Instead, I operate based on this assumption: if students take responsibility for making sure their work meets the most basic level of the rubric, other factors can then come into play and inspire them to achieve at higher levels. Those higher-level achievements are NOT graded, and are not part of the rubric they are given - but they emerge through the dialogue that I have with the students, and that the students have with each other looking at each other's work.

I think this will be more clear if I give examples. In all my classes, the students create a website called a "Storybook" which contains a set of stories, based on a topic of their choosing, which they research and then retell in their own words. They work on this project all semester long, so the project is segmented into a bunch of different pieces - writing segments, publishing segments, and so on. In the end, they produce websites like the ones you can see here:

Indian Epics

A typical rubric is something THEY must complete when they turn in the assignment for the week. A typical weekly assignment is like the one they turned in this week for Week 12: Week 12 assignment.

As you can see, this rubric is a kind of self-diagnostic intervention to make sure the project is ready to turn in. This is a huge help to me in the student taking responsibility for a certain level of effort before the story is passed off to me to read, and passed off to the other students.

Then, the application of rubrics becomes less formal. When students comment on each other's writing, I give them some prompts to help them make comments on each other's writing, but it is not a formal rubric per se:
Comment Guidelines.

In addition to getting comments back from 2-3 students per week (it's random, but it works to about 2-3 students per week on average), the students also get comments from me every week. Rather than evaluating their work with a rubric - which simply freaks them out and makes them feel "graded" - I give them RUNNING COMMENTS on their writing. These comments contain my reactions as I'm reading, along with grammar alerts (many of my students have serious writing deficits), plus cool web research suggestions, suggestions to look at other students' work, and so on. I spend 15-20 minutes per student per week on this comments process - and with approximately 80-90 students per semester, that means I spend a solid 20-30 hours per week commenting on their work. Clearly, it is the bulk of my work for the class each week, but it fits in my 40 hours per week time allotment.

That might sound kind of penny-pinching, but the results are phenomenal.

With regular weekly comments from me and other students in the class, ALL students improve their writing dramatically.

Accomplished students produce INCREDIBLE websites, which in turn serve as an inspiration to lagging students.

Lagging students make incredible PROGRESS from the start of the semester to the end, even if their final products do not compare to the work of the super-accomplished students.

So, I guess the main message I wanted to convey here is that you do not have to have a rubric that defines the higher achievements you hope for in your students. Instead, it is possible to use the rubric simply for the basic minimum (hence a much simpler rubric!), while then using something else - call it "Peer Models" or something if you want - to prompt higher achievement.

The benefit in the end is that the focus is not on me as the evaluator... instead, the focus is on the students as MODELERS, as ACHIEVERS... I am deeply involved by coaching the students with my comments, giving them tons of prompts to help them achieve more - but the actual definitions of achievement that they see around them are produced by the other students.

At least for me, it makes the classes a ton of fun to teach, with a completely different "feel" to them than something more strictly driven by yours truly, the so-called teacher.

Comment by Shawn Graham on November 16, 2007 at 10:48am
Hi Laura -

Thank you very much for your comments, and for the examples of your students' work! Fantastic stuff.

It never occurred to me to make the thing open ended, as you say... hmmm. How I wish I actually taught face to face! What you suggest I suspect would be easier with a class of students altogether in one place than with my usual gang of dispersed distance ed students - in practice. In principle, it should work. I need to think this through some more... but your model is very attractive!

I do wonder though - part of my rationale for a full rubric is that it takes (a bit of) the mystery out of how I grade. I agree that the process for the student is far more important than any number that I assign, but I find that students in general are more often interested in the final grade than in any of my comments or advice along the way... how do you end up with the final number or letter that ultimately must be awarded?

By the way, I swapped wikis for journals in one of my online classes, and that seems to be an assignment type that - for this group, at least - proves more satisfactory than the wikis. It seems the private v. public nature of the two assignment types is making the difference this time.
Comment by Laura Gibbs on November 16, 2007 at 11:03am
hi Shawn, I too teach 100% online - I have never seen any of my students face to face... that is how I evolved this system, in fact: how to give students the whole-hearted totally enthusiastic encouragement that I could do inperson in the classroom, but instead doing that ONLINE. and it works.. BETTER than in the classroom - I would never go back in the classroom now; teaching online is a gizillion times more effective, at least for me.

as for the grading, I understand exactly what you mean about taking the mystery out of it - that is essential: I do not give the students grades; they earn them. I use a points-based system which the students love, because it puts them in absolute control of their grade. since my courses are "Gen Ed" courses for people to graduate, but not part of their major, they are often quite content to just get a C or a B, so long as they pass the class, so they can in PLAN to get a C or a B, which is fine with me (so for example they can finish the class a month early if they don't mind just taking a grade of C, which is fine with me - my only interest is that they get a benefit out of what they do, and how much or how little they do is up to them).

every week there are 30 points of required work, plus a few points of extra credit in case they accidentally miss a required assignment - every week has the same "menu" you might call it:
Cherokee Week

the points are cumulative and the grade is simply based on total points accumulated: 410 points by the end of the semester is an A, 360 points is a B, 320 points is a C.

what has been brilliant about this is that the students know that they get their grade because of what they do (or don't do) - of the hundreds and hundreds of students who have taken these courses since 2002, I have not had one single grade complaint ever. I LOVE THAT. my job is not to evaluate or give grades: my job is just to build the environment in which the students do the work, and encourage them every step of the way. some students fail the course every semester, but that's simply because they don't have the 8 hours per week or time that it takes to do the work... and they know that it's about their lack of time, not anything I have done as the teacher "evaluating" them in some prejudicial way.

When I first started teaching this way, I thought the more capable and hard-working students would resent it - that I had "dumbed down" the class - but NOT AT ALL. the students who are great writers in class thrive on the accolates and sincere admiration the other students have for their abilities - that's really cool! they enjoy the peer accolades as much or more than any accolades they get from me... although they sure get accolades from me, too, because I am so thrilled by their work.

you are a gaming person, so check out this genius project in my Indian Epics class - it is the heroes of the Ramayana and the Mahabharata put into different gaming formats! I think it is FABULOUS!

Comment by Laura Gibbs on November 16, 2007 at 11:05am
whoops, last link didn't come through - I'll try again:

Sidekick Quest
Comment by Shawn Graham on November 16, 2007 at 3:41pm
Ah! I didn't realize! I'm going to mull over and digest what you've written. Lots of things to consider!

Fantastic project re Ramayana, I like the whole text-adventure approach - I've been using the Inform interactive-fiction platform (think word processor for writing 'choose your own adventures) to create text adventures which ideally would be used to teach historical literacy... an early project of mine lives here http://electricarchaeologist.wordpress.com/category/interactive-fiction/
Comment by Laura Gibbs on November 16, 2007 at 4:08pm
ha: the Canadians on the Nile looks so cool!

I think in a sense my class could be considered "baby steps" towards such a goal: my students learn the basics of creating a story and defining a narrator - but even that is a struggle for many of them, and it takes them all semester to really feel comfortable with writing in an actual VOICE as opposed to the impersonal, mind-numbing kind of writing they do for their other college classes.

if my class were not just a one-off kind of class that they take, usually in their final semester of school, just in order to graduate, I would love to build it into a larger project: after doing a Storybook project like the one they do now, they would really be ready to start to participate in a more focused, higher level, designing and/or contributing to the kind of interactive experience you have in mind. I think it would baffle them to try to do something like that in my class now, but after a semester of thinking about stories and storytellers, they would really get into a project of that sort - their creativity is really boundless, but focus, technique, planning, these are the elements they really need help to cultivate.

Comment by Laura Gibbs on November 17, 2007 at 7:34pm
Hi again, Shawn, I'm not sure if this discussion will go anywhere at classroom2.0 (another GREAT ning), but it's definitely something of interest to you I think:

2.0 Assessment
I need checklists, rubrics, or student planning sheets for some of the 2.0 tools we are using in the classroom. I want to give the students some direction and hold them accountable for what they are doing online (after a period of messing around, of course!). Has anyone designed "assessment" tools for Scratch, Alice, Animoto, Floorplanner, Toondoo, Civ IV, Age of Empires, TheSims, Sim City, etc?
Comment by dangelo on December 28, 2009 at 2:56pm
building for civilization thats right civilization u need to have good people also let me tell u that for a good health Viagra Online


You need to be a member of eLatin eGreek eLearn to add comments!

Join eLatin eGreek eLearn



© 2024   Created by Andrew Reinhard.   Powered by

Badges  |  Report an Issue  |  Terms of Service