Metacognition Knowing what you Know Improvement Method

The foundation of metacognition is knowing what you know and what you don’t know. This is based on a feeling. When you know (or think that you know) that you know something, there is a feeling of certainty. My students and I looked into this a few years ago. Our measurements showed us that first-year university students were able to ascertain whether or not they knew something was at a chance rate. After a few weeks (six) we were able to teach them to recognize that feeling with a greater degree of certainty.

The method we developed is quite simple. We had students come into the lab and engage in a “Pub Quiz”. They were given general knowledge questions to answer, and we calibrated it so that they would usually get about half of the questions correct. In addition to answering the questions – a question stem with four alternatives – they had to indicate whether or not they knew the answer was correct or were just guessing.

The scoring contingency that they were working on was as follows:

• Incorrect answer with a guessing response was worth 0
• Incorrect answer with a certain response was worth -6
• Correct answer with a guessing response was worth 1
• Correct answer with a certain response was worth 3

They got immediate feedback after each question with a new tally of their score. After the first session, we found that they were simply at chance performance when determining whether they recognized their own certainty or not (0.50 on an a’ signal detection calculation). We demonstrated this on a number of different groups of incoming students. After six weeks of repeating the pub quiz exercise, the students improved their signal detection score from 0.05 to between 0.85 – 0.90.

And they enjoyed the task.

Since the question was specifically based on how upper-middle aged children (age 10 - 12) could be taught this basic skill and I had made reference to this in my article, here it is. I had an app built that put the method I used out there and then, based on my own observations – with my grandchildren – I observed that they were able to increase this certainty feeling down to about my 11-year-old granddaughter. Unfortunately, after moving and leaving my job in the UK, I couldn’t afford to keep the app ungraded for the changes in operating systems so it doesn’t work anymore.

Pretty simple and effective measurement along with the added benefit of developing a little bit of self insight. Thanks for asking

For those interested in the gory details of what an a’ measure is, you can keep reading. I was a stats teacher for 20 years, so I can't really control myself here. If you keep reading and end up in tears, this is my disclaimer.

Signal detection theory is a method of determining how well someone can determine a signal amongst noise. The illustration that I have always used (and the best one I know of) has to do with a sonar operator on a submarine in the 1940s (I think computers do this now). It can also be applied to any job where a signal has to be detected through background noise - like a radiographer trying to determine something on a scan.

For a sonar operator, if a ping comes back to a sonar operator indicating another submarine in the vicinity, does the operator order a torpedo launch? It could be nothing more than a school of fish and that would be a waste of a torpedo - but the cost of missing another sub is pretty high. Signal detection theory was developed as a way to measure the ability of an operator to distinguish signal from noise. I’m not sure the story is actually true, but the illustration is good.

There are two signal detection measures that can be used to determine the effectiveness of an operator. The most common one is a d’ measure that tells you the distance between the peaks of the distributions of correct hits (detected a real submarine) and misses (thought a submarine was a school of fish (I hope there were not a lot of live trials for this one – not a great training method). You also get a rate of false alarms (firing a torpedo at a school of fish) and correct rejections (not wasting a torpedo on the fish).

The other measure that can be used is a framework that plots out the area that can be covered by these four possibilities (hits, misses, false alarms, and correct rejections) called a'.

I prefer the a’ measure because the number a' number itself means something. When the area of the possibilities is split in the middle like the illustration shows (a’ = 0.50), a person is guessing. If the a’ measure curves to the left toward the 100% hit corner, the detection is getting better with more subs detected and fish schools rejected. An opposite curve toward the bottom right means that the detection may be improving, but in the wrong direction with more and more of the subs being missed and the poor fish schools are being blown up.

There you go. Signal detection in a nutshell. This article gives you a two for one learning experience. Who could ask for more?