Jon_Knowles
Member
RVTrainee001 wrote:
“The video was informative and found the analyses interesting. Would be nice to see some actual session data contrasting the higher FoM and lower FoM sessions.â€Â
There is one session for two images. Since we never see the image that does not correspond to the event that happened, we can’t contrast the higher and lower FoM’s for each image (target).
Here is an example of a CAS session and a bit about the scoring during testing of the software.
Procedure:
The tasker and coder (Ed May) said to write my name at the top of a blank sheet of paper and to add the date and time.
Tag was: “Describe the photo I will be showing you in a few minutes.â€Â
I made a quick sketch (page 1) and, being new to this, asked if tasker wanted “moreâ€Â.
2nd Tag: “What do you see looking down on the arches (which I had drawn and named)?†(Page 2 data) (Note that the tasker has not seen either photo at this point.)
3rd Tag: “What do you see from 1000 feet up looking down?†(Page 3 data).
Session length: c. 3 minutes.
The words in the session (see pdf file) are:
P 1: 2 buildings arches
P 2: From top (of arches) Hills, mounds, gorge? / river?
P 3: hills/ mountains/ raised area, channel EOS = End of Session
Coding: Person (Coder) then moves the sliders on the coding sheet. (I am not displaying the sheet because it is better if (present and potential) viewers not know the 24 categories used on the coding sheet.)
The photo not the site itself is deemed the target in this setup and the viewer is asked to go to the photo. If data in the session is not visible in the photo, it is not to be credited by the Coder.
Coding time: c. 2 minutes.
A coin was then flipped to determine the outcome - either Target A or Target B. The computer returned a Figure of Merit (FoM) assessment. The FoM for Target A was .4626. The accuracy was .57 (how much of target is in the session data) and the reliability was .807 (how much of session data is in target). Multiplying the two gives the FoM.
This was deemed an “industrial strength†result and “investment worthyâ€Â. The outcome was target A, which corresponded to the .4626 FoM.
Note the high score even with a bare bones session like this one. Ed May suggests viewers draw or say their immediate responses. In the one CAS group trial we have run, viewers tended to have more words and sometimes more detailed drawings. We had perfect accuracy (4 out of 4) but passed 80% of the time. If neither Figure of Merit is .4519 or above, you pass. Ed May says he usually gets 70% passes with the software.
NOTE: TKR has a limit of c 192 KB per image. The pdf file for this session has 5 images and is 465 KB. I will post the pdf on my web site and link to it - unless there is another solution at TKR I'm not aware of.
“The video was informative and found the analyses interesting. Would be nice to see some actual session data contrasting the higher FoM and lower FoM sessions.â€Â
There is one session for two images. Since we never see the image that does not correspond to the event that happened, we can’t contrast the higher and lower FoM’s for each image (target).
Here is an example of a CAS session and a bit about the scoring during testing of the software.
Procedure:
The tasker and coder (Ed May) said to write my name at the top of a blank sheet of paper and to add the date and time.
Tag was: “Describe the photo I will be showing you in a few minutes.â€Â
I made a quick sketch (page 1) and, being new to this, asked if tasker wanted “moreâ€Â.
2nd Tag: “What do you see looking down on the arches (which I had drawn and named)?†(Page 2 data) (Note that the tasker has not seen either photo at this point.)
3rd Tag: “What do you see from 1000 feet up looking down?†(Page 3 data).
Session length: c. 3 minutes.
The words in the session (see pdf file) are:
P 1: 2 buildings arches
P 2: From top (of arches) Hills, mounds, gorge? / river?
P 3: hills/ mountains/ raised area, channel EOS = End of Session
Coding: Person (Coder) then moves the sliders on the coding sheet. (I am not displaying the sheet because it is better if (present and potential) viewers not know the 24 categories used on the coding sheet.)
The photo not the site itself is deemed the target in this setup and the viewer is asked to go to the photo. If data in the session is not visible in the photo, it is not to be credited by the Coder.
Coding time: c. 2 minutes.
A coin was then flipped to determine the outcome - either Target A or Target B. The computer returned a Figure of Merit (FoM) assessment. The FoM for Target A was .4626. The accuracy was .57 (how much of target is in the session data) and the reliability was .807 (how much of session data is in target). Multiplying the two gives the FoM.
This was deemed an “industrial strength†result and “investment worthyâ€Â. The outcome was target A, which corresponded to the .4626 FoM.
Note the high score even with a bare bones session like this one. Ed May suggests viewers draw or say their immediate responses. In the one CAS group trial we have run, viewers tended to have more words and sometimes more detailed drawings. We had perfect accuracy (4 out of 4) but passed 80% of the time. If neither Figure of Merit is .4519 or above, you pass. Ed May says he usually gets 70% passes with the software.
NOTE: TKR has a limit of c 192 KB per image. The pdf file for this session has 5 images and is 465 KB. I will post the pdf on my web site and link to it - unless there is another solution at TKR I'm not aware of.
