Annotation sessions can be used to conduct scientific studies.
Annotation sessions are a way to organize annotations of a volume. An annotation session has a start date, an end date and a set of members that belong to it. Once an annotation session was created, all annotations of the volume that are created after the start date (inclusive) and before the end date (exclusive) by one of the members will belong to the annotation session. Annotation sessions can be created by all users who have admin access to the volume. Each volume can have multiple annotation sessions. Only one annotation session can be active at the same time. Annotation sessions can be managed by volume admins on the volume edit page that you can reach with the button of the volume overview.
Existing annotation sessions can be used to filter volume annotation reports. A report restricted to an annotation session only includes annotations that belong to this session (i.e. created by a member between the start an end dates of the annotation session).
Similar to filtering volume annotation reports, annotation sessions can be used to filter the displayed annotations in the annotation tool, too. Simply select the filter tool ( ), choose the "session" filter, type the name of the annotation session and press enter. Now only annotations belonging to the annotation sessions are displayed. In the image annotation tool, this can be combined with the Annotation Cycle Mode to review all annotations of an annotation session.
Example: Jane has admin access, Joe and Jack both have editor access to the volume T. Joe already created the annotations a and b on 2016-10-17 and 2016-10-18 respectively. Jack only created one annotation c on 2016-10-18. Now Jane creates a new annotation session for volume T, which starts at 2016-10-18 and ends at 2016-10-20. She also adds Joe as a member to this new session.
Since annotation a was created before the annotation session started and c was created by Jack who is not a member of the annotation session, only annotation b belongs to the annotation session.
Annotation sessions are great for grouping existing annotations but the main use case for them are annotation studies. By annotation study we mean a survey where researchers want to measure statistics like the intra- or inter observer agreement (as described in [1]). For this purpose, annotation sessions offer some additional options that control the display of annotations during an annotation session. There are three possible scenarios for annotation studies:
In this scenario you want to measure the reproducibility of the annotations of each user. Each user is asked to annotate all or just a subset of the objects of interest of a given volume. After a certain delay (at least 48 h), each user is asked again to annotate the same objects of interest. During the second pass the users are not able to see their annotations of the first pass. By comparing the two sets of annotations for each user, the reproducibility or intra observer agreement of their annotations can be calculated. For this type of annotation study, the annotation sessions offer the hide own annotations ( hide own ) option. If this option is activated, all annotations of the own user that were created before the annotation session started are invisible while the annotation session is active. This will only affect users who are member of the annotation session.
Example: You want to measure the intra observer agreement of the user Jessica for objects of interest in the volume T. Meanwhile Jacob is already performing an annotation task on the same volume (e.g. manually annotating all laserpoints). His annotation task should not be affected by your study.
First make sure that you have admin access and Jessica has at least editor access to volume T. Now create an annotation session A for volume T (e.g. lasting 48 h from 2016-10-19 to 2016-10-21), add Jessica as member and activate the hide own annotations option. Then create a second annotation session B that starts with a certain delay after session A and has the same duration (e.g. from 2016-10-24 to 2016-10-26), again with Jessica as member and hide own annotations activated.
Now ask Jessica to annotate the objects of interest for 48 h starting with the 18th of October and then all over again starting with the 24th of October. While the sessions A or B are active, Jessica will not see any of her previously created annotations which could distort the results of your study. Jacobs annotation task, on the other hand, will not be affected by the annotation sessions.
Once both annotation sessions were finished you can export the two sets of annotations by creating new volume annotation reports for T and restrict one of them to annotation session A and the other to B. With one report for each annotation session you have the raw data to compare the annotations and compute the intra observer agreement using a method of your choice.
In contrast to the intra observer agreement to measure the reproducibility of annotations of a single user, the inter observer agreement can be used to measure the reproducibility of annotations between multiple different users. In other words this is a measure for how well different users find the same objects of interest and assign the same label to it. For this, all users are asked to annotate objects of interest in a volume at the same time. While the users are annotating, they must not see the annotations of the other users. By comparing the annotations of different users, the inter observer agreement can be calculated. This type of annotation study requires the hide other annotations ( hide other ) option of annotation sessions. If this option is activated, members of the annotation session will not see annotations of other users while the annotation session is active.
Example: You want to measure the inter observer agreement of the users Jasmine, Julia and Jake for objects of interest in the volume T. You need admin access and Jasmine, Julia and Jake need at least editor access to T.
First create a new annotation session for T (e.g. lasting 48 h), add the three users as members and activate the hide other annotations option. Now you ask the members to annotate objects of interest while the annotation session is active. Each user won't see the annotations of the other users.
Finally you can export a new volume annotation report for volume T which is restricted to your annotation session, similar to scenario 1. Use the exported user IDs to separate annotations of different users and calculate the inter observer agreement with a method of your choice.
To measure both the intra- and inter observer agreement, you just have to combine scenarios 1 and 2. Create two annotation sessions with a delay of at least 48 h in between, and activate the hide own annotations and hide other annotations options in both. Finally, add all users who should participate in the annotation study as members of the annotation sessions.
Activating both options of an annotation session may not be relevant for this scenario only. You might want to hide annotations of other users during an intra observer agreement study or the older own annotations during an inter observer agreement study, too, depending on your requirements.