Providing Feedback on Annotations

You can provide feedback on both published and draft datasets. Any user can provide feedback on published datasets, they do not have to be invited to the workspace; however only users who have been added to the private, draft workspace may provide feedback on them. Please see the following sections to learn more about both methods of providing feedback.

Providing feedback on published datasets

1. To provide feedback you will first need to select cells either on the embedding or in the Cell Labels side panel and then click the Provide feedback button to open the feedback window with three options: Agree, Disagree and Don't know.

2. To agree with an annotation, select Agree and either press Send My Feedback or first enter optional comments to include with your agreement.

3. To disagree with an annotation, select Disagree and then select from one of three options: whether the annotation should be split into two or more cell labels, merged with another annotation or general disagreement.

4. To disagree with an annotation and suggest it be split into two cell labels, select the cell label, click Disagree and then Disagree: Split. Then provide the optional cell label names and marker genes and the required explanation of why the cell label should be split.

5. To disagree with an annotation and suggest it be merged with another cell label, select the cell label, click Disagree and then Disagree: Merge. You will then need to select the cell label to merge it with and then click Continue Providing Feedback. You will be prompted to provide an explanation of why the labels should be merged and then click Send My Feedback.

6. To disagree and leave a comment, select the cell label, click Disagree and then Disagree with a comment. Provide the explanation of why you disagree with the annotation and click Send My Feedback.

7. To agree with the annotation but disagree with the metadata, select the cell label, click Disagree and then send your metadata refinements. Click Refine Metadata, enter your suggestions and click Send Changes.

8. If you are unsure about an annotation, select the cell label, click Provide Feedback and then select I Don't Know. You can optionally include an explanation or click Send My Feedback.

9. A summary of the feedback provided by all users can be viewed in the Cell Label side panel.

   

10. If needed, you can delete your feedback by clicking on the feedback you would like to delete in the side panel and click Delete Feedback in the pop-up window.

11. An overview of the feedback by each feedback category can be viewed on the embedding by selecting the relevant feedback type in the Plot Feedback dropdown menu.

12. Summary of feedback from all reviewers can be viewed by returning to the dataset page and clicking on Feedback Received. Additionally, you can download the feedback as a csv file by clicking on the Export CSV icon.

13. To sort the feedback by the most agreements, disagreements or I don't know responses, click on Sorting and select how you would like the feedback sorted.

14. To filter by the type of feedback or by the cell label click on the feedback type or cell label you would like to filter by.

Providing feedback on draft datasets

1. Only owners or admins can enter the internal review mode for draft datasets. To enter internal review mode, navigate to the Explore Dataset page and click Start Dataset Review. The dataset is now in Internal Review mode and cannot be edited until the review is cancelled.

2. To cancel the review, click the Stop Dataset Review button.

3. The feedback provided during the Internal Review can be copied over to the publication by clicking the checkbox when submitting the draft for review. The feedback will then be included on the publication. By default, this option is disabled.