Cultural heritage institutions and private collections such as the Library of Congress or the David Rumsey Map Collection are increasingly providing free online access to high-resolution scans of old maps. With the YUMA Map Annotation Tool, we want to facilitate collaborative scholarly annotation for such online resources. A central feature of our tool is the integration of semantic linking into the annotation process: annotations are semi-automatically enriched with context information from sources on the Linked Data Web. We argue that this semantic contextualization is, on the one hand, relevant for scholarly collaboration. On the other hand, we believe that it can be exploited to improve search and retrieval in the collection by providing additional structured metadata and geo-referencing information. In this paper we present current work in which we aim to verify our assumptions based on maps provided by the Library of Congress, and annotations collected from volunteer users in an ongoing crowdsourcing experiment.