Distributional Semantics and Compositionality
Call for Papers Shared Task Important Dates Program Committee Contact
A Workshop at ACL/HLT 2011, in Portland, Oregon.
Introduction
Any NLP system that does semantic processing relies on the assumption of semantic compositionality: the meaning of a phrase is determined by the meanings of its parts and their combination. However, this assumption does not hold for lexicalized phrases such as idiomatic expressions, which causes pain points not only for semantic, but also for syntactic processing, see (Sag et al. 2001). In particular, while distributional methods in semantics have proved to be very efficient in tackling a wide range of tasks in natural language processing, e.g., document retrieval, clustering and classification, question answering, query expansion, word similarity, synonym extraction, relation extraction, textual advertisement matching in search engines, etc. (see Turney and Pantel 2010 for a detailed overview), they are still strongly limited by being inherently word-based. While dictionaries and other lexical resources contain multiword entries, these are expensive to obtain, not available for all languages to a sufficient extent, the definition of a multiword varies across resources and non-compositional phrases are merely a subclass of multiwords. The workshop brings together researchers that are interested in extracting non-compositional phrases from large corpora by applying distributional models that assign a graded compositionality score to a phrase as well as researchers interested in expressing compositional meaning with such models. This score denotes the extent to which the compositionality assumption holds for a given expression. The latter can be used, for example, to decide whether the phrase should be treated as a single unit in applications. We emphasize that the focus is on automatically acquiring semantic compositionality. Approaches that employ prefabricated lists of non-compositional phrases should consider a different venue.

This event consists of a main session and a shared task.

References:
Ivan A Sag, Timothy Baldwin, Francis Bond, Ann Copestake, Dan Flickinger (2001): Multi-word Expressions: A Pain in the Neck for NLP. In Proc. of the 3rd International Conference on Intelligent Text Processing and Computational Linguistics (CICLing-2002), Mexico City, Mexico

Turney, P. and P. Pantel. (2010). From Frequency to Meaning: Vector Space Models of Semantics. Journal of Artificial Intelligence Research, 37, 141-188.


Call for Papers
For the main session, we invite submission of papers on the topic of automatically acquiring a model for semantic compositionality. This includes, but is not limited to:

Models of Distributional Similarity
Graph-based models over word spaces
Vector-space models for distributional semantics
Applications of semantic compositionality
Evaluation of semantic compositionality
Authors are invited to submit papers on original, unpublished work in the topic area of this workshop. In addition to long papers presenting completed work, we also invite short papers and demos:
- Long papers should present completed work and should not exceed 8 pages plus 1 page of references

- Short papers/demos can present work in progress or the description of a system, and should not exceed 4 pages plus 1 page of references.

As reviewing will be blind, please ensure that papers are anonymous. The papers should not include the authors' names and affiliations or any references to web sites, project names etc. revealing the authors' identity.


Shared Task
The organizers extracted candidate phrases from two large-scale freely available web-corpora, UkWaC and DeWaC (cf. http://wacky.sslmit.unibo.it/), containing respectively English and German POS tagged text. These data have been manually evaluated for compositionality with Amazon Turk. Workers were presented a sentence with a bolded target phrase and were asked to score how literal the phrase was between 0 and 10. 4-5 different, randomly sampled sentences from the WaCKy corpora for UK English and German were presented to 4 workers each.

Phrases consist of two lemmas and come in three grammatical relations:

ADJ_NN: adjective modifying a noun
V_SUBJ: noun as a subject of a verb
V_OBJ: noun as an object of a verb
Phrases were extracted semi-automatically. The relations were assigned by patterns and manually checked for validity. Phrases were selected in a way as to balance the data set while controlling for frequency.
Judgments: numerical and coarse
Scores were averaged over valid (non-spam) judgments per phrase and normalized between 0 and 100. These give the numerical scores that are used for the Average Point Difference score. For coarse-grained scoring, phrases with numerical judgments between 0 and 25 were assigned the label "low". Numerical judgments between 38 and 62 received the label "medium", judgments over 75 got the label "high". All other phrases were removed from the coarse judgment data.
Data
Format
Judgments are provided in a three-column-format:
relation, e.g. EN_V_OBJ
phrase in its WaCKy corpus base form, e.g. "provide evidence"
judgment, e.g. "38" or "high"
Volume
These are the total number of items by language and relation (items with coarse scores in parentheses):
English:

ADJ_NN: 144 (102)
V_SUBJ: 74 (56)
V_OBJ: 133 (96)
German:
ADJ_NN: 122 (95)
V_SUBJ: 64 (59)
V_OBJ: 108 (84)
The complete data was split into 40% training, 10% validation and 50% test.
Training and validation data can be downloaded here:
[TRAINING and VALIDATION SET].
The archive contains data sets for compositionality judgments for English and German as well as the official scoring scripts.
Participants of the task are free to choose whatever method and data resources they will use in their submission. Prefabricated lists of multiwords are not allowed. Since the data set is derived from the WaCkY corpora, participants are strongly encouraged to use these freely available text collections to build their models of compositionality, thus ensuring the highest possible comparability of results. Furthermore, since the WaCkY corpora are provided already POS-tagged and lemmatized, the workload on the participants' side is considerably reduced. This information (POS tags and lemmatization) may or may not be used by the participants. If needed, additional linguistic annotations or processing may also be added to the corpora. For obtaining the WaCky corpora, please email us (disco2011workshop @ gmail.com) for instructions to minimize load on the WaCky organizers. Of course, you can also directly contact the WaCky community at http://wacky.sslmit.unibo.it/doku.php?id=start.

For the challenge, participants submit their system's output on the test set to the task organizers, who score the systems and provide the official scores. Two scoring scripts are provided with the training set:

measuring the mean error by computing the average difference of system score and test data score;
binning the test data scores into three grades of compositionality (non-compositional, somewhat compositional, compositional), ordering the system output by score and optimally mapping the system output to the three bins.
The motivation for (a) is to reproduce the training data scores, the motivation for (b) is to give credit to systems that order the phrases correctly by compositionality but scale scores differently -- something that is easily 'fixed' in applications by appropriate thresholds. Additionally, a unified web application to calculate rank correlation coefficients (Spearman and Kendall) will be provided later.
Participants further submit a 4 page system description for publication in the workshop volume.

Interested in participating in the shared task? Please join the following Google group:
[DiSCo 2011 SHARED TASK]
Important Dates
Test data release: March 31, 2011
Regular paper submission deadline: April 1, 2011
Test data submission and system description deadline: April 8, 2011
Notification of acceptance: Apr 25, 2011
Camera-ready deadline: May 06, 2011
Post-ACL Workshop: June 24 , 2011



Program Committee
Enrique Alfonseca, Google Research, Switzerland
Tim Baldwin, University of Melbourne, Australia
Marco Baroni, University of Trento, Italy
Paul Buitelaar, National University of Ireland, Ireland
Chris Brockett, Microsoft Research, Redmond, US
Tim van de Cruys, INRIA, France
Stefan Evert, University of Osnabrück, Germany
Antske Fokkens, Saarland University, Germany
Silvana Hartmann, TU Darmstadt, Germany
Alfio Massimiliano Gliozzo, IBM, Hawthorne, NY, USA
Mirella Lapata, University of Edinburgh, UK
Ted Pedersen, University of Minnesota, Duluth, USA
Yves Peirsman, Stanford University, USA
Peter D. Turney, National Research Council Canada, Canada
Magnus Sahlgren, Gavagai, Sweden
Serge Sharoff, University of Leeds, UK
Anders Søgaard, University of Copenhagen, Denmark
Daniel Sonntag, German Research Center for AI, Germany
Diana McCarthy, Lexical Computing Ltd., UK
Dominic Widdows, Google, USA
Contact
Workshop Chairs:
Chris Biemann, San Francisco, USA
Eugenie Giesbrecht, FZI Research Center for Information Technology at the University of Karlsruhe, Germany
Emiliano Guevara, Institute for Linguistics and Scandinavian Studies, University of Oslo, Norway