These are the instructions for the NAACL 2015 review form as used by the reviewers in NAACL HLT 2015. Authors who have submitted a paper to NAACL 2015 can use this as a guide to understand the numeric scores in the reviews they have received.

Review Form

Please complete your review using the following guidelines for the scored categories.

Appropriateness (1-5)

Does this paper fit in NAACL-HLT?

The NAACL HLT 2015 conference covers a broad spectrum of disciplines aimed at: building intelligent systems to interact with humans using natural language; understanding computational and other linguistic properties of languages; and enhancing human-human communication through speech recognition, automatic translation, information retrieval, text summarization, and information extraction.

Both empirical and theoretical results are welcome; see the Call for Papers.

Clarity (1-5)

For a reasonably well-prepared reader, is it clear what was done and why? Is the paper well-written and well-structured?

Originality (1-5)

How original is the approach or problem presented in this paper? Does this paper break new ground in topic, methodology, or insight? A paper can score high for originality even if the results did not show a convincing benefit.

Soundness/Correctness (1-5)

Is the technical approach sound and well-chosen? Can one trust the claims of the paper – are they supported by proofs or proper experiments where the results of the experiments are correctly interpreted?

Impact of Ideas/Results (1-5)

How significant is the work described? If the ideas are novel, will they also be useful or inspirational? If the results are sound, are they also important?

Meaningful Comparison (1-5)

Do the authors place their work well with respect to existing literature? Are the references adequate? Are any experimental results meaningfully compared with appropriate prior approaches or other baselines?

If you feel references are inadequate be sure to include the relevant references in your comments.

Substance (1-5)

Does this paper have enough substance, or would it benefit from more ideas or results? Note that this question mainly concerns the amount of work; quality is evaluated in other categories.

Replicability (1-5)

Will members of the research community be able to reproduce or verify the results described in this paper? A lower score might be assigned if an insufficient amount of detail has been provided, if there is a highly subjective component to the setting of certain parameters, or if proprietary data have been used in the experiments. A low score here does not necessarily imply a low overall recommendation.

Members of the ACL community…

Recommendation (1-5)

There are many good submissions to NAACL-HLT 2015; how important is it to feature this one? Will people learn a lot by reading this paper or seeing it presented?

In deciding on your ultimate recommendation, please think over all your scores above. But remember that no paper is perfect, and remember that we want a conference full of interesting, diverse, and timely work. If a paper has some weaknesses, but you really got a lot out of it, feel free to fight for it. If a paper is solid but you could live without it, let us know that you would rather not see it in the conference. Remember also that the author has about a month to address reviewer comments before the camera-ready deadline.

Please do take the length of the submission into account. Rank short submissions relative to other short submissions, and full-length submissions relative to other full-length submissions. Acceptable short submissions include small, focused contributions; works in progress; negative results; opinion pieces and interesting application notes.

Reviewer Confidence (1-5)