In its first year, NAACL received 166 submissions. 43 papers were accepted, giving an acceptance rate of 26%.
With input from the NAACL and ANLP program committees, I chose to invite Len Talmy to give a plenary talk.
Providing a detailed country breakdown for submissions and acceptances is difficult, because many papers are multi-authored. However, a rough idea can be gained from the data we have on host countries of the first authors. This gives us a submission profile of:
93 from North American (56% of submissions)
54 from Europe (33% of submissions)
18 from Asia and (11% of submissions)
New Zealand
1 from Brazil ( 0% of submissions)
The acceptance profile is:
27 from North America (29% acceptance rate)
15 from Europe (28% acceptance rate)
1 from Asia ( 0% acceptance rate)
0 from Brazil ( 0% acceptance rate)
Apparently, more North Americans and Europeans than Asians decided to submit their better work to NAACL. Perhaps this is due to the fact that ACL is in Asia this year.
I invited program committee members to be session chairs, starting with area chairs, and filling the remaining slots with reviewers.
The program committee was hierarchical, with a senior program committee composed of 8 area chairs and a program chair, and 113 reviewers who reported to the area chairs. All members of the senior program committee are from North America (7 from the USA and 2 from Canada). The reviewers represent 20 different countries, and all are experienced reviewers. 72 (64%) of them are from North America; 30 (27%) are from Europe; and 11 (10%) are from Asia.
The lists of reviewers were coordinated, such that only in a few exceptional cases were reviewers asked to review for more than one area.
The area chairs cannot be thanked enough for the conscientious and painstaking jobs they performed. The reviewers were also excellent, providing careful and timely reviews of the papers they were assigned.
The NAACL areas are
They include areas traditionally well-represented at ACL conferences, and also areas we would like to expand. The feeling among the senior program committee is that having separate areas in Spoken Language and Multilingual Natural Language Understanding helped attract quality submissions in those areas. Unfortunately, the Cognitive Modeling and Human-Computer Interaction area was not as successful.
Following is breakdown of submissions and acceptances by area:
Submissions Acceptances Acceptance
Rate
Discourse, Dialogue, and Pragmatics 23 4 17.4
Semantics and the Lexicon 25 7 28.0
Syntax, Morphology, and Phonology 28 9 32.1
Generation and Summarization 17 3 17.6
Spoken Language 15 4 26.7
Corpus-Based and Statistical
Natural Language Processing 33 11 33.3
Cognitive Modeling and
Human-Computer Interaction 7 1 14.3
Multilingual Natural Language
Processing 18 4 22.2
In its first year, it was advantageous to co-locate NAACL with ANLP, an established conference. To coordinate the two conferences, submissions focusing on end-applications were invited to ANLP 2000, while submissions focusing on methodology were invited to NAACL 2000. Future NAACL conferences may encourage both types of submissions.
Although I do not endorse the split into “end-applications” and “methodology”, this split posed no particular problems for NAACL.
Reviewing was blind to reviewers and area chairs. Information about authors’ identities was given to program committee members in a few cases when it was needed to confirm conflicts of interest.
Papers were reviewed by 3 reviewers each. Once the reviews were complete, the reviewers discussed among themselves the papers for which there was disagreement. Based on the reviews and subsequent discussions, the area chairs identified the papers that appeared to be clear acceptances, clear rejections, and borderline cases.
At the senior program committee meeting held in Crystal City, Virginia, January 29, 2000, 9AM to 6PM, all of the papers that had been judged to be apparent borderline and accepted papers were discussed, and final acceptance decisions were made.
Extending Henry Thompson’s electronic review form for EACL ‘99, staff at the Computing Research Laboratory at NMSU developed software to enable electronic handling of the reviewing processes. Reviewers obtained the papers electronically from the Web; Area chairs were provided with an on-line system for assigning papers to reviewers; and the reviewers submitted their reviews electronically.
The electronic process probably required more of the reviewers’ time than a hardcopy process would have. Anticipating this, I kept the reviewer load low, with an average of roughly 4 papers per reviewer. As mentioned above, all of the reviewers are experienced reviewers.
ACL is at an awkward stage in its growth. It is too small to support office staff to handle submissions, distribution to reviewers, etc., but it is large enough that handling the submissions and reviews is a significant burden for the program chairs. Between the two conferences, NMSU had to handle almost 300 submissions, with little time to prepare. In addition, the time requirements for area chairs are quite significant. The electronic software made the tasks of the program and area chairs feasible. In my view, the reviewers had to spend a modest amount of extra time to deal with the software, which was mitigated by the lower than average reviewer load. Although specifying the software and overseeing its development was time-consuming, overall the savings in my and the area chairs’ time was significant.
I am grateful to the Computing Research Laboratory for donating significant internal administrative and technical support to ANLP-NAACL. Specifically, Linda Fresques worked on all aspects of the process, from managing the submissions to preparing the proceedings, and Eli Kane developed and maintained the review assignment and submission software. Lori Allen and Jacob Feddersen also helped with the submission process.
We received more positive than negative feedback about the software. We received negative feedback from two people.
In the instructions to reviewers, the reviewers were given the option to receive hardcopies of the papers. One person, Karen Sparck-Jones, took us up on this offer, after having been unsuccessful printing the papers from the Web. At she requested, she was also able to submit her reviews in e-mail. She asked that her comments be passed to the executive committee, so I append them below.
In addition, David Tram was negative about the software. I also append his comments below. In my opinion, Eli did an excellent job on the software given the time-frame available. Perhaps David’s comments will be useful to future program chairs.
Many thanks to Marie Meteer for her service and support in her role as General Chair. I would also like to express my gratitude to Kathy McCoy and Priscilla Rasmussen for their consistent support and guidance, and to the ACL and NAACL Executive Committees.