From the ground up

Our process in planning the conference has been informed by the theme we chose for this year’s NAACL – privacy and model bias. Wherever there was a decision to be made, we opted for the path which seemed to best ensure diverse representation, fair access to resources, and inclusivity.

To recruit a diverse set of people for the organizing committee, we sent a community-wide solicitation for volunteer Area Chairs (ACs) and Reviewers. These bodies are frequently, though not always (ACL 2018 also employed a wide call for volunteers), seeded with lists of recent past ACs and recent past reviewers. We went in the opposite direction, explicitly selecting ACs who had not been part of every recent *ACL conference and pairing them with more experienced chairs. The volunteers were all vetted by the PCs and assigned to tracks as either ACs or reviewers; ACs then filled out the rest of their committees through the usual networking process. With this process, we recruited a stellar set of 94 ACs – listed here – 40 of whom indicated being a first-time AC for NAACL. Each AC has been assigned as meta-reviewer for around 20-25 submissions. Bringing in so many new people also meant including many less experienced ACs and reviewers, which we felt was a plus for expanding the size and make-up of the organizing team. However, it also exposed how much of this process is governed by an unwritten set of cultural norms. Including grad students as reviewers (not all conferences do this), also a plus, made more transparent the need for a mentoring process for them. Emily’s post regarding reviewer’s code of conduct is a good example, but we propose to create more resources for reviewers.

Another choice we made with an eye towards bias-minimization was to follow the precedent of COLING 2018 in configuring the conference anonymization layers. Our review process is designed with extra layers of anonymity relative to other years: authors are anonymous to both reviewers and ACs, to minimize the influence of the author identity on the substance of the reviews. Additionally, reviewers are anonymous to each other during the whole review process. Having papers being anonymous to ACs was highly recommended to us by COLING 2018 PC chairs Emily and Leon. This means that while we have also recruited an outstanding set of reviewers, we won’t be able to reveal their names to you until the close of the reviewing process. Thanks in advance for your hard work, you anonymous superheroes!

In aiming for transparency and openness in the reviewer bidding process, we opted to allow reviewers to see full papers during bidding. Having this extra information is beneficial to all reviewers in making decisions about which papers they are best qualified to review, and especially beneficial to newer reviewers. Feedback from the community with respect to this point was about equally positive and negative. We will survey the community about this soon.

Finally, with regard to the overall conference timeline, there was a need to balance respect for ACs and reviewers time with external constraints from other conferences and the required sequence of events. Previous organizers all strongly advised against setting program deadlines close to the end of year, and an early January deadline is not friendly to regions typically underrepresented at NAACL, so our solution was to have an earlier submission deadline and then a longer period for paper checking and area assignment, such that reviewing could begin in early January. Given the limited benefit the NAACL 2018 PCs found in their analysis of author responses, we opted to fold that time into the review period to allow committee members more flexibility in when they did their work. Likewise, we opted for a two-phase submission process as a way to allow for a head-start on area assignment without pushing the main paper submission deadline too early.

Reviewing kicks off January 9th, and we’ll keep more posts coming to discuss desk rejects, our observations on the two-phase submission process, preserving anonymity in the review process, and more.