North American ACL North American Chapter of the ACL

Minutes for 2015

General Chair and Program Chairs Report

Slides from Business Meeting

Minutes from NAACL Board Meeting

  • Date: May 31, 2015
  • Location: NAACL-HLT 2015 in Denver, CO, USA

Attendance

  • Emily Bender (Board Member, via Skype)
  • Chris Callison-Burch (Past Chair)
  • Hal Daumé III (Chair)
  • Graeme Hirst (ACL Representative)
  • Ed Hovy (Board Member)
  • Matt Post (Board Member)
  • Joel Tetreault (Treasurer)
  • Michael White (Secretary)
  • Fei Xia (Board Member, via Skype)
  • Priscilla Rasmussen (ACL Business Manager)

Treasurer’s Report: Joel

  • Record NAACL sponsorship was gratefully acknowledged, including from many new sponsors such as Goldman Sachs. Priscilla noted that sponsorship levels are comparable to ICML if ACL/NAACL/EMNLP/EACL sponsorship is considered together.
  • Re co-locating with other conferences, the LSA Annual Meeting was mentioned as one possibility. On the plus side, there is now considerable interest in working with linguists on low-density languages, and a CL tutorial aimed at linguists could be attractive for those interested in industry jobs. On the minus side, the LSA meeting is usually in January and expectations for conference registration are much lower. To enhance affordability, a special 2-day LSA rate might be offered.
  • Co-location with ICML or AMTA was also discussed, with the need for considerable advance planning noted.
  • Joel suggested setting up a new Google account to manage planning documents via Google Drive.
  • ACTION ITEM Joel will work with other board members to establish contacts for possible 2018 co-location.
  • ACTION ITEM Mike will set up a new Google account to complement the public website.

Overview of NAACL 2015 reports: Hal

  • The board agreed with Hal’s proposed refinement of the best paper policy, as follows: “There will always be awards for at least one of each: best longer paper, best short paper and best student paper. Up to six runners-up in total will be publicized.”
  • The anti-harassment policy was reviewed and no changes were recommended.
  • ACTION ITEM Hal will put together a document to serve as a guide for new board members, including a reference to the anti-harassment policy to ensure new members are aware of it.

NAACL 2016: Hal

  • The preliminary NAACL 2016 dates were reviewed; it appears that it will work with the ACL schedule to have NAACL submissions after the new year.
  • After a discussion of the pros and cons of the author response period, it was agreed that at least for NAACL conferences the authors should be shown their preliminary overall scores so they can take these into account when deciding how to respond. It was also agreed that feedback from NAACL 2015 attendees would be gathered via the post-conference survey for consideration by the NAACL 2016 program co-chairs, Ani Nenkova and Owen Rambow.
  • The lack of local restaurants is likely to be an issue for NAACL 2016. Setting up some local transport may be an option for lunches; Priscilla suggests waiting until she has done a site visit before making any decisions to address this issue.

Overview of ACL 2017:Hal

  • Further consideration of a preliminary proposal to host ACL 2017 in Buenos Aires awaits the submission of a detailed proposal.
  • Priscilla is investigating North American options in Toronto, Minneapolis or Chicago. Alaska was mentioned as another option, though Priscilla noted the preference for central or eastern locations given the locations of NAACL 2015 and 2016.
  • The possibility of hosting NAACL 2018 (or beyond) in Central or South America was also mentioned, though Hal noted the risk of low attendance if ACL were at a similar distance in Europe or Asia the same year.

Backend in START subcommittee: Matt

  • Matt and CCB will be collecting suggestions (starting with their own long list) for ways of improving the process of working with START.
  • NAACL will take the lead on working with softconf to implement these changes for NAACL 2016; Graeme noted that the ACL exec might be convinced to contribute funds towards tasks that will benefit the ACL as a whole.

NAACL in S.A. subcommittee: Fei

  • Fei and Joel noted that helping to host ACL/NAACL in Central or South America is likely to be the most important way in which NAACL can help progress CL there. Meanwhile the Emerging Regions Fund remains active and helpful.
  • Fei clarified that according to the NAACL constitution, NAACL will serve members of the ACL in Central and South America until regional chapters are created for members there. As such, she noted that it does not make sense to consider changing the NAACL name until the question of regional chapters for Central and South America has been settled by ACL members there.
  • ACTION ITEM Fei will work on updating the contacts list started by Ted Pedersen for ACL members in Central and South America who might be interested in hosting a future conference.

Reviewing models subcommittee: Mike

The committee on reviewing models (Emily Bender, Hal Daumé and Michael White, chair) discussed the following topics by email, which were discussed further at the meeting:

  • Reviewer Feedback
    • Emily expanded on her candidacy statement where she noted that there is room to improve the way that reviewers’ considerable efforts for the NAACL-HLT conference are focused; in particular, a system of incentives to encourage reviewing that is fair, rigorous, and constructive (rather than superficial and/or unnecessarily mean) could include feedback to reviewers by area chairs, co­reviewers and even authors as well as recognition for outstanding reviewing.
    • After some discussion it was noted that authors, co-reviewers and area chairs could be encouraged to numerically rate reviews on a limited number of dimensions (such as thoroughness, insightfulness, fairness, constructiveness) and provide an overall rating. Naturally authors are apt to have a different perspective from co-reviewers and area chairs so it’s important to gather feedback from all parties.
    • The limited utility of the author response was briefly mentioned. If this is kept in future years, author feedback could encourage reviewers to more often update their reviews to reflect consideration of the author response.
    • To help preserve feedback anonymity, reviewers might only be shown their ratings averaged across the authors of all reviewed papers along with their ratings averaged across all co-reviewers and senior PC members.
    • Hal suggested that we should make it standard NAACL practice for the conference program chairs to ask their SPCs to nominate their best reviewers for recognition at the conference.
    • It was noted that ideally reviewer ratings should be aggregated over all ACL conferences, so that SPCs can take these into account when nominating best reviewers as well as throughout their decision making. Aggregated ratings could indicate the number of conferences, reviews and ratings they’re based on, over what time span.
    • To facilitate collection of ratings and tracking them over time, it was agreed that it would make sense to discuss integrating this capability into the START system, with the first step to inquire about costs with softconf. Ideally NAACL could obtain buy-in from the ACL as a whole, though NAACL could also begin the effort and let ACL join in later.
    • The recognition of best reviewers should be made more visible, appearing not only in the proceedings but also in the handbook and on the website.
    • Hal noted that additional data on reviewers (e.g. lateness) could be tracked via START, but also noted the potential dangers of keeping too much of such data.
  • Public Reviews
    • Hal suggested we consider making reviews of accepted papers public, as NIPS has done, suggesting it can be useful for readers new to the topic area. He also proposed making the submitted versions of papers public as well.
    • Emily expressed hesitation noting that many reviewers might feel obligated to produce more polished reviews and thus be less inclined to agree to review. She also noted the need for community feedback before taking the step of making submitted papers available.
    • It was suggested that reviewers could choose which comments to make public, which to share only with authors and which to make confidentially with the PC, thereby providing and opt-in system to ameliorate concerns about reviews needing polish. Authors might also be given the choice to opt-in to making the submitted version of their paper public.
    • Joel and CCB noted that the reviews, along with submitted and final versions of the reviewed papers, could provide useful data for those researching the writing process.
  • Summary Rejects
    • Hal suggested we consider allowing SPCs to do summary rejects, perhaps after consideration by a pool of special reviewers, in order to drastically cut down on the number of necessary reviews. Mike noted that this could enable more careful reviewing of papers that make it past this first cut.
    • Emily expressed a concern about unclear criteria for summary rejects and whether this might lead to profiling by e.g. institution or English fluency. For this reason, Hal decided he no longer considered it an idea worth pursuing.
  • Resubmitted papers
    • The idea of allowing authors to request that a resubmitted paper (rejected from an earlier conference) be reviewed by the same reviewers was discussed.
    • The idea was dropped after noting the practical difficulties of ensuring the same reviewers are available from conference to conference, and that TACL already serves the purpose of providing continuity of reviews.
  • ACTION ITEM Mike will lead discussions with Ani and Owen regarding which ideas to pursue for NAACL 2016, also coordinating with Min-Yen Kan (ACL Anthology Editor) and Drago Radev (ACL Secretary) as appropriate.

After the board meeting, further discussion took place at the conference and via email:

  • Hal shared data from NAACL 2013 showing that author feedback made very little difference in acceptance decisions and most reviews remained unchanged.
  • Michael Strube (ACL 2015 program co-chair) and Ed expressed opposition to any system for collecting reviewer ratings that was not opt-in and fully public.
  • Ani and Owen expressed concern about trying to make too many changes at once for NAACL 2016, adding that their priority is improving the way reviewers and SPC members are coordinated across paper areas.
  • Ani suggested focusing on enhancing the visibility of the stars reviewers receive when chosen as a best reviewer by an SPC member.
  • The idea of tracking these stars over time was discussed, with Ed suggesting the name ACL Reviewer Honor Roll for this site. If the Honor Roll were to only list those awarded stars (rather than those not awarded stars, and how often), it seemed unlikely to be objectionable to the membership.
  • At the business meeting, the following additional suggestions were received:
    • SPC meta-reviews should be made public, in part in order to let authors know that their response has been taken into consideration.
    • Reviewers could be given a review section for post-response comments.
    • Authors should be allowed to raise a red flag for factual errors in reviews.
    • If reviews are made public, authors should also be given the chance to make a public response.
    • The ACL Anthology could be extended to allow for public comments and responses.
  • David Jurgens will be running SemEval 2016 and is keen to try out both ratings of reviews and public comments. As such, the board decided to let SemEval trial these ideas and focus on making best reviewer awards more visible for NAACL 2016.
  • ACTION ITEM Mike and Matt will coordinate with David Jurgens on steps to take in this direction for SemEval 2016, including any necessary changes to START.

Elections: Mike

  • From the NAACL constitution:

Elections shall be conducted annually as follows: the Nominating Committee shall by the first of September preceding the end of a term of office nominate at least one person for each position to be filled, including any necessary new Nominating Committee member. The Secretary shall send to the Chapter Members notice of the nominations by September 15, at the same time requesting additional nominations. Additional nominations, supported by at least three Chapter Members, may be submitted by the fifteenth of October; evidence must be presented that the nominee will serve if elected. If there is more than one nominee for any position, the Secretary shall send ballots for that position to the Members. A majority vote of the ballots received by the fifteenth of December shall determine the results of such contests. Those elected shall take office on the first of January following.

  • Hal noted that the nominating committee has begun its work and that he recommends the committee suggest no more than two nominees per position.
  • Outgoing members:
    • Emily Bender (Board Member)
    • Hal Daumé III (Chair)
    • Ed Hovy (Board Member)
    • Michael White (Secretary)
  • Nominating Committee for 2015 elections:
    • Chris Callison-Burch
    • David Chiang
    • Mona Diab
    • Katrin Erk
    • Dan Gildea (Chair)
    • Julia Hirschberg
    • Rebecca Hwa
    • Anoop Sarkar

Decisions made over email

May 12, 2015

Matt Post reported on the decisions of the Emerging Regions Fund subcommittee as follows.

We received two applications for our 2015 call for funding proposals in emerging regions. They are both requests for $1,500 and focus on supporting STIL, the main Brazilian event in NLP. STIL is taking place in Natal, in the northeast of Brazil, on November 4–6. STIL is held joint with BRACIS 2015 (Brazilian Conference on Intelligent Systems), and ENIAC 2015 (Encontro Nacional de Inteligência Artificial e Computacional). We have funded this event in the past.

  • The first will fund undergraduate students for the IV Student Workshop on Information and Language Technology (TILic), which will be hosted by the main Brazilian event on NLP — the X Symposium in Information and Human Language Technology (STIL).
  • The second will be used to bring a North American invited speaker to STIL (last time, they brought Mike Riley and Andreas Stolcke).

We allocated $3,000 for this call. These were the only two responses, and they both look good to us, so Emily and I opted to fund them both.

May 4, 2015

The following three students were selected out of 14 applicants to receive a $2,000 scholarship to attend the 2015 Frederick Jelinek Memorial summer school at the University of Washington in Seattle, Washington (USA) from June 22, 2015 to July 2, 2015.

  • Keith Herbert (USA)
  • Alberto Manuel Orozco Camacho (MEX)
  • Roxana Lafuente (ARG)

The scholarship covers tuition, housing, a full meal plan for the dates of the summer school and approximately $300 towards travel reimbursements.

The selection committee was chaired by Matt Post, who was joined on the committee by Joel Tetreault, Michael White and Fei Xia. The committee decided to favor advanced undergraduate students in the selection process for whom the scholarship might be expected to have the greatest potential impact on their decision to pursue a careeer in human language technology.

The call for applications and scholarship selections followed an earlier decision by the whole board to direct $6,000 towards these scholarships. Given the limited funds available and the larger summer shool expenses in comparison to JHU summer schools in previous years, NAACL was unfortunately able to support far fewer students than previously.

March 23, 2015

Kevin Knight agreed to serve as general chair for NAACL 2016 with Ani Nenkova and Owen Rambow as program co-chairs.

February 17, 2015

The NAACL board voted unanimously to approve spending $10k for video recording the 2015 NAACL HLT conference.