Slides from NAACL 2016
Minutes from NAACL Board Meeting
- Date: June 12, 2016
- Location: NAACL-HLT 2016 in San Diego, CA, USA
Attendance
- Emily Bender (Chair)
- Hal Daumé III (Past Chair)
- Graeme Hirst (ACL Representative)
- Joel Tetreault (Treasurer)
- Matt Post (Board Member)
- Marie-Catherine de Marneffe (Board Member)
- Colin Cherry (Secretary)
- Fei Xia (Board Member, via Skype)
- Julia Hockenmaier (Board Member, via Skype)
- Priscilla Rasmussen (ACL Business Manager)
Ani Nenkova’s and Owen Rambow’s Report on NAACL 2016
They did three things differently this year:
- Draw a broader pool of reviewers
- Largest ever program committee - 685 reviewers, reviewing load: 3 average, 5 max
- Created a pool from ACL anthology past authors
- People who had 5 papers in last 10 years or 3 paper in last 5 years
- Invitations went to 1400 - got 685 acceptances
- Scripts were created for this - TODO we may want to capture them for the future
- Area chairs then got to pick their reviewers
- About 50 reviewers were silently excluded because no one picked them
- Flexible areas
- Wanted a more subtle approach to what an area is.
- Areas were defined by keywords - area chairs selected keywords and then grouped by cosine similarity
- No names were given to the resulting areas - people did complain about the missing names
- 3 matching tasks: pairs of area of chairs, matching of reviewers to paired area chairs, matching of papers to areas
- There were roughly 100 keywords
- Short aside discussion about how English wasn’t on the list of languages
- Keywords fell into 5 categories Linguistic Area, NLP area, Goal, Method, Language, Genre
- Some question of how to encourage people to pick an answer from each category
- Supporting reviewers
- Encourage ACs to make decisions together - keep areas small (40 papers)
- They recommend - keep this approach, work on instructing ACs, keep author feedback
- All papers got a meta-review, but this was kept hidden from the authors
- Area chairs were encouraged to provide justifications in meta review even when the decision was clear
- Not much one can do when a reviewer does not take part in the discussion
- Hard to exert pressure when you have a short period between submission and decision - need more time for discussion
- Author responses serve to instigate the discussion
- One side-effect of mixing short and long papers is that reviewers are harder on the short papers
- Owen and Ani are preparing written report including feedback from chairs and reviewers - TODO Collect and archive this report
- Can we associate keywords with people’s START profiles, so it’s permanent and not per-conference?
- Ani and Owen got keywords from a Google spreadsheet - would have been nicer to get the keywords within START
Treasurer’s Report (Joel)
- Joel led discussion of his report
- Not sure on how NAACL 2015 did - probably lost money
- Will likely have to cut back for 2017
- Video recording still has this special status as a conference thing we have to budget for / vote on each year
- Fold into conference budget?
- Argument for no: the recording does more for people not attending the conference, it is outreach
- LSA Summer Institute (which we typically fund) only happens ever second year
- Joel is encouraging us to seek out sponsorship from smaller / emerging players
- We already have 3 - 2 global sponsorship chairs and 1 local sponsorship chair (and Priscilla and Joel)
- Since we’ve gotten rid of the student lunch, we might consider doing a recruitment lunch
- Could we approach some companies specifically to fund the ERF? Many ERF applicants just want travel support.
- What about a Video Recording sponsor? Company could put their name on the page
- Priscilla’s sponsorship summary:
- We got $70k for the main conference
- Got a lot of 3-pack (NAACL, ACL, EMNLP) sponsors this year, many at gold level
- A number of silver sponsors approached us late
- We are doing a little bit better than last year, just with our standard sponsors
- Could we co-locate with other conferences (machine learning, linguistics)?
- Financial benefit of collocation depends heavily on cross-over
- Collocation is actually less work for Priscilla
- Is it possible to move when we commit to funds to after when we know about the results from a conference?
- Moving ERF is easy, but we don’t want it to be cut more often
- TODO: Joel volunteered to spearhead the NAACL Inc. sponsorship effort
- Priscilla did not like the idea of folding NAACL Inc. sponsorship into *ACL conference sponsorship
- Any time after ACL would be a good time to approach organizations for sponsorship, earlier is better
ACL Treasurer’s Report (Graeme)
- The ACL’s bookkeeping is currently in rough shape
- Some costs/credits have been misassigned, will be some time before we have a clear financial picture for NAACL 2015
- But why do we have a loss at all?
- Even with more people registering, costs still go up
- Line dancing was expensive, Denver is an expensive city
- Food is expensive, we are perhaps getting more than we need to fulfill our room requirements
- But there are perks (many intangible, such as leverage) to being a big customer for these hotels
- Regarding food
- Good to get everyone together, but it’s too bad there aren’t many chance for self-organized dinners out
- Conference is becoming all-inclusive
- We could drop breakfasts without losing leverage
- Past experiments with dropping breakfast have upset conference attendees
2016Q1 Report to the ACL, February 2016 (Emily)
The NAACL Chapter’s bank account held $90,017.12 as of December 31, 2015. This figure takes into account the surplus from ACL 2014, but does not yet reflect the impact of the NAACL 2015 conference. To give context for that balance, this is the highest level it has been since June 2012. The balance for 2014 was $34k and for 2013 it was $29k. In 2015, our expenditures were once again the sponsorship of external events: NACLO ($5k), two Emerging Regions Fund awards at $1.5k apiece ($3k) and the Summer Workshop ($6k) for a total of $14k. In short, 2016 begins on a very strong note for NAACL. That being said, we should continue to seek out new sources of sponsorship as many fields not previously connected with NLP, such as finance and gaming, were major sponsors for last year’s NAACL.
ACL 2017 (Priscilla / Emily)
- ACL 2017 will be in Vancouver in August - July 30
- Two handoff meetings coming up
NAACL 2018 & 2019 (Priscilla / Emily)
- Priscilla would prefer to negotiate two years at once, both at the same chain
- Collecting ideas for locations
- Priscilla has good contacts at Starwood, Hyatt, Marriott, Hilton - Starwood would like us to do New Orleans
- NAACL Board is in favour of negotiating another 2 years
- What Priscilla needs for a 2-year plan:
- Suggestions for general areas on the map (Boston and points north is an example area)
- Couple of states here, couple there, to give Priscilla some direction
- Hal has the results of a survey: Chicago, Boston, Austin, Toronto, New Orleans, Miami, Honolulu, Salt Lake, Vegas, Montreal, DC
- Feasibility of Latin / South America?
- Graeme and Priscilla have done some preliminary explorations
- No clear candidates at this point, don’t know about Mexico City
- Would definitely need a local to help bridge cultural, linguistic barriers
- Would make sense for NAACL to do a SA conference before ACL, NAACL has the mandate, and it is smaller
- What if we had a goal to host an Central/South American NAACL in 10 years?
- Is the ERF serving that goal? Could we do more?
- Would it make sense to send researchers from the NA universities to visit LA/SA?
- Would be Western tourist destinations be reasonable?
- Can scare off government sponsors, would still be more difficult than North America
Elections (Colin)
- Last year we had 25% voter turnout
- Nominations September 1
- Members notified of nominations on September 15
- Nominating Committee has been assembled:
- Rebecca Hwa
- Anoop Sarkar
- Mona Diab Chair
- Katrin Erk
- Julia Hirschberg
- Michael White
- Eduard Hovy
- Colin needs support running elections software
- Jim Harmond at Ohio State helped last year, Marie confirms that he will help again
- Should we have another board position?
- Could help with concerns about having enough people eligible to be Chair
- Could also help with expanding workload, sponsorship
- TODO: Emily to draft proposal to expand the board, so membership can vote on expansion in the next election
Latin America Report (Joel and Fei)
- Joel walked us through his slides
- Most popular request for support is travel support to attend NAACL
- Second most popular is funding to have top researchers visit
- What about a video classroom that could capture a broader audience?
- What is the expectation of the English language capability of the students we would encounter with an outreach visit?
- Are there Spanish- or Portuguese-speaking North American researchers that we could tap for outreach?
- Many researchers commented that our conference focus hinders their ability to get promotions, which depend on journal papers.
- We could promote TACL to this audience, which would also get them invited to the conference
- TACL is not currently indexed by Thompson Reuters - still not great for promotions
- This survey and mailing list already reflects big strides toward improving communication with LA
- What about sending the Jelinek summer school to LA?
- Machine learning summer school in Portugal? Seems to have a lot of *ACL people
- TODO: report back the results of the survey to the community
- TODO: reach out to the LA community to encourage them to submit a workshop to ACL in Vancouver
- Thamar Solario highlighted as someone who can help with organization
Report on Author Response (Marie-Catherine & Hal)
- Slides are here
- Sent a survey to area chairs to get an idea of usage of author responses
- Review text and discussion were most important factor in a borderline paper
- Area chairs are not really considering nitty-gritty scores - but they can be helpful to the reviewer to force them to think about the issues
- Best practices for an author response
- thank the reviewers, be positive
- prime expectations: it’s not going to be a big difference
- you maybe anonymous to the reviewers, but not to the area chair
- Structure the timeline to encourage discussion
- Very helpful to prime a discussion with a summary of positions by AC
- One concern that keeps coming up is can we provide a person’s score history
- What is person X’s histogram for an accepted/rejected paper
- Or with respect to other scores
Report on START (Matt and Colin)
- Putting together a list of rough edges and ideas that could improve experience for chairs, ACs, reviewers, submitters
- Rich at Softconf is located at UMD right above Hal
- Hal has a bunch of scripts that he uses to make START more helpful, probably others have scripts as well
- Proposal exists to have START cover smaller ACL-affiliated stand-alone events. This is orthogonal to technical improvements discussed here.
- CMT - Microsoft Conference Management toolkit - competitor that came up several times
Should NAACL joint the Computing Research Association (CRA)? (Julia)
- Slides are here
- Fees are large enough to be felt as a meaningful budget item
- TODO Hal: Talk to Lise Getoor re: AAAI’s benefits from being a CRA member
Plan for Business Meeting (Emily)
Issues from last business meeting:
- Public reviews - split 50-50
- Accessibility chair at NAACL
- Close captioning of video
- Priscilla: this cost $3,500 last year for a limited number of tracks
- Would there be a way to do this live for all tracks?
- Microsoft may be developing an AI-based solution
- Award categories?
- Best new data source?
- Best student paper?
- Best reproduction of a previous result?
- Most reproducible paper?
Minutes from NAACL Business Meeting
Question period
- Chris Callison-Burch (CCB): Supports idea of adding another board member.
- What other adjustments can we do to increase the coverage for chair?
- Emily: don’t necessarily want to do away with the constraint for board membership in the past 5 years
- One point raised: we should reach out to Caribbean (Cuba)
- Joel pointed out that Cuba (and the Caribbean) was included in the LA survey
- Question about video recording: why don’t we collection permissions from authors, audience?
- We handle the permissions through the copyright consent form when you submit your camera ready.
- If you do not want your talk recorded, communicate that concern to the board.
- Videos are currently licensed under creative commons
- TODO we should post our copyright agreement, release agreement on the website
- We are not currently getting permission for TACL papers
- Keynote speakers are not asked permission to be recorded.
- Audience members should be allowed to opt out from having questions recorded
- One answer: make an announcement at the beginning of the conference that you can request to have talks / comments deleted in post-production.
- Question about survey in Latin America / South America
- Language (research disseminated in English) is a big barrier to LA participation.
- Joel explained the survey (presented in Spanish and Portuguese).
- On Reviewing (Meg Mitchell):
- Seems like Best Reviewers list is celebrating people for doing things all reviewers should be doing
- Is there any potential to have authors provide reviewer feedback?
- How can we assure that we have high-quality reviews for every paper?
- How can we improve matching of papers to reviewers?
- Emily: This year’s new process (small, dynamic areas) is an attempt to improve review matching.
- Emily: Is there a way to collect and disseminate best practices for reviewing?
- Owen: He sees that the area chairs have a crucial role. We need to admit that this is a lot of work.
- Ani and Owen’s approach: reduce the size of each area, and then encourage area chairs to really interact with reviewers when making their decisions.
- Owen: All papers are important, not just borderline cases. Every decision should have a rationale.
- Emily: Making the role of the area chair explicit, making sure they understand it before signing up. Make sure they have time (reviewer discussion window, size of area) to do their job properly.
- Meg: What about feedback from authors?
- Hal: Anecdotal: a few years ago, authors could comment on how helpful a review is - strong correlation with acceptance, rejection.
- Hal: Data driven: NAACL 2013 - did a survey of authors, showing them them the review text and asking them for their review review (helpful, correctness). Delayed by about 6 months to give the authors a chance to cool down. Wound up much less correlated with accept-reject decisions.
- Hal: Another topic that comes up is maintaining institutional memory about reviewers; if we delay author feedback on review quality, then we need to have long-term institutional memory.
- Emily: Co-reviewers or area chairs could also review reviews.
- CCB: We now have a single logon for START. We no have a history for all reviews done by a particular user name. Can expose that history (score deviation from the mean) to the area chairs.
- Q: Reviewing is a lot of work. Author response in particular is a lot of work for both the authors and the reviewers. Can we give people less work, and ask them to do it better?
- Graeme: Most responses are not going to have an effect. The response making a serious difference is not going to be a regular event. Any response should be to a reviewer misconception.
- Owen: used to be critical of author response. PC Chair and Area Chair use the authors response (perhaps more often than the reviewers) - effect of author response is frequently invisible.
- Philip Resnik: author responses have turned into point-by-point rebuttals / promises. There are things that are not allowed: new data / new results. Can we go back to original intent? If there is a serious misconception point it out. It should be out of bounds to do a point-by-point rebuttal. A blank author response should mean that there were no misconceptions. It shouldn’t indicate that you aren’t serious if you don’t put up a response.
- Meg: in the computer vision community, you can give long responses. You can frequently need to run more experiments, get more numbers. Often times you’re just missing a baseline, which is easy to fix. Do we want to say that adding new information is on the table? CVPR - submit in November, hear back in March, a month for an author response
- Q: TACL should be our journal-like venue, not our conferences.
- Hal: I am known for being anti-author-response. He did a survey and asked of area chairs, asking what part of the reviewing procedure is mist important. They indicated that the author response was very useful.
- Hal: Response shouldn’t make a difference most of the time, but is it worth the time?
- Hal: We don’t currently allot enough time for response and discussion.
- Benjamin Van Durme, on behalf of Kevin Duh:
- This year, for EMNLP you can click a box to indicate whether you have an arXiv pre-print.
- Main issue is that we don’t have a way to share an arXiv link if it is added after submission
- Current policy is that reviewers are aware of an arXiv version, and can chose whether to view the paper on arXiv (breaking anonymity)
- You have to tell TACL that you pushed a paper to arXiv
- Owen: why should a reviewer or author have a choice to de-anonymize a submission
- Owen: arXiv should count as a publication
- Brian Roark:
- CHI (Computer Human Interaction) has one minute madness for all papers first thing in the morning (even full talks)
- CHI has a fringe conference called alt.chi, dangerous, not appropriate, humorous. NAACL could do the same thing.
Decisions made over e-mail
Feb 12 vote: Best paper policy
Voted on endorsement of the following policy:
Guidelines for committee membership:
The best paper committee should contain four to six experts from the NLP community. No two should represent the same subfield of NLP, defined broadly by “area chair”-like categories. All should be known in the community, but preferably represent different parts of the NAACL membership, for instance: academia, industry, government. The committee will be selected by the program chairs in consultation with the general chair, and may include area chairs but need not.
Guidelines for paper selection:
The best paper committee will select best papers for the conference. The precise number and names of these are specified elsewhere; these guidelines discuss what “best” means. When making selections, the committee should keep in mind that there are many ways for a paper to strongly contribute to the body of knowledge in our field. Examples are:
- Provide new algorithms, corpora, or technology
- Introduce a new insight into language or people
- Facilitate reproducibility via code, infrastructure, or data
- Provide new evaluation methods and standards
The papers selected by the committee should be good for different reasons. A paper need not be flawless in order to be best; being interesting and/or introducing a new idea or methodology to the community can be more important. The committee must write one or two sentences stating why each paper has been selected, for the program chairs to use during their announcement of the best paper awardees.
Hal, Marie-Catherine, Colin, Matt, Julia, Fei, Joel voting. Passes.
Posted here, but still not linked to main website as we have not yet addressed the concern of where is “elsewhere” in “The precise number and names of these are specified elsewhere.”
Feb 18 vote: Emerging Regions Fund (ERF)
Joel and Matt volunteered to manage the fund. Proposal to fund two proposals at $1,500.
Emily, Joel, Julie, Marie-Catherine, Matt, Colin, Fie, Graeme voting. Passes.
Feb 25 vote: Eliminate Best Student paper
Choice between (a) keeping IBM Best Student Paper as is and (b) eliminating Best Student Paper award, with IBM becoming a “Best Paper Sponsor”.
Colin, Graeme, Hal, Joel, Marie-Catherine, Matt, Emily, Fei voting. (b) carried and we asked Priscilla to contact IBM.
- IBM stopped sponsoring a best paper, but also sponsors NAACL, ACL and EMNLP at the Gold level. NAACL 2016 awarded two best long papers and one best short paper.
- Fei expressed a good argument against (b): in the future, if there are more papers from research labs and industry without students being the first authors, would getting rid of the best student paper category pose a new problem?
Feb 29 vote: Recording NAACL 2016 Talks
Approve $10-$15k for recording talks at NAACL 2016 with Weyond.com
Joel, Julia, Colin, Marie-Catherine, Fei voting. Passes.
Mar 28 vote: JSALT Summer School @ JHU
Should we fund student scholarships for the summer workshop?
If so, should it be a) 10k for 5 students, b) 10k split over 6k students.
Joel, Julia, Matt, Colin, Graeme, Marie-Catherine, Fei voting. Passes.
- We approve funding students the workshop at 10k for 5 students.
- Matt, Hal and Julia volunteered as reviewers.
April 15 vote: NACLO
- Should NAACL allocate $5000 to NACLO?
- Should NAACL provide a complimentary exhibit table to NACLO in San Diego?
Matt, Marie-Catherine, Joel, Colin, Fei, Emily voting. Both (1) and (2) pass.
May 10 Vote: Video recording for NAACL tutorials + main conference
Weyond recording of conference + tutorials cost $9912.50. Vote on whether to spend this much.
Colin, Matt, Marie-Catherine, Joel, Julia voting. Passes.
May 19 vote: Revisiting video recordings for 2016
Should we spend $9912.50 to record NAACL talks and tutorials in light of a potential NAACL 2015 loss?
Joel, Colin, Marie-Catherine, Matt, Julia voting. Passes.
May 23 vote: small additional expenditure for JHU Summer School students
Should we approve an additional $450 for a student’s travel? Their expenses were much higher than we had budgeted.
Marie-Catherine, Joel, Matt, Hal, Colin voting. Passes.