North American ACL North American Chapter of the ACL

Minutes for 2018

Slides from NAACL 2018

Minutes from NAACL Board Meeting

  • Date: June 3, 2018
  • Location: NAACL-HLT 2018 in New Orleans, LA, USA

Attendance

  • Julia Hockenmaier (Chair)
  • Emily Bender (Past Chair)
  • David Yarowsky (ACL Representative)
  • Joel Tetreault (Treasurer)
  • Colin Cherry (Secretary)
  • Philip Resnik (Board Member)
  • Ellen Riloff (Board Member)
  • Marie-Catherine de Marneffe (Board Member)
  • Marilyn Walker (Board Member)
  • Luke Zetttlemoyer (Board Member)
  • Priscilla Rasmussen (ACL Business Manager)

Report on ACL Professional Conduct Committee (Emily)

  • ACL has an anti-harassment policy
  • ACL has formed a committee to decide what should be done when complaints are submitted under this policy.
  • Part of their proposal will be to form a complaints committee that will be trained for mediation. Training will be carried out at ACL 2018 and EMNLP 2018, resulting in a 12-person committee
  • The committee will be responsible to the executive, and they can be dismissed by the committee if they are abusing that position.
  • Philip: Is this something we can outsource rather than training members of our organization?
    • Would create a greater distance in terms of liability for when things go wrong.
    • If such an industry exists, these people would be much more specialized.
    • Emily: ACL is in the lead on taking this initiative. Other organizations to follow.
    • One issue that ACL faces is that it is a very international organization, so it would be hard to hire, for example a US-based firm.
  • Julia: What if we worked with the ACM? Sought legal counsel.
    • Emily: I’ve been in touch with the people at NIPS, they’re behind us on this, but have been in touch with a lawyer.
  • Emily: Complaints are happening and the complainants are frustrated with the lack of procedure.

NAACL 2018 Test of Time Award (Heng Ji and Amanda Stent)

  • Slides on program across areas, test of time award.
  • Would like to continue the test of time award.
  • Initially started with just NAACL, but the nominations quickly expanded to include the entire *ACL organization - might be better to have ACL take over this process.
  • Committees were separate, but they converged as they voted, both agreed on the top paper.
  • The idea of asking NAACL area chairs to nominate papers was to make sure each area was represented, despite the prominence of areas over time.
  • Philip: What was the rationale for 2002 as the cut-off date?
    • Heng: Started with 2012, to make sure we had enough time to evaluate impact, and then stretched back 10 years.
    • Lyn: The fact that people wanted to nominate outside the constraints is a strong indicator of the fact that there is a real need/desire for awards like this to exist at the ACL level. She would like to see a NAACL award for just NAACL papers.
  • Julia: Weird to have a chapter organization assign an award to a paper presented at its parent organization.
  • This was a good experiment, a good proof of concept, but it’s time to do it properly.
  • Priscilla: Need to make sure that we have a policy that handles / excludes the workshops, since they are not consistently associated with NAACL.
  • Heng: Need a clear criteria about how many papers should be nominated from each area. Some areas didn’t nominate any.
  • Lyn: Getting the area chairs to do the nominations leads to natural variance.
  • Heng: Got some criticism in social media, but lots of possible feedback.
  • David: What is this going to look like next year?
    • Don’t move the beginning of the window, just extend it forward one year.
    • Priscilla: Do we include HLT?
  • Lyn: Not in favor of including HLT, NAACL was a sort of reaction to HLT.
  • TODO: Discuss this and nail down a policy later in this board meeting.

NAACL 2018 Program Committee (Heng Ji and Amanda Stent)

  • As a field was can do better than TPMS for assigning reviewers.
    • We need to be able to have area chairs assign papers to reviewers that they don’t know.
  • May also need tools for checking for submission overlap with TACL and other organizations.
  • We should be able to find near-duplicate submissions.
  • Julia: In discussion with ACL about handling this, in the context of our software infrastructure. Marti Hearst looking at a software revamp - not necessarily focusing on START yet.
  • Need to start collecting guest blog posts for the chairs blog.
  • Going to put detailed howto notes on their process into the ACL wiki.
  • Started tracking reviewer and chair demographics.
    • 72 area chairs, 2/3 male, 1/3 female.
    • 2/3 of reviewer pool is less than 0 to 4 years out of their PhD.
    • Purposely excluded PhD students - this seems very controversial to me (Colin).
  • Many of the 6s handed out were for arXiv papers.
  • Philip: The fact that you think you can guess is an indicator of how mature / familiar you are in your field.
  • ICLR de-blind papers when accept/reject decisions have been made.
  • Amanda: Strongly in support of double-blind review. Also in favour of making are chairs blind to author identity.
  • Emily: Problem in COLING is that if the area chair is blind to the COI, then everyone is blind to COI.
  • Recruit ad-hoc reviewers in advance.
  • NAACL needs local organization to assist Priscilla.
  • We had a healthy ad-hoc debate about the utility of junior and graduate student reviewers.
  • Lots of thought of preparation went into the new review form.
    • Got a lot of flack from the senior members of the community.
    • Structure of the review form was intended to teach junior reviewers how to review properly.
    • Julia: We’re doing something different for EMNLP
    • Interface didn’t allow for offline writing of review.
  • Potential questions to ask about the process:
    • David: How often did the are chairs override the mean score from the reviews?
    • How much discussion did the area chair solicit is another interesting statistic?
    • Was any effort made to seek out reviews with high-variance in scores?
  • Instructions to area chairs told them to look beyond the numbers, focus on diversity.
  • Emily: PC Blog is another thing we should do consistently
    • TODO: put a plan in place to store these.
  • Acceptance rate is 3% higher than it was last time.
    • We are getting complaints about that.
    • David: Competing against arXiv - broader pool is in our interest
    • Philip: In our field it’s important to emphasize that we have a very selective conference. Don’t want to get to the point where people don’t have journal papers, but also don’t have papers at a “selective conference”
    • Amanda: Interspeech has a higher acceptance rate.
    • Emily: COLING has an acceptance rate of 37%
    • Philip: If there is an A / B distinction in conferences, then NAACL needs to be retain an A status
    • Amanda: H-index is a much better measure for conference quality than acceptance rate. -Joeal: Conference acceptance rates: https://aclweb.org/aclwiki/Conference_acceptance_rates

New initiatives at NAACL 2018 (Marilyn Walker)

  • Industry track has led to submissions from a broader set of companies than we previously received.
    • The industry track sessions are being well-attended.
    • Philip: Maybe we want to guide industry presenters to emphasize how their problems and solutions are different from typical conference papers.
    • Elen: Are invited talks taking time away from oral presentations?
      • Lyn: Did lead to more posters.
      • Julia: Each plenary takes away many other (5-6?) slots.
      • Priscilla: We make up some time by starting earlier and running later.
  • Lyn: Distributed committees for workshops and tutorials are having trouble - lack of leadership.
  • Tutorial review process had no reviews in START.
  • Family-Friendly Chairs / Child Care
    • Did everything we could to spread the word - advertised early, encouraged workshops to advertise.
    • Got 3 children for 3 days - ages 4, 7 and 8.
    • Raised $16k in direct sponsorship for childcare.
    • Marie: Flying your kids to conferences is extremely expensive.
  • Philip: Pre-committing the money is not the right way to solve this problem. Can we look at reimbursing or subsidizing people who find their own solutions?
  • Priscilla: The first experiment was in July. NAACL’s timing conflicts with school being in session.
  • David: Issues are liability to us and finding acceptable services.
  • Philip / Priscilla: We need to find what we’re liable for.
  • Priscilla: ACL this year couldn’t get on-site facilities.
  • Colin: What about rolling pool of reimbursement funds, so a kid-light conference could potentially help subsidize another?
    • Priscilla: I would suggest unused funds go to the NAACL account where we can shield them.
  • TODO: Revisit reimbursement model of childcare later
  • TODO: Would like to add one topic to discuss later - one author submitted 24 papers.
  • TODO: Guidance to NAACL workshop and tutorial chairs, so we make sure our workshops/tutorials are handled well regardless of what happens at *ACL.

Visa issues (Priscilla)

  • Priscilla has a standard letter that she sends when someone needs one for visa purposes.
    • She doesn’t know how many people didn’t attempt to register because their visa was denied.
  • Priscilla has had 12 cancellations - only 3 were specifically because their visa was denied.
  • Last year, in Vancouver, there were 16 cancellations due to visa issues.
  • Marie: We need to make it clear to attendees that they should register, and that there is no financial risk in doing so.
    • “Make sure to register for the conference prior to requesting your visa.”
  • Philip: Add the phrase, “with proper documentation” for refunds
    • Not a super-popular idea, adds administrative overhead for Priscilla.
  • Julia: Book a return flight in advance.
  • Philip: Round-trip airfare is very hard to refund. Hotel reservation is low risk, but it’s unclear what value it adds to the application.
  • Luke: This is a big problem, we should do what we can, but not try to solve it.
  • Priscilla: Provide a little more advice on how to put together a good application (emphasize the importance of registering to the conference).
  • Consensus seems to be that there is no call to do something drastic, like only hold the conference in Canada.
  • Emily: There is a fair amount of chatter on social media about people not being able to attend.
  • TODO: We should make a thoughtful statement indicating that we are thinking about immigration issues.
  • Elen: When people are their visas rejected, are we giving them the possibility to do a remote presentation?
  • TODO: Look into viability of remote presentations when people cannot attend.

Conference Costs and Statistics (Priscilla)

  • 80% of registrations are early.
  • Sponsors at various levels are allowed to send 1-5 attendees.
  • Sponsorship policies are determined by an international *ACL committee.
  • Total people at NAACL: 1,334 - that is total bodies, including workshop-only and etc.
  • Julia: How much is the growth due to the industry track?
    • Priscilla: I don’t think much is.
  • Workshop and tutorial attendance are highly correlated with the degree to which they are associated with deep learning.
  • We’re over-subscribed with the social event.
  • Sponsorship has gone very well this year.
    • Added a new level of sponsorship (diamond level). -David: Is it possible to get big companies to contribute more? -Philip: This is an exercise in figuring out the right price-point. The reasonable thing to do is to have an informal conversation with the decision maker.
    • Emily: Might be useful to look at what other conferences (the rest of the market) are doing.
    • Priscilla: Other conferences have higher costs, but they aren’t part of a bigger organization. We’re constrained by being part of the ACL - seen as a single entity.
  • Elen: Does it make sense to do a 3-pack++ that adds in child care, etc? Avoid doing it piecemeal.
  • David: With the addition of AACL with this become a 4-pack? Asian chapter coming as of this year.
  • Lyn: Local sponsorship chairs went after anything between Houston and Atlanta, brought in Capital One. Got some small things from some local businesses. They were also supposed to help coordinate exhibitors - be a point of contact to answer questions, but that didn’t work out as planned. Had to forward questions on to Priscilla.
    • Have some local people come early to get the lay of the land at the hotel and help coordinate space.
    • Priscilla: Has gotten approval to hire a part-timer to be in charge of sponsors and exhibitors, managing the invoice, logos, logistics, booths etc.
  • Philip: Is the recruitment lunch a new thing? How does it work?
    • Priscilla: This has existed since 2016.
    • Lyn: Jason Baldridge and Alexis Palmer are the NAACL reps for sponsorship.
      • As GC, coordinating between international sponsorship and local sponsorship.
      • Went after sponsorships herself, going after specific sponsorship roles. It was a lot of work and frustrating.
      • Priscilla: Last year (2016) was the first year a GC brought in money.
  • Elen: Why should this fall on the GC? If the *ACL sponsorship is not sufficiently aggressive, we should make give the information that they should be more aggressive.
    • Julia: Everyone is re-inventing the wheel as they land in this role.
    • Priscilla: We still need the GC to go after smaller fish in their own hemisphere.
    • TODO: Create a recommendation for how to handle sponsorship solicitation.
    • Lyn worked from Chris’s list, add new companies to the list.
    • Priscilla: This is already on Marti’s agenda.
    • Lync: Creating a diversity sponsorship is a good idea
  • TODO: Where do our registration costs go?

Emerging Regions Fund (Marie and Joel)

  • 12 submissions this year.
  • Selected 3 recipients: $1,000 each.
  • Philip: We are giving money to support people to go to conferences not in our purview.
  • David: How much money do we give to people to come to NAACL?
    • A: None.
  • Joel: Suggestions for the future:
    • Increase to $4k
    • Two $1.5k to support organizations.
    • $1k to fund an individual
  • Julia: Should be some coordination between WiNLP and ERF.
  • Mandate of ERF is to support building up NLP in Latin America, including helping to organize local conferences.
  • TODO: What should be coordinated with WiNLP? They can handle bringing people to NAACL, while we help create local events.
  • Joel: Still want to try to host one of our conferences in Latin America.
    • Other major conferences have done it - it should be possible.
    • David: EMNLP 2020 could potentially be in Latin America
    • Joel: NAACL charter is to cover both continents.
    • Priscilla: Regarding Buenos Aires site visit - main local organizer was 4 hours away. Very hard to rely on a local organizer. Need to find the right person.
    • Julia: Might be worth talking to successful organizers of Latin America conferences.
    • Cities to consider: Cancun, San Juan, Mexico City.
    • TODO: Start an email thread on NAACL 2021-2022 and the possibility of Latin America.

Treasurer’s Report (Joel)

  • Priscilla estimates $25-50k for NAACL for ACL 2017.
  • Surpluses are shared with the SIGs as well, proportional to attendance.
  • Note that budget slide is before this year’s outlays.
  • TODO: Loss from 2015 hasn’t yet been deducted from our bank account - need to work it out with the David and Priscilla.
  • Joel: Can we merge with LSA or ICML?
  • Priscilla: Don’t want to get lost with respect to ICML. Also, don’t overestimate cross-attendance from co-located conferences.

JSALT Report (Ellen)

  • Funded 6 students out of 12 applicants
  • TODO: Try to figure out the value of the scholarship, follow up on earlier winners.
  • Joel: How did we advertise? A: Corpora list, ACL list, Twitter.
  • Julia: Another opportunity for the coordination with WiNLP and the BIG? This could be transformative for people in developing countries or poorer schools.
  • TODO: Start a discussion with WiNLP about next year’s scholarship.
  • Elen: How do we look for sponsorship for initiative directly?
  • Marie: Program officer at NSF has lots of money for students at workshops. NSF could be for US and we could cover outside US.
    • David: NSF already supports JSALT.

Elections (Colin)

  • First year electing 3 board members
    • Will alternate between 2 and 3 from here on in
  • 213 people voting out of 1242, so just above of 17% voter turnout
    • NAACL membership has grown by a factor of 1.30 since last year’s election
    • election participation has grown by a factor of 1.08
  • Colin to start looking into alternatives to ElectAssist

Where are our conference fees going (Priscilla)

  • Student fees have stayed fixed since 2014.
  • In the past 5 years, the regular fees have fluctuated by $50 depending on the venue.
  • Julia: Getting fees for NIPS - fees are $350. Don’t get much in terms of food.
  • Elen: What are the major sources of costs?
    • Lyn: Food and beverage are significant costs.
    • David: We make up the difference between us and NIPS with food.
    • Philip: This is a strategic decision to keep people together, networking and talking.
    • Priscilla: This year, dropped breakfast to 60% of attendees.
    • Consensus: Seems like this is the right amount of food.
  • Afternoon break is 70-75% of attendance.
  • Another big ticket item is AV
  • Elen: Will these changes impact our profitability this year?
  • Priscilla: Sponsorships are up to $130k beyond a $102k projection.
  • Social event was projected to be $110k, it’s holding there, but the increased bus and food count will hit us. We’ll be about $20k more.
    • Still less than the price of two poster session dinners.
  • Overall projected expenses are $790, and income is $925 - project between $50k and $100k surplus.
  • TODO: Sponsorship idea for logos on projectors.
  • Priscilla: We should be focusing on building up NAACL’s cushion to $100k
    • TODO: Call a vote to set a goal for a cushion of $100k over 5 years.
    • Philip: Potential policy - if you dip into the cushion, you have to tell everyone at the business meeting.
  • Luke: Consider an official hand-off procedure for named positions.
  • Priscilla: can we bring back the Walker Fund or an Aravind Joshi Travel Fund to support people from afar and/or students?

Minutes from NAACL Business Meeting

  • Date: June 4, 2018
  • Location: NAACL-HLT 2018 in New Orleans, LA, USA

General Chair Report (Marilyn Walker)

  • Largest NAACL, 1350 registrants.
  • Conference growing pains: area chairs, reviewers, size.
  • Post-conference survey is coming soon.

Program Chairs Report (Amanda Stent and Heng Ji)

  • Everything is on the chair’s blog.
  • Two corpora coming soon papers plus source code, reviews.
  • Made a proposal on how to improve the reviewing infrastructure.
  • TODO: need to save the PC blog for posterity, especially guest reviewers.
  • Went over demographics.
  • Acceptance rate by country, surprising amount of differentiation.
  • No difference in acceptance rate between industry and academia.
  • Reviewers give higher scores when they think they can identify the authors.
  • Emphasis on strengthening blind review.
  • Very few reviewers changed their scores with respect to the author response.

NAACL HLT 2018 Industry Track

  • Ethics in NLP panel was part of the industry track.
  • 91 total, 28 accepted (20 oral + 8 posters)

NAACL HLT 2018 Student Research Workshop

  • Two tracks: Research papers and thesis proposals.
  • Acceptance rate 39% (16/43 research papers, 4/8 thesis proposals submitted)
  • Lots of funding - not just registration, but travel.

NAACL HLT 2018 Widening NLP (WiNLP)

  • Second year; Non-archival workshop.
  • Acceptance rate: 68%
  • On-site mentoring and pre-submission mentoring
  • TODO: More visa denials then Priscilla is aware of: how do we track this better?
  • 13% of authors were denied visa despite early application + full funding
  • TODO: Registration is expensive for WiNLP attendees - whole discussion about student rate, reimbursement, need to be ready in advance for next year.
  • 32 mentors, mostly from industry.

Other NAACL 2018

  • Workshop and tutorial numbers look healthy.
  • Lyn: Raised issue of whether we should be deciding on workshops for all conferences all at once
    • is it reasonable to have all workshop proposals ready in time for NAACL?
  • Raised the issue of childcare - only had 3 children for 3 main days.
  • Costs were offset by direct company sponsorships.

NAACL Activities

  • JSALT Summer School scholarships ($10k)
  • North American Computational Linguistics Olympiad ($5k)
  • Emerging regions fund ($3k)

NAACL Treasurer’s Report

  • Raised issue of long term goal to carry a larger buffer balance.
  • See slides.

NAACL HLT 2019

  • NAACL HLT 2019 announced to be in Minneapolis, Minnesota (at a very nice hotel)

Open Floor Discussion

  • Started with the visa discussion.
    • Julia: Asked for a list of people who were denied visas.
    • Not sure invitation letter is effective enough.
    • Mostly people from developing companies who were denied visas.
    • Conference organizers could consider working more closely with embassy.
    • NAACL should be hosted in a place other than North America and Canada - offered to point us to someone in Mexico who is up to the task.
    • SRW had a student from Switzerland who couldn’t come.
    • Better to give letters that mention funding.
    • Moving the conference outside the US has the effect that it can prevent international students who are already inside the US and aren’t able to leave.
    • Suggestion to look into rolling offers for oral presentations at later conferences when people have their visas rejected
      • would require coordination to ACL.
    • Libby: for students who weren’t able to come, we had remote presenters for the poster session based on telepresence robots.
      • Also did live streaming to allow them to listen along live.
    • Lyn: $2k extra to do live streaming on top of video recording.
  • Michael Strube: Test of time procedure was somewhat overwhelming. Recommends we follow the AAAI model. Collection of papers from a single fixed year to vote on.
    • TODO: NAACL should try to lay down some standing procedures for this.
    • Michael Strube: Pick NAACL 2002, go through all their papers, and decide on one.
  • Salim Roukos: When a paper is presented, have it assessed by the audience, as feedback for our reviews. Could be done as an app.
    • Philip: Don’t release feedback on individual papers, just aggregate statistics.
    • Backchannel: Isn’t that was citation counts are?
    • Stephanie Lukin: Not everyone will necessarily use a voting-on-quality app - data will be biased.
  • Amanda: 77% of attendees downloaded the app.
  • Kristy Hollingshead: Really loved having the keynote speakers after the lunch break.
  • Accessibility issues: lack of water, delay on when meals will start at the social event.
    • TODO: Accessibility chair? Jason Eisner: How is the new anonymity policy working out?
    • Julia: No decrease in number of submissions due to anonymity policy.
    • Lyn: How many papers had to be withdrawn due to anonymity violations?
    • Heng: 10 papers were withdrawn (not many).
    • Lyn Emphasized the importance of supporting double-blind.
  • TODO: Let’s make sure we have lots of questions about the new review form on the post-conference survey.

Decisions made over e-mail

ACL 2020 Coordinating Committee

  • Committee to advise on the location and organization of ACL 2020 in North America
  • Members for 2018: Joel, Colin, Julia (January 10, 2018)

Honoring Aravind Joshi (February, 2018)

  • Discussion of how to honor Aravind Joshi’s passing at NAACL 2019
  • Idea of using clips from the interview Joel did earlier
  • David suggested to bring the idea of honoring all ACL fellows who have past each year to the ACL exec

Event Funding (March, 2018)

  • Motion: NAACL will allocate the following funds to the following events in 2018.
    • NAACL Emerging Regions Fund (ERF): $3,000.00
    • NAACL Summer School Scholarship at the Jelinek Memorial Workshop On Speech And Language Technology (JSALT): $10,000.00
    • North American Computational Linguistics Olympiad (NACLO): $5,000.00
  • 10 yes, 0 no, motion passes.

Area chair approval by exec (March, 2018)

  • This same issue was raised in 2017, and came up again in 2018
  • The ACL conference handbook states that area chairs should be proposed by the program chairs and approved by the exec
  • We again decided to ignore this rule
  • Julia contacted the ACL to have the handbook change to match what people are actually doing
    • It turns out that it was actually an outdated page

naacl.org up for renewal (April, 2018)

  • Colin proposed to renew naacl.org for 9 year, approved without a formal vote after a very positive discussion amongst the board
  • Details:
    • Domain Name Renewal: naacl.org: $314.91
    • +Web Forwarding for Domain Name naacl.org: $116.91
    • Total: $431.82
    • Service: networksolutions.com

WiNLP and early registration (April, 2018)

  • 4 questions came in from WiNLP:
    • Could we extend the early registration period by a few days? WiNLP and the student volunteer committee sent their notifications with only one weekend for participants to take advantage of the early registration fee.
    • Can NAACL can offer to cover registration fees (workshop or conference) for a small number of participants?
    • Can we instruct authors from developing countries to register with student pricing even if they are not students anymore (we will verify the locations against the list provided by the ACL membership site)
    • For participants who cannot front the money for registration, we would like to have a procedure for them to register by using the WiNLP existing funding. Should they register and mark payment as wire transfer which will be completed from the WiNLP account?
  • Led to an extremely lengthy discussion, where the end result was to leave the decision up to the GC, with the likely action being that 9 WiNLP attendees would be allowed to register with delayed (wire transfer) payment, and would be charged using the early student rate as a special case.
  • The board’s official position is that this should be seen as an exception, not as a general policy or a precedent for other or future conferences
  • TODO: Look into an NAACL (or ACL) level policy on how to make our conferences more accessible and affordable

NAACL 2019 General Chair

  • Jill Burstein selected as General Chair on May 25, 2018
  • Ted Pedersen as program co-chair on June 1, 2018
  • Christy Doran and Thamar Solorio as program co-chairs on June 7, 2018

Board voting infrastructure/confidentiality (June, 2018)

  • At least one board member privately raised a concern about anonymity of internal votes
  • Current voting process either involves an email to the chair / secretary or filling out a Google form with your name
  • TODO: look into better voting infrastructure for small, anonymous votes

NAACL Statement on visa issues (June, 2018)

  • Emily drafted a statement on visa issues in response community concerns
  • Ultimately shelved in favor of an ACL-level effort

Accessibility concerns raised at business meeting (June 2018)

  • Email thread summarized the concerns raised during the NAACL business meeting about accessibility issues at the conference.
  • Concerete actions:
    • Include an item on this for the post-conference survey
    • Encourage the creation of an Accessibility Chair for the next NAACL

Preparing to vote on a budget cushion (June, 2018)

  • Colin initiated a discussion on a budget cushion (minimum amount of cash funds to be held by NAACL before outlays)
  • Shelved momentarily due to there being higher-priority items to discuss

Move board discussions from email to Slack? (June, 2018)

  • Jill raised the idea of starting an NAACL slack channel
  • Discussed among the board, with little sign of support
    • People are happy with email
    • Concerns about Slack’s conversational style, and adding another channel to track
  • Idea was dropped without a vote