Some Lessons Learned from (Organizing) NAACL HLT 2007

  • Author: Candy Sidner, general chair
  • Date: June 4, 2007
  • Contents: “Lessons learned” reports: General chair, Workshops and tutorials, Demos, Publications
  1. Please note the report from the publications committee at the end of this document. It should be regarded closely.

  2. Overall, I think we should have dealt with invited speakers very early in the process (perhaps while we waited for the first set of papers to come in). It worked out all right in the end, but it was more challenging to pick speakers closer to the conference.

  3. With the exception of the treasurer, the sponsor chair and the exhibits chair, all my committees contained representatives of each of NLP, speech and IR. This tri-head committee arrangement is always awkward, although useful. I would recommend letting each committee know that with 3 chairs, they have to come to some terms about who does what, who watches over which things. In the individual reports below, you will notice that some committees felt that 3 tri-head arrangement was unnecessary. for Demos, 3 heads are not necessary but 2 to get the work done might be good. For publications, 3 people are needed (it’s a lot of work) but they don’t need to be spread across the 3 areas).

    It is wise to make sure the PC chairs are tenured, not junior, faculty as this position takes a lot of time and attention. There was discussion in the debriefing of NAACL HLT 07 about having a chair for cross discipline cutting papers. Seems like a good plan.

  4. Good publicity is a key to a good conference. The website is important to that. I recommend getting the website up as soon as there are local arrangements chairs. Just the site photo/icon, the conference name and dates is very helpful. Then add the organizing committee even if it’s done bit by bit. Finally we looked for ways to create alittle buzz once the papers started to appear but weren’t settled. We tried a using a frame iwht news-related items.

  5. Because of the increased concern on security of our borders, it is necessary to produce a letter that invites any paper givers, workshop paper givers, demo presenters and the like so that people outside the US can use them to get visas. In spite of that, most likely some presenter will be denined visa permission. Furthermore, there is nothing you can do about it (we even tried getting one of the NY senators, because HLT 07 was in NY, to address this. There was no response) . It is important that you ask the person to designate someone else to give their paper if that happens.

    Finally it would be wise to put a flashing warning on the website (where all the information about papers and workshops occurs) to indicate that visas are not guaranteed and to tell paper submitters to plan to find someone who can give their paper if the cannot come. To deal with the problem of a person not getting a visa to attend HLT, a suggestion was made to have papers in reserve–they would go in the proceedings but not be given unless needed. Before choosing this option, check with whoever is president of NAACL. We did not do this, so we’re not sure if it will work well.

  6. Getting sponsors early is a very good idea (before the end of the fiscal year) as sometimes potential sponsors have $$ at the end of the year to spend. It is important not only to send out letters asking for sponsorhips but to call likely sponsors personally.

    Priscilla has a DB of all of the sponsors for the past several years. This is a good place to start in looking for sponsors.

  7. I found doing a report once a month (and hence if needed asking committee groups to give me one) was very useful in keeping track of everything. I also suggest that if you choose the strucutre we did (of a poster/demo night with good food to keep people around), you make the decision early.

  8. The list of my committees (I combined workshops and tutorials which worked well): program committee, local arrangements, publicity, workshops&tutorials, treasurer, publications, demos, doctoralconsortium, sponsorship, exhibits.

  9. Many ACL policies apply to NAACL meetings. Get the ACL policies (they are online and review them).

  10. The START software for handling papers has some challenges in setting it up. Below is an email from Matt Stone about things he learned along the way. Also help can be had from NAACL board members for this.

    “(snip)

    SPC members cannot bid to have papers put in their tracks. We must do the assignment. We are assisted by keywords but much of this must be done by hand.

    Reviewers only have access to bidding on papers once papers have been assigned to tracks and the SPC member opens bidding. Reviewers have no way of seeing the overall pool of papers.

    We followed Jen CC and invited a couple SPC member per area, so each has a manageable workload of about 10 papers. Again, following the NAACL 06 lead, we have set it up so that each SPC member is a separate track.

    The complete bid information for each track is therefore a table of about 70-100 bits, “want” or “don’t” for each of 7-10 reviewers for each of 10 papers. Of which each reviewer will wind up being assigned 3-6.

    With this architecture, I don’t think a lot hinges on whether we do bidding. At best, reviewers will get the question “which half of these few papers do you want to review?” To be honest, I thought we’d basically decided not to do bidding once we went with a single track per SPC member rather than a single track per area.

    (The alternative architecture had its own major drawbacks, particularly the fact that SPC members would have had to keep track by hand outside the system of which papers were “theirs”, and would have to synchronize closely to manage the assignment of reviewers to papers, since this is a track-level operation. So this seemed slower and more error-prone, even if it meant less effort from us PC chairs to make sure reviewers got papers they liked.)

    (LL note - this is opposed to the alternative of each SPC member being in charge of more papers period, which is a model that has also been followed.)

    My two cents,”

  11. The sponsorship and exhibits chairs had to arrange for sponsorship logos for the conference (to acknowledge the sponsors). It’s always a challenge getting logos from sponsors, so keep in mind that this must be done and get them as soon as the sponsor agrees to funding.

  12. To deal with the problem of a person not getting a visa to attend HLT, a suggestion was made to have papers in reserve–they would go in the proceedings but not be given unless needed. Before choosing this option, check with whoever is president of NAACL.

  13. Because we did not have a full paper proceedings (it was optional and cost more), we had a conference booklet. There was criticism of that fact that the abstracts were not in this booklet (it grows the booklet considerably and will make it a longer task to organize). However, some people LIKED the booklet without abstracts because one got a better view of the day at a glance. So pick your poison!

Below are additional lessons learned comments from some of my committee chairs.

Marti Hearst (workshops and tutorials):

I would say for next time to keep track of both tutorial and workshop attendance to facilitate planning for next time.

(from the general chair): I would say that getting tutorial speakers to get their slides done on time to be photocopied is a priority (after picking the tutorial speakers from those who submit proposals!).

Bob Carpenter (Demos)

  1. Don’t need Tripartite committees Three people was too many for demos. With three people, we’d drop the ball on messages that went to all of us, or we’d have to spend time coordinating who was going to do what. And then the people who actually do the work (snip) don’t get the credit.

  2. Publication Software Impossible. Jason spent an entire work day wrestling with makefiles for the publication software. I think this is asking too much of volunteers. One person, preferably a paid professional, should handle publications. This worked very smoothly for the last ICSLP at which I had a paper, and it worked back in the day when the whole thing was just photocopied and bound together.

  3. Acceptance Rate too Low My overall advice is to figure out how to get more papers into the conference and workshops. The acceptance rate being so low makes the program committees conservative, the foci narrow, and means a lot of people just won’t be able to raise the funds to attend because they don’t have a paper. Consistency, if that even makes sense, is especially problematic with everyone reviewing so few papers.

Publications Committee

The following is a report from the Publications Committee for NAACL 2007: Yang Liu, Ronnie Smith, and Ellen Voorhees. The purpose of this report is to provide feedback concerning the publications process that we hope will be useful for future conference activities. We consider this report to be part of the public record of the ACL and agree to have any or all of this information disseminated to whoever may find it helpful.

First and foremost, in planning the timeline for producing the proceedings, a minimum of 3 weeks is needed after the final deadline for submission of all papers and companion workshop proceedings to put together the final documents for distribution to the publisher. It kept all three of us quite busy during that time.

Here are some specific comments related to the publications process. Some comments relate to the process in general, and some relate to use of the ACLPUB software. The ACLPUB comments mainly relate either to activities that are for the most part, not clearly documented, or else they are activities for which software modifications could improve the reliability and efficiency of the publications process.

  1. Prior to the final paper submission deadline, a publications chair should go ahead and interact with the head of the ACL anthology to get squared away on that process – which mainly consists of establishing the proper format for the URL’s. This is something that can be handled in advance of the 3 week time crush.

  2. Another important step very early in the process—in fact, this should be done when setting the initial timeline for the entire conference, is to have someone make contact with the publisher (currently Omnipress) and get agreement on their deadline, publishing procedures, and cost estimates (based on last year’s attendees, # of workshops, etc). The ACL Business Office (currently Priscilla Rasmussen) can help with this.

  3. Another important early step is to decide who is in charge of artwork, whether it be conference chairs, local chairs, or publication chairs.

  4. At about the time of final paper submission, a publications chair needs to get updated information from the publisher about exactly what is needed with respect to colors, artwork, ISBN, product delivery, etc.

  5. To engage in “truth in advertising” with the volunteers, it should be made clear that one of the publication chairs will be responsible for reviewing 1000+ proof pages in a 24 hour period arbitrarily decided by the publisher.

  6. Prior to the beginning of the publications process, the ACLPUB software should be updated so that the template files are set for the current conference. Otherwise, every workshop chair as well as the general publications chairs are all redundantly modifying the same information (or in the case of some workshop chairs, they are not and then the general publications chairs have to catch it, fix it, and rerun the software on their supposedly already finished “book”).

  7. If authors are sloppy and careless with capitalization and spelling when they submit their papers to START, the Publication Chairs have to go behind them and fix things. Even if authors are careful, the data still needs to be extensively edited for consistency and LaTeX-isms before producing the final table of contents, program and author lists. While this is documented, it probably needs more emphasis for a proceedings larger than a single workshop.

  8. Major hand-editing of the order file for setting the conference schedule is required for including short papers, demos and invited talks. As the software is currently configured, the short papers cannot be specified by paper number because they are separated from the full papers into a separate proceedings (“book”), and must instead have all title/author information fully hand-specified. If one of those titles contains a colon (:), that is a “reserved” symbol and will not be properly interpreted as a content symbol.

  9. There is also no ability with the software to list presentations within poster sessions with the author/title information in a nice format as currently the format requirements necessitate a separate time entry for each presentation.

  10. It is unclear what the URL variable in the meta file is acually supposed to be.

  11. The CDROM target for the final assembly uses a directory structure that is not what is used on the master CD’s that have been issued. The use of symbolic links in the CDROM target results in broken links in the CDROM master. Correcting for this requires renaming various files and hand editing others in each book.

  12. There is no support for making a hardcopy version of a companion proceedings that is constructed from multiple books.

  13. Pubchairs are responsible for creating ISBN barcodes for the proceedings. The makefile for the pubchair’s “print” option does fail when it can’t find a barcode.eps file, but that is the only mention of this.

We hope that this information will be of use and we of course are happy to communicate further with anyone who might benefit from our experience.

Sincerely, Yang Liu, Ronnie Smith, and Ellen Voorhees