Call For System Demonstrations

The system demonstration track at NAACL-HLT 2019 is a venue for papers describing system demonstrations, ranging from early prototypes to mature production-ready systems. Publicly available open-source or open-access systems are of special interest.

All accepted demos are published in a companion volume of the conference proceedings. We expect at least one of the authors to present a live demo during a demo session at NAACL 2019, with an accompanying poster.

This year, we will also present a Best Demo Paper Award. The winner will be chosen based on the contribution and completeness of the system, as assessed by reviewers and also based on the live demo at the conference.

Submissions will undergo a single-blind reviewing process. So papers may include author and affiliation information, and freely make references to previously published material, and URLs.

Submit a Demo

Submission Guidelines

All submissions should be made electronically via START. Submissions must include:

  • A paper describing the motivation and technical details of the system, including visual aids (e.g., screenshots, snapshots, or diagrams). See examples of recent accepted demo papers in NAACL 2018, ACL 2018, ACL 2017, EMNLP 2017, NAACL 2016, ACL 2015, NAACL 2015.

    • The papers must be no more than four pages, with unlimited pages for references. Please use the main NAACL 2019 paper style files (LaTeX, Word, Overleaf). Accepted papers will be given one additional page of content so that reviewers’ comments can be taken into account.

    • Any papers that do not follow the official style guidelines and page limits will be automatically rejected.

  • A short (max. 3 minutes) video demonstrating the system. This video will be used to evaluate the paper, but won’t be published unless requested.

    • A screencast with audio narration is a natural choice for demos which can be presented on a screen. Otherwise, a video of a user interacting with the system can be used (see this video demonstrating the sounding board system as an example).

    • The production quality of the video is not of interest. Hence, we encourage the videos to be simply a screencast of the software that is getting demoed, with zero to minimal editing efforts.

    • We recommend that you publish your video to YouTube or another website and include the link in your paper. If you prefer not to publicly upload a screencast, please submit the video (in MPEG4 format). The video must be included as supplementary material when you submit your paper through SoftConf.

In addition, we highly recommend that you also include a link to a website that hosts or demonstrates your system in the paper draft. However, we understand that this may not be possible for certain types of services or products.

Reviewing Policy

Reviewing will be single-blind, so you need not anonymize your paper and all the urls mentioned in it. Relevant papers that meet formatting requirements will be assessed on the basis of their relevance to the demo track, contribution, clarity, completeness and novelty. Submissions of identical or closely related work to multiple tracks at NAACL 2019 (comprising research, industry, demo) will be rejected by all tracks.

Areas of Interest

Areas of interest includes, but are not limited to, the following:

End-to-end systems:

  • Information extraction and question answering
  • Dialog or interactive systems
  • Machine translation for consumer or industry applications
  • Mobile applications
  • NLP and speech technologies to support accessibility and assistive devices
  • Technologies for the digital humanities
  • Intelligent input systems

Programs to automate linguistic analyses and aid NLP:

  • Crowd-sourcing systems for collecting textual annotations
  • Systems aiding research and development in NLP
  • Interfaces and resources to support linguistic annotation
  • Software architectures and reusable components
  • Software tools for evaluation or error analysis
  • Research toolkits implementing new or existing techniques
  • Tools for data visualization

NLP applications:

  • Systems supporting learning or education, for example:
    • Visual interactive aids for students
    • Tutorial agents to support real-time feedback for learning
    • Instructional aids for topics in computational linguistics
    • Systems to score or critique textual student responses
    • Systems to mine textual or behavioral data for educational purposes
    • NLP and speech technologies for language learning or correction
  • Systems in healthcare, for example:
    • System to aid medical reporting
    • Knowledge extraction systems for electronic health records
    • Patient Queries Answering system
    • Medical images to text
    • Medical transcriber

Important dates

Paper submissions due (incl. video)
 Monday February 11, 2019
Notification of acceptance Monday March 18, 2019
Camera ready papers due Wednesday April 3, 2019
All deadlines are 11.59 pm UTC -12h.

Contact Information

  • Email: naacl-2019-demo-track@googlegroups.com
  • Demonstration track co-chairs:
    • Waleed Ammar, Allen Institute for Artificial Intelligence, USA
    • Annie Louis, University of Edinburgh, Scotland
    • Nasrin Mostafazadeh, Elemental Cognition, USA
  • General chair: Jill Burstein (Educational Testing Service)

Please feel free to contact the demo chairs for any issues with your submissions or any questions regarding the relevance to the track.

Frequently Asked Questions

Should I submit to industry track or to the system demonstration track?

The NAACL 2019 industry track will feature papers related to challenges of real-world implementations of systems. Papers for the industry track also span a broad range of topics related to system implementation such as datasets, evaluation, scalability, and ethical issues. While demo papers can also involve these aspects, the focus is on a system that can be demonstrated to perform some task, analysis, or programming support. It is possible that the descriptions of some systems could be suitable for both the industry and demo tracks. Then the choice has to be made by the authors about the focus of the paper, and track participation. If the aim is to describe the system and its features, demo track may be more appropriate. If it is to discuss the challenges related to real-world application and deployment, then the industry track is suitable. Feel free to also read the FAQs on the industry track CFP page. Please do not submit the same work to both tracks.

Why is a video mandatory and how will it get reviewed?

The video of your demonstration is intended to aid reviewers in making a decision. The use is primarily to judge the completeness of the system and put a face on the described system. This will give the reviewers a better idea about the system features and characteristics, which can result in a more fair judgement.

Reviewers will be instructed that the interface quality is not the only aspect to consider in the video. Furthermore, the production quality of the video itself will not be a criterion in the reviewing process.

What technical support will be provided in the demo session?

The organizers will provide a monitor, internet access, and electric outlets. International visitors may need to bring an adaptor (details on the voltage range and plug type used in the US can be found here). Most demos are presented using a laptop, but some demos may use different types of computers (e.g., smartphones or embedded systems). Unfortunately, the organizers will not be able to provide laptops.

What are some free video and screencasting tools?

Here are some recommendations we received from fellow researchers in NLP and machine learning.

Windows/MacOS/Linux:

MacOS:

  • Built-in QuickTime Player

Linux: