PEPM'09
Savannah, Georgia, USA, January 19-20, 2009 ACM

Tool Paper Evaluation Criteria

ACM SIGPLAN 2009 Workshop on Partial Evaluation and Program Manipulation

PEPM has a special category of papers called tool demo papers. The main purpose of a tool paper is to display other researchers in the PEPM community a completed, robust and well-documented tool -- highlighting the overall functionality of the tool, the interfaces of the tool, interesting examples and applications of the tool, an assessment of the tool's strengths and weaknesses, and a summary of documentation/support available with the tool.

Tool demo paper submissions must satisfy the following requirements.

The body of the paper must be no longer than 4 pages in length using the two-column ACM Conference style (found here). The body of the paper should give an overview of the tool, the methodology associated with its use, a summary of how the tool has been applied and to what effect, and it should indicate what supporting artifacts (user manual, example repository, downloads, etc) are available. This material will be included in the PEPM proceedings for accepted tool demos. The paper should include the following two items:
an appendix (limited to six pages) that gives an outline of the proposed demo presentation (this material will NOT appear in the PEPM proceedings), and
a URL to a web-site giving documentation and further information about the tool.

Because tool papers have a different format and a different purpose than regular research papers, we would like to (a) describe in more detail the characteristics of a good tool paper, and (b) give the explicit evaluation criteria that will be used by the PEPM program committee when selecting tool papers.

Regular research papers may also describe the details of a particular tool, but note that there are important differences between the contents of a tool demo paper and a research paper that describes a tool. Research papers about tools clearly describe how aspects of the tool (e.g., its architecture, underlying algorithms, combinations of techniques, functionality, etc.) advance the state of the art. This should include a detailed comparison with related work. Moreover, research papers about tools should usually include the results/data of experimental studies that rigorously demonstrate the effectiveness of the claimed scientific advance. On the other hand, the main purpose of a tool demo paper is not to rigorously justify the scientific advances of the tool (although the tool demo paper may contain a concise list of advances or references to other research papers). Rather, a tool demo paper should simply provide concise summary of the current state of a tool and describe how researchers in the PEPM community might apply it effectively.

Below we list the explicit evaluation criteria that will be used by the PEPM committee when selecting tool papers.

Technical Foundations: In keeping with the overall goals of PEPM, described tools should be based on well-reasoned semantic principles. Even though the shorter length of tool papers will not allow authors to provide significant technical details of the theory underlying the tool, submissions should give a concise summary of the technical foundations and provide references to related work where more technical details are presented.
Novelty: In contrast with regular PEPM submissions, PEPM tool demo papers may include work that has been published elsewhere. In the ideal case, the technical foundations of the tool will have been published previously, and the submitted PEPM tool paper will report on follow-on work that has produced a robust tool that has been applied to interesting examples. The PEPM program committee will consider accepting tool demo papers that describe tools that have been demo'ed at other conferences/workshops if these conferences/workshops belong to a different community. For example, if a tool has been demo'ed at an operating system conference, the PEPM program committee might also consider accepting the same tool to be demo'ed at PEPM if it is determined that the demo could provide interesting and new insights to the PEPM community. If tool demo papers are submitted for tools that have demo'ed elsewhere, authors should include a statement in the paper that acknowledges previous demos and that justifies what the benefits would be for presenting the tool again for the PEPM audience. In summary, the "novelty" expected of a tool demo paper is not "never been published before or presented elsewhere", but instead "new useful and practical information provided to the PEPM community".
Stability: PEPM tool demo papers should describe tools with reasonably complete implementations. Tools where significant components are not fully implemented or tested and tools which have not been applied to interesting examples will not be considered. It is expected that tool demos will describe well-engineered internal architectures and interfaces.
Robustness: Tool demo papers will also be evaluated for evidence of tool robustness. This might include evidence that the tool has been applied to a variety of examples, or used by others outside of the research group that developed the tools.
Documentation: Tools to be demoed should also be presented on a web-site that includes documentation for installation and a user manual that describes the basic use of the tool, its options, and its application to at least on example.
Example Repository: Ideally, tools should include a repository of examples (e.g., contained in the tool distribution or available on the tool web site) along with a description of how to run the tool on the examples.
Publicly Availability: Preference will be given to tools that are freely available (e.g., downloadable from the tool web site) so that other researchers in the PEPM community can independently evaluate the tool. Exceptions may be made for tools from industry and commercial tools that cannot be made publicly available for business reasons.
Interesting Public Presentation: Evaluation of tool demo papers will include an assessment of the potential quality of the tool demo (e.g., completeness of tool implementation, depth of functionality, quality of user interfaces, application and assessment of interesting examples) as presented in the attached appendix described above that outlines the demo session.

Authors with further questions about the structure, contents, or suitability of a potential tool demo paper should contact the program chairs.