Location
Online
Event Website
https://hicss.hawaii.edu/
Start Date
3-1-2023 12:00 AM
End Date
7-1-2023 12:00 AM
Description
Large language models (LLMs) can synthesize code from natural language descriptions or by completing code in-context. In this paper, we consider the ability of LLMs to synthesize code, at inference time, for a novel API not in its training data, and specifically examine the impact of different API designs on this ability. We find that: 1) code examples in model training data seem to facilitate API use at inference time; 2) hallucination is the most common failure mode; and 3) the designs of both the novel API and the prompt affect performance. In light of these findings, we introduce the concept of a Synthetic API: an API designed to be used by LLMs instead of by humans. Synthetic APIs for LLMs offer the potential to further accelerate development of natural language interfaces to arbitrary tools and services.
Recommended Citation
Mullins, Ryan and Terry, Michael, "Synthetic APIs: Enabling Language Models to Act as Interlocutors Between Natural Language and Code" (2023). Hawaii International Conference on System Sciences 2023 (HICSS-56). 2.
https://aisel.aisnet.org/hicss-56/cl/text_analytics/2
Synthetic APIs: Enabling Language Models to Act as Interlocutors Between Natural Language and Code
Online
Large language models (LLMs) can synthesize code from natural language descriptions or by completing code in-context. In this paper, we consider the ability of LLMs to synthesize code, at inference time, for a novel API not in its training data, and specifically examine the impact of different API designs on this ability. We find that: 1) code examples in model training data seem to facilitate API use at inference time; 2) hallucination is the most common failure mode; and 3) the designs of both the novel API and the prompt affect performance. In light of these findings, we introduce the concept of a Synthetic API: an API designed to be used by LLMs instead of by humans. Synthetic APIs for LLMs offer the potential to further accelerate development of natural language interfaces to arbitrary tools and services.
https://aisel.aisnet.org/hicss-56/cl/text_analytics/2