Large language models have a fundamental limitation: they can only generate text based on patterns learned from training data. Ask ChatGPT about today's weather in Seoul, and it will either refuse or hallucinate—because it has no direct access to live weather feeds. Tool-calling solves this by giving LLMs the ability to invoke external APIs, retrieve real information, and act on it. Instead of guessing, the model can say, "Let me check the weather API first," then generate a properly formatted API call, parse the response, and answer with actual data.
This capability has transformed LLMs from static text generators into agents that can book flights, query databases, control smart-home devices, or fetch stock prices. The mechanism is straightforward: the model outputs a structured request (usually JSON) specifying an API endpoint and parameters. The system executes the call and returns the result to the model, which then continues its reasoning.
Most tool-calling benchmarks are English-only. If you want to evaluate how well an LLM handles API interactions, you're typically working with English-language tasks built around English-language APIs. This creates a significant blind spot.
Korean, Japanese, Arabic, and other non-English languages present different challenges: distinct syntactic structures, cultural context in query phrasing, and APIs that may have Korean-language documentation or return Korean-language responses. A model that performs excellently on English tool-calling benchmarks might fail spectacularly when asked to navigate Korean public APIs—not because it lacks reasoning ability, but because it was never evaluated or trained on such scenarios.
Korea has one of the world's most extensive open-data ecosystems. The Korean government operates the 공공데이터포털 (Public Data Portal), which hosts thousands of APIs covering everything from weather forecasts to hospital availability, air quality, public transportation schedules, and demographic statistics. These APIs are well-documented, publicly accessible, and—crucially—often return Korean-language data.
This makes Korean public APIs an ideal testbed for multilingual tool-calling research. They're real, diverse, and require genuine language understanding. A model that successfully navigates these APIs must understand Korean queries, map them to appropriate endpoints, handle Korean-language responses, and orchestrate multi-step workflows.
The paper makes two primary contributions. First, it introduces a benchmark for evaluating multi-step tool-calling capabilities over Korean open public APIs. This benchmark includes over 100 APIs and more than 1,000 multi-step tasks, covering domains like weather, transportation, public safety, and healthcare.
Second, it proposes a data-synthesis recipe—a systematic method for generating training and evaluation data at scale using LLMs themselves. The recipe dramatically reduces the cost and effort of creating tool-calling datasets, cutting annotation costs by up to 90% compared to manual data creation.
The authors evaluate several state-of-the-art LLMs on the benchmark, and the results are sobering: current models achieve less than 50% accuracy on the multi-step tasks. This gap between English and Korean performance highlights the urgent need for more multilingual tool-calling research.
Key Takeaway: Tool-calling is essential for LLMs to interact with real-world systems, but existing benchmarks are overwhelmingly English-centric. Korean public APIs provide a rich, realistic testbed for evaluating and improving multilingual multi-step tool-calling.
Tool-calling (also called function calling) is the process by which an LLM determines that it needs external information and generates a structured request to obtain it. The model might call a single tool—say, a weather API—or it might chain multiple calls together to accomplish a complex task.
The typical flow works as follows:
get_air_quality(city="Busan").Importantly, the model doesn't execute code itself—it generates the specification, and the surrounding system handles the actual HTTP requests.
Single-step tool-calling involves one API call to answer a query. "What's the temperature in Seoul?" maps to a single weather API call. Simple, linear, and easy to evaluate.
Multi-step tool-calling is more complex. The model must chain multiple API calls, where the output of one call often informs the parameters of the next. Consider this example from the paper:
"Find the nearest public hospital to my location and provide the number of available beds."
This requires at least three steps: 1. Call a geocoding API to convert the user's location description into coordinates. 2. Call a hospital directory API to find hospitals near those coordinates. 3. Call a bed availability API to check current bed counts at the identified hospital.
Each step depends on the previous one. If the geocoding call fails or returns ambiguous results, the subsequent steps are meaningless. The model must maintain context across multiple API interactions, handle intermediate responses, and make decisions based on partial information.
Real-world tasks are rarely single-step. Booking a trip requires checking flights, hotels, and local weather. Finding a restaurant involves location lookup, restaurant search, and possibly reservation availability. Monitoring public health might require pulling data from multiple government agencies.
Single-step tool-calling is a solved problem for most capable LLMs. Multi-step tool-calling remains challenging because it requires:
The benchmark includes tasks like:
Weather + timezone + geocoding: A user asks, "What is the weather in Seoul tomorrow?" The model must geocode "Seoul" to coordinates, determine the correct timezone for accurate "tomorrow" calculation, and then query the weather API.
Air quality across districts: "Get the air quality index for all districts in Busan and compare with the average of the last week." This requires querying an air quality API for each district, then querying a historical data API, then computing comparisons.
Public facility lookup: "Find public libraries near Gangnam Station that are open after 9 PM." The model must geocode the location, search for libraries, retrieve their operating hours, and filter by the time constraint.
These tasks seem simple to humans but expose significant weaknesses in current LLMs.
Key Takeaway: Multi-step tool-calling requires planning, state management, and contextual reasoning—not just the ability to format a single API request. This is where current LLMs, especially in non-English contexts, frequently fail.
Open public APIs are application programming interfaces provided by government agencies and public institutions, freely accessible to developers and researchers. Korea's Public Data Portal (data.go.kr) aggregates these APIs, providing unified documentation and access keys. They cover a vast range of services:
The paper's benchmark focuses on APIs that are genuinely useful for everyday queries. Weather APIs provide current conditions and forecasts—useful for the "What's the weather?" class of queries. Transportation APIs cover bus and subway systems, enabling queries like "When is the next bus to City Hall?" or "Is Line 2 operating normally?"
The benchmark also includes APIs from the Public Data Portal that handle more specialized queries—hospital information, public facility locations, and demographic statistics. This variety ensures the benchmark tests a broad range of tool-calling abilities, not just a single API pattern.
Korean public APIs present several challenges that make them a good stress test for LLMs:
Language asymmetry: The user query might be in Korean, the API expects parameters in Korean (like city names), but the model's internal reasoning might default to English. Handling this cross-lingual mapping is non-trivial.
Documentation complexity: API documentation is often verbose, inconsistent across endpoints, and may not follow RESTful conventions. Models fine-tuned on clean, synthetic API specs may struggle with real-world documentation.
Data format variability: Different APIs return data in different formats—XML, JSON, or CSV—with inconsistent field naming. A model must adapt to each API's quirks.
Parameter validation: Korean APIs often require specific parameter formats (like date formats or coordinate systems) that differ from what the model might assume based on English API training data.
If LLMs are to be genuinely useful for Korean speakers—or any non-English speakers—they must handle tool-calling in those languages. This isn't just about translation; it's about understanding culturally specific query patterns, navigating Korean-language API responses, and orchestrating multi-step workflows using Korean data sources.
The Korean open public API ecosystem provides a realistic, publicly accessible testbed for this research. Researchers can evaluate models without needing proprietary APIs or special access agreements.
The benchmark was designed with several principles in mind:
The benchmark includes over 100 Korean public APIs spanning weather, transportation, public safety, healthcare, and demographic domains. These APIs are real, publicly accessible services from the Korean government's Public Data Portal.
The benchmark contains over 1,000 multi-step tool-calling tasks. Each task is a Korean-language user query paired with a sequence of API calls that should be made to answer it. Tasks vary in complexity: some require two API calls, others require five or more.
Each task in the benchmark follows a structured format:
For example, a task might specify: Query = "내일 부산의 미세먼지 농도는?" (What's the fine dust level in Busan tomorrow?). The expected sequence: (1) Call air quality API with city=Busan and date=tomorrow, (2) parse the response, (3) formulate a Korean-language answer.
The primary metric is task success rate—the percentage of tasks where the model correctly orchestrates all required API calls and produces the correct final answer. A task is marked successful only if all intermediate steps are correct; a model that makes the right API calls but misformulates one parameter fails the task.
The evaluation methodology is automated: the benchmark system executes the model's API calls, checks them against expected sequences, and verifies final answers. This allows for consistent, reproducible evaluation across different models.
Key Takeaway: The benchmark is designed for realism and verifiability. It uses real Korean public APIs and requires complete, correct multi-step API sequences—not just partial credit for getting some steps right.
Building tool-calling benchmarks and training datasets manually is expensive and slow. Each task requires: designing a realistic query, determining the correct API call sequence, executing the calls to verify they work, and validating the ground-truth answer. Doing this for thousands of tasks across hundreds of APIs is impractical for most research teams.
The paper's data-synthesis recipe addresses this bottleneck by using LLMs to generate training data automatically.
The core idea is straightforward: instead of having humans write queries and API call sequences, use an LLM to generate them. The LLM is given a description of available APIs, their parameters, and the types of data they return. It's then prompted to generate realistic user queries and the corresponding multi-step API call sequences.
This approach leverages the LLM's understanding of both Korean language patterns and API usage. The generated data isn't perfect—it requires filtering and validation—but it's dramatically cheaper and faster than manual annotation.
The recipe involves several stages:
Stage 1: API description compilation. For each API, the system creates a structured description including endpoints, required parameters, return formats, and example calls. This is the "knowledge base" the LLM works from.
Stage 2: Task generation. The LLM is prompted with the API descriptions and asked to generate user queries that would require multiple API calls to answer. The prompt includes examples of good queries and instructions to vary the complexity and domain.
Stage 3: API sequence generation. For each generated query, the LLM produces the expected sequence of API calls, including specific parameter values. This is the "answer key" for the task.
Stage 4: Execution and validation. The system actually executes the generated API call sequences against the real APIs. If the calls succeed and return plausible data, the task is kept. If they fail (due to invalid parameters or non-existent endpoints), the task is discarded or regenerated.
Stage 5: Filtering and deduplication. The generated tasks are filtered for quality, removing duplicates, ambiguous queries, or tasks that are too similar to existing ones.
The paper reports that this recipe reduces annotation cost by up to 90% compared to manual data creation. Instead of paying annotators to design queries and manually trace API call sequences, the system generates candidate tasks in bulk and only requires human review for quality control.
This efficiency matters because tool-calling datasets need to be large to be useful for fine-tuning. A dataset of a few hundred examples is insufficient to teach a model the nuances of multi-step API orchestration. The synthesis recipe makes it feasible to generate tens of thousands of training examples.
Synthetic data has a well-known weakness: it can be repetitive, uncreative, or subtly wrong. The recipe addresses this through:
The validation stage is crucial. An LLM might confidently generate an API call sequence that references a non-existent endpoint or uses incorrect parameter formats. Executing the calls against real APIs catches these errors.
Key Takeaway: The data-synthesis recipe uses LLMs to generate training data at scale, with real API execution serving as a validation layer. This reduces annotation costs by up to 90% while maintaining data quality.
The paper evaluates several state-of-the-art LLMs, including both proprietary and open-source models. The models are tested in a zero-shot setting—they're given the API descriptions and the user query, then asked to produce the correct API call sequence and final answer.
The results are striking: state-of-the-art LLMs achieve less than 50% accuracy on the multi-step tool-calling benchmark. This is in stark contrast to their performance on English-language tool-calling benchmarks, where leading models often exceed 80% accuracy.
The performance gap suggests that current models have a significant weakness in multilingual multi-step tool-calling. They can handle the mechanics of API calls but struggle with the Korean-language reasoning and API navigation required by the benchmark.
The paper identifies several recurring patterns of failure:
Parameter hallucination: Models generate parameters that look plausible but don't exist in the actual API. For example, a model might call a weather API with city="Seoul" when the API expects a numeric city code.
Sequence errors: Models call APIs in the wrong order. For example, trying to query bed availability at a hospital before identifying which hospital to query.
Language mixing: Models generate API parameters in English when the API expects Korean, or vice versa. This causes API calls to fail or return incorrect results.
Context loss: In multi-step tasks, models sometimes forget the results of earlier API calls when making later ones. They might correctly query the air quality API for one district but then fail to carry that result forward when querying the next district.
Premature termination: Models sometimes stop after one or two API calls, having answered a simpler question than the one actually asked.
The contrast with English-language benchmarks is instructive. On English tool-calling benchmarks, models generally handle multi-step tasks reasonably well—they've seen enough English API examples during training to internalize common patterns. But Korean APIs are less represented in training data, and the syntactic differences between Korean and English create additional challenges.
This suggests that the bottleneck isn't tool-calling ability per se, but the intersection of tool-calling with non-English language understanding.
Key Takeaway: Current LLMs achieve under 50% accuracy on the Korean multi-step tool-calling benchmark, revealing a significant gap between English and non-English tool-calling performance. Common failure modes include parameter hallucination, sequence errors, and context loss.
The benchmark provides a concrete testbed for improving multilingual tool-calling. Researchers can fine-tune models on the synthetic training data generated by the recipe, then evaluate their performance on the benchmark to measure improvement. This creates a feedback loop: generate data → fine-tune → evaluate → identify remaining weaknesses → generate targeted data to address them.
The ultimate goal is LLMs that can serve Korean users effectively—answering questions about local weather, navigating public transportation, finding healthcare resources, and accessing government services. These are everyday tasks that require real API interactions, not just language generation.
A model that succeeds on this benchmark would be genuinely useful for Korean speakers. It could power AI assistants that help users navigate public services, access real-time information, and complete multi-step tasks.
The data-synthesis recipe isn't Korean-specific. The same approach could be applied to Japanese public APIs, European open data portals, or any other language with accessible public APIs. The paper's methodology provides a template for creating multilingual tool-calling benchmarks and training datasets.
This raises the possibility of cross-lingual transfer: could a model trained on Korean tool-calling tasks perform better on Japanese tasks, or vice versa? The answer likely depends on the linguistic similarity between the languages and the overlap in API structures.
The paper contributes to a growing body of research on making LLMs useful beyond English-speaking contexts. As AI assistants become integrated into daily life, they must handle the linguistic and cultural diversity of their users. Benchmarks like this one—rooted in real-world services and authentic language—are essential for measuring and driving progress.
The benchmark covers Korean public APIs, which have particular characteristics—they're government-run, follow specific documentation conventions, and serve a specific user base. The findings might not generalize to private APIs, APIs in other languages, or APIs with different documentation styles.
The benchmark also focuses on information retrieval tasks. It doesn't cover transactional tool-calling (like booking reservations or making purchases) where error handling and state management are more complex.
The data-synthesis recipe relies on LLMs to generate training data, which introduces potential biases. LLMs might generate queries that reflect their training data biases—overrepresenting certain topics (like weather) while underrepresenting others (like obscure government services). The filtering and validation stages mitigate this but don't eliminate it.
The recipe's generalizability is promising but untested. Applying it to other languages would require: publicly accessible APIs with documentation, LLMs capable of generating fluent queries in the target language, and validation infrastructure that can execute API calls. Each of these presents potential challenges.
The paper opens several research directions:
This paper makes two significant contributions to the field:
A benchmark for multi-step tool-calling over Korean open public APIs, containing over 1,000 tasks across more than 100 real APIs. This benchmark addresses the gap in non-English tool-calling evaluation.
A data-synthesis recipe that uses LLMs to generate training data at scale, reducing annotation costs by up to 90% while maintaining quality through real API execution validation.
The benchmark reveals that current LLMs are far from adequate for multilingual multi-step tool-calling, with state-of-the-art models achieving under 50% accuracy. This is a wake-up call: tool-calling capabilities that work in English don't automatically transfer to other languages.
The data-synthesis recipe provides a practical path forward. By making it feasible to generate large-scale training data, the recipe enables the fine-tuning needed to improve multilingual tool-calling performance.
The authors have released the benchmark and synthetic data to the research community. This is an invitation: use these resources to develop better multilingual tool-calling models, extend the methodology to other languages, and push the boundaries of what LLMs can do for non-English speakers.
The gap between English and Korean tool-calling performance won't close by itself. It requires deliberate effort, better benchmarks, and more diverse training data. This paper provides the tools—now it's up to the research community to use them.
Explore the benchmark and synthetic data to advance multilingual tool-calling research. The resources are publicly available, and the methodology can be extended to other languages and domains.
The paper introduces two things: (1) a benchmark for evaluating multi-step tool-calling over Korean open public APIs, containing over 1,000 tasks across 100+ real APIs, and (2) a data-synthesis recipe that uses LLMs to generate training data at scale, reducing annotation costs by up to 90%.
Korea has one of the world's most extensive open-data ecosystems, with thousands of publicly accessible government APIs. These APIs are real, diverse, and require Korean-language understanding—making them an ideal testbed for multilingual tool-calling research. The benchmark addresses a gap in non-English tool-calling evaluation.
The recipe is a method for automatically generating training data for tool-calling tasks. It involves: compiling API descriptions, using LLMs to generate realistic user queries and corresponding API call sequences, executing those calls against real APIs to validate them, and filtering the results for quality. This reduces annotation costs by up to 90% compared to manual data creation.
The benchmark uses task success rate as the primary metric. Each task requires a complete, correct sequence of API calls leading to the correct final answer. The evaluation is automated: the system executes the model's API calls, checks them against expected sequences, and verifies final answers. A task is only successful if all intermediate steps are correct.
State-of-the-art LLMs achieve less than 50% accuracy. Common failure modes include: generating invalid API parameters (parameter hallucination), calling APIs in the wrong order (sequence errors), mixing Korean and English in API parameters (language mixing), losing context across multiple API calls, and stopping prematurely after answering only part of the query.
Yes, in principle. The recipe requires: publicly accessible APIs with documentation, LLMs capable of generating fluent queries in the target language, and validation infrastructure to execute API calls. The authors suggest the methodology could be extended to other languages and public data ecosystems.
The benchmark includes over 100 Korean public APIs covering domains such as weather, transportation, public safety, healthcare, and demographic statistics. All APIs are real, publicly accessible services from the Korean government's Public Data Portal.
Yes. The authors have released the benchmark and the synthetic data generated by the recipe to the research community for further study and development.