The following test was performed using the newly released Northwind database sample that shipped with SQL Server 7.0. The standard eight tables were selected, plus one example query, the [Order Subtotals] query. Questions were selected randomly from the ELF regression test suite. Automatic (default) builds was used for the ELF sample and the English Wizard sample; for English Query, we used the pre-built sample interface that Microsoft ships with English Query.
The results clearly illustrate the overwhelming superiority of the ELF natural language database query system over its rivals. A + mark indicates a correct response. This condensed view of the results shows the error messages generated by the system, or a comment explaining why the SQL generated was wrong.
Total results: 16 correct for English Wizard (Linguistic Technology), 17 correct for English Query (Microsoft), 91 correct for ELF (English Language Frontend Software). To put this in perspective, if the 7 queries that were answered correctly by all 3 systems are discounted, then on the remaining questions, English Wizard scored 9.6%, English Query scored 10.7%, and ELF scored 90.3%.
| Query | ELF Response | EQ Response | EW Response |
|---|---|---|---|
| where are the suppliers from Germany located | + | Sorry, I didn't understand that. | + |
| show the names and complete address of the biscuit companies | + | Sorry, I didn't understand that. Please check your spelling or phrasing. If you capitalize proper names, it will be easier for me to understand you. |
INCORRECT: show employees, Discontinued 126 times (crosses with Order Details) |
| at which company does Ian work | INCORRECT: work=>worker=>employee (crosses with Employee table) | Based on the information I've been given about this database, I can't answer: "At which companies Ian does works?". I haven't been given any information on companies. |
INCORRECT: No rows returned. |
| who handles the specialty items | + | INCORRECT: No appropriate choice | I'm not familiar with the word: handles |
| show the domestic suppliers | + | Based on the information I've been given about this database, I can't answer: "How domestic are suppliers?". I haven't been given any information on domesticness. |
I'm not familiar with the word: domestic |
| show the New Orleans suppliers | + | INCORRECT: No answer because New Orleans is part of name, not whole name of company | + |
| show the New England suppliers | + | INCORRECT: Same problem as New Orleans | I'm not familiar with the words: New England |
| which company handles the specialty products | + | INCORRECT: No appropriate choice | I'm not familiar with the word: handles |
| which companies have Product Managers | + | Based on the information I've been given about this database, I can't answer: "Which companies have Product Managers?". I haven't been given any information on companies. |
+ |
| show the Product Managers | + | INCORRECT: | + |
| show the orders by Leverling to Hanover Sq | + | I need to know how to interpret the name "Leverling to Hanover Sq" | You must specify 2 values to select a range of values. |
| which products come in bottles | + | INCORRECT: No appropriate choice | I'm not familiar with the word: come |
| What are the names of our Canadian customers? | + | Based on the information I've been given about this database, I can't answer: "Which customers have countries?". I haven't been given any information on countries. |
I'm not familiar with the word: Canadian |
| Give the name and location of suppliers from Germany. | + | Sorry, I didn't understand that. | INCORRECT: Gives address field only, 270 times |
| Which are our Australian suppliers? | + | Based on the information I've been given about this database, I can't answer: "Which suppliers have countries?". I haven't been given any information on countries. |
I'm not familiar with the word: Australian |
| List the countries where suppliers are located, arranging the countries in alphabetical order. | INCORRECT: crosses on Orders table because of incorrect reference to ShipCountry | Suppliers aren't there. Customers are there. | I'm not familiar with the word: arranging |
| Products with names that start with "La". | + | INCORRECT: Wrong answer. Shows all containing, not starting with! | INCORRECT: Offers choice of Employee first or last name only |
| Suppliers who are not located in Canada | + | Based on the information I've been given about this database, I can't answer: "What are the unspecified things suppliers aren't in?". |
I'm confused by the word: Canada |
| Find the products that have between 10 and 20 units in stock | + | Sorry, I didn't understand that. | + |
| Records for customers who are located in Canada and whose names begin with the letter "M" | + | INCORRECT: Neither choice is correct. | I'm not familiar with the word: letter "M" |
| Suppliers who are located in Canada and whose names begin with the letters A-N. | + | Sorry, I didn't understand that. | "letters" must be numeric. |
| Suppliers who have a fax number | + | INCORRECT: show suppliers with or without faxes | + |
| Show the employees hired between May 1, 1992 and June 1, 1993 | + | + | + |
| Employees who live in the United Kingdom or employees who live in Seattle | + | I don't understand the phrase: "t_or I list every employee that lives in Seattle". | I'm not familiar with the words: United Kingdom |
| Orders placed before 1-Jan-93 | + | Based on the information I've been given about this database, I can't answer: "Which orders are placed before 1-Jan-93?"."Which orders are placed?" doesn't depend on 1-Jan-93. |
'1' is not the expected type. |
| Customers whose company names start with N-Z and who are located in either the United Kingdom or Paris | INCORRECT: close, but excludes customers in Paris whose Customer ID isn't PARIS (because there's one that is) | I don't understand the word "company" in the phrase "company start". | "N" must be numeric. |
| Orders that were placed during the month of February 93 | + | + | '(' required after builtin function "month of". |
| Find customers from Canada or the UK who have placed over 15 orders | + | I don't understand the phrase: "from UK". | '15' is not the expected type. |
| Suppliers who provide seafood products and who are from Singapore or Japan. | + | I don't understand the phrase: "from Singapore". | + |
| Find the customers who ordered the "Chef Anton's Cajun Seasoning" product | + | + | + |
| information on orders that were placed after 31-Mar-92, including the employee who made the sale and the customer who placed the order | + | Sorry, I didn't understand that. Please check your spelling or phrasing. If you capitalize proper names, it will be easier for me to understand you. |
'31' is not the expected type. |
| What's the average price of all our products | + | I haven't been given any information on prices. | I'm not familiar with the word: price |
| Give the name and id for each category. | + | INCORRECT: gives names but not Ids | INCORRECT: Offers only [Region ID] choices. |
| List the customers | + | + | + |
| Count the orders that have been placed for each seafood product | + | + | By "address", do you mean Customers, Employees, Suppliers? |
| Show the ship date and order subtotals since March of 1994 | + | I don't understand the phrase: "since March, 1994". | I cannot connect the table "Order Subtotals" to the other tables in your request. |
| Display the subtotal and shipping date of all orders | + | Sorry, I didn't understand that. | I cannot provided (sic) both summary and detail information in the same request. |
| List the suppliers in alphabetical order | + | I don't understand the phrase: "in orders". | + |
| Find the total number of Northwind suppliers | + | Sorry, I didn't understand that. | I'm not familiar with the word: Northwind |
| orders that were shipped today | + | + | What date does "today" refer to? |
| orders that were shipped during the past ten years | + | + | + |
| The number of orders that were shipped within the past 3100 days | + | + | What date does "last" refer to? |
| Find the total value of orders that have been shipped to each country | + | I haven't been given any information on values. | I'm not familiar with the word: value |
| Which products cost between $3 and $6? | + | Products don't have net costs. Line items have net costs. | I'm not familiar with the word: cost |
| Give the order id, product name, product id, price, quantity, discount and extended price for each purchase | + | Sorry, I didn't understand that. Please check your spelling or phrasing. If you capitalize proper names, it will be easier for me to understand you. |
I'm not familiar with the word: price |
| Show catalog information for the active products. | INCORRECT: "catalog" and "active" are not defined | I haven't been given any information on catalogs. | I'm not familiar with the word: catalog |
| the minimum price of all products in the Products table | + | I haven't been given any information on prices. | I'm not familiar with the word: price |
| all records with the current date | + | I don't understand the word "current" in the phrase "current date". | INCORRECT: show no records, only a count |
| What's the total number of orders we received this month | + | I haven't been given any information about people. | I'm not familiar with the word: received |
| all employees who have birthdays today | + | + | INCORRECT: interprets this as "who was born today"! |
| all employees who have birthdays on July 2 | + | + | + |
| All employee records that contain photos | INCORRECT: shows all employees and their photos (even if Photo field is empty) | I don't understand the word "employee" in the phrase "employee record". | INCORRECT: Wrong, interprets Photo as a True/False, which shows all records |
| Find the total number of customers in Canada or the United Kingdom who have placed orders, and group them by country | + | INCORRECT: Neither choice is correct. | I'm not familiar with the words: United Kingdom |
| Find the total value of orders shipped to each customer within each country | + | Sorry, I didn't understand that. | I'm not familiar with the word: value |
| Which employee sold the most units of tofu? | INCORRECT: shows employee who had the most tofu orders, not most total quantity | Based on the information I've been given about this database, I can't answer: "Which employees sold products?". |
I'm not familiar with the word: units |
| Subtotal and customer for orders shipped between 10/1/91 and 12/31/91, sorting on the value | + | Sorry, I didn't understand that. Please check your spelling or phrasing. If you capitalize proper names, it will be easier for me to understand you. |
I'm not familiar with the word: value |
| photos of employees whose last names start with "B" | + | I haven't been given any information on photos. | + |
| show photos of employees hired during 1991 | + | I haven't been given any information on photos. | + |
| which customers have ordered both Konbu and Filo Mix? | + | + | INCORRECT: Just plain stupid. Shows customers who have ordered a product which is named both "Konbu" and "Filo Mix" -- and of course no product has two names! |
| which products are more expensive than chai | + | Sorry, I didn't understand that. | I'm not familiar with the word: expensive |
| how much does chai cost | + | Products don't have net costs. Line items have net costs. | I'm not familiar with the word: cost |
| customers that ordered both chai and filo | + | Sorry, I didn't understand that. Please check your spelling or phrasing. If you capitalize proper names, it will be easier for me to understand you. |
I'm not familiar with the word: filo |
| how many products are there in each category | + | Products aren't in categories. Products are in orders. | Warning: due to a limitation of Microsoft Access the count displayed may include duplicates. |
| which customers have ordered every meat/poultry product | + | + | INCORRECT: shows every meat order, not those that ordered every meat product |
| which customers have never ordered seafood | + | + | I'm not familiar with the word: never |
| which customers ordered Longlife tofu but not filo mix | + | + | INCORRECT: shows customers that have ordered products which are named LongLife tofu, but not (at the sane time) named filo mix; in other words, all orders for Longlife tofu, ignoring the reference to filo mix |
| which customers always use Federal Shipping | + | Sorry, I didn't understand that. | I'm not familiar with the words: always use |
| which product costs the most | Sorry, unable to interpret the question. | I could not find a meaning for the noun "more". | I'm not familiar with the word: costs |
| which customers have placed more orders than average | + | + | I'm confused by the word "orders". |
| show the seafood products in reverse price order | + | I haven't been given any information about prices. | I'm not familiar with the words: reverse price |
| customers that have ordered from both Ma Maison and Tokyo Traders | + | Suppliers have not had customers orderring from them. Employees have had customers orderring from them. | INCORRECT: No records; shows orders from any company which is named both Tokyo Traders and Ma Maison, which is pretty darned unlikely |
| show company names of the suppliers that have more than 3 products | + | I don't understand the word "company" in the phrase "company name". | + |
| which orders were neither shipped to Canada nor sent via Speedy Express | + | INCORRECT: Neither choice is correct. | I can't relate "ship" to a seach value. |
| which orders were not both shipped to Canada and sent via Speedy Express | + | I don't understand the phrase: "via Speedy Express". | I'm confused by the word "ship". |
| how many customers have ordered every meat/poultry product | + | + | Misinterprets as "how many customers ordered EACH meat product" we'd give them that, but then they answer that question wrong by counting each customer once for each meat order, inflating the numbers |
| what percentage of customers have ordered every meat/poultry product | + | + | Warning: due to a limitation of Microsoft Access the count displayed may include duplicates. (Percentages inflated as in above query) |
| which customers bought products from every category | + | Based on the information I've been given about this database, I can't answer: "Which customers bought products?". |
INCORRECT: just shows every order |
| which customers ordered the fewest items | INCORRECT: shows who placed the fewest orders | Based on the information I've been given about this database, I can't answer: "Which items did customers order?". I haven't been given any information on items. |
I'm not familiar with the words: fewest items |
| show the names and complete address of the pear companies | + | I don't know what the companies are. | I'm not familiar with the word: pear |
| which of the clients that purchased tofu have also purchased chai? | + | Based on the information I've been given about this database, I can't answer: "Which clients have purchased products?". I haven't been given any information on clients. |
Error in CreateEWQueryDef: join expression not supported |
| Show the ship date and subtotals for all orders since March of 1991 | + | I don't know how to connect subtotals to orders or ship dates, so I can't answer this question. |
I cannot provided both summary and detail information in the same request. |
| how many customers in each country have ordered tofu? | + | Based on the information I've been given about this database, I can't answer: "What are the countries customers are in?". I haven't been given any information on countries. |
Warning: due to a limitation of Microsoft Access the count displayed may include duplicates. (As warned, it incorrectly includes duplicates) |
| which customers exclusively use Federal Shipping | + | Based on the information I've been given about this database, I can't answer: "Which customers use Federal Shipping?". |
I'm not familiar with the word: use |
| which customers use Federal Shipping exclusively | + | Based on the information I've been given about this database, I can't answer: "Which customers use Federal Shipping?". |
I'm not familiar with the word: use |
| customers that work at 12 Orchestra Terrace | + | Based on the information I've been given about this database, I can't answer: "Which customers work at shipping addresses?". |
INCORRECT: crosses customers with employees table |
| customers in the t2f area | + | INCORRECT: No appropriate choice | I'm not familiar with the word: t2f |
| count the orders for tofu versus those for chai | + | Sorry, I didn't understand that. | I'm not familiar with the word: versus |
| graph the number of tofu or chai orders | + | I didn't understand the meaning of "number of order". |
I'm not familiar with the word: graph |
| graph the number of Seattle employees against London | + | Based on the information I've been given about this database, I can't answer: "How many employees is Seattle the name of?". |
I'm not familiar with the word: against |
| graph the sum of subtotals for seafood against beverages | + | Sorry, I didn't understand that. | I'm not familiar with the word: against |
| graph the average subtotal for each category | + | Sorry, I didn't understand that. | Error processing query |
| graph the sum of subtotals for tofu, chai and konbu | + | I haven't been given any information on subtotals. | Error processing query |
| show the average number of products sold by each employee sales representative | INCORRECT: mistakenly referencing Customer contacts with title "Sales Representative" | I don't understand the word "sales" in the phrase "sales representative". | Error processing query |
| compare the average unit price showing employee and product | + | I don't understand the words "unit_price showing" in the phrase "unit_price showing employee". | INCORRECT: shows one number, not a crosstab |
| which products were shipped by Federal in the last 5 years | + | INCORRECT: "Federal" is not an Employee's firstname! | I'm not familiar with the word: Federal |
| list employees with home phones = (206) 555-8122 | + | INCORRECT: replaces '(206) 555-8122' with '(206)555-8122' which leads to no rows retrieved | I'm not familiar with the word: 206 |
| Find the total number of different customers in Canada or UK who have placed orders | + | Sorry, I didn't understand that. | Error processing query |
| find the total number of DISTINCT customers in Canada or the United Kingdom who have placed orders, and group them by country | + | INCORRECT: Neither choice is correct. | I'm not familiar with the words: United Kingdom |
| which suppliers have order dates that are newer than 600 months old | + | Sorry, I didn't understand that. | I'm not familiar with the words: newer |
| show the difference between discount and unit price | + | Customers don't have unit prices. Products have unit prices. | I'm not familiar with the word: difference |