Automatic item generation: what is it, and how can it help you

Traditional and standardized assessments have suffered noticeable wear and tear as they are being supplanted by other types of more modern evaluations that use technological tools that allow them to be automated. Automatic Item Generation is a clear example of this trend change and is also a significant player in the edTech revolution.

But what is this technique all about? How can it be implemented? What types of Automatic Item Generation are there?

In the following lines, we will answer these questions.

Automatic item generation definition

Automatic Item Generation (AIG) is a psychological and educational assessment technique to generate items (questions or tasks) automatically. 

Automatic item generation uses natural language algorithms and language processing techniques to create test items from a set of predefined rules. This technique is often used to develop adaptive tests, where the difficulty level of the items is adjusted based on the test taker’s responses.

How can automatic item generation help your institution?

Automatic Item Generation (AIG) can help your institution or company through the following benefits it provides:

• Efficiency: it allows the creation of many items quickly and efficiently.
• Objectivity: automatically generated items are usually more objective and less biased than manually created items.
• Adaptability: it allows the creation of adaptive tests, where the difficulty level of the items is adjusted according to the examinee’s responses.
• Accessibility: it can be helpful when items are needed in a specific language or for particular content unavailable on the market.
• Economy: It can reduce the costs associated with creating items manually.
• Scalability: it allows items to be created for many people efficiently.
• Personalization: it allows the creation of customized items for each examinee based on their previous answers.

Automatic item generation types

There are several types of Automatic Item Generation (AIG), each with different approaches and applications. Some of the most common types of AIG are:

AIG based on rules

In this approach, predefined rules are used to generate items from a data set.

Such rules have to do with the format of the questions themselves. For example, they could generate a multiple choice item with four answer options, only one of which is correct, to use present tense verbs or images or graphics to illustrate the items.

Topic-based AIG

In this type of Automatic Item Generation, a set of predefined topics is used to generate items.

An example of this would be the choice of 3 topics: history, science and mathematics.

A data set is then provided to the topic-based AIG tool. For example, a dataset with information on history and science.

Finally, the topic-based AIG tool uses predefined topics to generate items from the dataset.

Question (subject: history): Which events occurred first?

(A) The Industrial Revolution

(B) World War I

(C) The French Revolution

(D) The Discovery of America

Correct Answer: (D) The Discovery of America

Question (topic: science): Which planet in the solar system is closest to the sun?

(A) Mercury

(B) Venus

(C) Earth

(D) Mars

Correct answer: (A) Mercury

In this example, the topic-based AIG tool has used the topic “history” to generate an item about historical events and the topic “science” to generate an item about the solar system.

AIG based on item templates

This approach uses predefined templates to generate items from a data set.

First, a set of templates is defined to generate assessment items. For example, multiple choice questions with four answer options, of which only one is correct, or a fill-in-the-blank question with the right answer.

A data set is then provided to the template-based AIG tool. For example, a data set with history and science information.

Question (template 1): which of the following events occurred first?

(A) The Industrial Revolution

(B) World War I

(C) The French Revolution

(D) The Discovery of America

Correct Answer: (D) The Discovery of America

Question (template 2): Which planet in the solar system is closest to the sun?

__________ is the planet in the solar system that is closest to the sun.

Correct answer: Mercury

In this example, the template-based AIG tool has used template 1 to generate a multiple-choice item and template 2 to generate a fill-in-the-blank item.

AIG based on language models

This option uses language models to generate items from a data set.

First, a language model is trained with an extensive text data set—for example, a set of news items or a collection of books. Next, a dataset is provided to the language model-based AIG tool. 

Finally, the language model-based AIG tool uses the trained language model to generate items from the dataset, e.g., coherent and natural questions and answer choices.

Deep learning-based AIG

This may be the most complex of the Automatic Item Generation types, as deep learning uses neural networks to generate items from a data set.

Deep learning is a machine learning technique inspired by the structure and functioning of the human brain (in particular, the network of neurons). 

It is based on the idea that we can teach computers to learn by themselves from experience without explicitly programming the rules and features for them to look for.

How to implement Automatic Item Generation?

To implement Automatic Item Generation (AIG) at your institution, you need to take the following steps:

• Select an AIG approach that fits your purpose (such as those mentioned above) and the institution’s context.
• Collect and provide the necessary data for the AIG. This may include information about the content you want to assess and any other information relevant to the AIG.
• Train or use an existing IGA model. This may include training a language model or deep neural network or setting up a rule-based or template-based AIG tool.
• Evaluate the quality and effectiveness of the items generated. It is essential to ensure that the items are fit for purpose and the context of your institution and that they are consistent and fair to all examinees.
• Integrate the IGA into the institution’s assessment process. This may include using AIG-generated items in exams or assessments and integrating AIG into the item design and review process.

It is important to keep in mind that AIG is just one edTech tool and is not a magic bullet. It requires a careful planning and evaluation process to ensure its success and usefulness in your institution.
In this regard, you may find it useful to use one of SMOWL’s proctoring products, which will allow you to apply Automatic Item Generation (AIG) fully and securely in your assessments. Ask us for a free demo and see firsthand if the AIG + SMOWL combination is for you.