Generative vs Discriminative Models: What’s the Difference and Why It Matters

Introduction

Artificial Intelligence is an increasingly integral aspect of our society, being employed in a variety of ways from voice-based conversations to generating written, visual and audio material. Of the most compelling advancements is Generative AI, where algorithms or computer systems produce completely novel content.

In order for the workforce to catch up with these advances many people are enrolling in a Generative AI Training Course that provides hands on experience and insight into the inner workings of these systems.

The heart of these AI systems consists of generative and discriminative models. In this blog post, I will outline what each of these models are in layman’s terms with a few real world examples from my own experiences in this field.

What Are Generative Models?

AI models that learn how to generate new types of data based on historical data are called Generative Models (creating something based on past experience). When enough data has been processed, a generative model can produce new instances of similar items instead of only providing “yes/no” responses to a question.

For example, when a generative model is trained on thousands of house photos, then the model will figure out what a typical house looks like and can create an image of a house that didn’t exist before now.

How Generative Models Learn

Generative models learn how the overall structure and individual elements in the dataset go together; this gives generative models the ability to build new datasets using the same characteristics.

As an example of the above, if a generative model learned from all of the pictures of tens of thousands of dogs, then that generative model would have an understanding of the basic features of a dog such as ears, eyes, hair/skin, and certain body shapes, and it would produce its own images of what dogs would look like that appear like they were real, but they would not actually be replicas of any specific dog.

Examples of Generative Models (Without Math)

Text generation (for example, stories, emails)
Image generation (e.g., creating faces or works of art)
Music generation (creating new songs) The easiest way to think about a generative model is in terms of the knowledge behind how a dataset is generated, rather than solely from an output perspective.

What Are Discriminative Models?

Discriminative models are AI models that emphasize decision-making and prediction. These models look at the supplied data and determine which category the data fits within; however, these models do not create new content; rather, they focus on predicting with the highest possible accuracy.

An example of this application is when you receive an email; a discriminative model makes the determination if the email is spam or non-spam; it does not consider how the email was created; it simply makes a decision.

What Discriminative Models Learn

A discriminative model differentiates between classifications by determining the differences between them. The discriminable features in the data allow for a clear separation between classifications.

For example, with a spam filter, it is trained on the characteristics of spam (suspicious words, links, senders, etc.) so when a new e-mail arrives, it can predict if that e-mail will be spam or would go to your inbox because it does not fit the training pattern.

Therefore, a discriminative model is essentially a set of decision boundary rules that an applicant uses to assign a label to incoming data. Discriminative Models are used in the following places:

E-mail filtering systems to prevent unsolicited materials from entering your inbox.
Face recognition systems that open your phone or verify your identity
Credit approval systems that determine whether to approve or disapprove a loan.

Summary:

A discriminative model indicates the decision made regarding the assignment of a label to given data.

Main Difference Between Generative Models & Discriminative Models

Aspect	Generative Models	Discriminative Models
Purpose	Generate new data based on the patterns of existing data, Common in Generative AI applications.	Predict outcomes and make decisions about data via label assignments to input data.
What the Model Learns	Understanding how data is created – Structure, Patterns and Relationships of data features.	How to classify (distinguish) one type of data category from another.
Learning Focus	Generation focuses on the overall (complete) distribution of the data	Classification focuses on identifying the “lines” that separate or divide different data categories.
Output	Generative Models generate new content e.g. text, audio, images, synthetic data etc…	Discriminative Models generate labels or predictions (for example) yes/no, or category name.
Creativity Level	High, due to the ability of generative models to produce creative, original and diverse content.	Low, due to the emphasis on accuracy of predictions as opposed to creativity.
Data Requirements	Generative Models are designed to work with Unlabelled & very Limited labelled Data.	Discriminative Models Require Well Labelled Data to perform effectively.
Commonly Used Examples	Generative Models: Image Generators, Text Generators, Chatbots, Music Generators.	Discriminative Models: Spam Filters, Face Recognition Systems, Credit Approval Systems.
Recommended Use Cases	Generative Models are most useful for content creation, data simulation and other Creative A.I. Applications.	Discriminative Models are best for Classification, Detection, Prediction and Decision-Making Tasks.

How Generative Models Work (High-Level View)

Learning Data Distributions

Generative models research massive collections of information to learn how information typically develops. This includes learning which traits occur together frequently and the relationship these qualities have with one another.

An example of a generative model would be one trained on images of vehicles. A generative model may recognize that most vehicles are constructed with the following characteristics: wheels located at the base, windows positioned between the wheels, and lights on both ends of the vehicle. A generative model also learns how vehicles may differ in colour, shape and size. By learning these patterns, a generative model gets an idea of what is considered to be the “shape” of the vehicle category as a whole versus specific examples of a vehicle.

In other words, generative models learn what is referred to as realistic and normal within the dataset.

Generative Models Generate New Data

Once trained, generative models can produce data where the generated data has not been seen before but appears to be genuine.

Examples of such capabilities include:

Text Generators can write new paragraphs regarding topics never before encountered.
Image Generators can produce pictures of people and places not found in real life.
Music Generators can make original songs from information learned about music.

Generative Models Provide Opportunities for Creativity and Simulation

Generative Models Provide Many Advantages

Advantages set Generative Models apart from other techniques; they can produce all forms of media, such as; text, images, sound, and video.
They are beneficial if the amount of labeled training data is small because they can learn from unmarked or raw data.
They can model real environments, so businesses can create simulations where the true risks or costs are very high.

Healthcare is an excellent representation of using Generative Models to generate synthetic patient records mirroring real medical trends. This protects a patient’s privacy and allows researchers to develop and test their systems safely.

How Discriminative Models Work (High-Level View)

Learning Decision Boundaries

Discriminative Models determine what would be considered a decision boundary. A decision boundary is considered the method or line a model uses to know where one category ends and the other begins.

For instance, in Image Classification, the model determines which visual features are used to classify an image of a cat versus an image of a dog, and as the model improves, it determines clearly where the two categories separate based on the determined features.

Basically, a model can identify the following:

Which features are the most important;
How much each feature is worth; and
How to combine them to reach the correct decision.

Focus on Prediction Accuracy

Discriminative Models are focused on achieving high levels of accuracy. They learn from training data, and for the most part, provide the correct answers as often as possible and do not perform data re-creation or general structure of data, instead concentrating solely on finding aspects that will ultimately lead to correctly making predictions. The accuracy of a model has a major influence on the cost of an operation, so they are very valuable for operations where any mistake could be expensive.

Some areas that Discriminative models perform well are medical diagnostics and fraud detection — even small improvements to accuracy can result in significant improvements in productivity or profitability.

Advantages of Discriminative Approaches

The popular use of Discriminative models is due to the following practical benefits:

High accuracy rates when there is ample labeled training data.
Faster training times than most generative models.
Simpler to evaluate models, because the predicted output is compared to a known correct answer.

Due to their many advantages, Discriminative models are typically the first choice in enterprise and business applications.

Real-World Examples and Use Cases

The Importance of Generative Models When:

Making content: By presenting information with a user generated method of content development (for blogs, images, and videos), users can produce high-quality content in a short amount of time compared to an in-house method.
Augmenting data: Generative models produce additional training examples when there are limited examples available which will result in increased performance of AI models.
Creating conversational agents: The use of Generative models allows chatbot and virtual assistant developers to provide non-linear, natural sounding responses that mimic the tone and feel of a human.

The Importance of Discriminative Models When:

Detecting medical conditions: Discriminative models have a high level of accuracy when detecting health-related conditions based on either patient data or images.
Approving loans: Discriminative models evaluate multiple parameters to determine whether to approve a loan application or not.
Identification of Image Objects: security and autonomous systems using trained machine learning models that classify images or videos into object types.

Generally speaking, most people start out learning how to use Artificial Intelligence by studying or joining Artificial Intelligence Training Course and by applying models that discriminate between different types of objects, then as their experience and interest grows they may choose to move on to building models that generate data instead of merely classifying it. The distinction between these two kinds of models allows individuals to choose the best methodology/strategy to address any particular issue.

Advantages & Limitations of Each Model

Knowing the positive and negative aspects of both generative vs discriminative models allow for selection of appropriate method for your AI Project.

Generative Models

Benefits:

The ability to generate new types of information (text, image/audio, etc.). This makes them useful for generating content or simulating things and data to augment the existing dataset.
Great flexibility and creativity to investigate things which are not already in the training data.
Generative models work well with “unlabeled” data, thereby eliminating the need for an extensive amount of label data to work with.

Drawbacks:

More complex models to train and tune.
More substantial data requirements/Computational Power to operate efficiently.
The products generated can be difficult to predict.

Example: In healthcare, generative models are used to produce synthetic records of patients for studies. Still, due care must be used to ensure that the results from these types of models do not produce unrealistic information.

Discriminative Models

Benefits:

If appropriate amounts of labelled data are available, they provide you with extremely accurate results.
Discriminative models are generally much easier to develop and train.
There are various measures of success available that are easy to compare – Accuracy, Precision, and Recall.

Drawbacks:

Discriminative Models do not have the capability of producing a new dataset.
Discriminative models rely heavily on being provided an appropriate labelled dataset.

Example: In the Fraud Detection arena, Banks use discriminative models to classify transactions. While the algorithms are very reliable at classifying transactions, they do not generate new or simulative examples/scenarios.

How to Decide: Generative or Discriminative Models

Types of Problems

Want predictions? – Discriminative
Want to create your own data? – Generative

Availability and Quality of Data

Limited amount of labeled data – Can use generative modelling
A lot of cleanly labelled data – Good to use discriminative modelling

How to choose between performance and interpretability

Discriminative models usually have a better explanation of the decisions made by the model, whereas generative models tend to be considered a “black box.”

Conclusion

Generative and discriminative models have been pivotal in AI systems over time. Although generative models focus primarily on creating new information, while discriminative models focus on making decisions based on existing data, I believe that this represents only half the puzzle of how generative & discriminative models provide value because there is no absolute answer as to which model type has more value than another; it depends on the problem being solved with that model type.

Understanding the difference between these two model types helps engineers and business executives, students, and everyday people make informed choices regarding their use of AI, and therefore it is expected that as the industry continues to expand throughout the world in the future, understanding what distinguishes these two model types will be a crucial aspect of success in the digital economy.

Table of contents

Generative vs Discriminative Models: What’s the Difference and Why It Matters

Introduction