How To Calculate The Outlier Of A Data Set

Outliers are data points that are far from other data points. In other words, they're unusual values in a dataset. Outliers are problematic for many statistical analyses because they tin can cause tests to either miss significant findings or misconstrue real results.

Unfortunately, there are no strict statistical rules for definitively identifying outliers. Finding outliers depends on subject area-area knowledge and an understanding of the data collection process. While there is no solid mathematical definition, in that location are guidelines and statistical tests you lot can use to find outlier candidates.

In this post, I'll explicate what outliers are and why they are problematic, and present diverse methods for finding them. Additionally, I close this post by comparing the different techniques for identifying outliers and share my preferred approach.

Outliers and Their Impact

Outliers are a simple concept—they are values that are notably different from other data points, and they can cause problems in statistical procedures.

To demonstrate how much a single outlier tin affect the results, let's examine the properties of an example dataset. It contains 15 summit measurements of human males. One of those values is an outlier. The table beneath shows the hateful height and standard deviation with and without the outlier.

Throughout this mail service, I'll be using this case CSV dataset: Outliers.

With Outlier	Without Outlier	Departure
2.4m (seven' 10.5")	1.8m (v' 10.8")	0.6m (~2 feet)
two.3m (7' 6")	0.14m (5.v inches)	two.16m (~7 anxiety)

From the table, it'south easy to meet how a single outlier can distort reality. A single value changes the mean height past 0.6m (2 feet) and the standard deviation by a whopping 2.16m (7 feet)! Hypothesis tests that use the mean with the outlier are off the mark. And, the much larger standard departure volition severely reduce statistical ability!

Earlier performing statistical analyses, you should identify potential outliers. That's the subject of this post. In the adjacent post, nosotros'll move on to figuring out what to practice with them.

There are a variety of ways to find outliers. All these methods employ dissimilar approaches for finding values that are unusual compared to the residue of the dataset. I'll start with visual assessments so move onto more than belittling assessments.

Let'due south find that outlier! I've got five methods for you to try.

Sorting Your Datasheet to Notice Outliers

Sorting your datasheet is a unproblematic but constructive fashion to highlight unusual values. Simply sort your data sheet for each variable then look for unusually loftier or low values.

For example, I've sorted the example dataset in ascending gild, every bit shown beneath. The highest value is conspicuously dissimilar than the others. While this approach doesn't quantify the outlier's caste of unusualness, I similar it considering, at a glance, you'll observe the unusually high or low values.

Graphing Your Data to Identify Outliers

Boxplots, histograms, and scatterplots can highlight outliers.

Boxplots brandish asterisks or other symbols on the graph to indicate explicitly when datasets incorporate outliers. These graphs use the interquartile method with fences to discover outliers, which I explain later. The boxplot below displays our example dataset. It'southward clear that the outlier is quite different than the typical data value.

You can as well utilise boxplots to discover outliers when you have groups in your information. The boxplot beneath shows a different dataset that has an outlier in the Method 2 grouping. Click here to learn more nigh boxplots.

Histograms also emphasize the being of outliers. Await for isolated bars, as shown beneath. Our outlier is the bar far to the right. The graph crams the legitimate data points on the far left.

Click here to larn more near histograms.

About of the outliers I discuss in this postal service are univariate outliers. Nosotros look at a information distribution for a unmarried variable and find values that fall exterior the distribution. However, you tin can apply a scatterplot to discover outliers in a multivariate setting.

In the graph beneath, nosotros're looking at two variables, Input and Output. The scatterplot with regression line shows how nearly of the points follow the fitted line for the model. However, the circled point does not fit the model well.

Interestingly, the Input value (~fourteen) for this observation isn't unusual at all because the other Input values range from x through 20 on the X-axis. Also, notice how the Output value (~50) is similarly inside the range of values on the Y-centrality (10 – 60). Neither the Input nor the Output values themselves are unusual in this dataset. Instead, it's an outlier because it doesn't fit the model.

This type of outlier can exist a problem in regression analysis. Given the multifaceted nature of multivariate regression, there are numerous types of outliers in that realm. In my ebook virtually regression analysis, I detail various methods and tests for identifying outliers in a multivariate context.

For the rest of this post, we'll focus on univariate outliers.

Using Z-scores to Detect Outliers

Z-scores can quantify the unusualness of an observation when your information follow the normal distribution. Z-scores are the number of standard deviations to a higher place and below the hateful that each value falls. For example, a Z-score of 2 indicates that an observation is two standard deviations higher up the boilerplate while a Z-score of -2 signifies information technology is 2 standard deviations below the mean. A Z-score of zilch represents a value that equals the mean.

To calculate the Z-score for an observation, accept the raw measurement, subtract the mean, and split past the standard deviation. Mathematically, the formula for that process is the following:

The further abroad an observation's Z-score is from zero, the more than unusual information technology is. A standard cut-off value for finding outliers are Z-scores of +/-3 or further from naught. The probability distribution below displays the distribution of Z-scores in a standard normal distribution. Z-scores across +/- three are so farthermost y'all can barely come across the shading under the bend.

In a population that follows the normal distribution, Z-score values more farthermost than +/- iii accept a probability of 0.0027 (2 * 0.00135), which is about 1 in 370 observations. However, if your data don't follow the normal distribution, this approach might not be accurate.

Z-scores and Our Example Dataset

In our case dataset below, I display the values in the example dataset along with the Z-scores. This arroyo identifies the same ascertainment as being an outlier.

Note that Z-scores tin be misleading with small datasets because the maximum Z-score is express to (n−one) / √ north.*

Indeed, our Z-score of ~3.six is correct well-nigh the maximum value for a sample size of xv. Sample sizes of 10 or fewer observations cannot have Z-scores that exceed a cutoff value of +/-3.

Also, note that the outlier'south presence throws off the Z-scores because information technology inflates the mean and standard divergence as we saw before. Notice how all the Z-scores are negative except the outlier's value. If we calculated Z-scores without the outlier, they'd be different! Be aware that if your dataset contains outliers, Z-values are biased such that they appear to be less extreme (i.eastward., closer to zero).

For more than data nearly z-scores, read my post, Z-score: Definition, Formula, and Uses.

The z-score cutoff value is based on the empirical rule. For more information, read my post, Empirical Rule: Definition, Formula, and Uses.

Related posts: Normal Distribution and Agreement Probability Distributions

Using the Interquartile Range to Create Outlier Fences

You can use the interquartile range (IQR), several quartile values, and an adjustment factor to calculate boundaries for what constitutes minor and major outliers. Minor and major denote the unusualness of the outlier relative to the overall distribution of values. Major outliers are more farthermost. Analysts also refer to these categorizations as mild and extreme outliers.

The IQR is the middle 50% of the dataset. Information technology's the range of values between the 3rd quartile and the get-go quartile (Q3 – Q1). We tin can take the IQR, Q1, and Q3 values to calculate the following outlier fences for our dataset: lower outer, lower inner, upper inner, and upper outer. These fences determine whether data points are outliers and whether they are mild or extreme.

Values that fall inside the 2 inner fences are not outliers. Let'southward run into how this method works using our example dataset.

Click here to larn more virtually interquartile ranges and percentiles.

Calculating the Outlier Fences Using the Interquartile Range

Using statistical software, I can determine the interquartile range along with the Q1 and Q3 values for our example dataset. We'll need these values to calculate the "fences" for identifying pocket-sized and major outliers. The output beneath indicates that our Q1 value is ane.714 and the Q3 value is ane.936. Our IQR is 1.936 – 1.714 = 0.222.

To calculate the outlier fences, practice the following:

Take your IQR and multiply information technology by i.5 and 3. We'll use these values to obtain the inner and outer fences. For our example, the IQR equals 0.222. Consequently, 0.222 * i.5 = 0.333 and 0.222 * iii = 0.666. Nosotros'll apply 0.333 and 0.666 in the following steps.
Summate the inner and outer lower fences. Accept the Q1 value and subtract the two values from footstep i. The 2 results are the lower inner and outer outlier fences. For our example, Q1 is one.714. And then, the lower inner fence = i.714 – 0.333 = ane.381 and the lower outer fence = i.714 – 0.666 = 1.048.
Calculate the inner and outer upper fences. Take the Q3 value and add the 2 values from stride one. The ii results are the upper inner and upper outlier fences. For our example, Q3 is i.936. And then, the upper inner fence = 1.936 + 0.333 = ii.269 and the upper outer contend = ane.936 + 0.666 = 2.602.

Using the Outlier Fences with Our Example Dataset

For our instance dataset, the values for these fences are one.048, i.381, 2.269, and 2.602. Almost all of our data should fall between the inner fences, which are i.381 and ii.269. At this point, we look at our information values and determine whether whatsoever qualify as being major or minor outliers. 14 out of the 15 data points autumn within the inner fences—they are not outliers. The 15^thursday data signal falls outside the upper outer fence—it's a major or extreme outlier.

The IQR method is helpful because it uses percentiles, which practice non depend on a specific distribution. Additionally, percentiles are relatively robust to the presence of outliers compared to the other quantitative methods.

Boxplots apply the IQR method to decide the inner fences. Typically, I'll use boxplots rather than calculating the fences myself when I want to use this approach. Of the quantitative approaches in this post, this is my preferred method. The interquartile range is robust to outliers, which is clearly a crucial property when you're looking for outliers!

Related postal service: What are Robust Statistics?

Finding Outliers with Hypothesis Tests

You can use hypothesis tests to notice outliers. Many outlier tests exist, simply I'll focus on one to illustrate how they work. In this post, I demonstrate Grubbs' test, which tests the following hypotheses:

Nix: All values in the sample were fatigued from a single population that follows the aforementioned normal distribution.
Alternative: One value in the sample was not drawn from the same commonly distributed population as the other values.

If the p-value for this test is less than your significance level, you can reject the null and conclude that one of the values is an outlier. The analysis identifies the value in question.

Let's perform this hypothesis test using our sample dataset. Grubbs' exam assumes your information are fatigued from a usually distributed population, and it can find only one outlier. If you lot doubtable you have additional outliers, apply a dissimilar test.

Grubbs' outlier test produced a p-value of 0.000. Because it is less than our significance level, we tin can conclude that our dataset contains an outlier. The output indicates it is the high value we found before.

If you lot use Grubbs' exam and find an outlier, don't remove that outlier and perform the analysis again. That process can cause you to remove values that are non outliers.

Challenges of Using Outlier Hypothesis Tests: Masking and Swamping

When performing an outlier exam, you either need to choose a process based on the number of outliers or specify the number of outliers for a test. Grubbs' exam checks for merely one outlier. However, other procedures, such every bit the Tietjen-Moore Test, require you to specify the number of outliers. That'due south hard to do correctly! After all, you're performing the examination to find outliers! Masking and swamping are 2 problems that can occur when you specify the incorrect number of outliers in a dataset.

Masking occurs when you specify likewise few outliers. The additional outliers that exist can bear upon the exam then that information technology detects no outliers. For example, if you specify 1 outlier when there are two, the examination can miss both outliers.

Conversely, swamping occurs when you specify too many outliers. In this case, the test identifies too many data points equally beingness outliers. For example, if you lot specify two outliers when there is only i, the test might determine that there are two outliers.

Because of these problems, I'm not a big fan of outlier tests. More on this in the next section!

My Philosophy virtually Finding Outliers

As you saw, at that place are many means to identify outliers. My philosophy is that you must use your in-depth knowledge about all the variables when analyzing information. Part of this knowledge is knowing what values are typical, unusual, and incommunicable.

I find that when you lot have this in-depth cognition, it's best to apply the more straightforward, visual methods. At a glance, data points that are potential outliers will popular out under your knowledgeable gaze. Consequently, I'll often use boxplots, histograms, and good old-fashioned data sorting! These simple tools provide enough information for me to find unusual data points for further investigation.

Typically, I don't use Z-scores and hypothesis tests to find outliers because of their various complications. Using outlier tests can exist challenging considering they commonly presume your data follow the normal distribution, and then in that location's masking and swamping. Additionally, the existence of outliers makes Z-scores less extreme. Information technology's ironic, merely these methods for identifying outliers are really sensitive to the presence of outliers! Fortunately, as long every bit researchers apply a uncomplicated method to display unusual values, a knowledgeable analyst is likely to know which values need farther investigation.

In my view, the more formal statistical tests and calculations are overkill considering they can't definitively identify outliers. Ultimately, analysts must investigate unusual values and utilise their expertise to determine whether they are legitimate data points. Statistical procedures don't know the subject thing or the data collection process and can't make the concluding decision. You lot should non include or exclude an ascertainment based entirely on the results of a hypothesis test or statistical measure.

At this stage of the analysis, we're only identifying potential outliers for further investigation. It'south just the first step in treatment them. If we err, we desire to err on the side of investigating as well many values rather than likewise few.

In my next mail service, I'll explicate what you're looking for when investigating outliers and how that helps y'all determine whether to remove them from your dataset. Not all outliers are bad and some should not be deleted. In fact, outliers can be very informative about the subject-area and data collection procedure. It'southward important to understand how outliers occur and whether they might happen once again equally a normal part of the procedure or report surface area.

Read my Guidelines for Removing and Handling Outliers.

If you lot're learning about hypothesis testing and like the approach I apply in my blog, bank check out my eBook!

Reference

Ronald E. Shiffler (1988) Maximum Z Scores and Outliers, The American Statistician, 42:1, 79-80, DOI: 10.1080/00031305.1988.10475530