Supervised Learning - Foundations Project: ReCell¶

Problem Statement¶

Business Context¶

Buying and selling used phones and tablets used to be something that happened on a handful of online marketplace sites. But the used and refurbished device market has grown considerably over the past decade, and a new IDC (International Data Corporation) forecast predicts that the used phone market would be worth \$52.7bn by 2023 with a compound annual growth rate (CAGR) of 13.6% from 2018 to 2023. This growth can be attributed to an uptick in demand for used phones and tablets that offer considerable savings compared with new models.

Refurbished and used devices continue to provide cost-effective alternatives to both consumers and businesses that are looking to save money when purchasing one. There are plenty of other benefits associated with the used device market. Used and refurbished devices can be sold with warranties and can also be insured with proof of purchase. Third-party vendors/platforms, such as Verizon, Amazon, etc., provide attractive offers to customers for refurbished devices. Maximizing the longevity of devices through second-hand trade also reduces their environmental impact and helps in recycling and reducing waste. The impact of the COVID-19 outbreak may further boost this segment as consumers cut back on discretionary spending and buy phones and tablets only for immediate needs.

Objective¶

The rising potential of this comparatively under-the-radar market fuels the need for an ML-based solution to develop a dynamic pricing strategy for used and refurbished devices. ReCell, a startup aiming to tap the potential in this market, has hired you as a data scientist. They want you to analyze the data provided and build a linear regression model to predict the price of a used phone/tablet and identify factors that significantly influence it.

Data Description¶

The data contains the different attributes of used/refurbished phones and tablets. The data was collected in the year 2021. The detailed data dictionary is given below.

  • brand_name: Name of manufacturing brand
  • os: OS on which the device runs
  • screen_size: Size of the screen in cm
  • 4g: Whether 4G is available or not
  • 5g: Whether 5G is available or not
  • main_camera_mp: Resolution of the rear camera in megapixels
  • selfie_camera_mp: Resolution of the front camera in megapixels
  • int_memory: Amount of internal memory (ROM) in GB
  • ram: Amount of RAM in GB
  • battery: Energy capacity of the device battery in mAh
  • weight: Weight of the device in grams
  • release_year: Year when the device model was released
  • days_used: Number of days the used/refurbished device has been used
  • normalized_new_price: Normalized price of a new device of the same model in euros
  • normalized_used_price: Normalized price of the used/refurbished device in euros

Importing necessary libraries¶

In [1]:
# Installing the libraries with the specified version.
# uncomment and run the following line if Google Colab is being used
# !pip install scikit-learn==1.2.2 seaborn==0.13.1 matplotlib==3.7.1 numpy==1.25.2 pandas==1.5.3 -q --user
In [2]:
# Installing the libraries with the specified version.
# uncomment and run the following lines if Jupyter Notebook is being used
# !pip install scikit-learn==1.2.2 seaborn==0.11.1 matplotlib==3.3.4 numpy==1.24.3 pandas==1.5.2 -q --user

Note: After running the above cell, kindly restart the notebook kernel and run all cells sequentially from the start again.

In [3]:
import numpy as np
import pandas as pd

import matplotlib.pyplot as plt
import seaborn as sns

import pylab
import scipy.stats as stats

import statsmodels.stats.api as sms
from statsmodels.compat import lzip

sns.set()

from sklearn.model_selection import train_test_split

import statsmodels.api as sm

from sklearn.metrics import mean_absolute_error, mean_squared_error, r2_score

import warnings
warnings.filterwarnings("ignore", category=FutureWarning)

tab20_blue = '#1f77b4'
tab20_orange = '#ff7f0e'
tab20_green = '#2ca02c'
tab20_red = '#d62728'
tab20_puple = '#9467bd'
tab20_pink = '#e377c2'
tab20_grey = '#7f7f7f'
tab20_yellow = '#bcbd22'
tab20_teal = '#17becf'
sns.set_style("white")

Loading the dataset¶

In [4]:
data = pd.read_csv('used_device_data.csv')
In [5]:
df = data.copy()

Data Overview¶

  • Observations
  • Sanity checks
In [6]:
df.shape
Out[6]:
(3454, 15)
In [7]:
df.info()

<class 'pandas.core.frame.DataFrame'>
RangeIndex: 3454 entries, 0 to 3453
Data columns (total 15 columns):
 #   Column                 Non-Null Count  Dtype  
---  ------                 --------------  -----  
 0   brand_name             3454 non-null   object 
 1   os                     3454 non-null   object 
 2   screen_size            3454 non-null   float64
 3   4g                     3454 non-null   object 
 4   5g                     3454 non-null   object 
 5   main_camera_mp         3275 non-null   float64
 6   selfie_camera_mp       3452 non-null   float64
 7   int_memory             3450 non-null   float64
 8   ram                    3450 non-null   float64
 9   battery                3448 non-null   float64
 10  weight                 3447 non-null   float64
 11  release_year           3454 non-null   int64  
 12  days_used              3454 non-null   int64  
 13  normalized_used_price  3454 non-null   float64
 14  normalized_new_price   3454 non-null   float64
dtypes: float64(9), int64(2), object(4)
memory usage: 404.9+ KB
In [8]:
df.sample(5)
Out[8]:
brand_name os screen_size 4g 5g main_camera_mp selfie_camera_mp int_memory ram battery weight release_year days_used normalized_used_price normalized_new_price
1381 Huawei Android 20.32 yes no 5.0 1.0 32.0 4.0 4800.0 329.0 2014 591 4.564452 5.704249
2253 Others Android 10.29 no no 8.0 0.3 16.0 4.0 2400.0 154.0 2013 1068 4.393585 5.395172
1450 Lava Android 12.70 yes no 13.0 8.0 16.0 4.0 2500.0 128.0 2015 792 4.303119 4.794136
2678 Sony Android 12.70 yes no 19.0 5.0 64.0 4.0 2870.0 168.0 2018 703 4.502694 6.155537
2129 Oppo Android 15.34 yes no 13.0 25.0 64.0 4.0 3600.0 156.0 2018 563 4.839768 5.349391
In [9]:
df.describe(include="all").T
Out[9]:
count unique top freq mean std min 25% 50% 75% max
brand_name 3454 34 Others 502 NaN NaN NaN NaN NaN NaN NaN
os 3454 4 Android 3214 NaN NaN NaN NaN NaN NaN NaN
screen_size 3454.0 NaN NaN NaN 13.713115 3.80528 5.08 12.7 12.83 15.34 30.71
4g 3454 2 yes 2335 NaN NaN NaN NaN NaN NaN NaN
5g 3454 2 no 3302 NaN NaN NaN NaN NaN NaN NaN
main_camera_mp 3275.0 NaN NaN NaN 9.460208 4.815461 0.08 5.0 8.0 13.0 48.0
selfie_camera_mp 3452.0 NaN NaN NaN 6.554229 6.970372 0.0 2.0 5.0 8.0 32.0
int_memory 3450.0 NaN NaN NaN 54.573099 84.972371 0.01 16.0 32.0 64.0 1024.0
ram 3450.0 NaN NaN NaN 4.036122 1.365105 0.02 4.0 4.0 4.0 12.0
battery 3448.0 NaN NaN NaN 3133.402697 1299.682844 500.0 2100.0 3000.0 4000.0 9720.0
weight 3447.0 NaN NaN NaN 182.751871 88.413228 69.0 142.0 160.0 185.0 855.0
release_year 3454.0 NaN NaN NaN 2015.965258 2.298455 2013.0 2014.0 2015.5 2018.0 2020.0
days_used 3454.0 NaN NaN NaN 674.869716 248.580166 91.0 533.5 690.5 868.75 1094.0
normalized_used_price 3454.0 NaN NaN NaN 4.364712 0.588914 1.536867 4.033931 4.405133 4.7557 6.619433
normalized_new_price 3454.0 NaN NaN NaN 5.233107 0.683637 2.901422 4.790342 5.245892 5.673718 7.847841
In [10]:
df.duplicated().sum()
Out[10]:
0
In [11]:
df.isnull().sum()
Out[11]:
brand_name                 0
os                         0
screen_size                0
4g                         0
5g                         0
main_camera_mp           179
selfie_camera_mp           2
int_memory                 4
ram                        4
battery                    6
weight                     7
release_year               0
days_used                  0
normalized_used_price      0
normalized_new_price       0
dtype: int64

Exploratory Data Analysis (EDA)¶

  • EDA is an important part of any project involving data.
  • It is important to investigate and understand the data better before building a model with it.
  • A few questions have been mentioned below which will help you approach the analysis in the right manner and generate insights from the data.
  • A thorough analysis of the data, in addition to the questions mentioned below, should be done.

EDA¶

Questions:

  1. What does the distribution of normalized used device prices look like?
  2. What percentage of the used device market is dominated by Android devices?
  3. The amount of RAM is important for the smooth functioning of a device. How does the amount of RAM vary with the brand?
  4. A large battery often increases a device's weight, making it feel uncomfortable in the hands. How does the weight vary for phones and tablets offering large batteries (more than 4500 mAh)?
  5. Bigger screens are desirable for entertainment purposes as they offer a better viewing experience. How many phones and tablets are available across different brands with a screen size larger than 6 inches?
  6. A lot of devices nowadays offer great selfie cameras, allowing us to capture our favorite moments with loved ones. What is the distribution of devices offering greater than 8MP selfie cameras across brands?
  7. Which attributes are highly correlated with the normalized price of a used device?
In [12]:
def histogram_and_boxplot(data, feature, figsize=(10, 5), kde=False, bins=None, box_color=tab20_orange, 
                          hist_color=tab20_blue, showmeans=True, title_fontsize=16, xlabel_fontsize=14, 
                          ylabel_fontsize=14, rotation=0, fontsize=14, labelsize=12):
    """
    Boxplot and histogram combined

    data: dataframe
    feature: dataframe column
    figsize: size of figure (default (10,5))
    kde: whether to show the density curve (default False)
    bins: number of bins for histogram (default None)
    box_color: color of boxplot (default "tab20_orange")
    hist_color: color of histogram (default "tab20_blue")
    title_fontsize: font size for the title (default 16)
    xlabel_fontsize: font size for the x-axis label (default 16)
    ylabel_fontsize: font size for the y-axis label (default 16)
    rotation: rotation of x-axis labels (default 0 degrees)
    fontsize: font size of the axis tick labels (default 16)
    labelsize: font size of the annotation labels (default 16)
    """
    fig, (ax_box, ax_hist) = plt.subplots(
        nrows=2,
        sharex=True,
        gridspec_kw={"height_ratios": (0.25, 0.75)},
        figsize=figsize,
    )  
    
    sns.boxplot(data=data, x=feature, ax=ax_box, showmeans=showmeans, color=box_color) 
    
    sns.histplot(data=data, x=feature, kde=kde, ax=ax_hist, bins=bins, color=hist_color
                 ) if bins else sns.histplot(data=data, x=feature, kde=kde, ax=ax_hist, color=hist_color)

    mean = data[feature].mean()
    median = data[feature].median()
    ax_hist.axvline(mean, color=tab20_green, linestyle="--", label=f'Mean: {mean:.2f}')
    ax_hist.axvline(median, color=tab20_orange, linestyle="-", label=f'Median: {median:.2f}')
    ax_hist.legend(fontsize=labelsize)
    ax_box.set(title=f'Distribution of {" ".join(feature.split("_")).lower()}', xlabel='', ylabel='')

    ax_box.tick_params(axis='x', rotation=rotation, labelsize=fontsize)
    ax_box.tick_params(axis='y', labelsize=fontsize)
    ax_hist.set_xlabel(feature, fontsize=xlabel_fontsize)
    ax_hist.set_ylabel('Count', fontsize=ylabel_fontsize)
    ax_hist.tick_params(axis='x', rotation=rotation, labelsize=fontsize)
    ax_hist.tick_params(axis='y', labelsize=fontsize)
    plt.show()
In [13]:
def labeled_barplot(data, feature, perc=False, n=None, palette="tab20", figsize=(10, 5), 
                    rotation=90, fontsize=14, labelsize=12, title_fontsize=16, 
                    xlabel_fontsize=14, ylabel_fontsize=14):
    """
    Barplot with percentage at the top

    data: dataframe
    feature: dataframe column
    perc: whether to display percentages instead of count (default is False)
    n: displays the top n category levels (default is None, i.e., display all levels)
    """

    total = len(data[feature])
    unique_count = data[feature].nunique()
    plot_count = unique_count if n is None else min(n, unique_count)
    plt.figure(figsize=(max(plot_count + 2, figsize[0]), figsize[1]))
    plt.xticks(rotation=rotation, fontsize=fontsize)
    order = data[feature].value_counts().index[:plot_count]
    ax = sns.countplot(
        data=data,
        x=feature,
        order=order,
        palette=palette
    )

    for p in ax.patches:
        if perc:
            label = "{:.1f}%".format(100 * p.get_height() / total) 
        else:
            label = p.get_height() 

        x = p.get_x() + p.get_width() / 2
        y = p.get_height()

        ax.annotate(
            label,
            (x, y),
            ha="center",
            va="center",
            size=labelsize,
            xytext=(0, 5),
            textcoords="offset points"
        ) 

    ax.set_title(f'Percentage of {" ".join(feature.split("_")).lower()} Distribution', fontsize=title_fontsize)
    ax.set_xlabel(feature, fontsize=xlabel_fontsize)
    ax.set_ylabel('Count' if not perc else 'Percentage', fontsize=ylabel_fontsize)
    ax.tick_params(axis='x', rotation=rotation, labelsize=fontsize)
    ax.tick_params(axis='y', labelsize=fontsize)

    plt.show()
In [14]:
def labeled_barplot_side_by_side(data, feature1, feature2, perc=False, n=None, palette="tab20", 
                                 figsize=(10, 5), rotation=90, fontsize=14, labelsize=12, 
                                 title_fontsize=16, xlabel_fontsize=14, ylabel_fontsize=14):
    """
    Barplot with percentage at the top for two features side by side

    data: dataframe
    feature1: first dataframe column
    feature2: second dataframe column
    perc: whether to display percentages instead of count (default is False)
    n: displays the top n category levels (default is None, i.e., display all levels)
    palette: color palette for the plots (default is "tab20")
    figsize: size of the figure (default is (10, 5))
    rotation: rotation of x-axis labels (default is 90 degrees)
    fontsize: font size of the axis tick labels (default is 16)
    labelsize: font size of the annotation labels (default is 16)
    title_fontsize: font size of the title (default is 16)
    xlabel_fontsize: font size of the x-axis label (default is 16)
    ylabel_fontsize: font size of the y-axis label (default is 16)
    """

    fig, axs = plt.subplots(1, 2, figsize=figsize)
    
    for ax, feature in zip(axs, [feature1, feature2]):
        total = len(data[feature])
        unique_count = data[feature].nunique()
        plot_count = unique_count if n is None else min(n, unique_count)
        
        order = data[feature].value_counts().index[:plot_count]
        sns.countplot(
            data=data,
            x=feature,
            order=order,
            palette=palette,
            ax=ax
        )
        
        for p in ax.patches:
            if perc:
                label = "{:.1f}%".format(100 * p.get_height() / total)
            else:
                label = p.get_height()

            x = p.get_x() + p.get_width() / 2
            y = p.get_height()

            ax.annotate(
                label,
                (x, y),
                ha="center",
                va="center",
                size=labelsize,
                xytext=(0, 5),
                textcoords="offset points"
            )
        
        ax.set_title(f'Percentage of {" ".join(feature.split("_")).lower()}', fontsize=title_fontsize)
        ax.set_xlabel(feature, fontsize=xlabel_fontsize)
        ax.set_ylabel('Count' if not perc else 'Percentage', fontsize=ylabel_fontsize)
        ax.tick_params(axis='x', rotation=rotation, labelsize=fontsize)
        ax.tick_params(axis='y', labelsize=fontsize)

    plt.tight_layout()
    plt.show()

What does the distribution of normalized used device prices look like?

In [15]:
histogram_and_boxplot(df, "normalized_used_price", kde=True)
No description has been provided for this image

Observations of the distribution of normalized used device prices:

  • The normalized used price is approximately normal with a slight left skew.
  • The mean (4.36) is slightly less than the median (4.41).
  • Most of the devices fall between 3.5 and 5.
  • There are some outliers, mostly on the lower end.
In [16]:
histogram_and_boxplot(df, "normalized_new_price", kde=True)
No description has been provided for this image
In [17]:
histogram_and_boxplot(df, "screen_size", kde=True, bins=8)
No description has been provided for this image
In [18]:
histogram_and_boxplot(df, "main_camera_mp", kde=True, bins=10)
No description has been provided for this image
In [19]:
histogram_and_boxplot(df, "selfie_camera_mp", kde=True, bins=10)
No description has been provided for this image
In [20]:
histogram_and_boxplot(df, "int_memory", kde=True,bins=10)
No description has been provided for this image
In [21]:
histogram_and_boxplot(df, "battery", kde=True)
No description has been provided for this image
In [22]:
histogram_and_boxplot(df, "weight", kde=True)
No description has been provided for this image
In [23]:
histogram_and_boxplot(df, "days_used", kde=True)
No description has been provided for this image

What percentage of the used device market is dominated by Android devices?

In [24]:
labeled_barplot(df, "os", perc=True)
No description has been provided for this image

Observations of Android in the used device market:
Andriod phones make up an overwhelming majority of the used device market with 93.1%. Given the very low percentages of other operating systems we may need to make special considerations when determining if the same factors that drive Android resale value also drive the price for other brands. We may want to look into why there are so many more Androids in this data set. Are there more Androids available or was our data collection biased. Is there a greater demand for Android devices or just higher available. Further analysis is need to determine the driver of the demand and availability of other devices.

Percentage of market by brand:

In [25]:
labeled_barplot(df, "brand_name", perc=True, palette="tab20")
No description has been provided for this image

Distribution of RAM broken out by brand name: The amount of RAM is important for the smooth functioning of a device. How does the amount of RAM vary with the brand?

In [26]:
plt.figure(figsize=(14, 8))
sns.violinplot(x='brand_name', y='ram', data=df, palette="tab20", inner="box")
plt.title('Distribution of RAM by Brand')
plt.xlabel('Brand')
plt.ylabel('RAM (GB)')
plt.xticks(rotation=90)
plt.show()
No description has been provided for this image

Observations of RAM variance:

  • One plus as a wide range in variation of RAM from about 3GB to 14GB
  • Apple and many other brands have a narrow range of RAM suggesting the ram is consistent across most devices within brand. Most seem to have one primary configuration then there are very small percentages or outliers of devices that have higher or lower amount of RAM indicated by only having one density peak and some long tails.

Percentage of phones with 4g and 5g:

In [27]:
labeled_barplot_side_by_side(df, '4g', '5g', perc=True, rotation=None)
No description has been provided for this image
In [28]:
labeled_barplot(df, "ram", perc=True)
No description has been provided for this image
In [29]:
labeled_barplot(df, "release_year", perc=True)
No description has been provided for this image
In [30]:
sns.pairplot(data=df[['days_used', 'battery', 'weight', 'normalized_new_price', 'normalized_used_price' ]])
plt.suptitle('Pair plots of continuous variables', fontsize=16)
plt.subplots_adjust(top=0.95)
plt.show()
No description has been provided for this image
In [31]:
sns.pairplot(data=df[['release_year', 'screen_size', 'main_camera_mp', 'selfie_camera_mp', 'int_memory', 'normalized_used_price']])
plt.suptitle('Pair plots of discrete variables', fontsize=16)
plt.subplots_adjust(top=0.95)
plt.show()
No description has been provided for this image
In [32]:
num_cols = df.select_dtypes(include=np.number).columns.tolist()
plt.figure(figsize=(12, 7))
sns.heatmap(df[num_cols].corr(), annot=True, vmin=-1, vmax=1, fmt=".2f", cmap="Spectral")
plt.suptitle('Correlation heat map', fontsize=16)
plt.subplots_adjust(top=0.9)
plt.show()
No description has been provided for this image

Observations¶

  • The strongest predictor for used price is the new price. The correlation of 0.83 suggests the as new price increases so does the used price.
  • Other features that have a strong positive correlation are battery (0.61), screen size (0.61), main camera mp (0.59), ram (0.52) and release year (0.51). For front and back camera quality, battery capacity, ram and screen size the bigger/higher the value the higher the resale value.
  • The main negative correlation is the days used (-0.36) indicating the more days used device is used the lower the resale value.

A large battery often increases a device's weight, making it feel uncomfortable in the hands. How does the weight vary for phones and tablets offering large batteries (more than 4500 mAh)?

In [33]:
plt.figure(figsize=(12, 8))
large_battery_devices = df[df['battery'] > 4500]
sns.violinplot(x='brand_name', y='weight', data=large_battery_devices, palette="Paired", inner="box")
plt.title('Weight Distribution of Phones and Tablets with Batteries > 4500 mAh', fontsize=16)
plt.xlabel('Brand Name')
plt.ylabel('Weight (grams)')
plt.xticks(rotation=90)
plt.show()
No description has been provided for this image

Observations¶

  • There are several brands that have limited data or do not offer a large capacity battery of 4500 mAh like Oppo, Spice, etc
  • Of those that do have large capacity offerering that have the largest variations in weight include Samsung, acer, levano, LG and Asus
  • The brands that have the lowest average weight with large capacity battery are Honor, infinix, Xiaomi, and Asus.
In [34]:
plt.figure(figsize=(8, 8))
sns.scatterplot(data=df, x='weight', y='battery', hue='os')
plt.legend(loc="upper left")
plt.title('Distribution of battery and weight by operating system', fontsize=16)
plt.show()
No description has been provided for this image
In [35]:
df['screen_size_inches'] = df['screen_size'] * 0.393701
df['device_type'] = df['screen_size_inches'].apply(lambda x: 'Tablet' if x > 7 else 'Phone')
large_screen_devices = df[df['screen_size'] > 6]
In [36]:
plt.figure(figsize=(8, 8))
sns.scatterplot(data=df, x='screen_size_inches', y='battery', hue='os')
plt.legend(loc="upper left")
plt.title('Distribution of screen size in inches and weight by operating system', fontsize=16)
plt.show()
No description has been provided for this image

Bigger screens are desirable for entertainment purposes as they offer a better viewing experience. How many phones and tablets are available across different brands with a screen size larger than 6 inches?

In [37]:
plt.figure(figsize=(14, 8))
sns.countplot(data=large_screen_devices, x='brand_name', hue='device_type', palette='tab20')
plt.title('Number of Phones and Tablets with Screen Size > 6 inches Across Brands')
plt.xlabel('Brand')
plt.ylabel('Count')
plt.xticks(rotation=90)
plt.show()
No description has been provided for this image

Observations:¶

  • Samsung has the most phones with large screens but there are a significant count of devices that have a large screen in the others categroy.
  • The phones with the least number of large screen devices are Apple and Infinix.

A lot of devices nowadays offer great selfie cameras, allowing us to capture our favorite moments with loved ones. What is the distribution of devices offering greater than 8MP selfie cameras across brands?

In [38]:
df_melted = df.melt(id_vars=['os'], 
                    value_vars=['main_camera_mp', 'selfie_camera_mp'], 
                    var_name='camera_type', 
                    value_name='megapixels')

plt.figure(figsize=(15, 7))
sns.boxplot(data=df_melted, x='megapixels', y='os', hue='camera_type', palette='tab20')
plt.title('Distribution of Main and Selfie Camera Megapixels by OS', fontsize=16)
plt.xlabel('Megapixels', fontsize=14)
plt.ylabel('OS', fontsize=14)
plt.legend(title='Camera Type', fontsize=12, title_fontsize=14)
plt.tight_layout()
plt.show()
No description has been provided for this image

Observations of camera quality:

  • Android has the highest megapixel cameras but the range of quality ranges greatly across devices and brands.
  • iOS has the highest mean megapixels for main cameras across all devices.
  • All brands favor higher quality main camera over selfie camera.
In [39]:
plt.figure(figsize=(15, 5))
sns.boxplot(data=df.sort_values("4g"), x="release_year", y="normalized_used_price", hue="4g", palette="tab20")
plt.show()
plt.figure(figsize=(15, 5))
sns.boxplot(data=df.sort_values("5g"), x="release_year", y="normalized_used_price", hue="5g", palette="tab20")
plt.show()
No description has been provided for this image
No description has been provided for this image
In [40]:
plt.figure(figsize=(15, 5))
sns.scatterplot(data=df.sort_values("4g"), x="normalized_new_price", y="normalized_used_price", hue="4g", palette="tab20")
plt.show()
plt.figure(figsize=(15, 5))
sns.scatterplot(data=df.sort_values("5g"), x="normalized_new_price", y="normalized_used_price", hue="5g", style='5g', palette="tab20")
plt.show()
sns.lmplot(data=df, x='normalized_new_price', y='normalized_used_price', col='os', hue='os', palette="tab20");
plt.show()
No description has been provided for this image
No description has been provided for this image
No description has been provided for this image

Data Preprocessing¶

  • Missing value treatment
  • Feature engineering (if needed)
  • Outlier detection and treatment (if needed)
  • Preparing data for modeling
  • Any other preprocessing steps (if needed)
In [41]:
df.isnull().sum()
Out[41]:
brand_name                 0
os                         0
screen_size                0
4g                         0
5g                         0
main_camera_mp           179
selfie_camera_mp           2
int_memory                 4
ram                        4
battery                    6
weight                     7
release_year               0
days_used                  0
normalized_used_price      0
normalized_new_price       0
screen_size_inches         0
device_type                0
dtype: int64
In [42]:
df_clean = df.copy()

df_clean['main_camera_mp'] = df['main_camera_mp'].fillna(value=df.groupby(['release_year'])['main_camera_mp'].transform("median"))
df_clean['selfie_camera_mp'] = df['selfie_camera_mp'].fillna(value=df.groupby(['release_year'])['selfie_camera_mp'].transform("median"))
df_clean['int_memory'] = df['int_memory'].fillna(value=df.groupby(['brand_name', 'release_year'])['int_memory'].transform("median"))
df_clean['ram'] = df['ram'].fillna(value=df.groupby(['brand_name', 'release_year'])['ram'].transform("median"))
df_clean['battery'] = df_clean['battery'].fillna(value=df.groupby(['brand_name'])['battery'].transform("median"))
df_clean['weight'] = df_clean['weight'].fillna(value=df.groupby(['brand_name'])['weight'].transform("median"))

df_clean.isnull().sum()
Out[42]:
brand_name               0
os                       0
screen_size              0
4g                       0
5g                       0
main_camera_mp           0
selfie_camera_mp         0
int_memory               0
ram                      0
battery                  0
weight                   0
release_year             0
days_used                0
normalized_used_price    0
normalized_new_price     0
screen_size_inches       0
device_type              0
dtype: int64
In [43]:
df_clean['selfie_camera_mp'] = np.ceil(df_clean['selfie_camera_mp'])
df_clean['main_camera_mp'] = np.ceil(df_clean['main_camera_mp'])
df_clean['screen_size_inches'] = np.ceil(df_clean['screen_size_inches'])
df_clean['ram'] = np.ceil(df_clean['ram'])
df_clean['weight'] = np.ceil(df_clean['weight'])
df_clean.drop(columns=['device_type', 'screen_size'], axis=1, inplace=True)
In [44]:
df_clean.info()

<class 'pandas.core.frame.DataFrame'>
RangeIndex: 3454 entries, 0 to 3453
Data columns (total 15 columns):
 #   Column                 Non-Null Count  Dtype  
---  ------                 --------------  -----  
 0   brand_name             3454 non-null   object 
 1   os                     3454 non-null   object 
 2   4g                     3454 non-null   object 
 3   5g                     3454 non-null   object 
 4   main_camera_mp         3454 non-null   float64
 5   selfie_camera_mp       3454 non-null   float64
 6   int_memory             3454 non-null   float64
 7   ram                    3454 non-null   float64
 8   battery                3454 non-null   float64
 9   weight                 3454 non-null   float64
 10  release_year           3454 non-null   int64  
 11  days_used              3454 non-null   int64  
 12  normalized_used_price  3454 non-null   float64
 13  normalized_new_price   3454 non-null   float64
 14  screen_size_inches     3454 non-null   float64
dtypes: float64(9), int64(2), object(4)
memory usage: 404.9+ KB
In [45]:
df_clean.describe().T
Out[45]:
count mean std min 25% 50% 75% max
main_camera_mp 3454.0 9.535611 4.659290 1.000000 5.000000 8.000000 13.000000 48.000000
selfie_camera_mp 3454.0 6.716850 6.842379 0.000000 2.000000 5.000000 8.000000 32.000000
int_memory 3454.0 54.528474 84.934991 0.010000 16.000000 32.000000 64.000000 1024.000000
ram 3454.0 4.062536 1.290937 1.000000 4.000000 4.000000 4.000000 12.000000
battery 3454.0 3132.577446 1298.884193 500.000000 2100.000000 3000.000000 4000.000000 9720.000000
weight 3454.0 182.698031 88.348437 69.000000 142.000000 160.000000 185.000000 855.000000
release_year 3454.0 2015.965258 2.298455 2013.000000 2014.000000 2015.500000 2018.000000 2020.000000
days_used 3454.0 674.869716 248.580166 91.000000 533.500000 690.500000 868.750000 1094.000000
normalized_used_price 3454.0 4.364712 0.588914 1.536867 4.033931 4.405133 4.755700 6.619433
normalized_new_price 3454.0 5.233107 0.683637 2.901422 4.790342 5.245892 5.673718 7.847841
screen_size_inches 3454.0 6.299942 1.479394 3.000000 6.000000 6.000000 7.000000 13.000000
  • It is a good idea to explore the data once again after manipulating it.
In [46]:
num_cols = df_clean.select_dtypes(include=np.number).columns.tolist()
plt.figure(figsize=(12, 7))
sns.heatmap(df_clean[num_cols].corr(), annot=True, vmin=-1, vmax=1, fmt=".2f", cmap="Spectral")
plt.suptitle('Correlation heat map', fontsize=16)
plt.subplots_adjust(top=0.9)
plt.show()
No description has been provided for this image
In [47]:
for colname in  ["normalized_new_price", "normalized_used_price"]:
    plt.hist(df[colname], bins=20)
    plt.title(colname)
No description has been provided for this image

Feature Engineering

  • The goal is to predict used price.

-- Step 1 Encode categorical features -- Step 2 Split data into train (70%) and test data (30%) -- Step 3 Build a linear regression model and test performance.

In [48]:
d = df_clean.copy()
In [49]:
brand_mean_target = round(d.groupby('brand_name')['normalized_used_price'].mean(), 2)
d['brand_target_mean'] = d['brand_name'].map(brand_mean_target)
d.drop('brand_name', axis=1, inplace=True)
d.sample(20)
Out[49]:
os 4g 5g main_camera_mp selfie_camera_mp int_memory ram battery weight release_year days_used normalized_used_price normalized_new_price screen_size_inches brand_target_mean
2448 Android yes no 13.0 5.0 16.0 4.0 3300.0 170.0 2016 581 4.129873 5.438601 6.0 4.47
828 Others yes no 8.0 2.0 16.0 4.0 2800.0 170.0 2015 1094 4.085472 4.385894 6.0 4.31
1970 Android yes no 8.0 5.0 16.0 4.0 3100.0 169.0 2018 660 3.956231 4.698296 6.0 4.43
1388 Android no no 1.0 1.0 256.0 1.0 1350.0 130.0 2013 575 3.028199 4.096675 4.0 4.65
135 Android yes no 13.0 8.0 32.0 3.0 4230.0 168.0 2018 664 4.457714 5.437253 7.0 4.76
3440 Android yes no 12.0 10.0 256.0 12.0 4300.0 196.0 2019 489 5.200153 6.509499 7.0 4.47
1106 Android yes no 13.0 8.0 32.0 4.0 3000.0 164.0 2018 641 4.480400 5.293104 6.0 4.67
2184 Android yes no 13.0 5.0 16.0 4.0 2410.0 140.0 2014 790 4.630545 5.677028 6.0 4.76
448 Android no no 2.0 1.0 16.0 4.0 3400.0 298.0 2014 920 4.094345 4.707546 8.0 4.22
1444 Android yes no 5.0 2.0 16.0 4.0 2500.0 169.0 2013 607 4.528613 5.590875 5.0 4.17
736 Android yes no 13.0 5.0 32.0 4.0 4100.0 159.0 2016 599 4.388257 5.337826 6.0 4.51
1593 Android no no 8.0 2.0 32.0 4.0 3000.0 172.0 2014 859 3.954508 5.196949 7.0 4.38
1774 Android yes no 5.0 2.0 16.0 4.0 2460.0 126.0 2013 910 4.236712 5.308466 5.0 4.30
310 Android yes no 8.0 16.0 32.0 3.0 4000.0 175.0 2019 482 4.486612 4.861903 7.0 4.30
2490 Android yes no 5.0 2.0 16.0 4.0 6000.0 490.0 2015 822 4.725084 5.832527 10.0 4.47
233 Android yes no 13.0 5.0 32.0 2.0 3020.0 146.0 2019 201 4.220096 4.496359 6.0 4.67
1281 Android yes no 13.0 8.0 128.0 4.0 3340.0 169.0 2018 496 4.815107 5.389802 7.0 4.65
1602 Android no no 8.0 2.0 16.0 4.0 9000.0 615.0 2014 954 4.824466 5.560566 11.0 4.38
2486 Android no no 5.0 2.0 32.0 4.0 2600.0 490.0 2015 556 4.729421 5.293807 10.0 4.47
367 Android yes yes 8.0 13.0 128.0 8.0 4000.0 169.0 2020 282 5.678499 6.780728 7.0 4.47
In [50]:
display(d.isna().sum())
X = d.drop('normalized_used_price', axis=1)
Y = d['normalized_used_price']
X = sm.add_constant(X)
X = pd.get_dummies(X, columns=X.select_dtypes(include=["object", "category"]).columns.tolist(), drop_first=True)
X = X.astype(float)
X.head()

os                       0
4g                       0
5g                       0
main_camera_mp           0
selfie_camera_mp         0
int_memory               0
ram                      0
battery                  0
weight                   0
release_year             0
days_used                0
normalized_used_price    0
normalized_new_price     0
screen_size_inches       0
brand_target_mean        0
dtype: int64
Out[50]:
const main_camera_mp selfie_camera_mp int_memory ram battery weight release_year days_used normalized_new_price screen_size_inches brand_target_mean os_Others os_Windows os_iOS 4g_yes 5g_yes
0 1.0 13.0 5.0 64.0 3.0 3020.0 146.0 2020.0 127.0 4.715100 6.0 4.67 0.0 0.0 0.0 1.0 0.0
1 1.0 13.0 16.0 128.0 8.0 4300.0 213.0 2020.0 325.0 5.519018 7.0 4.67 0.0 0.0 0.0 1.0 1.0
2 1.0 13.0 8.0 128.0 8.0 4200.0 213.0 2020.0 162.0 5.884631 7.0 4.67 0.0 0.0 0.0 1.0 1.0
3 1.0 13.0 8.0 64.0 6.0 7250.0 480.0 2020.0 345.0 5.630961 11.0 4.67 0.0 0.0 0.0 1.0 1.0
4 1.0 13.0 8.0 64.0 3.0 5000.0 185.0 2020.0 293.0 4.947837 7.0 4.67 0.0 0.0 0.0 1.0 0.0
In [51]:
x_train, x_test, y_train, y_test = train_test_split(X, Y, test_size=0.3, random_state=1)
print("Number of rows in train data =", x_train.shape[0])
print("Number of rows in test data =", x_test.shape[0])

Number of rows in train data = 2417
Number of rows in test data = 1037

Model Building - Linear Regression¶

In [52]:
olsmodel = sm.OLS(y_train, x_train).fit()
display(olsmodel.summary())
OLS Regression Results
Dep. Variable: normalized_used_price R-squared: 0.840
Model: OLS Adj. R-squared: 0.839
Method: Least Squares F-statistic: 785.0
Date: Sat, 13 Jul 2024 Prob (F-statistic): 0.00
Time: 01:24:00 Log-Likelihood: 83.201
No. Observations: 2417 AIC: -132.4
Df Residuals: 2400 BIC: -33.97
Df Model: 16
Covariance Type: nonrobust
coef std err t P>|t| [0.025 0.975]
const -57.6243 8.939 -6.446 0.000 -75.153 -40.096
main_camera_mp 0.0206 0.001 14.409 0.000 0.018 0.023
selfie_camera_mp 0.0136 0.001 12.424 0.000 0.011 0.016
int_memory 8.175e-05 6.74e-05 1.213 0.225 -5.04e-05 0.000
ram 0.0215 0.005 4.022 0.000 0.011 0.032
battery -9.406e-06 7.13e-06 -1.320 0.187 -2.34e-05 4.57e-06
weight 0.0010 0.000 7.265 0.000 0.001 0.001
release_year 0.0291 0.004 6.579 0.000 0.020 0.038
days_used 3.727e-05 3.07e-05 1.214 0.225 -2.29e-05 9.75e-05
normalized_new_price 0.4213 0.012 36.329 0.000 0.399 0.444
screen_size_inches 0.0591 0.008 7.085 0.000 0.043 0.075
brand_target_mean 0.0202 0.020 1.012 0.312 -0.019 0.059
os_Others -0.0695 0.029 -2.387 0.017 -0.127 -0.012
os_Windows 0.0315 0.037 0.860 0.390 -0.040 0.103
os_iOS -0.0778 0.046 -1.692 0.091 -0.168 0.012
4g_yes 0.0463 0.015 2.984 0.003 0.016 0.077
5g_yes -0.0051 0.032 -0.160 0.873 -0.068 0.057
Omnibus: 229.420 Durbin-Watson: 1.904
Prob(Omnibus): 0.000 Jarque-Bera (JB): 432.474
Skew: -0.634 Prob(JB): 1.23e-94
Kurtosis: 4.640 Cond. No. 7.40e+06


Notes:
[1] Standard Errors assume that the covariance matrix of the errors is correctly specified.
[2] The condition number is large, 7.4e+06. This might indicate that there are
strong multicollinearity or other numerical problems.

Observations:¶

  • The coefficient for the intercept (const) represents the expected value of the dependent variable (used device price) when all the predictor variables are set to zero.
  • The predictor coefs represents the average change in the dependent variable for a one unit change in the predictor holding all other variables constant.
  • The std err (standard error) reflects the level of accuracy of the coefficients.
    • The lower the std err the higher the level of accuracy.
  • For each independent feature there is a p-value (probability value) that represents the significance level of the coeffecient. - $H_o$ : Independent feature is not significant ($\beta_i = 0$) - $H_a$ : Independent feature is that it is significant ($\beta_i \neq 0$)
    • Any p-value less than 0.05 is considered statistically significant, meaning we can reject the null hypothesis and that there is significant evidence to show that the feature does have an impact on prediction of the used device price.
  • The confidence interval (last two columns in coefficient rows) represents the range in which the coefficients are likley to fall (with 95% probability)

Model Performance Check¶

In [53]:
def adj_r2_score(predictors, targets, predictions):
    r2 = r2_score(targets, predictions)
    n = predictors.shape[0]
    k = predictors.shape[1]
    return 1 - ((1 - r2) * (n - 1) / (n - k - 1))


def mape_score(targets, predictions):
    return np.mean(np.abs(targets - predictions) / targets) * 100


def model_performance_regression(model, predictors, target):
    """
    Function to compute different metrics to check regression model performance

    model: regressor
    predictors: independent variables
    target: dependent variable
    """

    pred = model.predict(predictors)
    r2 = r2_score(target, pred)
    adjr2 = adj_r2_score(predictors, target, pred) 
    rmse = np.sqrt(mean_squared_error(target, pred))
    mae = mean_absolute_error(target, pred)
    mape = mape_score(target, pred)

    df_perf = pd.DataFrame(
        {
            "RMSE": rmse,
            "MAE": mae,
            "R-squared": r2,
            "Adj. R-squared": adjr2,
            "MAPE": mape,
        },
        index=[0],
    )

    return df_perf
In [79]:
olsmodel_train_perf = model_performance_regression(olsmodel, x_train, y_train)
In [80]:
olsmodel_test_perf = model_performance_regression(olsmodel, x_test, y_test)
In [81]:
print("Training Performance")
display(olsmodel_train_perf)
print("Test Performance")
display(olsmodel_test_perf)

Training Performance
RMSE MAE R-squared Adj. R-squared MAPE
0 0.233783 0.183622 0.839579 0.838443 4.411459 
Test Performance
RMSE MAE R-squared Adj. R-squared MAPE
0 0.23797 0.183983 0.842991 0.840372 4.49641

Observations:

  • The training adjusted R-squared is 0.84. The model is not underfitting.
  • The train and test RMSE (root mean squared error) and the MAE (mean absolute error) are comparable. The model is not overfitting.
  • MAE suggest the model can predict the used device price within a mean error of 0.18 on the test data.
  • MAPE (mean absolute percentage error) of 4.5 on test data means that the model's predictions are, on average, within 4.5% of the actual used device price. This shows the model is very accurate in forcasting the price of used devices.

Checking Linear Regression Assumptions¶

  • In order to make statistical inferences from a linear regression model, it is important to ensure that the assumptions of linear regression are satisfied.

    • We will be checking the following Linear Regression assumptions:

      1. No Multicollinearity

      2. Linearity of variables

      3. Independence of error terms

      4. Normality of error terms

      5. No Heteroscedasticity

TEST FOR MULTICOLLINEARITY¶

In [56]:
from statsmodels.stats.outliers_influence import variance_inflation_factor
def checking_vif(predictors):
    vif = pd.DataFrame()
    vif["feature"] = predictors.columns
    vif["VIF"] = [variance_inflation_factor(predictors.values, i) for i in range(len(predictors.columns))]
    return vif
In [82]:
vif = checking_vif(x_train)
vif
Out[82]:
feature VIF
0 const 3.508742e+06
1 main_camera_mp 1.947758e+00
2 selfie_camera_mp 2.519080e+00
3 int_memory 1.247329e+00
4 ram 2.144870e+00
5 battery 3.838267e+00
6 weight 6.101573e+00
7 release_year 4.536312e+00
8 days_used 2.579905e+00
9 normalized_new_price 2.733247e+00
10 screen_size_inches 6.859834e+00
11 brand_target_mean 1.660658e+00
12 os_Others 1.434433e+00
13 os_Windows 1.027725e+00
14 os_iOS 1.138248e+00
15 4g_yes 2.308747e+00
16 5g_yes 1.822183e+00
In [58]:
def treating_multicollinearity(predictors, target, high_vif_columns):
    """
    Checking the effect of dropping the columns showing high multicollinearity
    on model performance (adj. R-squared and RMSE)

    predictors: independent variables
    target: dependent variable
    high_vif_columns: columns having high VIF
    """
    adj_r2 = []
    rmse = []

    for cols in high_vif_columns:
        train = predictors.loc[:, ~predictors.columns.str.startswith(cols)]
        olsmodel = sm.OLS(target, train).fit()
        adj_r2.append(olsmodel.rsquared_adj)
        rmse.append(np.sqrt(olsmodel.mse_resid))

    temp = pd.DataFrame(
        {
            "col": high_vif_columns,
            "adj_r2": adj_r2,
            "rmse_after": rmse,
        }
    ).sort_values(by="adj_r2", ascending=True)
    temp.reset_index(drop=True, inplace=True)

    return temp

Removing Multicollinearity¶

To remove multicollinearity

  1. Drop every column one by one that has a VIF score greater than 5.
  2. Look at the adjusted R-squared and RMSE of all these models.
  3. Drop the variable that makes the least change in adjusted R-squared.
  4. Check the VIF scores again.
  5. Continue till you get all VIF scores under 5.
In [59]:
col_list = vif[(vif['VIF'] > 5) & (vif['feature'] != 'const')]['feature'].head(2).tolist()
res = treating_multicollinearity(x_train, y_train, col_list)
display(res)
col adj_r2 rmse_after
0 weight 0.835027 0.237126
1 screen_size_inches 0.835201 0.237001
In [60]:
col_to_drop = res["col"][0]
x_train2 = x_train.loc[:, ~x_train.columns.str.startswith(col_to_drop)]
x_test2 = x_test.loc[:, ~x_test.columns.str.startswith(col_to_drop)]

vif = checking_vif(x_train2)
print("VIF after dropping:", col_to_drop)
display(vif)

VIF after dropping: weight
feature VIF
0 const 3.401353e+06
1 main_camera_mp 1.866595e+00
2 selfie_camera_mp 2.496045e+00
3 int_memory 1.247326e+00
4 ram 2.141976e+00
5 battery 3.431984e+00
6 release_year 4.394034e+00
7 days_used 2.572847e+00
8 normalized_new_price 2.725345e+00
9 screen_size_inches 3.214652e+00
10 brand_target_mean 1.658241e+00
11 os_Others 1.320344e+00
12 os_Windows 1.027209e+00
13 os_iOS 1.129024e+00
14 4g_yes 2.286757e+00
15 5g_yes 1.819929e+00

Observations:¶

  • All VIF values are less than 5.
  • There are no more multicollinearity in the data.
In [61]:
olsmod1 = sm.OLS(y_train, x_train2).fit()
display(olsmod1.summary())
OLS Regression Results
Dep. Variable: normalized_used_price R-squared: 0.836
Model: OLS Adj. R-squared: 0.835
Method: Least Squares F-statistic: 816.3
Date: Sat, 13 Jul 2024 Prob (F-statistic): 0.00
Time: 01:24:01 Log-Likelihood: 56.912
No. Observations: 2417 AIC: -81.82
Df Residuals: 2401 BIC: 10.82
Df Model: 15
Covariance Type: nonrobust
coef std err t P>|t| [0.025 0.975]
const -46.2631 8.895 -5.201 0.000 -63.707 -28.820
main_camera_mp 0.0185 0.001 13.064 0.000 0.016 0.021
selfie_camera_mp 0.0128 0.001 11.658 0.000 0.011 0.015
int_memory 8.26e-05 6.81e-05 1.213 0.225 -5.09e-05 0.000
ram 0.0201 0.005 3.718 0.000 0.010 0.031
battery 7.437e-06 6.81e-06 1.092 0.275 -5.92e-06 2.08e-05
release_year 0.0234 0.004 5.320 0.000 0.015 0.032
days_used 4.894e-05 3.1e-05 1.579 0.114 -1.18e-05 0.000
normalized_new_price 0.4259 0.012 36.383 0.000 0.403 0.449
screen_size_inches 0.1033 0.006 17.894 0.000 0.092 0.115
brand_target_mean 0.0146 0.020 0.727 0.467 -0.025 0.054
os_Others -0.0099 0.028 -0.349 0.727 -0.065 0.046
os_Windows 0.0374 0.037 1.012 0.311 -0.035 0.110
os_iOS -0.0477 0.046 -1.031 0.303 -0.139 0.043
4g_yes 0.0353 0.016 2.262 0.024 0.005 0.066
5g_yes 0.0030 0.032 0.095 0.925 -0.060 0.066
Omnibus: 216.522 Durbin-Watson: 1.909
Prob(Omnibus): 0.000 Jarque-Bera (JB): 393.139
Skew: -0.616 Prob(JB): 4.27e-86
Kurtosis: 4.545 Cond. No. 7.28e+06


Notes:
[1] Standard Errors assume that the covariance matrix of the errors is correctly specified.
[2] The condition number is large, 7.28e+06. This might indicate that there are
strong multicollinearity or other numerical problems.

Observations:¶

  • After removing weight we can see that const has changed and the std err value is slightly lower.
  • The confidence interval is also slightly smaller which indicates improved accuracy.
  • before: const -57.6243 8.939 -6.446 0.000 -75.153 -40.096
  • after : const -46.2631 8.895 -5.201 0.000 -63.707 -28.820
  • before: R-squared: 0.840
  • after : R-squared: 0.836
  • The adjusted R-squared value has dropped only a small amount so dropping the column did not have much impact.
    • before: Adj. R-squared: 0.839
    • after : Adj. R-squared: 0.835

Dealing with high p-value variables¶

  • Some of the dummy variables in the data have p-value > 0.05. Variables that have a value greater than 0.5 are considered insignificant and should be dropped.
  • p-values may change after dropping a variable. We will drop one at a time starting with the highest p-value.
  • Then we will create a new model without the dropped feature and check p-values again repeating the process till there are no insignificant features in our model.
In [62]:
def remove_insignificant_features(predictors):
    selected_features = predictors.columns.tolist()
    removed_features = []
    while len(selected_features) > 0:
        x_train_aux = predictors[selected_features]
        model = sm.OLS(y_train, x_train_aux).fit()
        p_values = model.pvalues
        max_p_value = max(p_values)
        feature_with_p_max = p_values.idxmax()
    
        if max_p_value > 0.05:
            selected_features.remove(feature_with_p_max)
            removed_features.append(feature_with_p_max)
        else:
            break
    
    print('Removed features:', removed_features)
    print('Remaining features: ', selected_features)
    return selected_features
In [63]:
selected_features = remove_insignificant_features(x_train2.copy())

Removed features: ['5g_yes', 'os_Others', 'brand_target_mean', 'os_iOS', 'os_Windows', 'battery', 'int_memory', 'days_used']
Remaining features:  ['const', 'main_camera_mp', 'selfie_camera_mp', 'ram', 'release_year', 'normalized_new_price', 'screen_size_inches', '4g_yes']
In [64]:
x_train3 = x_train2[selected_features]
x_test3 = x_test2[selected_features]
In [65]:
olsmod2 = sm.OLS(y_train, x_train3).fit()
display(olsmod2.summary())
OLS Regression Results
Dep. Variable: normalized_used_price R-squared: 0.836
Model: OLS Adj. R-squared: 0.835
Method: Least Squares F-statistic: 1749.
Date: Sat, 13 Jul 2024 Prob (F-statistic): 0.00
Time: 01:24:01 Log-Likelihood: 53.286
No. Observations: 2417 AIC: -90.57
Df Residuals: 2409 BIC: -44.25
Df Model: 7
Covariance Type: nonrobust
coef std err t P>|t| [0.025 0.975]
const -40.5048 6.806 -5.951 0.000 -53.851 -27.159
main_camera_mp 0.0188 0.001 14.060 0.000 0.016 0.021
selfie_camera_mp 0.0130 0.001 12.139 0.000 0.011 0.015
ram 0.0210 0.005 4.510 0.000 0.012 0.030
release_year 0.0206 0.003 6.107 0.000 0.014 0.027
normalized_new_price 0.4285 0.011 39.377 0.000 0.407 0.450
screen_size_inches 0.1076 0.004 28.380 0.000 0.100 0.115
4g_yes 0.0385 0.015 2.555 0.011 0.009 0.068
Omnibus: 216.248 Durbin-Watson: 1.911
Prob(Omnibus): 0.000 Jarque-Bera (JB): 396.969
Skew: -0.612 Prob(JB): 6.30e-87
Kurtosis: 4.563 Cond. No. 2.85e+06


Notes:
[1] Standard Errors assume that the covariance matrix of the errors is correctly specified.
[2] The condition number is large, 2.85e+06. This might indicate that there are
strong multicollinearity or other numerical problems.
In [66]:
olsmod2_train_perf = model_performance_regression(olsmod2, x_train3, y_train)
olsmod2_test_perf = model_performance_regression(olsmod2, x_test3, y_test)
In [67]:
print("Training Performance")
display(olsmod2_train_perf)
print("Test Performance")
display(olsmod2_test_perf)

Training Performance
RMSE MAE R-squared Adj. R-squared MAPE
0 0.236695 0.185567 0.835559 0.835012 4.452555 
Test Performance
RMSE MAE R-squared Adj. R-squared MAPE
0 0.240362 0.185432 0.839819 0.838572 4.526207

Observations after removing features with high p-values¶

  • There are no insignificant features in the model
  • The adjusted R-Squared is 0.836 meaning the model can explain up to 84% of the variance in the used price.
  • The adjusted R-Squared in olsmod1 (after removing variables with multicollineariety) was 0.835. This very small difference indicates the variables that were dropped were not impacting the model.
  • RMSE and MAE values are very close for the training and testing data sets, indicating the model is not overfitting.

TEST FOR LINEARITY AND INDEPENDENCE¶

  • Linearity describes a straight line relationship between two variables. Predcitor variables must have a linear relationship with the dependent variable
  • If the error terms (or residuals) are not independence then the confidence intervals of the coefficeint estimates will be narrower and could cause the conclustion that a parameter is statistically significant when it is not.
In [68]:
df_pred = pd.DataFrame()

df_pred["Actual Values"] = y_train
df_pred["Fitted Values"] = olsmod2.fittedvalues
df_pred["Residuals"] = olsmod2.resid

df_pred.head()
Out[68]:
Actual Values Fitted Values Residuals
3026 4.087488 3.867893 0.219594
1525 4.448399 4.582682 -0.134283
1128 4.315353 4.288759 0.026593
3003 4.282068 4.255228 0.026840
2907 4.456438 4.459759 -0.003321
In [69]:
plt.figure(figsize=(5, 5))
sns.residplot(data=df_pred, x="Fitted Values", y="Residuals", color="purple", lowess=True)
plt.xlabel("Fitted Values")
plt.ylabel("Residuals")
plt.title("Fitted vs Residual plot")
plt.show()
No description has been provided for this image

Observation:¶

  • If there is a pattern in the plot between fitted and residuals it would indicate non-linearity in the data which would mean the model is not capturing the non-linear effects.
  • In this plot there is no pattern indicating that the assumptions of linearity and independence are met.

TEST FOR NORMALITY¶

  • Error terms or residuals should be normally distributed. If they are not, confidence intevals may be too narrow and inaccurate, making coefficient estimation more difficult. Non-normality suggests outliers need to be checked to improve the model.
In [70]:
plt.figure(figsize=(5, 5))
sns.histplot(data=df_pred, x="Residuals", kde=True)
plt.title("Normality of residuals")
plt.show()
No description has been provided for this image
In [71]:
plt.figure(figsize=(5, 5))
stats.probplot(df_pred["Residuals"], dist="norm", plot=pylab)
plt.show()
No description has been provided for this image
In [72]:
stats.shapiro(df_pred["Residuals"])
Out[72]:
ShapiroResult(statistic=0.9717327952384949, pvalue=1.93208428849325e-21)

Observations for Normality¶

  • The histogram of residuals has a bell shape characteristic of normality but it does look a little left skewed.
  • The residuals mostly follow a straight line in the q-q plot.
  • The p-value in the Shapiro test is very small which indicates that the residuals may not be normal and this could lead to issues such as unreliable confidence intervals and biased coefficient estimates in the model.
  • Based on the histogram and q-q plot we will consider the assumption of normality to be met.

TEST FOR HOMOSCEDASTICITY¶

  • If the variance of the residuals is symmetrically distributed across the regression line, then the data is said to be homoscedastic.
  • If the variance is unequal for the residuals across the regression line, then the data is said to be heteroscedastic.
  • The presence of non-constant variance in the error terms results in heteroscedasticity. Generally, non-constant variance arises in presence of outliers.
In [73]:
name = ["F statistic", "p-value"]
test = sms.het_goldfeldquandt(df_pred["Residuals"], x_train3)
lzip(name, test)
Out[73]:
[('F statistic', 1.0520632950072462), ('p-value', 0.18969074531364338)]

Observation on Test for Homoscedasticity¶

  • The p-value is greater than 0.05 so the residuals are homoscedastic and the assumption is met.
In [74]:
# predictions on the test set
pred = olsmod2.predict(x_test3)

df_pred_test = pd.DataFrame({"Actual": y_test, "Predicted": pred})
df_pred_test.sample(10, random_state=1)
Out[74]:
Actual Predicted
1995 4.566741 4.373350
2341 3.696103 3.942322
1913 3.592093 3.774153
688 4.306495 4.120242
650 4.522115 5.133677
2291 4.259294 4.399627
40 4.997685 5.444454
1884 3.875359 4.101330
2538 4.206631 4.066285
45 5.380450 5.276652

Final Model¶

In [75]:
x_train_final = x_train3.copy()
x_test_final = x_test3.copy()
In [76]:
olsmodel_final = sm.OLS(y_train, x_train_final).fit()
display(olsmodel_final.summary())
OLS Regression Results
Dep. Variable: normalized_used_price R-squared: 0.836
Model: OLS Adj. R-squared: 0.835
Method: Least Squares F-statistic: 1749.
Date: Sat, 13 Jul 2024 Prob (F-statistic): 0.00
Time: 01:24:01 Log-Likelihood: 53.286
No. Observations: 2417 AIC: -90.57
Df Residuals: 2409 BIC: -44.25
Df Model: 7
Covariance Type: nonrobust
coef std err t P>|t| [0.025 0.975]
const -40.5048 6.806 -5.951 0.000 -53.851 -27.159
main_camera_mp 0.0188 0.001 14.060 0.000 0.016 0.021
selfie_camera_mp 0.0130 0.001 12.139 0.000 0.011 0.015
ram 0.0210 0.005 4.510 0.000 0.012 0.030
release_year 0.0206 0.003 6.107 0.000 0.014 0.027
normalized_new_price 0.4285 0.011 39.377 0.000 0.407 0.450
screen_size_inches 0.1076 0.004 28.380 0.000 0.100 0.115
4g_yes 0.0385 0.015 2.555 0.011 0.009 0.068
Omnibus: 216.248 Durbin-Watson: 1.911
Prob(Omnibus): 0.000 Jarque-Bera (JB): 396.969
Skew: -0.612 Prob(JB): 6.30e-87
Kurtosis: 4.563 Cond. No. 2.85e+06


Notes:
[1] Standard Errors assume that the covariance matrix of the errors is correctly specified.
[2] The condition number is large, 2.85e+06. This might indicate that there are
strong multicollinearity or other numerical problems.
In [77]:
olsmodel_final_train_perf = model_performance_regression(olsmodel_final, x_train_final, y_train)
olsmodel_final_test_perf = model_performance_regression(olsmodel_final, x_test_final, y_test)
In [78]:
print("Training Performance")
display(olsmodel_final_train_perf)
print("Test Performance")
display(olsmodel_final_test_perf)

Training Performance
RMSE MAE R-squared Adj. R-squared MAPE
0 0.236695 0.185567 0.835559 0.835012 4.452555 
Test Performance
RMSE MAE R-squared Adj. R-squared MAPE
0 0.240362 0.185432 0.839819 0.838572 4.526207

Actionable Insights and Recommendations¶

Model Performance

  • High R-Squared value of 0.836) indicates that 83.6% of the variance in the used data prices can be explained by this model. The high number indicates it is a strong relationship between the model features and the price making it a good predictor.
  • The low RMSE and MAE on both training and test data suggest the model's predictions are accurate and reliable.

Significant Features

  • Main Camera Megapixes, Selfie Camera Megapixels, Ram, Release Year, New Price and Screen Size are all contributing positively to the used price.
  • The higher the values for the significant features the higher the used device price.

Insignificant Features

  • Battering, internal memory and days used all have a higher p-values and were removed as they were deemed insignificant

Recomendations¶

  • Use significant features identified to set competitive prices for used devices. Focus on the main and selfie camera resolutions, ram and release year.
  • Highlight the significant features in the product listings and marketing materials to attract buyers.
  • Prioritize bringing in and selling new models with higher initial prices, larger screens and advance cameras.
  • Continuously monitor model performance and update when new data is available to ensure accuracy.