Appending data

Sometimes, we may be analyzing multiple datasets that have a similar structure or samples of the same dataset. While analyzing our datasets, we may need to append them together into a new single dataset. When we append datasets, we stitch the datasets along the rows. For example, if we have 2 datasets containing 1,000 rows and 20 columns each, the appended data will contain 2,000 rows and 20 columns. The rows typically increase while the columns remain the same. The datasets are allowed to have a different number of rows but typically should have the same number of columns to avoid errors after appending.

In pandas, the concat method helps us append data.

Getting ready

We will continue working with the Marketing Campaign data from Kaggle. We will work with two samples of that dataset.

Place the marketing_campaign_append1.csv and marketing_campaign_append2.csv files in the data subfolder created in the first recipe. Alternatively, you could retrieve all the files from the GitHub repository.

How to do it…

We will explore how to append data using the pandas library:

1. Import the pandas library:

   import pandas as pd

2. Load the .csv files into a dataframe using read_csv. Then, subset the dataframes to include only relevant columns:

   marketing_sample1 = pd.read_csv("data/marketing_campaign_append1.csv")

   marketing_sample2 = pd.read_csv("data/marketing_campaign_append2.csv")

   marketing_sample1 = marketing_sample1[['ID', 'Year_Birth','Education','Marital_Status','Income', 'Kidhome','Teenhome','Dt_Customer', 'Recency','NumStorePurchases', 'NumWebVisitsMonth']]

   marketing_sample2 = marketing_sample2[['ID', 'Year_Birth','Education','Marital_Status','Income', 'Kidhome','Teenhome','Dt_Customer', 'Recency','NumStorePurchases', 'NumWebVisitsMonth']]

3. Take a look at the two datasets. Check the first few rows and use transpose (T) to show more information:

   marketing_sample1.head(2).T

       0    1

   ID    5524    2174

   Year_Birth    1957    1954

   …        …        …

   NumWebVisitsMonth    7    5

   marketing_sample2.head(2).T

       0    1

   ID    9135    466

   Year_Birth    1950    1944

   …        …        …

   NumWebVisitsMonth    8    2

4. Check the data types as well as the number of columns and rows:

   marketing_sample1.dtypes

   ID    int64

   Year_Birth    int64

   …          …

   NumWebVisitsMonth    int64

   marketing_sample2.dtypes

   ID    int64

   Year_Birth    int64

   …          …

   NumWebVisitsMonth    int64

   marketing_sample1.shape

   (500, 11)

   marketing_sample2.shape

   (500, 11)

5. Append the datasets. Use the concat method from the pandas library to append the data:

   appended_data = pd.concat([marketing_sample1, marketing_sample2])

6. Inspect the shape of the result and the first few rows:

   appended_data.head(2).T

       0    1

   ID    5524    2174

   Year_Birth    1957    1954

   Education    Graduation    Graduation

   Marital_Status    Single    Single

   Income    58138.0    46344.0

   Kidhome    0    1

   Teenhome    0    1

   Dt_Customer    04/09/2012    08/03/2014

   Recency    58    38

   NumStorePurchases    4    2

   NumWebVisitsMonth    7    5

   appended_data.shape

   (1000, 11)

Well done! We have appended our datasets.

How it works...

We import the pandas library and refer to it as pd in step 1. In step 2, we use read_csv to load the two .csv files to be appended into pandas dataframes. We call the dataframes marketing_sample1 and marketing_sample2 respectively. We also subset the dataframes to include only 11 relevant columns. In step 3, we inspect the dataset using the head method to see the first two rows in the dataset; we also use transform (T) along with head to transform the rows into columns due to the size of the data (i.e., it has many columns). In step 4, we use the dtypes attribute of the dataframe to show the data types of all columns. Numeric data has int and float data types while character data has the object data type. We inspect the number of rows and columns using shape, which returns a tuple that displays the number of rows and columns respectively.

In step 5, we apply the concat method to append the two datasets. The method takes in the list of dataframes as an argument. The list is the only argument required because the default setting of the concat method is to append data. In step 6, we inspect the first few rows of the output and its shape.

There’s more...

Using the concat method in pandas, we can append multiple datasets beyond just two. All that is required is to include these datasets in the list, and then they will be appended. It is important to note that the datasets must have the same columns.