How to combine multiple dataframe rows together with different column sizes in python pandas

In this tutorial, we will look at how to combine multiple dataframe rows together with different column sizes using the Python Pandas library.

First, let’s start by creating two sample dataframes.

1
import pandas as pd
2

3
df1 = pd.DataFrame({'A': ['A0', 'A1', 'A2'],
4
                    'B': ['B0', 'B1', 'B2'],
5
                    'C': ['C0', 'C1', 'C2'],
6
                    'D': ['D0', 'D1', 'D2']})
7

8
df2 = pd.DataFrame({'A': ['A3', 'A4', 'A5', 'A6'],
9
                    'B': ['B3', 'B4', 'B5', 'B6'],
10
                    'C': ['C3', 'C4', 'C5', 'C6'],
11
                    'D': ['D3', 'D4', 'D5', 'D6'],
12
                    'E': ['E3', 'E4', 'E5', 'E6']})
13

14
print(df1)
15
print(df2)

This will print out the following dataframes:

1
   A   B   C   D
2
0  A0  B0  C0  D0
3
1  A1  B1  C1  D1
4
2  A2  B2  C2  D2
5

6
   A   B   C   D   E
7
0  A3  B3  C3  D3  E3
8
1  A4  B4  C4  D4  E4
9
2  A5  B5  C5  D5  E5
10
3  A6  B6  C6  D6  E6

Now, let’s say we want to combine these two dataframes together such that they are stacked on top of each other, with the rows of df2 coming after the rows of df1. In other words, the final dataframe should look like this:

1
   A   B   C   D   E
2
0  A0  B0  C0  D0  NaN
3
1  A1  B1  C1  D1  NaN
4
2  A2  B2  C2  D2  NaN
5
3  A3  B3  C3  D3  E3
6
4  A4  B4  C4  D4  E4
7
5  A5  B5  C5  D5  E5
8
6  A6  B6  C6  D6  E6

To do this, we can use the pd.concat() function. This function can be used to concatenate dataframes along either the rows or columns. We want to concatenate along the rows, so we set the axis parameter to 0.

1
df_combined = pd.concat([df1, df2], axis=0)
2
print(df_combined)

This will print out the combined dataframe as shown above. Notice that the resulting dataframe has NaN values for the missing values in the E column of df1.

Alternatively, we can also use the df1.append() method to achieve the same result. This method can be used to append rows of one dataframe to the end of another dataframe.

1
df_combined = df1.append(df2)
2
print(df_combined)

This will also print out the combined dataframe as shown above.

Note that both the pd.concat() and df.append() functions will align the dataframes based on their columns. If the dataframes have different column names, the resulting dataframe will have NaN values for the missing values.

If you want to specify which columns should be used to align the dataframes, you can use the join parameter of the pd.concat() function. For example:

1
df_combined = pd.concat([df1, df2], axis=0, join='inner')
2
print(df_combined)

This will only include the columns that are present in both df1 and df2, and will print out the following dataframe:

1
   A   B   C   D
2
0  A0  B0  C0  D0
3
1  A1  B1  C1  D1
4
2  A2  B2  C2  D2
5
3  A3  B3  C3  D3
6
4  A4  B4  C4  D4
7
5  A5  B5  C5  D5
8
6  A6  B6  C6  D6

I hope this tutorial helps! Let me know if you have any questions.