Pandas is a powerful and versatile library in Python used for data manipulation and analysis. It provides data structures and functions designed to make working with structured data effortless. In this comprehensive guide, we'll explore the various functionalities of Pandas through practical examples and demonstrations.
Section 1:
Getting Started with Pandas Pandas is an open-source library that can be installed using Python's package manager pip. To install Pandas, simply execute the following command:
pip install pandas
Once installed, Pandas can be imported into Python scripts or notebooks using the import statement:
import pandas as PD
Section 2:
Loading Data One of the fundamental tasks in data analysis is loading data from various sources. Pandas provides functions to read data from different file formats such as CSV, Excel, and HTML.
1. Reading CSV Files
CSV (Comma-Separated Values) files are widely used for storing tabular data. Pandas provides the read_csv() function to load data from CSV files into a DataFrame.Example : use services.csv file.
df = pd.read_csv("services.csv")
The read_csv() function automatically detects the delimiter and header of the CSV file and loads the data into a DataFrame.
2. Reading Excel Files
Excel files can also be loaded into Pandas using the read_excel() function. This function accepts the path to the Excel file as input and returns a DataFrame containing the data.
df1 = pd.read_excel("LUSID Excel - Setting up your market data.xlsx")
The read_excel() function allows you to specify the sheet name or index if the Excel file contains multiple sheets.
3. Reading Data from Web
Pandas can read data directly from web pages using the read_html() function. This function parses HTML tables on a web page and returns a list of DataFrames, each corresponding to a table on the page.
url_data = pd.read_html("https://www.basketball-reference.com/leagues/NBA_2015_totals.html")
df3 = url_data[0]
The read_html() function relies on external libraries such as lxml and BeautifulSoup to parse HTML, so make sure to install these dependencies before using it.
Section 3:
Exploring Data Once the data is loaded into Pandas DataFrames, the next step is to explore and understand its structure. Pandas provides several functions for inspecting the data, including head(), tail(), info(), and describe().
1. Viewing Data
The head() and tail() functions allow you to view the first few rows and last few rows of the DataFrame, respectively.
df.head()
df.tail()
These functions are useful for quickly inspecting the contents of the DataFrame and understanding its structure.
2. Understanding Data
The info() function provides a concise summary of the DataFrame, including the number of non-null values, data types, and memory usage.
df.info()
The describe() function generates descriptive statistics for numerical columns in the DataFrame, such as count, mean, standard deviation, minimum, and maximum.
df.describe()
These functions help in gaining insights into the structure and distribution of the data, which is essential for data analysis and modeling.
Section 4:
Data Manipulation Pandas provide powerful tools for manipulating and transforming data. This section covers common data manipulation tasks such as selecting columns, filtering data, creating new columns, and handling missing values.
1. Selecting Columns
Columns in a DataFrame can be selected using square brackets or dot notation.
df['status']
This selects the 'status' column from the DataFrame and returns it as a Series object.
df[['status']]
To select multiple columns, pass a list of column names within double square brackets.
2. Filtering Data
Data can be filtered based on specific conditions using logical operations.
df[df['Age'] > 18]
This filters the DataFrame to include only rows where the 'Age' column is greater than 18.
3. Creating New Columns
New columns can be created by performing operations on existing columns.
df['new_col'] = 0
This creates a new column named 'new_col' and initializes all values to 0.
df["new_col1"] = df['PassengerId'] + df['Pclass']
This creates a new column named 'new_col1' by adding the values of 'PassengerId' and 'Pclass' columns element-wise.
Section 5:
Indexing and Slicing Indexing and slicing are essential operations for accessing and manipulating data in Pandas DataFrames.
1. Indexing
Data can be accessed using index-based selection methods like iloc[] and label-based selection methods like loc[].
df.iloc[0:2, [0, 1, 2]]
This selects the first two rows and the first three columns of the DataFrame using integer-based indexing.
df.loc[0:2, ['PassengerId', 'Survived', 'Pclass']]
This selects the first three rows and the specified columns using label-based indexing.
2. Slicing
Pandas supports slicing operations to select subsets of data from DataFrames.
df[0::2]
This selects every other row starting from the first row.
df[['PassengerId', 'Survived', 'Pclass']][0:2]
This selects the specified columns and the first two rows.
Section 6:
Grouping and Aggregation Grouping and aggregation are common operations in data analysis for summarizing and analyzing data.
1. Grouping Data
Data can be grouped together based on specific criteria using groupby() function.
df.groupby('Sex').mean()
This groups the data by the 'Sex' column and calculates the mean of numerical columns for each group.
2. Aggregation
Aggregate functions like sum(), mean(), and count() can be applied to grouped data.
df.groupby('Sex').mean()['Age']
This calculates the mean age for each gender group.
Conclusion:
Pandas is a versatile library that simplifies the process of data analysis and manipulation in Python. In this comprehensive guide, we've covered the essential functionalities of Pandas, including data loading, exploration, manipulation, indexing, and aggregation. By mastering Pandas, data analysts and scientists can efficiently handle and analyze structured data, making it an invaluable tool in the field of data science. Whether you're working with CSV files, Excel sheets, or web data, Pandas provides the tools you need to extract insights and derive value from your data. With its intuitive interface and powerful capabilities, Pandas is a must-have library for anyone working with data in Python.
Next part we deep dive into more depth and also plot visualizations using pandas….Soon.
