4. Data Manipulation with Pandas

4. Data Manipulation with Pandas

The Pandas library is one of the most powerful and widely used tools in the Python universe for data manipulation and analysis. The name "Pandas" is derived from "Panel Data", an economic term for data sets that include observations over time for the same individuals. Developed by Wes McKinney, the library was designed to facilitate working with "relational" or "labeled" data, which is fundamental in the Machine Learning and Deep Learning process, as it allows efficient manipulation of large data sets, cleaning, their transformation and analysis.

Introduction to Pandas Objects

Pandas has two main data structures: Series and DataFrames. A Series is a one-dimensional array capable of storing any type of data with axis labels, known as indices. A DataFrame is a two-dimensional structure, a kind of table, which is essentially a collection of Series with a common index.

Installation and Import

Before you start working with Pandas, you need to install it using the pip package manager:

pip install pandas

After installation, you can import Pandas usually with the alias pd:

import pandas as pd

Data Loading

One of the first tasks when working with Pandas is to load data for analysis. Pandas supports reading a variety of file formats, including CSV, Excel, JSON, HTML, and SQL. For example, to load a CSV file, we use the read_csv:

df = pd.read_csv('path_to_your_file.csv')

Data Exploration

Once the data is loaded into a DataFrame, we can start exploring it using methods such as head(), which displays the first rows of the DataFrame, and tail(), which shows the last few lines. Methods such as describe() provide a statistical summary of numeric columns.

Selection and Indexing

Selecting and indexing data is crucial for data manipulation. Pandas offers several ways to select a subset of data from a DataFrame. We can select specific columns using square bracket notation:

specific_series = df['column_name']

To select lines, we can use the loc method for label-based selections, or iloc for integer position-based selections.

Data Cleansing

Cleansing data is an important part of the Machine Learning preparation process. With Pandas, we can handle missing values ​​using methods like dropna(), which removes rows or columns with missing values, and fillna(), which fills those values ​​with a specified value. Additionally, we can remove duplicates with drop_duplicates().

Data Transformation

Transforming data is another common operation. We can add or remove columns, apply functions to entire rows or columns, and perform grouping operations. The apply() method is particularly useful for applying a function to a column:

df['nova_coluna'] = df['existing_coluna'].apply(uma_funcao)

For grouping operations, the groupby() method is essential, allowing you to group data and apply aggregation functions such as sum(), mean()< /code> and count().

df['column'].plot(kind='hist')