Mathew K AnalyticsLesson 15 · Finance and Stock Market Analytics
Assessing Financial Data Quality: Techniques and Best Practices for Accurate Analysis
Financial data quality is essential for accurate stock market analysis. Real-world data can have errors, missing values, or inconsistencies. You will learn…
- CourseFinance and Stock Market Analytics
- Lesson15 of 16
- Video23 min
- FormatJupyter notebook · 21 code cells
What you'll learn
Data
No separate download needed — the notebook creates or downloads everything it uses.
📓 Full notebook
Download .ipynbAssessing Financial Data Quality#
- Financial data quality is essential for accurate stock market analysis.
- Real-world data can have errors, missing values, or inconsistencies.
- You will learn how to inspect, validate, and clean stock data from real sources.
- The goal is to ensure reliable analytics and sound financial decisions.
- By the end, you will be able to spot and fix major data quality issues in stock datasets.
import pandas as pd
import numpy as np
import yfinance as yf
import warnings
warnings.filterwarnings('ignore')
Understanding Our Data Sources#
- We use real daily stock price data for major tech companies from yfinance.
- Each row contains stock OHLCV: Open, High, Low, Close, and Volume.
- Data comes in a 'long' format: each company and date is a row.
- It is easy to confuse reshapingdo not flatten or pivot unless necessary.
- Common mistake: Overwriting index or losing original tickers.
tickers = ['AAPL','MSFT','GOOGL','AMZN','TSLA']
ohlcv = yf.download(tickers, period='1y', auto_adjust=True, progress=False)
ohlcv = ohlcv.stack(future_stack=True).rename_axis(['Date','Ticker']).reset_index()
ohlcv.columns.name = None
print(ohlcv.shape)
print(ohlcv.head(3))
print(ohlcv.columns.tolist())
print('Unique tickers:', ohlcv['Ticker'].unique())
print('Date range:', ohlcv['Date'].min(), 'to', ohlcv['Date'].max())
missing = ohlcv.isna().sum()
print('Missing values by column:')
print(missing[missing>0])
nan_rows = ohlcv[ohlcv.isna().any(axis=1)]
print('Rows with any missing data:')
print(nan_rows.head(2))
ohlcv['Date'] = pd.to_datetime(ohlcv['Date'])
invalid_dates = ohlcv[ohlcv['Date'].isna()]
print('Rows with invalid Date:', invalid_dates.shape[0])
dups = ohlcv.duplicated(subset=['Date','Ticker'])
print('Number of duplicate rows:', dups.sum())
ohlcv_clean = ohlcv.drop_duplicates(subset=['Date','Ticker'])
print('Rows after removing duplicates:', ohlcv_clean.shape[0])
descriptive = ohlcv_clean.describe(include='all')
print(descriptive)
zero_vol = ohlcv_clean[ohlcv_clean['Volume'] == 0]
print('Rows with zero trading volume:', zero_vol.shape[0])
print('Any negative close prices:', (ohlcv_clean['Close'] < 0).any())
print('Any negative volume:', (ohlcv_clean['Volume'] < 0).any())
strange = ohlcv_clean[(ohlcv_clean['High'] < ohlcv_clean['Low']) | (ohlcv_clean['Open'] < 0) | (ohlcv_clean['Low'] < 0)]
print('Rows with bad OHLC logic:', strange.shape[0])
print('Inspecting a suspicious row for business understanding:')
if strange.shape[0]>0:
print(strange.head(1))
Intermediate Data Validation Checks#
- Are all business holidays and weekends excluded from trading data?
- Does every ticker have consistent records for each valid date?
- Is there a pattern in missingnesssuch as for new IPOs or technical outages?
pivot = ohlcv_clean.pivot(index='Date', columns='Ticker', values='Close')
date_coverage = pivot.notnull().sum(axis=1)
print('Number of tickers reported per day (any missing):')
print(date_coverage.value_counts())
piv = ohlcv_clean.pivot(index='Date', columns='Ticker', values='Close')
weekends = piv.index[piv.index.weekday >= 5]
print('Weekend entries:', len(weekends))
missing_per_ticker = piv.isna().sum()
print('Missing days per ticker:')
print(missing_per_ticker)
Quick Debugging for Data Cleaning#
- Check total row count before and after cleaning.
- Always summarize full and missing data per variable.
- Save cleaned data to file and visually inspect if needed.
ohlcv_clean.to_csv('ohlcv_clean.csv', index=False)
print('Cleaned data saved to ohlcv_clean.csv')
Best Practices for Financial Data Quality#
- Validate every field against business logic and expected range.
- Track all changes madeduplicates removed, rows dropped, or values replaced.
- Never delete raw data without keeping a backup.
- Document all assumptions and cleaning rules applied.
def clean_ohlcv(df):
df = df.copy()
df = df[df['Volume'] > 0]
df = df[df['Close'] > 0]
df = df[df['High'] >= df['Low']]
df = df.drop_duplicates(subset=['Date','Ticker'])
return df
ohlcv_final = clean_ohlcv(ohlcv)
print('Final cleaned rows:', ohlcv_final.shape[0])
np.random.seed(42) # reproducibility
random_sample = ohlcv_final.sample(n=5)
print('Random sample of high-quality records:')
print(random_sample)
End-to-End Example: From Raw Data to Quality Analytics#
- Raw OHLCV data is assessed, cleaned, and ready for analytics.
- Analysts can now trust summary statistics, trends, and investment models.
- Try applying these quality checks on any new stock data.
# Example: Compute average closing price per ticker after cleaning
avg_close = ohlcv_final.groupby('Ticker')['Close'].mean().round(2)
print('Average Close Price per Ticker:')
print(avg_close)
Continue Your Journey#
- Want to dive deeper into real-world finance analytics?
- Watch more on our channel and subscribe for hands-on Python finance tutorials!
Found this useful?
All lessons, notebooks and datasets here are free. If they helped you, a coffee keeps new lessons coming.
Related lessons
14 minLesson 18
Handling Missing Data in Time Series: Proven Techniques for Accurate Analysis
23 minLesson 15
Assessing Data Quality in Market Research
27 minLesson 15
Assessing Retail Data Quality in Python for E-commerce Analytics | Step-by-Step Guide
29 minLesson 59