Fluctuations in demand, especially for perishable products, is often a major budget leak, because it is difficult to predict demand accurately, timely, and regularly. Some retailers report up to 40% stock unsold, resulting in high values of forecasting accuracy errors.
The implementation of machine learning for predicting demand data addresses the problem with accuracy of forecasts, frequency of reporting, and general manual burden. With that, company heads get more visibility on their sales and supplies, answering the recurring questions more swiftly:
- How often inventory needs to be replenished?
- What are the required stock levels for each SKU?
- How to calculate demand for different selling points?
- How to manage inventory when anomalies occur?
In this article, we’ll share our experience with developing machine learning analytics modules for demand and sales forecasting, and elaborate on frequent concern for what those systems are capable of, and what you’ll need to develop one for your specific case.
Inventory Demand Forecasting: Preconditions for Using Machine Learning
The profit a business can generate always depends on how accurately you understand the current demand. However, demand forecasting and planning might become complicated when you operate more than one selling point, there are different items in the stock, and some of them don’t have sales data at all.
To calculate the demand for a single venue, we have to take into account multiple factors such as:
- Seasonality – past changes in demand which correlate with season changes or different time periods, f.e. day of the week, or other periods of time .
- Geography – the difference in demand between regions of sales.
- Marketing – or the impact of your marketing campaigns on sales.
- Competition – the impact of competitor actions, like price changes, ongoing marketing campaigns on your own demand.
- Global economy – such as the response to overall economic events and the buyers ability, inflation, etc.
- Product types – or the correlation between all the aforementioned factors with each separate product type.
If you perform demand forecasting in some manual form with the help of spreadsheets, you might already have a set of data that you use for calculations. So at this point, it already makes sense to think about automating the whole process if you have a number of preconditions:
1. Demand data availability
Sales data is the most available piece of information that allows us to start working with forecasting models. But, demand forecasting requires more than just historical sales. And if you already set up some processes and infrastructure to collect data on your internal and external factors, it can be used for machine learning easier than collecting it from scratch. A three-month period of historical data for one item is enough to kick start forecasting with ML. And the more data you’ll have, the better.
Additionally, if you already have historical data for multiple years from the past, it will allow you to start developing an ML solution and already bring value. Since predicting inventory level implies we need your specific sales, which can’t be sourced elsewhere or bought on a dataset market.
2. Data usability
For each product, the seasonality cycle plays a crucial role in predicting demand. Because, for instance, if you sell tents for hiking, its seasonal growth in demand appears to be during the summer period, with peaks in a certain month in your area. If we take the data for only 5 months for training the ML model, the prediction of the machine learning model won’t give adequate results, since we need a year’s data as a bare minimum, to calculate the seasonality.
For products that don’t fluctuate in demand seasonally, say forks and spoons, a three-month data will be enough to start training the model and producing forecasts. But it always depends on the item itself and other external factors like competition, or amount of holidays for a given period.
3. Prediction frequency
Different business types will require different frequency of forecasting. Conducting analysis on a monthly or even weekly basis can be a challenging task in terms of manual forecasting.
In this case, it’s a no-brainer that using machine learning frees up the resources, because it suggests an automated pipeline that gathers data and provides demand forecasts for given periods. One thing to keep in mind is that, the shorter the forecasting period, the less accurate it will be, since it’s impossible to generate enough data to cover all the required time-changes.
4. Multiple venue demand forecasting
For entrepreneurs that own multiple selling points that are set across different regions, forecasting demand will require a lot of human resources. Since seasonality, geography, and competition will all be different.
Setting up a system that sources data scattered across multiple databases, and presents analytics through a single dashboard, reduces the cost of demand forecasting itself. In its turn, a unified analytical solution presents a 360 view on your venues, inventory and sales activities allowing more flexibility in terms of planning and inventory management.
Based on our experience building a demand forecasting module for our client, SmartTab, we want to point out that the use of ML for demand forecasting is a large competitive advantage. Since your business obtains a better visibility and frees the human resources for development activities, rather than focusing on operational things.
From a technical standpoint, creating forecasts for multiple venues in most of the cases doesn’t require developing separate models for each specific case. The same goes for forecasting each separate product because we only need a suitable set of data that we can aggregate and properly feed to the model.
4 Key Factors Affecting ML Demand Forecasting
Now, speaking of the actual ML models that perform forecasting, we need to discuss what external and internal factors can impact the work of the model. Since the ML model will derive its prediction from past events, its prediction accuracy rate is unlikely to be 100%. Although, by understanding the following things we can mitigate those risks on the stage of developing the actual model.
1. PRODUCT TYPES AND MODELING ERRORS
The product type is an important factor to consider for the demand model. For example, for a perishable item that has an actual demand of 100 cases, the prediction of selling 90 cases is preferred over the prediction of 110 cases. Missing the sales of 10 cases is a better result than wasting 10 cases, even though the actual error is the same percentage.
2. REGIONAL IMPACTS ON MODEL PERFORMANCE
Predictive models are strongly influenced by regional factors that include customer behavior and cultural determinants. They also include the following:
- Marketing campaigns may be regionally specific and have a different impact that depends on where a customer is located.
- Holidays may vary between regions, which might be a consideration for adjusting the model.
- Legal issues/laws may limit the use of certain data in different regions.
3. NEW COMPETITORS ON THE MARKET
Demand forecasting is a dynamic concept. The more competitors and product alternatives are present in the market, the harder demand forecasting becomes. The competition level contains sub-factors, such as the number of alternative products and competitors.
So, it is a very good idea to add this information dynamically to your demand forecasting model.
4. ECONOMIC SITUATION
The state of the economy influences businesses and demand forecasting models. To put it more bluntly: periods of economic decline are likely to cause lower demand for expensive products, though sales of low-priced goods may go up. Therefore, an economic situation as well as trends aren’t external factors and should be considered when building AI models.
Sales Forecasting vs Demand Forecasting
Sales and demand forecasting are closely related terms, but use different sets of data. While sales forecasting requires only historical data generated through actual transactions, demand forecasting may also include weather reports, customer surveys, web analytics, social media scraping, etc. Which means, sales data is more available and easier to collect.
Businesses-wise, it can make sense to start gradually with the development of a sales forecasting system, if there is not enough qualified demand data. Although, for both types of forecasting systems, it’s essential to understand that they are vulnerable to anomalies or unpredictable situations. It means that machine learning models should be upgraded according to current reality.
As the demand forecasting model processes historical data, it can’t know that the demand has radically changed. For example, if last year, we had one demand indicator for a certain type of consumer product, it can change next year due to economic turbulence, or supplier countries being cut off from exports.
In that case, there might be several ways to get an accurate forecast. Here are the five most common ways:
- Collect data about new market behavior. Once the situation becomes more or less stable, develop a demand forecasting model from scratch.
- Apply a feature engineering approach. By processing external data, news, a current market state, price index, exchange rates, and other economic factors, machine learning models are capable of making more up-to-date forecasts.
- Upload the most recent data and provide it with the highest weights during model prediction. The period of a loadable dataset might vary from one to two months, depending on the products’ category. In this way, we can detect shifts in demand patterns and enhance forecast accuracy in a timely manner.
- Apply the transfer learning approach. If there is any gathered historical data, we can use it to predict demand in the context of the current crisis.
- Apply the information cascade modeling approach. We can forecast how people will make buying decisions according to the behavior patterns of most people.
During AI app development, AI engineers analyze historical data for forecasting. This forecasting cannot predict the disruption caused by a global pandemic, a war, a cataclysm, for example. Such an event requires the future recalibration of the machine learning models.
But keep in mind that after the demand situation normalizes after the pandemic/war/ etc – you need to adjust your model back, since in other cases – the model can remember the pandemic’s pattern and predict it for the next short-time period (e.g. next year).
Our machine learning engineers will be happy to work on your forecasting project. Tell us about your business needs and we’ll find the best solution to meet them.
How to Start Forecasting Demand: Bare Minimum Required
Now, let’s discuss the practical steps to implementing machine learning for demand or sales forecasting. The time frame needed to analyze the data and start the work on the project is usually set between 1.5-2 months for creating a POC. At this stage, the results of the POC of your demand forecasting model can already be used in operational activities while the development is proceeding.
If you are interested in the general flow of machine learning and AI projects, you can read our dedicated material where we explain our approach to requirement analysis and stages of development.
Read also:
AI Application Development GuideBut for now, we’ll focus on the demand forecasting specifics. So, before embarking on demand forecasting model development, you should understand the workflow of ML modeling. This offers a data-driven roadmap of how to optimize cooperation with software developers.
STEP 1. BRIEF DATA REVIEW
The first step when initiating the demand forecasting project is to provide the client with meaningful insights. The process includes the following steps:
- Gather available data
- Briefly review the data structure, accuracy, and consistency
- Run a few data tests and pilots
- Look through a statistical summary
In our experience, a few days is enough to understand the current situation and outline possible solutions. This is a part of data science exploration that comes with an involvement of the client to gather the data and provide the most relevant corporate storage.
STEP 2. SETTING BUSINESS GOALS AND SUCCESS METRICS
Before coming to the stage of developing a demand forecasting solution, a software development team needs to agree with the client/business owner on the success metrics for the model’s results evaluation. Success metrics offer a clear definition of what is “valuable” within demand forecasting. A typical message might state:
“I need a machine learning solution that predicts demand for […] products, for the next [week/month/a half-a-year/year], with […]% accuracy.”
This statement example will help you to identify what your success metrics will look like. You are expected to consider the following information:
- Product Types / Categories
What types of products/product categories will you forecast?
Different products/services should be considered and predicted independently for most cases. For example, the demand forecast for perishable products and subscription services coming at the same time each month will likely be different.
- Time Frame
What is the length of time for the demand forecast?
Short-term forecasts are commonly done for less than 12 months – 1 week/1 month/6 months.
These forecasts may have the following purposes:
- Uninterrupted supply of products/services
- Sales target setting and evaluating sales performance
- Optimization of prices according to market fluctuations and inflation
Long-term forecasts are completed for periods longer than a year. The main purposes of long-term forecasts may include the following:
- Long-term financial planning and funds acquisition
- Decision-making regarding the expansion of business
- Annual strategic planning
- Accuracy
What is the minimum expected percentage of demand forecast accuracy for making informed decisions?
Implementing retail software development projects, we were able to reach an average accuracy level of 95.96% for positions with enough data. The minimum required forecast accuracy level is set depending on your business goals.
Examples of metrics to measure the forecast accuracy are MAPE (Mean Absolute Percentage Error), MAE (Mean Absolute Error), or custom metrics.
STEP 3. DATA UNDERSTANDING & PREPARATION
Regardless of what we’d like to predict, data quality is a critical component of an accurate demand forecast. The following data could be used for building forecasting models:
When building a forecasting model, the data is evaluated according to the following parameters:
- Consistency
- Accuracy
- Validity
- Relevance
- Accessibility
- Completeness
- Detalization
In reality, the data collected by companies often isn’t ideal. It usually needs to be cleaned, analyzed for gaps and anomalies, checked for relevance, and restored. That’s why data science consultants can be involved at this stage.
Data understanding is the next task once preparation and structuring are completed. It’s not modeling yet but an excellent way to understand data by visualization. Below you can see how we visualized the data understanding process:
This visualization demonstrates data decomposition, extracting trends, and seasonal or other factors from input data. It’s divided into several graphs:
- The 1st graph is an original timeline (time series visualization)_
- The 2nd, 3rd, and 4th graphs separately represent seasonality, trends, and noise for further analysis and forecasting
STEP 4. MACHINE LEARNING MODELS DEVELOPMENT
There are no “one-size-fits-all” forecasting algorithms. The choice of machine learning models depends on several factors, such as business goal, data type, data amount and quality, forecasting period, etc.
However, there are multiple approaches to that, and they are often combined under the hood of a single forecasting system. The following approaches are the used ones for demand forecasting and proved to be the most efficient:
- ARIMA/SARIMA
- Exponential Smoothing
- Regression models
- Gradient Boosting
- Long Short-Term Memory (LSTM)
- Ensemble Models
- Transformer-based Models
The choice of the approach will depend on the actual requirements. For the SmartTab project, we used a time series approach with a combination of Gradient Boosting and KNN models. The major components to analyze include trends, seasonality, irregularity, and cyclicity. With that, we can predict how much revenue will be earned within the next upcoming year with the daily granularity for each venue and total venue chain.
Although, the topic of using ML in analytical systems is much wider and covers not only retail cases. So if you want a much deeper insight into various models and approaches, read our dedicated article.
STEP 5. Productionalization
Once we have the data, we can start the training, validation, and improvement procedures. This process is done iteratively until the model achieves the maximum desired results.
A forecasting model can be used in its raw form, presenting its predictions in some sort of table, or sending mails with analytics for a certain period. However, more commonly we would also develop a front-end part of the application. Which is basically a dashboard that presents insights and visualizations through a single interface, and allows the user to query different reports, share them with the stakeholders, customize visualization types, etc. After testing and approving, the product is deployed into the production environment.
One thing to mention is that to keep the forecasting capabilities up to date, the model will require constant updates with new data. These can be your daily transactions, or inventory turnover. For this purpose, we recommend setting up an automated pipeline that on a regular basis aggregates recent data and updates the ML model with it accordingly. It will help to keep reliability of the developed demand forecasting model on a regular basis.
Leverage ML Demand Forecasting for Your Business
Once you make a decision to run a project on demand forecasting development, pay attention to proficiency of a hired team. Demand forecasting projects fall into the category of machine learning and data science, which requires in-depth domain expertise for processing data and choosing the right approach to solve your specific problem.
MobiDev provides an all-around service for developing demand forecasting systems of different scales and complexity, customization options, and integrations. We have multiple projects in our portfolio and years of experience in software development, and are ready to start the discussion concerning your project requirements.
Contact us below or book a call with a MobiDev expert to discuss your project needs.
