How do you choose the right data for AI and ML?

I am assuming the purpose of the AI/ML is to build a model for a prediction use case and the question is about how to select training data. An important consideration for me is to select data obtained when conditions were "similar" to that of the conditions under which I will use the model. For example, if I'm trying to model, say, credit card defaults for sub-prime borrowers today, I would not train the model on defaults for prime borrowers from ten years ago. No amount of cross-validation will help one get good predictions in actual usage if the model is trained on data that is generated under conditions fundamentally different from that conditions that the model is to be used in.

One thing that I've found very helpful is understanding the domain. Whether it is finance, biology, or agriculture, knowing the basics of the domain we are working in can make the process of finding the best data smoother. Domain knowledge can serve as a bridge between our goals and the attributes we need to identify in the potential data input.

Yes, to provide a solution one should understand the problem statement. Gain knowledge from existing solution if it’s a optimisation problem what’s already been tried, if it’s new use-case talk to domain expert or someone who understands problem area learn about factors that contribute, accordingly decide data points to collect or additional source that can add value.

One should think in terms of metrics which can be business and model based and then think about the data. Identifying the connection between business metrics and model metrics is vital to achieving alignment with organizational goals. For instance, if a company aims to increase customer retention by 10%, this overarching business metric might translate into an ML model metric like improving the prediction accuracy of customer churn by a certain percentage. In this scenario, the ML model must focus on identifying the key features that can predict customer churn, such as engagement patterns or purchase history.

Having the right data for a machine learning task is critical to achieving optimal results. In my profession, I specialize in natural language processing and collaborate intimately with annotators to create the ideal dataset for model training. The significance of working with annotators cannot be stressed enough, especially when it comes to furnishing them with comprehensive guidelines to obtain the desired dataset.

Data Sources - 1. Solid AI foundation begins with diverse data sources: internal databases, APIs, external datasets. 2. Don't overlook alternative avenues like parsing services for extracting data from PDFs and images. This approach enriches AI models, granting them a competitive edge in real-world applications. Data Quantity - Beyond reliable sources, data quantity is key for AI accuracy 1. Linear models thrive with around 100 data points per feature—a sturdy foundation. 2. Complex domains like feedforward neural networks require datasets with at least 10 times parameters (weights and biases) for robust learning. Class balance matters - 1. In classification, balanced class representation is vital—aim for a 2:1 ratio.

What is your minimum viable dataset ? Collecting, cleaning and refining data is the most time consuming part of the process, so make sure you invest wisely. A quick win approach is to start with a simple naive model leveraging minimum data to make a decent prediction. From there you will have a performance baseline and a functional demo that you can communicate early with your business stakeholders and then negotiate additional time and ressources top enrich the dataset. You will have a better understanding of the incremental value of additional data. This will establish your credibility and multiply your impact.

In a world of rapid data generation, validating data sanctity and continuity (fill rate) is crucial. External data partners may promise great features but often fall short when data is unavailable for the complete population. Even if features display strong predictive strength, low fill rates make them unsuitable for production scenarios. It is highly advisable to verify the fill rate provided by external data partners before considering their data for use.

The data sources should be investigated and properly documented, so we shall know where the data is coming from, how reliable the data is (e.g. accuracy of measurement), frequency of obtaining the data vs. frequency of predictions we make. The data shall represent the whole population i.e. training data shall cover the scenarios of inference we will have in production. Same applies for the test data, both on evaluation and monitoring phases. Specific KPIs shall be set for the data consistency and size, so we can control the inflow and determine if the data pipeline failed. Implement data KPIs on data quality, monitoring if the distribution is within the parameters, or input data changed so we proactively will see the model will degrade.

By understanding your needs. Talk to Business, understand what they really want to achieve. Reach out to SMEs with those needs, and concisely discuss the goals. Most of the time, they will guide you. If an SME doesn't exist, look for research in the similar field and check their data source. Study the data to see if that would serve the purpose.

Data cleaning and preprocessing is the second step after getting the right data. And this is arguably the most important step. I have a lot of experience with time series data. Such data have a lot of noise, outliers, and inconsistencies. And while cleaning, it is very easy to make data leaks that will destroy your model in production. Make sure to do the following steps: 1. Remove outliers. In time series setting, the median filter works well. 2. Interpolate or fill the data gaps where possible. Interpolation is usually nice, but make sure you avoid data leaks. 3. Remove inconsistencies in data, for instance, the places, where two data signals contradict each other. Such data can make your model perform badly.

Following advance feature engineering steps can help enhance model's accuracy - 1. Interaction features - Create new features by combining existing features. For example - sales_value = price * quantity 2. Polynomial features - These features help capture non-linear relationships better 3. Time-based Features 4. Text Features - Convert text data into numerical features using techniques like TF-IDF

Paying attention to join columns and types is vital when prepping your data. Incorrect joins can lead to hard-to-trace errors, skewing your analytics. Pre-validate your keys for the intended type of join (inner, left, right, or full) to maintain data integrity and optimize performance. In big data scenarios like Apache Spark, efficiency is paramount. Modularize your data filtering steps and perform them before joins. This minimizes data shuffling across nodes, speeding up queries and reducing memory use. Early filtering ensures your joins are not just accurate, but also computationally efficient.

Even though the data is right, if it is not preprocessed right, it will not give any reliable results. There are no shortcuts. Basic things to check are missing data, messy data like strings in an integer column, negative age (yes, it happens), and outliers. Especially in labguage models, the type of preprocessing can depend on what you will do next. Make sure that you know your data well, and familiarize yourself with the assumptions/pitfalls of the technique you are using, very similar to checking for normality and transforming if the data is not normal in an ANOVA test.

It is important to understand how outliers affect the model. Be sure to treat them correctly. One can extrapolate missing values smartly towards enrichment. Feature engineering to understand the right application of data variables is also essential. When using sensitive information, it is usually recommended to transform the data through methods like encryption. Finally, audit the data (could be simple univariate/bivariate) to understand it. This not only helps understand business metrics and data fundamentally, but also provides direction on data wrangling. Data cleaning and pre processing is definitely a key step in any model building process and the foundation to your business goals.

When splitting the data for train, validation and test, we shall ensure that all sets represent the population. That the distributions are the same etc. The data split should avoid any data leakage, especially when dealing with time series. Important to maintain balanced data, as with highly imbalanced data the results could be good at first glance, like accuracy 0.95, but in reality it will mean that the rare class(es) is not predicted at all. So important to check other metrics as well. And address the imbalance with over/undersampling, like SMOTE (of course training set only). Some models like XGboost can do it on their own, both with class weights and individual row weights. The exact technique to use depends on the data and problem.

Split Your Data: Dividing your data into distinct subsets is crucial to ensuring your model's effectiveness. Start by creating three main subsets: training, validation, and testing. The training set—comprising the largest portion—nourishes your model's learning process. The validation set helps fine-tune your model's hyperparameters and gauge its performance, while the testing set evaluates the final model's real-world predictive power. If you solve a classification problem, make sure you classes are balanced between the data splits.

It is better to name folds as training validation and holdout. As in different places name "test" might be used as validation or holdout. It is also important to represent with splits your business logic, when you can use random split, time based split, group based split, or more complicated cases

Normally, a train_test split is used, which enables the data to be split into a user-defined ratio for training and testing. In cases of class imbalance in classification tasks, usually SMOTE or an equivalent is used to change this. It is important to correctly utilize SMOTE in the scripts, as it should only be implemented on the training data, to not corrupt the test data. Another alternative to issues with class imbalance in classification could be to artificially create cases for training using GAN. An excellent book worth reading for this scenario is Jannes Klaas, "Machine Learning for finance" 2019, as it provides code examples and discussions on use cases.

The great tools for visualization are matplotlib and Plotly. If you have high dimensional data, reduce the dimensions using PCA and see if there are any clusters in data. If so, it will help a lot to understand your data. Matplolib is often sufficient, but if you work on time series data, plotly can be useful to go through the plot by just scrolling the plot back and forth. Other useful plots are heatmap, seaborn kde plot, seaborn regplot, matplolib scatter plots. These will alow you to understand correlations within your data.

In many ways, exploring and visualizing your data is more important than the actual machine learning algorithms we deploy daily. It is also a critical step that I have seen skipped (or missed) several times in favor of a plug-and-chug modeling approach to data science. Understanding the data's distribution is critical for choosing the correct modeling approach. Regularly, I suggest simple plots that are easy to understand and communicate to a non-technical audience: density plots, scatter plots, histograms, bar charts. Further, descriptive statistics help guide our deeper analysis. Combined with visualizations, they help us identify relationships and outliers that can drastically affect either our data or our models. They are critical.

This step is usually considered as a part of data evaluation and definitely before data split, as you may find some behaviour you need to be aware of.

Visualisation is a powerful outcome. It helps not only technical people but also less savvy business owners understand outcomes and decide on next steps. It also enables continuous monitoring of data patterns and model results in a very automated and effective manner.

Model monitoring and retraining is an essential part. But in the beginning, you do not need to spend a lot of time on that. Just retraining it periodically will do the job. You can define a threshold within a scheduler, say, once a week, and retrain your model. Later, in the advance setting, when all other important things are solved, implementing data drift detection and automatic model retraining is very useful. Tools like evidently, mlflow and prefect/mage for orchestration will help you out.

A strong SoP for your model- frequency of retraining, process of continuous data enrichment, stable technical infrastructure and metrics performance monitoring will surely ensure quality models built for the long term. As legislation evolves, the data science community will have to learn and incorporate the same.

Another key thing to consider, which is easily brushed under the carpet in the name of speed/agility is to vet your model for biases. If it's a language model, consider replacing the names in your test data with those of another geopolitical region. What if you swap all the pronouns out? You want to be aware of any systemic biases your model has, and ensure that you're not going to be exacerbating harm when using it. If you have a customer service bot, you are going to have a vested interest in ensuring it treats a potential new client the same, regardless of their name being Janice, Sadiq, Mark or Mei Mei. Tools like nlptest by JSL or evaluation sets like BIGBench from Google are useful starting points.

One of the most important and hard part of AI ML is feature engineering step, it takes a lot of effort to be created. Also big deal of today's AI is how fair and biased the data is. With regulations upcoming it is also important to be able to answer about transparency and license of the data used.

In starting out in machine learning, there is always an urge to tryout more and more fancy and complex tools to show off the skillset of the craftsman. Bigger is always better, right? Not in this scenario. Usually the more compact and area-focussed tools provide better results, when compared to complex setups like for an example, GANs and ANNs in general. The main reason lies in the build and train time vs the output produced (also to be considered, the re-training needed to main a complex model). Usually, in business data insights and the analysis level is based on a 80/20 ratio (or another similar ratio), which means that to squeeze out the last 20% of knowledge or the last 20% of performance, normally isn't worth the investment.