DEV Community: nuvo

allowNestedData, dataHandler, No-Code Data Pipeline Release

nuvo — Fri, 12 May 2023 08:52:54 +0000

The moment has arrived to reveal what our engineering team has been occupied with during the previous months. Let's delve into the March Product Updates!

We are excited to announce the latest updates and features. In this blog, we will cover our top five feature highlights of March:
Our data importer can now process nested JSON files
Our dataHandler feature brings a lot of new possibilities and allows you to handle even more complex data transformation scenarios than before
We reduced the loading time for option and column mapping by up to 90%
We extended the i18nOverrides functionality for boolean drop-downs and introduced new pre-written translations
And the big news: We released a new product, our automated no-code Data Pipelines

Let’s dive in!

Parsing nested JSON files (allowNestedData)

Multi-dimensional/grouped data also called nested data is something a lot of companies struggle with when importing and reformatting customer data. Breaking up these dimensions into a two-dimensional structure that can be displayed as a table takes up significant time and effort for engineering teams and is a key challenge brought to us by our clients.

Our new feature allowNestedData finally solves this issue by allowing to de-nest .json files based on a pre-defined set of rules. The de-nesting process involves replacing arrays with underscores ""_ and objects with periods "." to facilitate the display of data in a 2D table.

Check out our documentation for more information.

dataHandler for complex data transformation scenarios

Our new dataHandler feature allows to solve complex data manipulation scenarios, such as transposing data, merging and splitting columns, joining sheets, de-nesting data, and more. Unlike our other Cleaning Functions, that either iterate through every entry or are applied on only one column, the dataHandler function works on the entire data at once.

The function contains two parts: The headerStep (runs before the “Header Selection” step) and the reviewStep (runs before the “Review Entries” step) providing access to the original data and its metadata as well as enabling you to add/remove columns and rows. The dataHandler can be configured to run based on the your needs.
More details on this powerful tool as well as Code Sandboxes to test sample use cases can be found in our documentation.

Up to 90% loading time reduction for Columns and Category Matching‍

To improve our user experience during the “Match Columns” step, especially with target data models including category columns with a large amount of dropdown options, we updated our matching module and were able to reduce the matching time up to 90%.

Install the latest version and try it out with your own sample file, we'd love to hear your feedback!

Improvement of i18n functionality

For enabling you to use custom UI texts or support multiple languages, we provide you with an i18nOverrides option in the SettingsAPI. We now also added the i18n support for the “Yes” and “No” labels of the boolean dropdown fields via the keys “txt_boolean_yes” and "txt_boolean_no". In addition, we extended our list of pre-written translations to reduce your time and efforts for implementing our importer even more!

Find more information about this feature in our documentation.

Do you want to implement our data importer in a language that is not on the list yet? Feel free to reach out to us and we are happy to add it!

New Release – nuvo's no-code Data Pipelines

(drum roll, please) We are excited to announce the official release of our new no-Code Data Pipelines solution!

Automating the mapping, transformation and importing of customer and supplier data based on a pre-defined schedule or event was a huge pain point that a lot of our users told us about.

Providing a secure and scalable solution for this that enables anyone in the organization to setup data pipelines, manage, observe and fix them accordingly was something we wanted to solve for all our users.

After long months of development, we are happy to release our No-Code Data Pipeline Solution and are very proud to also be ranked as Top 3 Product of the Day on Product Hunt!

Our Data Pipeline Product allows to:

Connect to a dedicated input source and output source to define where the data should come from and be sent to
Build powerful data transformations using Excel-like formulas or JavaScript Code injection
Run, observe and fix your data pipelines using our simple and intuitive dashboard Intrigued to learn more? Check out our [Start Guide](https://docs.getnuvo.com/dp/start/! or get in touch with us for a short demo and test access!

If you want more details about any of the new features, don't hesitate to book a call with our team.

How To Choose the Best Data Ingestion Tool for Your Business

nuvo — Fri, 12 May 2023 08:17:33 +0000

Let’s take a quick run through everything to do with data ingestion, including how to choose between all the great data ingestion tools out there.

The key to making reliable data-driven decisions is maintaining a high-quality data intake. If your data is messy, scattered, or incomplete, the conclusions you draw from it will be, too. A carefully-planned data ingestion process, where data is properly validated, stored, and secured, will save you time and money and enable fast, smart decision-making.

Why is Data Ingestion so Challenging?

When receiving data from customers, partners, or suppliers, you naturally come across a wide variety of schemas. Customers will call headers differently, format entries in a different way, or simply store required information in multiple sources.

This makes manual data imports from a range of sources a time-consuming, tedious, and expensive task prone to human error. In our data-rich world, data ingestion tools are becoming increasingly popular. These tools automatically extract data from a range of sources, convert it into the format you need to work with, and transfer it all to the desired location. Data ingestion tools ensure that clean data flows from the original source into your data, reporting, and analytics system.

Why is Data Ingestion Important? And is it Relevant to You?

That might all sound good in theory but how can you know if data ingestion tools are really relevant for you in practice? Let’s walk through some of the key benefits to help you figure that out.

Improve data quality

Messy data is no good to anyone. The top data ingestion tools can automatically streamline and sort your data before it gets stored in your database.

More effective data management

As data ingestion tools clean your data on its way into your database, there are far fewer inaccuracies or duplications which leads to more efficient data use overall.

Speed

Data ingestion software allows you to automate your processes which means you can extract data from sources and ingest it into your data system quickly. As minimal human oversight is needed, and your team’s time can be better spent on other tasks.

Scalability

Data ingestion only really becomes a challenge when your organization scales up. If you’re working with a low number of customers, then manually ingesting data is quick and easy. However, as soon as your data volume increases, so does your workload. That’s where data ingestion technologies really show their value.

Cost-effectiveness

As data ingestion tools allow you to automate your processes, you save time and money you would otherwise spend on repetitive manual processes.

Easy to use

Data ingestion tools like nuvo are built to be used by anyone, not just those with existing tech knowledge. User-friendly interfaces with simple, intuitive functionalities are easy to master and don’t require long, boring training sessions. Win-win.

Faster customer onboarding

For any business, but particularly low-touch SaaS, it’s important to give the customer value as soon as you can. Integrating real-time data ingestion tools into your onboarding flow helps you process new data immediately which speeds up the customer journey and reduces churn.

Simplified data cleansing

When you work with data, you need it to be clean and easy to transform. Choosing the right data ingestion software automates data cleansing and simplifies data transformation processes meaning you can trust your data from day one.

Quicker data transformation

Real-time data ingestion removes the need for batch-processing. Using a data ingestion tool means that data can be cleansed, validated, filtered, enriched, and normalized as soon as it arrives.

Faster, smarter decisions

As data ingestion tools allow for immediate inbound data transformation, internal teams can make faster decisions and generate more leads.

Focus on your business

As inbound data can be transformed so fast, it allows teams to move away from validating and cleaning data back to tasks that directly impact the bottom-line. The ROI on data ingestion tools quickly becomes clear.

Factors to Evaluate when Setting up Your Data Ingestion Process

If you’re convinced and ready to choose a data ingestion tool, there are some key factors to consider when making your choice. You need to think about interface, format, security, interoperability, frequency, and user-friendliness.

Interface

Is your data ingestion process customer-facing, or will your in-house experts handle it? These approaches call for strongly differentiated user interfaces and allow a different level of complexity.

Format

What type of data do you have to ingest? Does your data have a somewhat repetitive structure and can be fully automated at some point, or do you have heavily different data schemas to ingest?

Security

Are you working with highly sensitive data? In that case, you clearly need to identify who can access your data and at what point.

Interoperability

How well does your data ingestion process play with others? Make sure the one you choose is compatible with all your data sources.

Frequency

Do you need real-time data ingestion or would you prefer to use a scheduled, or event-based approach? If real-time processing is key, look for software that performs that function.

User-friendliness

In most cases, it’s important to save engineering resources for more pressing tasks. By choosing a data ingestion tool that’s easy for non-technical team members to use, you’ll save everyone a lot of headaches and use your time and team more efficiently.

Steps of Ingesting Data

Let’s discover what are the steps of importing data into your desired destination and why these steps are so important.

Data Importing

The first step in a good data ingestion process is importing the data from various sources such as databases, files, or APIs. It's essential to ensure that the data is accurate, complete, and relevant to the business needs. This step involves identifying the source, connection, and format of the data, and importing it into the system.

Data Mapping

Once the data is imported, it's essential to map it to the target schema or format. The mapping process involves aligning the data fields from the source to the target schema. Mapping ensures that the data is consistent and in the right format for analysis or processing.

Data Validation and Cleaning

Data validation and cleaning are critical to the data ingestion process as they ensure the data is accurate, high quality, complete and trustworthy. This part involves removing any inconsistencies, errors, or duplicates in the data, and verifying it against predefined rules, constraints, or standards. This final step makes certain that the data is suitable for further analysis or processing.

And there you have it! We hope that you have now a much better understanding of the data ingestion process and can make an informed decision on the tool you need. If you’d like to learn more about how to create a seamless data ingestion experience with nuvo, contact us today and get a free demo.

How nuvo Importer SDK 2.0 Transforms the Way You Onboard Data

nuvo — Fri, 12 May 2023 08:10:30 +0000

Data import use cases vary across industries, from e-commerce to construction, HR, fintech, or energy. However, dealing with messy and ever-changing customer data is a common challenge for B2B software companies.

With this in mind, our AI-assisted nuvo Data Importer was built to provide our clients with a seamless, scalable, and secure data import experience covering even the most complex edge cases. And foremostly, it offers the highest data privacy and security standards on the market. During the past years, we have continuously advanced our solution based on new challenges, formats, and industry-specific use cases to provide our clients with the most powerful import solution.

Now, after months of research and development, we proudly announce the release of Importer SDK 2.0!

This version introduces many new powerful features that extend advanced data validation and cleaning capabilities, increase customizability, and speed up implementation.

Here are seven of the most notable new features:

Start the importer at any step, at any event, and with your preferred data format using the Dynamic Import feature
Transpose and de-nest data; merge and split columns; join sheets; and perform many other complex data manipulations with the dataHandler feature
Set up conditional dropdown fields with the Value-Based Dropdown feature
Parse nested JSON files with allowNestedData
Up to 95% loading time reduction during columns and option matching using our new architecture
Fully customize your importer with only a few lines of code using the Simplified Styling feature
Support multiple languages even faster with our default translations using our language property via the Simplified Translation feature Let’s dive into how these new features transform the way you onboard data. ‍

Start the Importer at Any Step, at Any Event, and with Your Preferred Data Format

Providing your users with the best possible data import experience inevitably comes with high flexibility requirements.

The Dynamic Import feature allows you to start the importer at any step, at any event, and with your preferred data format. This enables you to cover various use cases like allowing your users to import complex file structures such as .txt and zip files or to use nuvo not only as an importer but also as a data management UI, where users can edit their existing data.

Moreover, you can start the import by fetching data from any API instead of uploading a file manually. This is game-changing when you want to enable your users to migrate data from their CRM, ERP, PIM, or other applications to yours.

Sounds too good to be true? Check out the documentation or reach out to us to learn how the Dynamic Import feature can cover your use case.

Perform Complex Data Manipulations

Dealing with customer data can be a nightmare for onboarding and engineering teams due to its varying structure, size, and format. During the past years, we have seen many wild edge cases across industries. Hence, developing a feature that can deal with even the messiest input data became a high priority – dataHandler.

The dataHandler feature allows for solving complex data manipulation scenarios, such as transposing data, merging and splitting columns, joining sheets, de-nesting data, and more. Unlike our cleaning functions that iterate through every entry or access only one column at a time, the dataHandler functions (headerStep and reviewStep) work on the entire data at once.

This gives you complete control over the input data directly after the upload and gives you access to modify the data after the mapping step. In addition, it allows you to automatically add, delete, and modify columns and rows in the data, helping you to manage different input file structures and giving greater flexibility for data manipulation.

Whether you need to transform a few columns or an entire dataset, the dataHandler functions provide the flexibility and power required to do the job efficiently.

Go to our documentation for more details and some sample code sandboxes to test it out!

Conditional Rendering of Dropdowns

Dropdown or category fields are often used in data import and migration processes. But mapping and validating data against hundreds or even thousands of different dropdown options or categories can be challenging, frustrating, and highly error-prone for the user.

Additionally, when the dropdown fields depend on the values of certain columns or other conditions, complexity multiplies for both the user and the engineering team setting up an importer.

With our Value-Based Dropdown feature, you can now control which options are displayed in a dropdown column based on the value(s) of other columns in the same row. We achieve this by allowing you to link dropdown options with other columns by using specific operators such as AND, OR, GTE (greater than or equal to), LTE (less than or equal to), and others.

By applying these operators, you can define even complex conditions that determine whether or not to display a given dropdown option. Once the logic is defined, dropdown options are automatically updated based on the values in the linked columns in the “Review Entries” step.

Learn more about it in our documentation.

Parsing Nested JSON Files

Multi-dimensional or grouped data (also called nested data) is something a lot of companies struggle with when importing and reformatting customer data. Breaking up these dimensions into a two-dimensional structure that can be displayed as a table takes up significant time and effort for engineering teams and is a key challenge our clients brought us.

Our new feature allowNestedData solves this issue by allowing you to de-nest .json files based on pre-defined rules. The de-nesting process involves replacing arrays with underscores "_" and objects with periods "." to facilitate the display of data in a 2D table.

Find more information in our documentation.

Up to 95% Loading Time Reduction During Columns and Option Matching

You hopefully noticed a significant reduction in loading time during the “Match Columns” step already after the March release. For Importer SDK 2.0, we enhanced the matching module and mechanism further. With the SDK 2.0, we process the column headers on the backend side by default, which reduces the matching time up to 95% in comparison to our January version.

Additionally, we have added an optional functionality (processingEngine === “node”) to reduce the mapping time even further by processing also the uploaded spreadsheet content on the backend side. Please be aware that migrating to SDK 2.0 does not automatically apply this option. By default, SDK 2.0 won't process the spreadsheet content of your users.

Install the latest version and try it out with your own sample file. We'd love to hear your feedback!

Furthermore, check out the new optional SDK 2.0 functionality in our documentation to even further speed up the mapping process by allowing the option mapping on the backend side.

Simplified Styling and Simplified Translation

Speed and ease of implementation while maintaining flexibility for customization are key aspects of nuvo Data Importer. To speed up the implementation even further, we have significantly simplified the styling options, allowing you to fully customize and white-label the importer using only a handful of properties within the global class.

Additionally, we implemented a simple way to change the UI language, that ables you to implement multiple-language support significantly faster. You can apply nine different languages by only changing the language key within the settings. Of course, you can always override the default text or add additional languages by using our i18nOverrides functionality.

If your language still needs to be included inside the language property, please reach out to us, and we are happy to add it.

Check out our translation guide for more details.

Simplified styling with only a handful of properties
And that’s a wrap! We hope these new features help you to say goodbye to non-scalable import scripts and manually cleaning or reformatting customer files.

Do you want to hear more?

Join nuvo’s Co-Founders Michael Zittermann (CEO) and Ben Hartig (CTO) on Tuesday, May 16 at 10:30 am CEST for a 30-minute session on the new capabilities and find out how they help you to create the best possible data import experience.

Transform Your Data Import Process

nuvo — Thu, 13 Apr 2023 11:19:08 +0000

Importing data is one of the first meaningful interactions your customer will have with your product. Let’s explore how to make it as seamless as possible.

Data onboarding generally is the starting point for interactions between B2B software applications and their customers. It is the starting point of interaction between your application and customers. Studies show that 86% of customers are willing to pay a higher price for a better UX, which can increase customer satisfaction.
‍

External data often comes in files such as Excel, CSV, and JSON that lack a clear organization, making it challenging to work with. In fact, data cleaning and mapping can take up to 80% of a data scientist’s time, leaving little room for analysis and insight. This problem is not limited to small businesses; even large companies struggle with the data onboarding process due to the sheer volume of data. Furthermore, the high and recurring manual effort for internal onboarding and engineering teams can pose severe bottlenecks in onboarding new customers in time.

Data onboarding is challenging due to the lack of standardization in data formats, even within a single file type like CSV. This makes it difficult to create automated processes to clean and map data. Additionally, data quality issues such as incomplete, inconsistent, and inaccurate data can lead to flawed and faulty conclusions, affecting decision-making, particularly for those companies that rely on data for compliance purposes.

To address these challenges, many companies are trying to implement a scalable approach that allows automating the data cleaning and mapping processes using machine learning. nuvo Data Importer is one of the solutions that can help companies streamline their data onboarding processes. It offers a range of cleaning functions that help automate the data cleaning and mapping process by reducing the time and resources required for data onboarding.

The nuvo Data Importer’s ML algorithms and flexible cleaning functions can nearly cover every use case by transforming data and displaying customized messages to the user during the import process. Similarly, companies can reduce the time and resources required for data onboarding by almost 90% using the nuvo Data Importer.

What exactly are cleaning functions?

Specific import workflow events trigger cleaning functions and serve as callback functions. Users can receive feedback and automatically transform imported data to meet the required format using these functions. This enables them to achieve faster and higher-quality data submissions without requiring direct user interaction.

For example, cleaning functions can compare retrieved and imported data, identify duplicate entries, display errors, enrich entries, establish dependencies between columns, merge or split multiple columns, call third-party APIs for verification purposes, and many more tasks.

Review Entries Step

nuvo offers different types of cleaning functions to provide high flexibility in validating and automatically reformatting data. Now we will take a closer look at two functions: onEntryInit and columnHook.

onEntryInit

This function enables iteration through all imported entries/rows by providing access to every value within each iteration. Users can use this function to establish column-cross dependencies between two or more columns, such as merge, split, or value-based Regular Expressions (Regex) checks. In the example below, the validation of countries is contingent upon the continent selected.

     onEntryInit={(row, index) => {
     if (row.continent) {
        const matchFound = data.find((set)=> set.country === row.country);
         // check if selected country belongs to the continent
        if (matchFound?.continent !== row.continent) {
           return {
             country: {
               info: [
                 {
                   message: "Country does not belong to " + row.continent,
                   level: "warning"
                 }
               ]
             }
           };
        }
    }
 }}

CodeSandbox Link

‍

columnHook

All the data in a particular column can be accessed inside the columnHook function. This enables you to apply custom data transformation and provides feedback to the user within the UI in the form of custom info, warnings, and error messages. The example below shows how columnHooks can conduct server callbacks to validate the incoming data with the backend and check for duplicate entries. In the case of a duplicate, a custom error message is shown, allowing the user to edit the record prior to importing it.

 columnHooks={{
     email: async (values) => {
       let registeredEmails;
       await fetch(
         "https://my-json-server.typicode.com/comdocks/nuvo/customers"
       )
         .then((response) => response.json())
         .then((json) => {
           registeredEmails = json.map((row) => row.email);
         });

       const duplicateErrors = [];

       values.forEach((entry) => {
         //warn user if email already exists
         if (registeredEmails.includes(entry.value)) {
           duplicateErrors.push([
             {
               info: [
                 {
                   message:
                     "Duplicate entry. The email address already exists.",
                   level: "error"
                 }
               ]
             },
             entry.index
           ]);
         }
       });

       return duplicateErrors;
     }
 }}

CodeSandbox Link

‍

The onEntryInit and columnHook functions provide high flexibility in validating and automatically reformatting data, making data onboarding faster and more efficient. Companies can greatly benefit from using nuvo Data Importer, which enables better decision-making and increases customer satisfaction.

If you’re looking to streamline your data onboarding process and optimize your data-driven decision-making, try out nuvo Data Importer. With its powerful cleaning functions and machine learning algorithms, it can cover nearly every use case and significantly reduce the time and resources required for data onboarding.

‍
Want to see a live demo or receive a free trial? Book a call with the team for a quick demo.

‍
Source: UserTesting. (2019). The ROI of CX: Why you can’t afford to ignore customer experience. Retrieved from https://www.usertesting.com/blog/the-roi-of-cx-why-you-cant-afford-to-ignore-customer-experience/

Introducing nuvo No-Code Data Pipelines

nuvo — Thu, 30 Mar 2023 09:30:13 +0000

A revolutionary approach for automating external data onboarding by utilizing AI to automatically map, validate, clean, and import data

‍Since starting our Data Onboarding journey in 2020, we've been able to support countless clients in handling messy CSV files. While being able to minimize the pain of importing spreadsheet data, the desire for a fully automated solution was clearly in the room. We found that there is a need for processes that require manual input and fully automated solutions; however, the trend is definitely shifting to the latter.

‍

How it all started‍

We found FMCG wholesalers updating millions of rows of product data overnight, ESG software constantly ingesting client data for real-time analytics, or even car manufacturers regularly onboarding supplier data in various formats. Our Importer SDK released in the recent year led to massive improvements in mentioned use cases and saved hundreds of hours of development time. The non-technical UI and fast implementation time allowed engineering teams to create powerful import scripts while significantly shortening onboarding cycles for their client and in-house team.
‍
Still, whenever your software or internal processes require a constant stream of external data, the manual requirements of our Importer SDK unveiled some room for improvement—more and more feedback accumulated in the direction of a fully automated data import process. The vast opportunity, combined with our vision to transform data to make it universally understandable, led to our engineering team coming up with big news.

‍Our No-Code Data Pipelines‍

We're excited to announce the launch of our newest product addition, nuvo No-Code Data Pipelines, designed to help businesses quickly automate the onboarding of external data. At nuvo, we understand the pain points of importing data from external sources like Excel and CSV files. This process typically requires custom development and a lot of manual reformatting efforts. Our team recognized the need for a solution that simplifies connecting to external data sources, and we're proud to offer a product that addresses this problem.

With nuvo's No-Code Data Pipelines, you can quickly and easily build, launch, and run pipelines that connect to sources such as CSV files, (S)FTP, and HTTPS. Our no-code user interface empowers everyone in your organization to set up and automate data transformations and import external data on a predefined schedule.

So, how does it work?‍

Select Connectors: Choose your Input Source and link it to your desired Output Target.
AI Column Mapping: Our AI algorithm automatically maps input columns to your output schema.
Data Transformation: Use Excel-like formulas, or inject custom code to enable powerful transformations.
Schedule Pipeline: Determine when your pipeline should run.
Monitor Runs: Overview all executions and fix broken pipelines if necessary. ‍ At nuvo, we're committed to empowering businesses to work more efficiently by providing innovative solutions to their data challenges. Our product offers a simple and intuitive way to automate data onboarding without the need for extensive technical knowledge or custom development. Our No-Code Data Pipelines will help businesses streamline their data onboarding process and improve productivity.

We want to thank our existing clients and partners for supporting our journey to build a suite of data transformation solutions that empower companies to turn every piece of data into value.

Contact us today and start your 14-day free trial to automate your CSV data imports fully! We cannot wait to hear your feedback on the product and are happy to answer any questions!

All the best,
Michael, CEO of nuvo

Best Practices for Master Data Management (MDM)

nuvo — Tue, 28 Mar 2023 09:22:10 +0000

Master Data Management is crucial for maintaining high-quality data and making informed decisions. Let's delve into it.

Modern digital businesses generate huge amounts of data. The level of insight you can glean from that data depends largely on how you manage it. Good data management practices keep data quality high which enables you to conduct meaningful data analyses, make better decisions for your organization, and adhere to data standards and governance rules. Prioritizing how you manage your master data will save you time, effort, stress, and money in the long run. So let’s get into it.

‍

What is Master Data Management? ‍

Master data is the essential information you need to run your organization. It can include customers, accounts, products, suppliers, vendors, locations, and any non-transactional data your business needs to function. If, for example, a customer buys a particular product on a certain date, the customer and product info are classed as master data, whereas the date on which the customer bought it would be considered transactional data.

‍

As master data is at the core of everything your organization does, it’s important to observe some master data management (MDM) best practices. It’s your duty to properly manage and protect your data, and a huge database is of little to no use unless it’s structured in a way that enables meaningful analysis of high-quality data. As your master data is the most business-critical information you have, getting master data management right from the start is a key factor to business success.

‍

Types of Master Data‍

Understanding and managing data is all about storing it in the right context. There’s a lot of data that adds context to your business, but not all contextual data is master data. As we’ve already mentioned, master data refers to key factors that enable your business to run, like customer data, supplier locations, and product details. What you consider master data will depend on your specific business needs and goals, but the most common types of master data are:

People — this is data related to people or organisations like customers, suppliers, vendors, employees, or similar.
‍Products — this is data related to products or services your organization sells.
‍Financials — this is data related to your finances such as ledgers and sales records.
‍Locations — this is data related to stores, corporate offices, distribution centers, warehouses or similar. ‍

Why is Master Data Management so Important?

It’s incredibly useful to have a single source of truth that can be used across your organization. All teams should have one access point where they can find relevant, business-critical information like customer IDs, product lists, and distributor info.

‍

Master data management is important because it allows proper maintenance of this crucial data on a rolling basis. This means it’s easier to weed out inconsistencies or discrepancies which keeps your data quality high. Using this high-quality data, you can then make better data-driven decisions based on reliable figures, hypotheses, and extrapolations.

‍

What are the core functions of master data management? ‍

Master data management supports smart, smooth internal operations but how exactly does it do this?

‍

1. Data standard setting

Setting standards for your data is one of the most challenging aspects of master data management so it’s worth doing plenty of advance research and planning. One of the most fundamental MDM best practices is to set data standards that agree with other data types across your company. Different departments will have different needs, so the data standards you set should be adaptable while still maintaining the uniformity needed for standardization.

‍

2. Data governance‍

Data governance sets the internal rules for how you gather, store, use, and dispose of data. It also dictates who can access what, when, and why. If you work with big data, a data governance strategy is a non-negotiable necessity. Setting strong master data governance best practices and policies for your company allows you to have a clear overview of data use in your organization.

‍

3. Data integration‍

As we’ve covered in a previous article about data migration, data integration allows data to flow back and forth freely within your operations. This means you need data fields that can map easily to each other across different parts of the organization where naming conventions may differ. Data transfer from one application to another may throw errors and end up being anything but seamless, but MDM can help you anticipate potential errors through smart data integration policies.

‍

Before you start, consider how to define your data integration policies and how to manage integrations between different applications. Using a tool that helps you automatically reformat data to meet the target data schema will help with this. You should also try to prevent errors and duplicates by setting up server callbacks to validate importer data against the target database.

‍

4. Data stewardship‍

Data stewardship is how you maintain the quality of your data and make sure your master data management system can work effectively. The majority of large companies will hire a data steward to specifically manage this task as bad data makes consolidation and integration difficult and creates problems for the long-term management of master data. For smaller organizations, it’s important to think about which roles should manage data stewardship, who can access, change, and create master data, and how to manage master data-related tasks.

‍

Master Data Management Best Practices‍

When you’re setting up your master data management policies, it’s important to take all the different factors and best practices into account. Product master data management best practices will likely look slightly different to supplier master data management best practices, for example. The same goes when you’re setting up customer or vendor master data, or any other number of master data that may be necessary to your organization. Regardless of how flexible you need your systems to be, however, there are some hard and fast master data management best practices that should always be applied. The following will help you keep data quality high internally, and through exchanges with external stakeholders like customers or vendors. Let’s explore.

‍

Educate your stakeholders‍

Master data management is a team effort. While certain roles and teams will oversee MDM operations, it’s crucial that all stakeholders understand how to maintain and benefit from your data. By educating all those who contribute to your databases, you’ll ensure widespread adherence to the policies you’ve set up in advance and divide the work among teams so that certain roles don’t get overloaded. This also makes sure you’re retaining knowledge within the organization and not at risk of losing it by limiting the information to one role or team. It’s important to remove barriers-to-entry here, so making data validations and cleaning processes for both internal and external stakeholders as easy and intuitive as possible is key. This means that every stakeholder can easily contribute to ensuring that imported or updated data aligns with data standards. Setting up smart customer onboarding systems and data ingestion processes will also make it easier for you to oversee data quality when you’re working with external stakeholders and spot inconsistencies before they become problematic.

‍

Set up data validation and cleaning processes‍

Implementing automated processes that ensure data is validated and cleaned before it even enters your database will be a game changer for your master data management. Automations can turn frustrating, time-consuming tasks into quick checks. Good automations will also alert you to bad data or mismatched fields, which allows you to fix any issues before they become problems. In practice, this looks like setting up automations that validate and meet the target data schema, ensuring high data quality by checking cross-column dependencies, validating data with external APIs, and using import tools that provide immediate actionable feedback to the user so they can see, clean, and reformat data errors on the spot rather than having to restart the process.

‍

Server callbacks allow you to validate data against your existing database. That means you’ll see errors, avoid duplicate entries, and be able to reformat data then and there before it enters your database. It’s important to use the best tools at your disposal and integrate them into a policy that adheres to your internal master data management best practices.

‍

Understand your data protection responsibilities‍

Since GDPR came into force in May 2018, companies have been required to ensure all data related to natural persons like customers or employees is properly managed. As much of the affected data could be considered master data, this means that GDPR changed how MDM functioned in many businesses. It became even more necessary than before to have an easy-to-manage, well-structured, and transparent data system. Any organization managing people-related master data needs to have a clear understanding of what’s needed for GDPR-compliance and in many cases it’s a good idea to hire a data protection officer.

‍

Use AI-assisted data imports‍

AI can be the perfect complement to well thought-out master data management systems. AI-assisted data imports can automate data validation and cleaning to make sure that all transferred data complies with your pre-defined data standards, rules, and policies. This will again save you time that you might have to otherwise spend manually verifying data, matching fields, or correcting import errors. Once your schema and automations are in place, it’s good to work with a tool that remembers your set-up and applies the same logic every time a similar file is imported. This facilitates a smoother customer onboarding experience which is essential to reducing potential churn. This is also why your importer should be able to handle customer data edge cases without having to create custom import scripts. The easier the experience is on all sides of the import process, the better it is for everyone.

‍

Managing your master data doesn’t have to be a daunting task if you have the right processes. It is however, a crucial one that requires thoughtful planning, best practice implementation, and smart support from well-chosen tools. By educating internal stakeholders, setting up simple processes for external stakeholders, and prioritizing GDPR-compliance, you’re already off to a good start. nuvo’s Data Importer allows you to automate and scale data transfers with external parties in a secure way. Data imports are only one part of a broader master data management system, but getting imports right can help ensure high quality data along the length and breadth of your processes.

‍

Set yourself up for success and contact us today to find out how nuvo can support your master data management goals

Product Launch!!

nuvo — Fri, 17 Mar 2023 11:06:58 +0000

Exciting news! 🎉 We launched our No-Code Data Pipelines on Product Hunt!

If you're tired of the frustration and expense of onboarding external data, we have the solution for you.

✅ With our data pipelines, you can easily connect to your customers' systems and seamlessly ingest, map, validate, and clean data without relying on your engineering team. No more impracticable updates or unknown structures - we've got you covered!

We're grateful to be able to bring this solution to you, and we would appreciate your support as we continue to grow. Sign up for our open beta and test it out for free! Your feedback is valuable to us as we work to improve our product.

🙏 Thank you for your support! ➡ https://www.producthunt.com/posts/nuvo-no-code-data-pipelines

Data Migration: Strategy, Steps and Best Practices

nuvo — Mon, 06 Mar 2023 09:10:10 +0000

Successful data migration requires smart planning, an effective strategy, and the right tools. Let’s have a look at how to make your data migration process seamless.

Data migration may sound simple to the uninitiated—it’s just moving data, right?—but what might seem self-explanatory from the outside is actually a deeply complex process that requires careful planning. After all, any successful data migration requires watertight security, pinpoint accuracy, and smooth transition. That’s why, for many companies, setting up a data migration strategy and process can be a daunting task. There are so many things that can go wrong.

For lots of organisations, an internal lack of knowledge around data is the first major obstacle. Data literacy is important for adequate data management and to prevent overwhelm when new tools or systems require data to be migrated. Vendor applications and their data structure can often be incompatible with each other and while workarounds can bridge the gap for a while, these are rarely viable as long-term solutions. Data silos often see teams working with incomplete or inaccurate datasets which render meaningful data analysis impossible. The absence of an SSOT (single source of truth) aggregated data system also means that the organisation cannot accurately gauge its data needs due to the lack of a proper overview. This also gets in the way of data mapping because it’s hard to match fields when the criteria vary so widely. The absence of a universal schema can also lead to data duplication and complex validation thresholds.

All of this can make data noisier, harder to work with, and difficult to migrate should the need arise. However, picking the right data migration tools and software can significantly ease the process. So now we know the risks and challenges companies face when migrating data, let’s look at how it can be achieved successfully.

But first, let’s review the basics.

What is Data Migration?

Data migration is the process of moving data between formats, applications, or locations. A company might decide to migrate their data if, for example, they start using a new product that requires different file formats or if they choose to shift from an on-prem data centre to the cloud.

Data migration can be complex and delicate, especially when large quantities of data—Big Data—are involved. It’s rarely simply a question of moving data from A to B. The entire data migration process involves a series of steps where teams need to select, prepare, extract, and transform data before permanently moving it between locations or apps. That’s why having a data migration plan from the start is essential.

What’s the Difference Between Data Migration and Data Integration?

Data migration and data integration both involve moving data, but differ in fundamental ways. Data migration is the process of repackaging and moving data and data integration is the process of sharing data between systems. Migration tends to be a one-way journey, whereas integration allows data to flow back and forth. Integration can be important in verticals like e-commerce where many simultaneous processes require data to move flexibly between different internal systems.

What About Data Migration Versus Data Replication?

Data migration is a one-time process where data is transformed and moved from one place to another, as in a database migration. Data replication is when you copy the data source at a particular point in time without altering the original. This allows you to have several working copies of the same data available at once. This supports seamless access to data without slowing down servers or other users’ access. It also means that multiple users can work with the data which can then be synced to update at the source.

How Are Data Migration and Data Conversion Different?

The key difference between data migration and data conversion is that migration transfers while conversion transforms. Data migration moves data from an origin point to a destination, whereas data conversion is the process of extracting and transforming data into a desired format.

The 2 Most Common Data Migration Strategies

As with any iterative process, there are many different approaches to data migration. However, most organisations will opt for one of two more popular data migration strategies, depending on their specific internal needs. Let’s have a look at each one and their main differences.

Big Bang Migration

As its name suggests, Big Bang Migration happens all at once. While it’s appealing to many organisations to get all data migration steps completed during a period of planned downtime, the company needs to be in a position where it can absorb the effect of having its main systems offline for a period of time during the ETL (extract, transform, load) process. This adds pressure and increases the speed requirements of the data migration process. While all data migration strategies need to be planned out in advance, it’s of utmost importance that Big Bang data migration goes through a dry run so that teams can iron out any crinkles in the system before they undertake the real migration.

Trickle Migration

For organisations that can’t operate with data systems offline for any period of time, trickle migration might be a better option. As this data migration strategy runs in phases, both the old and new systems run in parallel until such time as the data migration is fully completed. This means downtime is limited and migration can run continuously for an extended period of time. Trickle migration adds complexity given that it happens over a longer period of time and requires multiple systems to run at once. Planning each step in advance is essential to minimise risk and ensure smooth migration.

Step-by-Step Best Practices for Data Migration

Regardless of which strategy you choose, there are some data migration best practices that should be observed. Knowing exactly what you’re working with ahead of time will lessen the risk of error and make sure that you can feel confident in the approach you’ve chosen.

So let’s run through a few data migration best practices that you should bear in mind.

Plan the migration carefully.
Planning is the name of the game when it comes to data migration. As data security is key, you need to know exactly how you’re going to migrate your data, when you’re going to do it, what potential risks are, and how you plan to mitigate them.

Clean up the data before migrating.

As if you were moving to a new apartment, it’s important to sort and clean before you move from one place to another. Cleaning up data in advance saves you a lot of work in the long run, and increases your chances of a smooth, successful migration. Instead of running into unexpected roadblocks, you’ll know exactly what data you’re dealing with and how to best approach it.

Test the migration process.

Running through the process before you start is also an important data migration best practice. Testing the migration process allows you to plan more wisely, understand the advantages and limitations of your data migration software and tools, and can help you plan your workload. You wouldn’t fly in an untested aircraft, so don’t take the same level of risk with your data. To test the migration process, begin with a small batch of data and monitor its progress closely. When you feel confident that everything works smoothly, you can scale up your approach to migrate the whole system.

Migrate data in small batches.

Even with Big Bang data migration which happens during a planned one-off window, it’s generally considered smart to migrate data in small batches. This means that if something does go wrong, it will be easier for you to find and fix the problem without risking the whole system or adding unnecessary frustration or risk to the data migration process.

Monitor the migration process.

Once your planned data migration steps are running, monitoring progress is key. As the process is complicated, you need to be on-hand and aware so that you can step in if it throws errors. If you have properly cleaned your data and planned your process ahead of time, problems should be minimal.

Validate the data after the migration.

Data is only useful if it is high-quality, cleverly organised, and efficiently maintained. Therefore, data validation after the migration process is completed is an important step that must not be overlooked. Unvalidated data is useless as good data analysis needs accuracy.

Why nuvo might be the data migration tool you need

We started this article by outlining the challenges that companies face when contemplating data migration. As you now know, successful data migration needs a lot of planning and a smart choice of tools. So set yourself up for success and eliminate any potential frustrations by choosing the right tools and getting it right from the very first step.

nuvo can make your data migration faster and easier while reducing cost and the likelihood of mistakes. AI-supported column matching makes data import and mapping a breeze. Advanced data validation alerts users to errors that need to be corrected before they become a problem. Automate data cleaning and enable easy transformation to the format your system requires. Getting your data right from onboarding will make it easier to work with, make analysis more meaningful and enable seamless migrations whenever you need them. Contact us to find out more.

5 Things to Consider When Building a Data Import Tool

nuvo — Wed, 22 Feb 2023 13:07:53 +0000

The lack of a reliable, error-free data import solution can result in frustration for both employees and customers. It can even result in lost business.

Reliable data import is a vital necessity across an organization’s departments, whether in a B2B or B2C context. In our more and more data-driven world – especially in the software and technology sector- the first thing that your customers do and experience (sometimes even before they access your solution) is data uploading and importing.

Unfortunately, this seemingly simple procedure is often fraught with errors due to a lack of universal standards, technical limitations, and technical complexities.

Data importing has not received the attention it should, despite its importance in rapid onboarding procedures and the overall end-user experience.

Why is data import critical?

Importing data from legacy systems is often needed to onboard a new customer. Data import tasks are either carried out in-house by employees or they are outsourced to the customer by requiring them to upload a file to a website or app.

The data import process has high relevance in the customer onboarding experience. Correctly imported data means higher-quality data and time-savings for the business.

Errors in the data import procedure can lead to frustration. Considering that 86% of customers would pay more for a better onboarding experience, improving the data import step is vital for any business.

Data import problems to consider
As data import is a highly complex and specific task, the process can be error-prone resulting in various obstacles that have to be faced along the way.

1. Export/import format

There is no universal standard for how data must be exported or imported. Software and tools export data in CSV files, (comma-separated values), Excel workbooks, raw text data, database structured text, and at least a dozen other formats.

Standard formats have been attempted, but it is still up to the vendor to implement those standards in their software.

This variety of formats means that programmers must either program data import solutions for each type of format or opt for the two or three most popular ones and exclude the rest. This leads to incompatibility for customers who can’t provide their data in the required format.

2. Regional data errors

Perhaps one of the most frustrating data import errors for programmers is the high variance in data formats from region to region. For example, Germany uses the comma as a decimal point which means that “Comma-Separated Value” files are actually separated by a semi-colon, and this plays havoc with data import functionality.

A pretty common example of this challenge is the difference in date formats. In contrast to the European way of displaying a date, US date formats state the month before the day. What can lead to small misunderstandings in daily doing can become a relevant source for data quality issues and lead to significant errors when it comes to automatically imported data. Oftentimes, the import even fails, leading to frustration and high manual validation and cleaning efforts.

3. Data errors

The exported data itself might contain errors. The source of the data might not have done a great job at validating user input resulting in illogical errors such as a birth date that is 200 years in the past or an annual household income of $30 for a Fortune 500 CEO.

Building a robust tool that catches data errors while importing is a mammoth task!

4. Data type errors

Computers need to determine the type of data before being able to do anything logical with it.

This is best understood using Excel as an example.

Although Excel might display a date as “01 Jan 2023,” that date is actually stored as the number “44936”.

By telling Excel that the type of data above is a date and not a number, it knows what to do when performing calculations with it.

Data imports can become a convoluted mess when the import tool gets confused about the type of data being imported.

5. Lack of resources

In general, the challenges mentioned above are only a selection of issues that usually lead to an immense amount of resources required to build a solution that might or might not work every time, for every case, and which will be specific to one particular system. This results in “lock-in” and, if a company wants to change platforms, the entire data tool would need to be programmed again.

Apart from the setup, resources required for continuous advancement and maintenance of the tool are often neglected when calculating the ROI to have a make or buy decision. In fact, companies that decide to go with an in-house built solution first, often end up overturning the decision realizing that a growing customer base, a scale across use cases, and advancements to the core product continuously add requirements to the data importer and make the original solution not usable earlier than expected.

Building a data import tool is not a one-off cost. The tool needs to be maintained to keep up with dependencies, making it an ongoing cost.

The solution for easy data imports
There are two aspects to a complete data import solution:

The file format

The technology used to understand the data itself
Despite the plethora of formats that data can be exported in, two mainstays exist—CSV and Excel files. No matter what other formats exist, virtually all software can export into CSV, and there is wide support for Excel files as well.

The second part of the solution lies squarely in the area of Artificial Intelligence.

Only through using sophisticated AI algorithms is it possible to sift through the complexities, determine the type of data being imported, and understand its content so that it is imported correctly.

The resources required to build such a tool are enormous, even if that tool is meant to work with just a single in-house system. Building one that is 100% flexible and so prevents vendor lock-in is virtually impossible unless it is your business’s primary offering.

Nuvo has engaged the necessary resources and created a 100% flexible, problem-free, reliable data import tool that can be used to import any data from any system using CSV and Microsoft Excel files, with zero errors.

Should you build your own data tool or buy one?

The choice is ultimately up to you if you want to build your own data import tool or not. If you do, you will need to consider the five things above and plan for them.

Buying a data import tool makes more sense from an economic and efficiency viewpoint, but only if the tool is reasonably priced and brings value to your business.

Homegrown CSV Importer vs nuvo Importer – Is a pre-built data import library worth the investment?

nuvo — Wed, 22 Feb 2023 13:02:11 +0000

You have built your own CSV importer, but as your organization, your product and its requirements grow, you ask yourself if switching from your homegrown solution to a pre-built library could be beneficial.

As you grow and onboard more and more clients, your importer increases in complexity and chances are, you can’t afford to keep up with the cost, talent, and time it’ll take to build and maintain your own CSV import solution. Data import cases will increase in complexity, requirements especially performance requirements will rise accordingly and at some point, a data importer without smart features just won’t be able to do the job successfully on a long-term basis.

In general, deciding on build vs buy is crucial, to begin with:

The focus of your developer resources should always be on the product’s core features. The implementation time and the maintenance of secondary/support functionalities such as importer, login, sign up, etc. shall be reduced as far as possible.

However, even for companies that decided to build a CSV importer in-house at some point, replacing the homegrown importer at a later point is worth considering.

Here are some signs that should get you thinking about switching:

New requirements on security & compliance when tapping into new markets
Increased implementation and maintenance efforts for advanced as well as smart data mapping and cleaning features
High implementation services for customers
Limitations in adoption and customer onboardings due to long development time
There is a risk of brain drain in case a key developer leaves your team or company

In general, software libraries that offer 24/7 support with a quick response time (24h), are well maintained, and frequently updated (especially in regards to major version releases of common frontend frameworks) can be a valuable solution to replace your homegrown importer.

To provide some numbers and a potential guideline that can support in deciding to make or buy, we created some scenarios covering different levels of import complexities:

Is an off-the-shelf CSV importer worth the investment?

Deciding to buy instead of build or replace a homegrown component that already took internal resources should always be driven by a positive ROI calculation considering the investment for setup but also long-term maintenance of the solution.

As the import cases and their complexity can vary significantly, it is important to distinguish and consider different use cases separately.

To provide some reference data and support decision making we will look at four different import scenarios, including their requirements, and discuss their pros and cons:

1. Simple CSV importer for internal use only

As a first case, we consider a basic and straightforward data import case. The data is imported by internal teams only, the target data model is rather basic, containing around 5 – 10 columns, no complex validations, and the column mapping is conducted using a simple dropdown view without automation via AI or any other algorithm.

This case serves the minimum requirements, usually has no specific requirements regarding styling or UI, and is implemented quite fast.

Experienced software engineers estimate around 2 months of development time, requiring a team of 3-4 engineers working on it.

The simplicity that allows the fast implementation of the importer, unfortunately, comes with some drawbacks that need to be addressed as they will significantly affect the use and manual effort related to data imports:

Simple and static importers require effortful preparation of the data, including manual mapping and cleaning with every file imported.
The performance of the importer is often neglected in this simple data import case and can lead to errors and failure of the importer as soon as larger datasets are imported.
Maintenance of static importers is high as with every change in the target data schema, the importer has to be adapted, requiring continuous developer resources.

2. Advanced CSV Importer

The second scenario we consider is a more advanced data importer that can be provided to non-technical, external uses as for customer self-service imports.

To ensure a smooth experience for your clients, the advanced CSV importer should fulfill some additional requirements and offer features such as:

Advanced UI that guides non-tech users through the workflow seamlessly
Basic data mapping enabled by fuzzy matching
Data cleaning functions and data validation features

The efforts to develop a CSV importer as described above can be estimated at around 3 months of development time for an Engineering team of 3-4 developers.

Even though this more advanced CSV importer can enable self-service imports for your clients and takes manual efforts from internal teams due to smart features and data validation and cleaning functions, however, there are drawbacks that should be considered making a decision of in-house development of this advanced importer:

Maintaining high performance with the upload of large data is something most homegrown importers struggle with.
Maintenance requirements of the data importer provided to clients for self-service are high and continuous developer resources are needed to update the importer with changing target data schemas should be considered with the decision to keep an advanced CSV importer in-house.

3. AI-assisted, high-performance CSV importer

The third scenario we are looking at is an AI-assisted, high-performance importer that is optimized for large data, is characterized by a simple and advanced UI and includes AI-assisted smart features for automated data mapping, validation, and cleaning.

Being able to provide a CSV import solution equipped with all features mentioned above will not only provide a seamless data onboarding experience to your clients but will also gain significant time savings for your internal teams.

The development time of importers on this level can be estimated at around 8 months (considering a development team of 3-4 developers) and more to be added, as additional features will be required with a growing customer base and increased import complexity over time.

4. nuvo Importer

Implementing a plug-and-play import library is considered the fourth scenario in this article. The nuvo Importer is a component that is embedded in the application’s front-end using one of the following frontend frameworks: React, Angular, Vue.js, and plain JavaScript.

The component enables you to guide your users through our data onboarding workflow, including uploading and selecting a file, choosing the preferred sheet and the header row, matching the imported columns to the columns of your target data model, and cleaning the imported data.

Getting started usually takes a couple of minutes only and provides you with a data importer offering:

Fully custom styling
Compliance to the highest security and data privacy standards
AI-supported column matching
Advanced data validation and cleaning as well as 24/7 customer support, maintenance, and continuous feature and performance improvements

To conclude, make vs buy or maintaining homegrown vs replacing with an off-the-shelf solution is a decision that has to be considered carefully and depends on the requirements and resources available.

As both variables can develop and vary over time, reconsidering a decision that was once made, can often be worth the investment.