Handling hundreds of thousands of data rows with peak performance

Overview

Shareline is a custom software development company working with clients to create bespoke solutions. Recently, one of their clients was the National Food Chain Safety Office (NFCSO) in Hungary. The department they worked with was responsible for tracking the purchase and use of resources, buyers and sellers, and the countries of origin and destination with an overall goal to protect environmental sustainability according to the EU laws.

The NFCSO was looking for web-based software to help users supply data required by the organization. For their own purposes, they also needed back office functionalities such as reporting and cataloging. Combining all of that, the software had to support cross-checked validations to ensure the consistency of the data entered on both sides.

On the user side, the software's main features are the data-providing user interface and the ability to check their status in the government's publicly available catalog. For the organization, the main features are reporting and data validation. They need to be able to enter, verify, store, and display data.

The Challenge

The system the NFCSO was previously using was Excel-based, so Shareline had to make Excel-based uploads available. But when an Excel file is uploaded, all the data has to be loaded into a data grid and validated before the user closes and moves on. And since some users are large production companies with tons of data, the main technical challenge was to find a web-based data grid that could handle huge data amounts with acceptable performance in a single browser window on a standard PC.

Shareline's first move was to find potential data grid candidates and evaluate them. They found that while some did very well in user experience, most of them failed when it came to handling large amounts of data. They needed something that could handle hundreds of thousands of rows, and because of the volume, the data had to be cross-checked. And to be cross-checked, it had to be kept in memory.

They did find a few viable options and tried to integrate them into a proof of concept. They created a very simple solution with hard-coded data and no backend at all, then started performance tests on different types of hardware.

This phase was all about experimenting. Based on the requirements, it was very important to find an older, low-end notebook to check how the solution would perform. All in all, Handsontable turned out to be the best data grid they tested.

The Solution

To make the transition to the new system as smooth as possible for its users, Shareline had to duplicate Excel's key features like auto-fill and inline editing. Handsontable excelled in this area and had these features out of the box.

For Shareline, Handsontable's main feature is its customizability. It's why they could create an app that mimicked Excel, even allowing them to implement a tab-based UI. It also enabled them to create other required features like drop-down lists that contained special field types.

Handsontable's flexibility also helped streamline Shareline's work process. The JavaScript data grid made it super easy to implement changes. Before they reached the final version of the software, they tested different concepts and designs. And because Handsontable was so adaptable, they didn't have to start from scratch and restart the development every time.

The Result: great performance with large amounts of data and custom business logic

Handsontable's effect on the user experience of the app can be summed up in one word: performance. Shareline was able to provide real-time data validation for hundreds of thousands of rows in a single browser on a standard PC, which to them is a major accomplishment.

They also point to a few key benefits of implementing Handsontable:

• Having the ability to create an Excel-like user interface and functionality with good performance
• The addition of custom business logic to the UI, an important advantage over using Excel
• Extensive, real-time validation to provide accurate data
• Being able to avoid user training

Overall, their client NFCSO was extremely satisfied with Shareline's solution meeting all the organization's requirements and expectations. The software company has also gained a lot of experience using Handsontable in the process. So now they have a tool that excels in supporting real-time inline editing with large amounts of data within the browser and lets them implement custom business logic. And they expect to find a place for it in other solutions in the future.

Handsontable is the best grid for real-time editing of large amounts of data, so if your project requires this, it's the one to go with.