Acceleration of Data Cleaning & Preparation at Scale

Summaries

• High Level

Data harvesting is annually growing at 23 percent where 60 to 80% of analytics time is spent cleaning and processing data.

• This bottleneck increases annual losses because unprocessed data quickly loses value.
• Intensity greatly reduces the time needed to clean and process data.

• User Level

The Intensity API is used by software engineers to rapidly clean and process data.

1. A workflow script is user-created to transform source data into target data
• Workflow scripting commands are grouped:
• Change, Clean, Conceal, Confirm, Convert, Correct, Elminate, Extract, Halt, Preserve, Put, Remove, Set, Swap, Transfer, & Transform.
• Delimited arguments are passed to each command.
2. The API is called by passing a source filename or buffer, a target filename or buffer, and a workflow script or buffer.
• A flag can usually be passed to immediately return while a related thread is still processing.
• An API method is provided to write processed data if thread processing has been completed.
• An API method is provided to return a count of:
  • Total available threads, free threads, running threads, & waiting threads.
3. Cleaning and processing often completes in a fraction of a second.
• This is due to optimization of compiled C and C++ source code, extensive direct memory access, and asynchronous threading of processing.
4. We will work with clients to enhance Intensity to provide the most powerful data cleaning and processing API.
• Enhancement will include user-defined signature analysis of source data to select an existing user-defined workflow script.

Problem

Enterprises generate enormous data, but cleaning and organizing it is such a bottleneck that much of its potential revenue is never realized.

• Lost revenue opportunities
• Higher operational costs
• Slower decision-making
• Missed competitive advantage

Solution

Reducing cleaning and organizing bottlenecks permit realization of more revenue.

• More revenue opportunities
• Lower operational costs
• Faster decision-making
• Realized competitive advantage

Product

Intensity is a high-velocity data engine that eliminates cleaning and preparation bottlenecks, allowing enterprises to turn raw data into insights and revenue in real time.

• Created from scratch in C/C++ for extreme speed, stability and ease of use.
• 100% ownership of Intensity code base, scripting language, API, and products.
• No use of 3rd-party libraries or open-source components.
• Created without use of AI tools.

Market

• $447.68 billion global big data analytics market size [estimated for 2026].
• $3.87 billion global data wrangling product market size [estimated for 2026].
• 97,000+ big data companies worldwide.

Opportunities Lost

• 23% average annual increase in harvested data since 2021.
• 60 to 80% of analytics time is spent preparing data.

Opportunities Realized

Intensity was designed and created to achieve the highest speed of data transformation.

• Intensity workflow scripts are created by software engineers to rapidly transform raw data into target data.
• The Intensity application can be used to create and test Intensity workflow scripts.
• Intensity API methods are called by custom software to pass a workflow script, input file and target file.
• API methods asynchronously thread out data processing.
• Most API methods can immediately return while processing continues on a unique thread.
• The design enables repeat API calls to made to simultaneously process a variety of data.
• API methods are provided to complete processing stages, and determine available, used and held threads.

Technical Notes [click]

Reach out for further information

Richard Evers, CEO/Founder, revers@midnightblue.ca
Waterloo, ON, Canada