How To Scroll And View Millions Of Records

How To Scroll And View Millions Of Records

How To Scroll And View Millions Of Records

Introduction I was asked recently what kind of approach to take when one needs to display large volumes of data to a user in a linear manner. In particular, what if you had millions of records (or even just hundreds, same concepts apply!), and for whatever reason, needed to allow the user to be able to scroll through these. I have handled this before, using some well traveled patterns. So, I thought, it would be useful to put it down as a short article and share the knowledge. Background When dealing with anything more than a screenful of data, we need to consider how to present this data to the user. How will the user see the data and how can they navigate it? Clearly, anything more than a screen's worth of data is too much for the poor human eye to take in at once and process with accuracy, so we need to implement some system of managing things behind the scenes. These basic concepts work the same for web as they do on desktop, as they do on mobile (and perhaps other interfaces we haven't encountered yet!). This article looks at these (which you may know already nut have not thought too much about) and gives some examples and things to look out for. Core Concept 1 - tracking user position When a user requests data, they do it from a particular point of view, or position. The position taken, depends on the perspective and context of the data. Some examples.

Read Also:
The 5 pillars of digital transformation businesses need to set themselves up for success

With the above examples, we can see the perspective of where the user stands in relation to the data they want to see, and we want to present to them, can be variable. Therefore, a critical part of presenting large amounts of data to a user is understanding and tracking their user position relevant and in context to the overall dataset we are dealing with. Core Concept 2 - data portals When a user requests data, and we need to know a few things.

We have seen this pattern on the desktop and the web on many occasions - it's the concept of data paging. In the example above (from the wonderful datatables .net), we can see the users required view is a filter on any field that contains 'lo', their current position is 'page 1', and they only wish to view a volume of 10 records at one time. Back-end scrolling view-port management It is clear and obvious from the front-end point of view, what's happening, or more correctly, what we are instructing to happen, but how is it handled on the back-end? To explain, let's look at a simple example of a dataset of 15 records. Lets assume that the user starts out at the first record, and wants to move through in blocks/views/sets of 5 records at a time. In step 1, we get records 1..5, step 2 records 6..10, and finally in step 3 records 11..15. When the user clicks on the 'prev/next' buttons on the front-end, they send an instruction to the back-end saying 'I am here at position X, please give me Y number of records from that position'. Step 1 Starting position = Record 0, from here, give me 5 records = Rec 1..5 Step 2 Starting position = Record 5, from here, give me 5 records = Rec 6..10 Step 3 Starting position = Record 10, from here, give me 5 records = Rec 11..15 This translates to the back-end very simply, depending on the back-end you are using - here are two examples.

Read Also:
Inside the Health Data Industry's Opaque Diagnosis: A Q&A With Author of 'Our Bodies, Our Data'

Check the documentation for whatever other back-end you are using - there should be an equivalent methodology to allow you to implement this functionality. Now, having pointed you to this common method of extracting a limited view of data from a structured dataset, I must caution you not to expect to rely on it every time. Always question the use case, the volume of data in question, the index fields you have to work with, etc. It may be the case that in very very large sets of data a map/reduce pattern may work well, or it may be that you have data pre-loaded for particular presentations of data that you can serve up, or it may be that the offset/limit slows down with a particular volume of data etc....

 



HR & Workforce Analytics Summit 2017 San Francisco

19
Jun
2017
HR & Workforce Analytics Summit 2017 San Francisco

$200 off with code DATA200

Read Also:
Self-taught artificial intelligence beats doctors at predicting heart attacks
Read Also:
DeepMind’s first NHS health app faces more regulatory bumps

M.I.E. SUMMIT BERLIN 2017

20
Jun
2017
M.I.E. SUMMIT BERLIN 2017

15% off with code 7databe

Read Also:
DeepMind’s first NHS health app faces more regulatory bumps

Sentiment Analysis Symposium

27
Jun
2017
Sentiment Analysis Symposium

15% off with code 7WDATA

Read Also:
What is data profiling and how does it make big data easier?

Data Analytics and Behavioural Science Applied to Retail and Consumer Markets

28
Jun
2017
Data Analytics and Behavioural Science Applied to Retail and Consumer Markets

15% off with code 7WDATA

Read Also:
Artificial intelligence is now the real thing

AI, Machine Learning and Sentiment Analysis Applied to Finance

28
Jun
2017
AI, Machine Learning and Sentiment Analysis Applied to Finance

15% off with code 7WDATA

Read Also:
If your data lake turned into a data swamp, it might be time to take the next step

Leave a Reply

Your email address will not be published. Required fields are marked *