How To Scroll And View Millions Of Records

How To Scroll And View Millions Of Records

How To Scroll And View Millions Of Records
Introduction I was asked recently what kind of approach to take when one needs to display large volumes of data to a user in a linear manner. In particular, what if you had millions of records (or even just hundreds, same concepts apply!), and for whatever reason, needed to allow the user to be able to scroll through these. I have handled this before, using some well traveled patterns. So, I thought, it would be useful to put it down as a short article and share the knowledge. Background When dealing with anything more than a screenful of data, we need to consider how to present this data to the user. How will the user see the data and how can they navigate it? Clearly, anything more than a screen’s worth of data is too much for the poor human eye to take in at once and process with accuracy, so we need to implement some system of managing things behind the scenes. These basic concepts work the same for web as they do on desktop, as they do on mobile (and perhaps other interfaces we haven’t encountered yet!). This article looks at these (which you may know already nut have not thought too much about) and gives some examples and things to look out for. Core Concept 1 – tracking user position When a user requests data, they do it from a particular point of view, or position. The position taken, depends on the perspective and context of the data. Some examples.

Read Also:
SaaS Data Migration & Data Integration

With the above examples, we can see the perspective of where the user stands in relation to the data they want to see, and we want to present to them, can be variable. Therefore, a critical part of presenting large amounts of data to a user is understanding and tracking their user position relevant and in context to the overall dataset we are dealing with. Core Concept 2 – data portals When a user requests data, and we need to know a few things.

We have seen this pattern on the desktop and the web on many occasions – it’s the concept of data paging. In the example above (from the wonderful datatables .net), we can see the users required view is a filter on any field that contains ‘lo’, their current position is ‘page 1’, and they only wish to view a volume of 10 records at one time. Back-end scrolling view-port management It is clear and obvious from the front-end point of view, what’s happening, or more correctly, what we are instructing to happen, but how is it handled on the back-end? To explain, let’s look at a simple example of a dataset of 15 records. Lets assume that the user starts out at the first record, and wants to move through in blocks/views/sets of 5 records at a time. In step 1, we get records 1..5, step 2 records 6..10, and finally in step 3 records 11..15. When the user clicks on the ‘prev/next’ buttons on the front-end, they send an instruction to the back-end saying ‘I am here at position X, please give me Y number of records from that position’. Step 1 Starting position = Record 0, from here, give me 5 records = Rec 1..5 Step 2 Starting position = Record 5, from here, give me 5 records = Rec 6..10 Step 3 Starting position = Record 10, from here, give me 5 records = Rec 11..15 This translates to the back-end very simply, depending on the back-end you are using – here are two examples.

Read Also:
Generali's Elena Rasa Takes a Data-Driven Approach to Excellence

Check the documentation for whatever other back-end you are using – there should be an equivalent methodology to allow you to implement this functionality. Now, having pointed you to this common method of extracting a limited view of data from a structured dataset, I must caution you not to expect to rely on it every time. Always question the use case, the volume of data in question, the index fields you have to work with, etc. It may be the case that in very very large sets of data a map/reduce pattern may work well, or it may be that you have data pre-loaded for particular presentations of data that you can serve up, or it may be that the offset/limit slows down with a particular volume of data etc….

Read Full Story…

 

Leave a Reply

Your email address will not be published. Required fields are marked *