Jump to content

UniDbGrid - fetching considerations


arilotta

Recommended Posts

Hi all,

as far as I have read from the forum the only way to deal with big datasets and uniDbGrids is paging.

This works quite well, but I would prefer another option, let me explain better.

 

Without paging, the web server fetches the whole dataset and it gives back it to the client:

 

DB server --> Web server (Unigui) : whole dataset

Web Server (UniGUI) --> Browser: whole dataset

 

This is the worst situation, as it implies a lot of overhead for all the actors (DB Server, Web Server, 

Browser, network, etc.). It should not be used for large datasets.

 

 

With paging, the web server fetches the whole dataset (this is why the option FetchAll shall be set),

and it gives back to the client (browser) just the current page. Therefore:

 

DB server --> Web server (Unigui) : whole dataset

Web Server (UniGUI) --> Browser: partial dataset/ curent page

 

This is better, as it minimizes the overhead on the client side (browser), but the Web Server (UniGUI) still

has to deal with a large dataset that has to be fetched entirely from the DB server and to be kept in memory.

 

 

The last option, that I really miss with UniGUI, is the way that VCL DBGrid works, but it is common to find it

on many websites. This is the reason why I'm asking about it, I'm confident there should be a way to work it out

with UniGUI.I think that it could be called "incremental fetch".

 

Practically, UniGUI should fetch just the first N rows and show them on the UniDbGrid component, letting the user know

that there are more rows than the ones actually visible. It could be a button on the paging bar called "Load more rows".

If the user clicks on that button, UniGUI should fetch N additional rows from the dataset, and then display N*2 total

rows. This approach can be iterated till the whole dataset is fetched.

Alternatively, without using the "Load more rows" button, it could be possible to do it in the VCL way:

when the user scrolls down the grid and he reaches the last line, the "give me more lines" could be automatically

fired.

The "incremental fetch" solution would be the best in my opinion, as the user does not really care to know the number of pages in the grid,

nor he needs to skip randonly on different pages. Usually he does want quickly the first N rows, and then the possibility

to look up another N rows for a fe times. Then, if the data is not found, he would change the actual filter/selection.

This would bring the following situation:

 

DB server --> Web server (Unigui) : partial dataset

Web Server (UniGUI) --> Browser: partial dataset

 
 
 
Sorry for the long post, hoping that I was clear, thank you all.
Andrea
  • Upvote 2
Link to comment
Share on other sites

Hi!
Nice to see people who think about optimization.

Unfortunately (and I should know) it is not possible.
Data extraction deals with TDataSet. But this component does not have Paging.
FireDac, when properly configured, can extract data page by page, but there's still a Library (ADO), TDBF, and other data access components that have no idea about paged data retrieval.
And there are methods of Tdataset, which negates the whole optimization data, for example: 1) Locate the method that should go through the whole dataset (millions of records), 2) the method SetFilter, which should select only 100 records out of a million.
That is, to produce the final dataset component, the server must make a selection of all data.
It has nothing to do with UniGUI. If you can do paging sample in VCL, then it will turn out in UniGUI. But Grid has nothing to do with it. DbGrid does not have to know anything about the components, which provide him data.
Everyone can organize such, but it's almost impossible to make universal.

Sorry for my YandexTranslator-English, but I hope that main idea is clear.

Link to comment
Share on other sites

Hi Volk65, I think you are right regarding the fact that not all TDataSet descendents support incremental fetching.

For example, the TClientDataSet or TVirtualTable, which are not bound to a DB server, obviously do not support it.

Even the general TDataSet does not support it.

But I think there are many other specialized descendent that support it instead.

I am using DevArt components to connect to Oracle DB, and I can finely tune the following fetching options:

 

- FetchAll (boolean: fetch all rows or not )

- FetchRows (number or rows to fetch in a bunch)

 

FireDac, as you say, can do it.

Implementing the "incremental fetch" with UniDbGrids, and using a TDataSet that does not support it, would give the same result

as today.

But if used with TOraQuery (or FireDac), would give the same user experience as VCL in a web application: amazing !

 

Of course using the Locate (or Last command) would break the rule, but it is something that the developer should be aware of,

as the behaviour is well known. It is sufficient to avoid local filtering options with big datasets and this kind of "incremental fetching grids"

(and reopen the queryinstead with changed conditions).

 

To make a comparison with VCL, it is the same as using the vertical scrollbar on a DBGrid filled with 1 million rows, and dragging it

to the bottom: the system hangs.

It is up to the developer to block such a bad behaviour.

 

Thank you for sharing your opinion.

Anyone else ?

Andrea

Link to comment
Share on other sites

HI,

Farshad already known my ancient "partial fetching" request for Dbgrid; this one is the same but not only at Dbgrid visual level; Andrea want to optimize record's number present in memory.

The problem is that Tdataset  partial fetchrow add records fetched to others already fetched scrolling the dbgrid; so, If I go to last records, all records are fetched :-)

The best idea is (as devexpress servermode) that records in memory are always the twice of row visible in grid. So I have always 100 records in memory (dbgrid with 50 rows), even if I go to the last records....
But this involve a quering mechanism at sql level for paging...

;-)

 
 

Link to comment
Share on other sites

Hi Stefano, I see your concerns about scrolling (down) the grid and the consequent fetching of additional rows, but this should be limited in one of the following ways:

 

- the users scrolls down the grid to the last row, and they can fetch and additional page (N rows) using a dedicated button "Load more rows" that could be 

  placed on the navigation bar, instead of the current previous/next/first/last page buttons

- the user scrolls down the grid to the last row, and automatically (ajax event ?) and additional page (N rows) is fetched from the opened query and added to the UniDbGrid

 

So no way for the end user to move to the last row and fetch all the rows from the dataset as a whole.

After a few page fetches without finding what they are looking for, the users would tipically refine the filter to get the data he really wants.

Memory occupation, CPU, network usage minimized on the DB side and web server (UniGUI).

 

Andrea

Link to comment
Share on other sites

Hi all, I've developed a small app that shows better the idea, it is published at the following address (I will keep it online for a week):

 

http://188.11.1.245:8746/FetchingDemo.dll

 

- In the main page, it is possible to choose:

    - the grid page size

    - the maximum number of rows to fetch (this limits the rows manipulating the SQL in the query)

    - whether fetch all the rows or not

- The actual table contains more than 6 million records

- Clicking the "Open" button the secondary form with the UniDbgrid (FRMGrid) is opened with the settings specified in the main form

- The grid contains a new button in the paging bar: "More pages..."; by clicking it an additional page (pagesize rows) is fetched from

  the opened query if available, and the grid and the paging bat is updated accordingly

- The secondary form contains also a couple of buttons to show the rows fetched so far (Fetched row count) and the possibility to

  select a row given its absolute position (RecNo)

 

Now let's tets the following scenarios:

1. pagesize:10; #rows: 100000; fetch all rows: TRUE

    This is the current suggested configuration for large datasets: pagination+fetch all. 

    It takes more or less 5 seconds on my web server to fetch 100000 rows from the DB server, to paginate them and to send the fisrt page back

    to the browser. The memory consumption on the web server is huge, as all the 100000 are kept in memory.

    It is almost impossible to use this approach to fetch 1 million rows.

2. pagesize:10; #rows: 100000; fetch all rows: FALSE

    This is the implementation I'm suggesting for big datasets. 

    It takes nothing to open the secondary form (in fact it is opened immediately), and the memory consumption is minimized

    as only the first 10 rows are fetched from the DB server. If the user wants to fetch additional pages, it can click on the

   "More pages..." button and fetch them one by one (it takes nothing to fetch additional pages, 10 at a time).

    Memory consumption is minimized on DB server and web server, network traffic highly reduced, web site more responsive....

 

In my original post, I told I would prefer additional rows fetched from the DB to be appended to the bottom of the grid rather than

being put in an additional page, but I was not able to achieve this behaviour with my limited knowledge of Ext JS...

For the moment, I think that this solution is acceptable for my needs.

 

I would really like to know your opinion.

 

Andrea

Link to comment
Share on other sites

Hi all, I've developed a small app that shows better the idea, it is published at the following address (I will keep it online for a week):

 

http://188.11.1.245:8746/FetchingDemo.dll

 

- In the main page, it is possible to choose:

    - the grid page size

    - the maximum number of rows to fetch (this limits the rows manipulating the SQL in the query)

    - whether fetch all the rows or not

- The actual table contains more than 6 million records

- Clicking the "Open" button the secondary form with the UniDbgrid (FRMGrid) is opened with the settings specified in the main form

- The grid contains a new button in the paging bar: "More pages..."; by clicking it an additional page (pagesize rows) is fetched from

  the opened query if available, and the grid and the paging bat is updated accordingly

- The secondary form contains also a couple of buttons to show the rows fetched so far (Fetched row count) and the possibility to

  select a row given its absolute position (RecNo)

 

Now let's tets the following scenarios:

1. pagesize:10; #rows: 100000; fetch all rows: TRUE

    This is the current suggested configuration for large datasets: pagination+fetch all. 

    It takes more or less 5 seconds on my web server to fetch 100000 rows from the DB server, to paginate them and to send the fisrt page back

    to the browser. The memory consumption on the web server is huge, as all the 100000 are kept in memory.

    It is almost impossible to use this approach to fetch 1 million rows.

2. pagesize:10; #rows: 100000; fetch all rows: FALSE

    This is the implementation I'm suggesting for big datasets. 

    It takes nothing to open the secondary form (in fact it is opened immediately), and the memory consumption is minimized

    as only the first 10 rows are fetched from the DB server. If the user wants to fetch additional pages, it can click on the

   "More pages..." button and fetch them one by one (it takes nothing to fetch additional pages, 10 at a time).

    Memory consumption is minimized on DB server and web server, network traffic highly reduced, web site more responsive....

 

In my original post, I told I would prefer additional rows fetched from the DB to be appended to the bottom of the grid rather than

being put in an additional page, but I was not able to achieve this behaviour with my limited knowledge of Ext JS...

For the moment, I think that this solution is acceptable for my needs.

 

I would really like to know your opinion.

 

Andrea

 

Andrea,

 

I like your solution.

 

Can you post the code that would put the "More pages" button on the page bar and advance the dataset?

 

Thanks in advance.

Link to comment
Share on other sites

Glad to post the code (MOGSY, my name's Andrea :-)

Here you can download the whole app:

 

https://drive.google.com/file/d/1p09s2H3-gky58SZ8VglohjtZjJD81sM6/view?usp=sharing

 

- to customize the grid (hide refresh button and add "More pages..." button) I've used some code that I've found on this forum and it's not mine:

GRD.ClientEvents.UniEvents 

Ext.toolbar.Paging [pagingBar]

pagingBar.afterCreate

 

function pagingBar.afterCreate(sender)
{
  sender.items.items[10].hide();
  sender.add([{
               xtype: 'button',
               text: 'More pages...',
               handler: function() {             
                 ajaxRequest(sender, 'morePages',[]);                 
               }
              }]);
}
 
- to support the incremental fetching your tdataset descendent (and your RDBMS) should support the following propertes:
* FetchRows (or similar, number of rows to fetch each time)
* RecordCount
* RecNo
 
I use Oracle and ODAC components from DevArt, in a previous post Volk65 said that even FireDac components should work,
you can check the following link:
 
Link to comment
Share on other sites

 

Glad to post the code (MOGSY, my name's Andrea :-)

Here you can download the whole app:

 

https://drive.google.com/file/d/1p09s2H3-gky58SZ8VglohjtZjJD81sM6/view?usp=sharing

 

- to customize the grid (hide refresh button and add "More pages..." button) I've used some code that I've found on this forum and it's not mine:

GRD.ClientEvents.UniEvents 

Ext.toolbar.Paging [pagingBar]

pagingBar.afterCreate

 

function pagingBar.afterCreate(sender)
{
  sender.items.items[10].hide();
  sender.add([{
               xtype: 'button',
               text: 'More pages...',
               handler: function() {             
                 ajaxRequest(sender, 'morePages',[]);                 
               }
              }]);
}
 
- to support the incremental fetching your tdataset descendent (and your RDBMS) should support the following propertes:
* FetchRows (or similar, number of rows to fetch each time)
* RecordCount
* RecNo
 
I use Oracle and ODAC components from DevArt, in a previous post Volk65 said that even FireDac components should work,
you can check the following link:
 

 

 

Andrea,

 

Thank you so much, Sir!

Link to comment
Share on other sites

Please sign in to comment

You will be able to leave a comment after signing in



Sign In Now
×
×
  • Create New...