arilotta Posted January 10, 2018 Share Posted January 10, 2018 Hi all, as far as I have read from the forum the only way to deal with big datasets and uniDbGrids is paging. This works quite well, but I would prefer another option, let me explain better. Without paging, the web server fetches the whole dataset and it gives back it to the client: DB server --> Web server (Unigui) : whole dataset Web Server (UniGUI) --> Browser: whole dataset This is the worst situation, as it implies a lot of overhead for all the actors (DB Server, Web Server, Browser, network, etc.). It should not be used for large datasets. With paging, the web server fetches the whole dataset (this is why the option FetchAll shall be set), and it gives back to the client (browser) just the current page. Therefore: DB server --> Web server (Unigui) : whole dataset Web Server (UniGUI) --> Browser: partial dataset/ curent page This is better, as it minimizes the overhead on the client side (browser), but the Web Server (UniGUI) still has to deal with a large dataset that has to be fetched entirely from the DB server and to be kept in memory. The last option, that I really miss with UniGUI, is the way that VCL DBGrid works, but it is common to find it on many websites. This is the reason why I'm asking about it, I'm confident there should be a way to work it out with UniGUI.I think that it could be called "incremental fetch". Practically, UniGUI should fetch just the first N rows and show them on the UniDbGrid component, letting the user know that there are more rows than the ones actually visible. It could be a button on the paging bar called "Load more rows". If the user clicks on that button, UniGUI should fetch N additional rows from the dataset, and then display N*2 total rows. This approach can be iterated till the whole dataset is fetched. Alternatively, without using the "Load more rows" button, it could be possible to do it in the VCL way: when the user scrolls down the grid and he reaches the last line, the "give me more lines" could be automatically fired. The "incremental fetch" solution would be the best in my opinion, as the user does not really care to know the number of pages in the grid, nor he needs to skip randonly on different pages. Usually he does want quickly the first N rows, and then the possibility to look up another N rows for a fe times. Then, if the data is not found, he would change the actual filter/selection. This would bring the following situation: DB server --> Web server (Unigui) : partial dataset Web Server (UniGUI) --> Browser: partial dataset Sorry for the long post, hoping that I was clear, thank you all. Andrea 2 Quote Link to comment Share on other sites More sharing options...
Volk65 Posted January 10, 2018 Share Posted January 10, 2018 Hi!Nice to see people who think about optimization.Unfortunately (and I should know) it is not possible.Data extraction deals with TDataSet. But this component does not have Paging.FireDac, when properly configured, can extract data page by page, but there's still a Library (ADO), TDBF, and other data access components that have no idea about paged data retrieval.And there are methods of Tdataset, which negates the whole optimization data, for example: 1) Locate the method that should go through the whole dataset (millions of records), 2) the method SetFilter, which should select only 100 records out of a million.That is, to produce the final dataset component, the server must make a selection of all data.It has nothing to do with UniGUI. If you can do paging sample in VCL, then it will turn out in UniGUI. But Grid has nothing to do with it. DbGrid does not have to know anything about the components, which provide him data.Everyone can organize such, but it's almost impossible to make universal.Sorry for my YandexTranslator-English, but I hope that main idea is clear. Quote Link to comment Share on other sites More sharing options...
arilotta Posted January 11, 2018 Author Share Posted January 11, 2018 Hi Volk65, I think you are right regarding the fact that not all TDataSet descendents support incremental fetching. For example, the TClientDataSet or TVirtualTable, which are not bound to a DB server, obviously do not support it. Even the general TDataSet does not support it. But I think there are many other specialized descendent that support it instead. I am using DevArt components to connect to Oracle DB, and I can finely tune the following fetching options: - FetchAll (boolean: fetch all rows or not ) - FetchRows (number or rows to fetch in a bunch) FireDac, as you say, can do it. Implementing the "incremental fetch" with UniDbGrids, and using a TDataSet that does not support it, would give the same result as today. But if used with TOraQuery (or FireDac), would give the same user experience as VCL in a web application: amazing ! Of course using the Locate (or Last command) would break the rule, but it is something that the developer should be aware of, as the behaviour is well known. It is sufficient to avoid local filtering options with big datasets and this kind of "incremental fetching grids" (and reopen the queryinstead with changed conditions). To make a comparison with VCL, it is the same as using the vertical scrollbar on a DBGrid filled with 1 million rows, and dragging it to the bottom: the system hangs. It is up to the developer to block such a bad behaviour. Thank you for sharing your opinion. Anyone else ? Andrea Quote Link to comment Share on other sites More sharing options...
Stemon63 Posted January 11, 2018 Share Posted January 11, 2018 HI, Farshad already known my ancient "partial fetching" request for Dbgrid; this one is the same but not only at Dbgrid visual level; Andrea want to optimize record's number present in memory. The problem is that Tdataset partial fetchrow add records fetched to others already fetched scrolling the dbgrid; so, If I go to last records, all records are fetched :-) The best idea is (as devexpress servermode) that records in memory are always the twice of row visible in grid. So I have always 100 records in memory (dbgrid with 50 rows), even if I go to the last records....But this involve a quering mechanism at sql level for paging... ;-) Quote Link to comment Share on other sites More sharing options...
arilotta Posted January 11, 2018 Author Share Posted January 11, 2018 Hi Stefano, I see your concerns about scrolling (down) the grid and the consequent fetching of additional rows, but this should be limited in one of the following ways: - the users scrolls down the grid to the last row, and they can fetch and additional page (N rows) using a dedicated button "Load more rows" that could be placed on the navigation bar, instead of the current previous/next/first/last page buttons - the user scrolls down the grid to the last row, and automatically (ajax event ?) and additional page (N rows) is fetched from the opened query and added to the UniDbGrid So no way for the end user to move to the last row and fetch all the rows from the dataset as a whole. After a few page fetches without finding what they are looking for, the users would tipically refine the filter to get the data he really wants. Memory occupation, CPU, network usage minimized on the DB side and web server (UniGUI). Andrea Quote Link to comment Share on other sites More sharing options...
arilotta Posted January 17, 2018 Author Share Posted January 17, 2018 Hi all, I've developed a small app that shows better the idea, it is published at the following address (I will keep it online for a week): http://188.11.1.245:8746/FetchingDemo.dll - In the main page, it is possible to choose: - the grid page size - the maximum number of rows to fetch (this limits the rows manipulating the SQL in the query) - whether fetch all the rows or not - The actual table contains more than 6 million records - Clicking the "Open" button the secondary form with the UniDbgrid (FRMGrid) is opened with the settings specified in the main form - The grid contains a new button in the paging bar: "More pages..."; by clicking it an additional page (pagesize rows) is fetched from the opened query if available, and the grid and the paging bat is updated accordingly - The secondary form contains also a couple of buttons to show the rows fetched so far (Fetched row count) and the possibility to select a row given its absolute position (RecNo) Now let's tets the following scenarios: 1. pagesize:10; #rows: 100000; fetch all rows: TRUE This is the current suggested configuration for large datasets: pagination+fetch all. It takes more or less 5 seconds on my web server to fetch 100000 rows from the DB server, to paginate them and to send the fisrt page back to the browser. The memory consumption on the web server is huge, as all the 100000 are kept in memory. It is almost impossible to use this approach to fetch 1 million rows. 2. pagesize:10; #rows: 100000; fetch all rows: FALSE This is the implementation I'm suggesting for big datasets. It takes nothing to open the secondary form (in fact it is opened immediately), and the memory consumption is minimized as only the first 10 rows are fetched from the DB server. If the user wants to fetch additional pages, it can click on the "More pages..." button and fetch them one by one (it takes nothing to fetch additional pages, 10 at a time). Memory consumption is minimized on DB server and web server, network traffic highly reduced, web site more responsive.... In my original post, I told I would prefer additional rows fetched from the DB to be appended to the bottom of the grid rather than being put in an additional page, but I was not able to achieve this behaviour with my limited knowledge of Ext JS... For the moment, I think that this solution is acceptable for my needs. I would really like to know your opinion. Andrea Quote Link to comment Share on other sites More sharing options...
Servant5166 Posted January 22, 2018 Share Posted January 22, 2018 Hi all, I've developed a small app that shows better the idea, it is published at the following address (I will keep it online for a week): http://188.11.1.245:8746/FetchingDemo.dll - In the main page, it is possible to choose: - the grid page size - the maximum number of rows to fetch (this limits the rows manipulating the SQL in the query) - whether fetch all the rows or not - The actual table contains more than 6 million records - Clicking the "Open" button the secondary form with the UniDbgrid (FRMGrid) is opened with the settings specified in the main form - The grid contains a new button in the paging bar: "More pages..."; by clicking it an additional page (pagesize rows) is fetched from the opened query if available, and the grid and the paging bat is updated accordingly - The secondary form contains also a couple of buttons to show the rows fetched so far (Fetched row count) and the possibility to select a row given its absolute position (RecNo) Now let's tets the following scenarios: 1. pagesize:10; #rows: 100000; fetch all rows: TRUE This is the current suggested configuration for large datasets: pagination+fetch all. It takes more or less 5 seconds on my web server to fetch 100000 rows from the DB server, to paginate them and to send the fisrt page back to the browser. The memory consumption on the web server is huge, as all the 100000 are kept in memory. It is almost impossible to use this approach to fetch 1 million rows. 2. pagesize:10; #rows: 100000; fetch all rows: FALSE This is the implementation I'm suggesting for big datasets. It takes nothing to open the secondary form (in fact it is opened immediately), and the memory consumption is minimized as only the first 10 rows are fetched from the DB server. If the user wants to fetch additional pages, it can click on the "More pages..." button and fetch them one by one (it takes nothing to fetch additional pages, 10 at a time). Memory consumption is minimized on DB server and web server, network traffic highly reduced, web site more responsive.... In my original post, I told I would prefer additional rows fetched from the DB to be appended to the bottom of the grid rather than being put in an additional page, but I was not able to achieve this behaviour with my limited knowledge of Ext JS... For the moment, I think that this solution is acceptable for my needs. I would really like to know your opinion. Andrea Andrea, I like your solution. Can you post the code that would put the "More pages" button on the page bar and advance the dataset? Thanks in advance. Quote Link to comment Share on other sites More sharing options...
MOGSY Posted January 22, 2018 Share Posted January 22, 2018 Hi Stefano Could you please explain how to achieve the fetch you mentioned please. Thanking you in advance. Quote Link to comment Share on other sites More sharing options...
arilotta Posted January 22, 2018 Author Share Posted January 22, 2018 Glad to post the code (MOGSY, my name's Andrea :-) Here you can download the whole app: https://drive.google.com/file/d/1p09s2H3-gky58SZ8VglohjtZjJD81sM6/view?usp=sharing - to customize the grid (hide refresh button and add "More pages..." button) I've used some code that I've found on this forum and it's not mine: GRD.ClientEvents.UniEvents Ext.toolbar.Paging [pagingBar] pagingBar.afterCreate function pagingBar.afterCreate(sender) { sender.items.items[10].hide(); sender.add([{ xtype: 'button', text: 'More pages...', handler: function() { ajaxRequest(sender, 'morePages',[]); } }]); } - to support the incremental fetching your tdataset descendent (and your RDBMS) should support the following propertes: * FetchRows (or similar, number of rows to fetch each time) * RecordCount * RecNo I use Oracle and ODAC components from DevArt, in a previous post Volk65 said that even FireDac components should work, you can check the following link: http://docwiki.embarcadero.com/RADStudio/Berlin/en/Fetching_Rows_(FireDAC) Quote Link to comment Share on other sites More sharing options...
MOGSY Posted January 22, 2018 Share Posted January 22, 2018 Hi Andrea, Thanks you very much, much appreciated specially the fast response. regards Mogsy Quote Link to comment Share on other sites More sharing options...
Servant5166 Posted January 22, 2018 Share Posted January 22, 2018 Glad to post the code (MOGSY, my name's Andrea :-) Here you can download the whole app: https://drive.google.com/file/d/1p09s2H3-gky58SZ8VglohjtZjJD81sM6/view?usp=sharing - to customize the grid (hide refresh button and add "More pages..." button) I've used some code that I've found on this forum and it's not mine: GRD.ClientEvents.UniEvents Ext.toolbar.Paging [pagingBar] pagingBar.afterCreate function pagingBar.afterCreate(sender) { sender.items.items[10].hide(); sender.add([{ xtype: 'button', text: 'More pages...', handler: function() { ajaxRequest(sender, 'morePages',[]); } }]); } - to support the incremental fetching your tdataset descendent (and your RDBMS) should support the following propertes: * FetchRows (or similar, number of rows to fetch each time) * RecordCount * RecNo I use Oracle and ODAC components from DevArt, in a previous post Volk65 said that even FireDac components should work, you can check the following link: http://docwiki.embarcadero.com/RADStudio/Berlin/en/Fetching_Rows_(FireDAC) Andrea, Thank you so much, Sir! Quote Link to comment Share on other sites More sharing options...
Recommended Posts
Join the conversation
You can post now and register later. If you have an account, sign in now to post with your account.