+- TACTIC Open Source (http://forum.southpawtech.com)
+-- Forum: TACTIC Open Source (http://forum.southpawtech.com/forumdisplay.php?fid=3)
+--- Forum: TACTIC Discussion (http://forum.southpawtech.com/forumdisplay.php?fid=4)
+--- Thread: Thoughts about speeding up check-in process (/showthread.php?tid=156)
Thoughts about speeding up check-in process - listy - 07-21-2020
Hi All!
Me again, with the same annoying things as always
Thinking about speeding up check-in process i got some things that may help to get better user experience for web or other Ui's/api calls.
1: User should wait only for the actual Upload process to server. Forcing user to wait for all server-side stuff is bad experience.
2: If user want proper naming - request it before check-in (pretty fast and take about 100ms maximum).
3: Lock version and/or snapshot on server-side before checkin. For example, creating long-lasting template snapshot, analogue to virtual snapshot but not checked-in yet. Something similar - cnahge-lists in the Perforce.
This can resolve versioning mess, file you throwed and got naming already "waited" by server, and when file uploaded to server it can be processed properly.
Any more thoughts, Anyone?
RE: Thoughts about speeding up check-in process - remkonoteboom - 07-22-2020
There are really two aspects to this. 1) The speed of the check-in itself and 2) the perception to the user
The amount of time a check-in task should not really matter to the user if they don't perceive it. From any UI, this can be greatly mitigated by making sure that all check-ins are asyncronous (javascript) or in a separate thread (python). Now in the check-in widget, it seems that the uploads are asynchronous, but server.simple_checkin is not. For API calls, we have added methods preceeded by a "p_" which uses promises. There is no p_simple_checkin and there should. In this way, check-in could proceed and the use could continue working as soon a "publish" button is pressed while everything happens in the background. So the first thing to do is to add a p_simple_checkin function and to update the check-in widget to use those. In python, you should always check-in using a separate thread and provide the user progress.
As you noticed in your comment in GitHub, the largest 3 factors in the speed of the check-in are:
1) upload time
2) generation of thumbnails
3) md5 generation
There is nothing we can do about #1 from a TACTIC point of view. In TACTIC, the table generates thumbnails dynamically as needed. The reason for this was because we encountered many situations where a large number of files were ingested and thumbnail generation took an enormous amount of time. So we delayed generation of these until a user actually needs to see the thumbnail.
The MD5 generation was previously used for interfaces to be able to "quickly" determine if a file that was about to be checked in was actually different that the last version. There is a issue in Github to make this optional.
RE: Thoughts about speeding up check-in process - listy - 07-22-2020
Hi! Thanks for Your answer!
I indeed use Python threads, but there is more bottlenecks. I'll try to describe:
When we do Upload we start and finish transaction - it takes time, on a 2.5ghz > 30ms~ each.
Doing 10 checkin in a row, like 10 files in the same process, only version changes, in this case i HAD to wait until all server routine done. Except if i force 10 files simple_checkin() simultaneously i will get total mess with their versioning.
Server cores. Doing 8 checkins on a server with 4 cores will create more bottleneck than checkin them one by another, so if server has it's own queue it wouldn't be the problem.