+- TACTIC Open Source (http://forum.southpawtech.com)
+-- Forum: TACTIC Open Source (http://forum.southpawtech.com/forumdisplay.php?fid=3)
+--- Forum: TACTIC Discussion (http://forum.southpawtech.com/forumdisplay.php?fid=4)
+--- Thread: Is there a way to speed up tactics? (/showthread.php?tid=157)
Pages:
1
2
RE: Is there a way to speed up tactics? - kanta - 07-24-2020
(07-23-2020, 04:52 PM)remkonoteboom Wrote: I am sorry, kanta. I mean to reply to your post, but accidentally overwrite it. There is no way to undo. At any rate, my response is repeated here:
Quote:I also tried'load balancing'.
What was the result of this? It should look something like the attached.
Nothing happens when I hit the'load balancing' button.
There is no text like the image you attached.
[root@q361-0706 ~]# top
top - 14:13:53 up 80 days, 21:58, 1 user, load average: 0.62, 0.36, 0.24
Tasks: 159 total, 2 running, 157 sleeping, 0 stopped, 0 zombie
%Cpu0 : 0.0 us, 0.0 sy, 0.0 ni,100.0 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0 st
%Cpu1 : 0.0 us, 0.0 sy, 0.0 ni,100.0 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0 st
%Cpu2 : 0.0 us, 0.0 sy, 0.0 ni,100.0 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0 st
%Cpu3 : 88.8 us, 11.2 sy, 0.0 ni, 0.0 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0 st
KiB Mem : 16086852 total, 3333240 free, 2600172 used, 10153440 buff/cache
KiB Swap: 4194300 total, 4178940 free, 15360 used. 12207224 avail Mem
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
7638 tactic 20 0 2928376 705844 9624 S 100.0 4.4 1234:05 python3
7456 tactic 20 0 2998552 839300 9596 S 1.3 5.2 1240:44 python3
I think the reason of slow Tactic is Tactic process which executed by python3 /opt/tactic/tactic/src/bin/startup.py 8082 reached to CPU 100%. Is there possible to be slow due to the carrying out Tactic inner code?
RE: Is there a way to speed up tactics? - remkonoteboom - 07-24-2020
I am not sure what that python3 process is doing. But, it is a clue.
1) Does it stay at 100% always or just when you pressed load balancing?
2) Are others using the system while your are looking at this?
If 100% on that single CPU is continuous, that means everyone, including your load balancing test, is bottlenecked through a single CPU. My guess, at the moment, is that load balancing is not on. If this is the case, then, you are not utilizing all of your CPUs and would explain the slow behavior.
Can you do:
Quote:# ps -ef | grep python
Which produces something like:
Quote:tactic 335 24850 0 15:08 ? 00:00:01 python3 /opt/tactic/tactic/src/bin/startup.py 8081
tactic 765 24850 0 15:10 ? 00:00:01 python3 /opt/tactic/tactic/src/bin/startup.py 8082
tactic 1078 24850 0 15:11 ? 00:00:01 python3 /opt/tactic/tactic/src/bin/startup.py 8083
root 24843 1 0 Apr21 ? 00:00:00 su - tactic -m -c /usr/bin/python /opt/tactic/tactic/src/bin/monitor.py
Also, can you post the load balancing part in the TACTIC apache conf file, which is in: /etc/httpd/conf.d/tactic.conf
It should be something like:
Quote:# Using the ProxyPass directives
<Proxy balancer://tactic>
BalancerMember http://localhost:8081/tactic
BalancerMember http://localhost:8082/tactic
BalancerMember http://localhost:8083/tactic
</Proxy>
ProxyPass /assets !
ProxyPass /context !
ProxyPass /doc !
ProxyPass /tactic balancer://tactic/
ProxyPass /projects balancer://tactic/
ProxyPass / balancer://tactic/
RE: Is there a way to speed up tactics? - EricTsuei - 07-30-2020
I am running at a 104 core 64GB server, using windows.
the speed seems same as your speed. it's really slow.
I also increase the balancer member to 40, it's better while you have multiple users. but it used so much memory.
but each user load pages is still slow. but it's more stable with more balancer member.
RE: Is there a way to speed up tactics? - remkonoteboom - 07-30-2020
Is that correct? 104 Cores? And with 40 load balanced processes? That is a huge amount. I know you have a lot of projects going, but this is still quite a lot of processes.
Would you be able to post a view of the Windows Task Manager on the server? I would really like to know what the bottleneck may be.
RE: Is there a way to speed up tactics? - EricTsuei - 07-31-2020
(07-30-2020, 03:58 PM)remkonoteboom Wrote: Is that correct? 104 Cores? And with 40 load balanced processes? That is a huge amount. I know you have a lot of projects going, but this is still quite a lot of processes.
Would you be able to post a view of the Windows Task Manager on the server? I would really like to know what the bottleneck may be.
sure,
I am now running in ESXi virtual machine, and assign 96 core to windows server, save 8 core for stable running ESXi.
[attachment=70]
[attachment=71]
[attachment=72]
[attachment=73]
RE: Is there a way to speed up tactics? - remkonoteboom - 07-31-2020
To be honest, I have never seen an installation that required so many processes. This includes full scale CG feature films and multiple simultaneous full episodic television projects. Of course, you may have a very large number of users that simultaneously hit the system or are running large number of automated processes.
When under load, how many of those have python processes at 100%?
RE: Is there a way to speed up tactics? - EricTsuei - 08-03-2020
NO, I am just doing this to see if this can help improve user load speed.
but, seems it's more stable, but won't get more speed.
and we are not so large scale studio, but we are producing 2 CG feature films and 4 other TV episodes.
RE: Is there a way to speed up tactics? - kanta - 08-04-2020
According to your help, the speed of Tactic problem has solved.
The problem was considered due to the speed of Tactic engine, however, it was setting issue. Current loading speed is just 2sec after modifying setting. (previous speed was 15sec)
I really thank you for your help.
I shared this problem and solution if it would be useful to other users who has same issue.
------------------------------
After modification (there was no change in the computer hardware)
-----------------------------------------
Modification
1.
the Python processes are running for a long time , and it's causing sluggishness. We set the process_time_alive setting to 30 in /opt/tactic/tactic_data/config/tactic-conf.xml to prevent this in the future:
<process_time_alive>30</process_time_alive>
It means the processes will be refreshed every 30 minutes automatically. Users will not notice.
2.
We also increased load-balancing to 6 TACTIC processes (previously it was 3) since your machine had enough capabilities.