Check stuck thread in weblogic

Check stuck thread in weblogic. Possibly due to stuck threads. This is not intended to be a comprehensive guide to all Stuck Thread issues. * Stuck Thread Timer Interval: Enter the number of seconds, after which WebLogic Server periodically scans threads to see if they have been continually working for the length of time specified by Stuck Thread Max Time. Similarly, if statistics indicate that fewer threads did not reduce throughput, WebLogic Apr 9, 2019 · 1. May 15, 2020 · How Weblogic determine a thread to declare as hogging? A thread declared as Stuck if it runs over 600 secs (default configuration which you can increase or decrease from admin console). Nov 25, 2014 · Weblogic monitors its threads and marks them as "stuck" after a certain period of time if they are not returning from a method like in a socket listener loop in thread's run method. By default, WebLogic Server considers a thread to be "stuck" after 600 seconds of continuous use. Expand Environment and click on Servers in the left menu; Click on the Admin server or any managed server you want to check the stuck threads; Click on the Monitoring tab in the server properties; click on WebLogic Server automatically detects when a thread in an execute queue becomes “stuck. i. It notifies and attaches thread dump. cmd or setDomainEnv. On EBS 12. Increased database connection pool. Consider Paul's rocket mass heater . x find that some or all the OACore Managed servers are frequently going into Warning status with struck threads. For example, the following thread is stuck waiting to process POST data: Cause Nov 26, 2011 · I've got the below code working on Weblogic Application Server 10. We are facing issue in our production environment with multiple blocking threads and while checking the thread dumps have found the below stack trace where in 14 blocking threads were present with same description. This ensures that proxy and business services should not depend on the WebLogic Server execute thread pool for execution and WebLogic Server has enough free or idle threads to process any ad-hoc requests from Service Bus without leading to stuck thread problems. Due to this issue, users see a lot of performance issues and sometimes cannot even access the EBS instance when this issue Jul 28, 2015 · If WL console is unresponsive then it means that you Admin server thread is stuck, so you need to kill you Admin server Java process and restart it . (StuckThreadMaxTime) of “600” seconds in “server-failure-trigger”. 0): setDomainEnv. 3. I'm using WebLogic 11g. Default (self StuckThreadMaxエラー. Jan 5, 2021 · In Weblogic, how can I change Work Manager's Stuck-Thread-Action using WLST and Jython. Use WLST and threadDump() The first option has usually my preference. 3. You can tune a server’s thread detection behavior by changing the length of time before a thread is diagnosed as stuck, and by changing the frequency with which the server checks for stuck threads. It may be a small cluster of threads or all of them. 0 and later: Stuck Threads Issue in WebLogic. java weblogic. Apr 19, 2013 · at oracle. In order to get this fixed you are trying to increase the thread pool following the steps as per the document in the link below. Maximum Capacity: from 15 to 500. Aug 31, 2012 · outputStream. on JDBC tab make sure all datasources are in RUNNING state. To configure WebLogic Server stuck thread detection behavior. By Default, the WLS For example, if you have four processors, then four threads can be running concurrently with the number of stuck threads. Nov 26, 2011 · Does this message imply that my thread is aborted, or can it continue after this ? <[STUCK] ExecuteThread: '0' for q ueue: 'weblogic. Edit the config. To determine the amount of stuck threads, see Detecting "Stuck" Threads. flush(); The problem we're seeing in WebLogic is lots of STUCK threads, so we dumped the thread stack and found some threads getting stuck at org. 11 1. There could be situations where these the requests waiting in queue could take up to 30-40 minutes to be Oracle Business Intelligence. First, set the use81-style-execute-queues element to true in the configuration file. connector. so at the end you will have a single log file which has around 20 - 25 seconds worth of action on the app server. xml file and reboot the server. 205. Tuning the Execute Thread Detection Behavior. If the Configurator does not respond, restart it. This document provides comprehensive guidance on how to tune WebLogic Server parameters, monitor its health, and troubleshoot common problems. Make sure you export weblogic. When facing slowdown conditions, you will normally see May 10, 2023 · In this Document. EDITED: Another solution which is useful when you are trying to read from a socket and you are not getting any response from the server is the following: An RMISocketFactory instance is used by the RMI Work Manager minimum and maximum constraints should be set based on the maximum load. To launch a long running task, you need to set the LONGRUNNING_HINT to true in the ManagedTask that is launched. If you want a thread to run till the server or application stops, you have to mark your class with a @LongRunning (weblogic. 1, there are 3 socket reader threads by default. The stack-trace in the log file looks as follows: <Error> <WebLogicServer> <BEA-000337> <[STUCK] ExecuteThread: '12' for queue: 'weblogic. and it looks like Dyntrace doesnt see this in its Weblogic JXM editor WebLogic Server automatically detects when a thread in an execute queue becomes “stuck. 7. The application is shutdown by bringing it into admin mode. Note: This recommendation is highly application-dependent. Also, consider setting the WebLogic Server “Stuck Thread Max Time” value from its default 600 seconds to a larger value such as 1200 seconds. Becouse your processing takes 20min and default setting of StuckThreadMaxTime is set to 600 seconds, you will see such errors in logs. e stuck status=false like below. By Default, the WLS comes with 600 secs. At the time of stuck threads. Goal. xml file, setting the use81-style-execute-queues element to true. set StuckThreadMaxTime with bigger value (its placed on weblogic console in: domain > Environment > Servers > Admin Server > Configuration/Tuning) edited Mar 11 Jul 1, 2018 · 0. In this video, we will cover Troubleshooting of weblogic server, running with Warning HealthCheckout my below videos as well :Weblogic HTTP Extended Logging Jan 8, 2012 · There are many ways to do that. Default execute queue from the Administration Console and reboot the server. The only thing you can do is to figure if you can somehow detect such situations while they occur to then collect thread dumps and to log the results. For example: When large BI report is running and the time it Nov 6, 2009 · We are able to configure by using "Work Managers" in WebLogic and putting a limit on the number of threads for that servlet. rmi. Jan 15, 2019 · The following thread was extracted from a thread dump taken on a JVM instance running WebLogic Server. ke. Does it show the sockets in ESTABLISHED state? or some other state like TIME_WAIT or CLOSE_WAIT ? How long have you configured your stuck-thread-timeout? The default is 600 seconds, so does this operation take longer than 10 minutes? Sep 3, 2013 · Refresh the page every 3-5 seconds. Check JMS connections are not too high. Mar 16, 2012 · After that change the Weblogic system status is complaining about the thread pool "Thread pool has stucked threads". Jun 29, 2018 · One way to check if you have stuck threads is through Weblogic Admin console. Admin -url t3://weblogicadminconsoleurl:7001 -username username -password password THREAD_DUMP Weblogic Admin Console. Stuck Threads are threads that are blocked, and can't return to the threadpool for a certain amout of time. However, it also requires careful tuning to optimize its performance and avoid issues such as stuck threads, deadlocks, or memory leaks. x domain, you need to include the use81-style-execute-queues sub-element of the server element in the config. All Work Managers belonging to the application are shut down, and behave as described above. I suggest to investigate why the thread is taking so long and if it's possible to. In this post I will deconstruct this thread and describe the data it contains and the potential issues it may illuminate. For example, if you have four processors, then four threads can be running concurrently with the number of stuck threads. Then take the decision according to it. 3) Optimize the program and reduce the processing time. I do not w To use user-defined execute queues in a WebLogic Server 10. In WL 9 and 10, WebLogic allocates 33% of server threads to act as socket readers by default. 0. If you need to script some kind of monitoring tool, WLST is extremely powerful. Note: This recommendation is highly application-dependent Jan 22, 2024 · Oracle WebLogic Server - Version 12. For example, if we put maximim thread limit as 3, then if there are 10 parallel requests; 7 requests are in queue. Jan 16, 2023 · Weblogic Admin utility. ). Step 1: Go to the weblogic console and check for managed server status like oacore. Default value for “StuckThreadMaxTime” is 10 minutes. servlet. Info: maximum thread constraint constraintName is reached Description: maximum thread constraint constraintName is reached. 163. 6 and later: How Stuck Thread Timeout and JTA timeout values are related? Mar 5, 2024 · Oracle WebLogic Server - Version 12. For example you could use the nagios plugin check_http to check the status of a particular server. Stuck Thread Handling WebLogic Server checks for stuck threads periodically. OBI 12C Weblogic and Em not opening. You can read more about how weblogic 12 handles threads here: https://docs. 解決策: 次の手順に従って、「スタック・スレッド最大時間」設定の値を大きくします: WebLogicコンソールにログオンします If the WLSAgent cannot be used then NCPA will need to be installed on the WebLogic server as per the Installing NCPA documentation. The definition of "world" depends on the garbage collector being used and context. During a GC sweep, the GC will "stop the world" to perform its cleanup. Oct 12, 2012 · If you see Stuck threads like below in the WebLogic server log then follow the steps mentioned to fix this issue. If possible I would try: Find which query or queries are getting the threads stuck and check execution time. out in the Dec 22, 2019 · 3. extensions. WebLogic Server automatically detects when a thread in an execute queue becomes "stuck. ” Because a stuck thread cannot complete its current work or accept new work, the server logs a message each time it diagnoses a stuck thread. do", which is more than the configured time (StuckThreadMaxTime) of " 600" seconds Aug 3, 2023 · Oracle WebLogic Server - Version 10. Jul 5, 2017 · 3. Aug 16, 2017 · HI All ,we are facing the thread stuck in the web logic frequently and we have to restart the planning services to resolve the issue. Changes. Admin url –: -username -password THREAD_DUMP. oracle. – > Stuck Threads are only flags, there to warn you that this thread is taking too long. Enter a title that clearly identifies the subject of your question. Download samurai which checks for the blocked threads , deadlock threads . Before officially collecting garbage, GC will invoke the object's finalize (). OS: Centos Weblogic: WebLogic Server Version: 10. This need not be changed usually. In you have stuck threads but the WebLogic Console is still available, you can go to Environment, Servers and select a server. When the system is under stress, STUCK threads are seen in WebLogic logs at regular intervals. We are using weblogic 10. jdbc. WebLogic Serverは、設定した期間、作業状態が継続した(アイドルでない)場合、スレッドをスタック状態と診断します。 サーバーのスレッド検出動作は、スレッドがスタック状態と診断されるまでの時間や、サーバーがスタック・スレッドをチェックする頻度を Dec 25, 2020 · Is Weblogic 12c ThreadPoolRune JMX Counter available in Dyntrace v1. If you have not already done so, stop the WebLogic Server instance. Services > Data Sources > EPMSystemRegistry > Configuration > Connection Pool. Scenario 1: Every alert we need to check the log file and investigate based on Jul 10, 2018 · I could see the health show "Warning" and the reason "ThreadPool has stuck threads". ServletRequestImpl Jul 24, 2020 · Stuck threads will reduce the ebs performance and it’s very important activity to DBA’s. 6. Problem can happen any time and any load condition, with no pattern to the occurrences. Aug 3, 2023 · In this Document. Use jstack <pid> (or jstack -F <pid> if the process is hung) Use VisualVM. Note that it is possible that some of your threads are doing legitimate work for over 10 minutes with Mar 27, 2020 · # Node Manager needs to be running to run this script. Here check for “Heap Free Percent” (It should be more percentage free) Also, check for count of the “Hogging Thread Count” which should be either 0 or very less. 1. This note is intended to be one example of how to identify the cause of a "STUCK" thread in the WebLogic Server or Managed Server. Your application threads are probably started like this: new Thread(new MyThreadClass())). sh depending on the OS. it will give output like below. internal. Then, explicitly create the weblogic. Select the Configuration -> Tuning tab in the right pane. We are using a builtin feature ' Use JMX/PMI extension editor to add custom metrics without coding. LongRunning Dec 16, 2023 · Click to get started! My Oracle Support provides customers with access to over a million knowledge articles and a vibrant support community of peers and Oracle experts. If the server is close to out of memory, all requests should slow down. May 17, 2009 · Socket Reader Threads accept the incoming request from the Listen Thread Queue and put it on the Execute Thread Queue. In the body, insert detailed information, including Oracle product and version. rnel. You can configure Node Manager or a third-part high-availability solution to restart the server instance for automatic failure recovery. No, you can't. from here. Mar 8, 2024 · Oracle WebCenter Content: Imaging - Version 12. It could be that the stuck threads are generated by a problem in the code, by the unavailability of an external system, or it can even be expected behavior, in the case of some long-lasting operations in the Nov 7, 2016 · I'm having a Weblogic server with one Admin Server and 3 managed servers in a single cluster. admin) is aware that the thread has crossed 10 mins (600 sec = 10 min, not 15) This is a configurable value. Worst case you have to give the customers/operators clear instructions how to gather dumps themselves to make them available to you. Default (self-tuning)' has been busy for "641" seconds working on the request Feb 21, 2014 · 2. msgpack. 0 version and JDK 1. References. 2. Then you should look for you want to check is a stuck thread or long running Apr 19, 2011 · 1. Stuck threads may result if the server is nearing out of memory. Potentially, you could go to the weblogic admin console and restart the application. Any help with the below is much appreciated. Check if the finalizer thread is blocked or waiting. To determine the amount of stuck threads, see Tuning the Execute Thread Detection Behavior. lookup(TemplateRegistry:198), see dump below. Jan 29, 2015 · Tuning = stuck thread reporting. One of the them is working fine. Step 2: if It will show the oacore server in “warning state” then we need to check. <Error> <WebLogicServer> <BEA-000337> < [STUCK] ExecuteThread: ‘1’ for queue: ‘weblogic. Problem. “StuckThreadMaxTime” parameter). ") #11g #webcenter #weblogic #weblogic-portal. Oracle WebLogic Server - Version 12. e. Send a SIGQUIT, either via CTRL + \ ( CTRL + Break on Windows) or with a kill -3 <pid>. Mar 9, 2023 · This MBean provides information about the server’s state and performance, including information about threads. If all application threads are stuck, a server instance marks itself failed, if configured to do so, exits. We need to monitor stuck thread count. ' to get the hung threads for the traditional WebSphere. Aug 17, 2011 · In threads tab make sure health is OK and there are no stuck threads i. Dec 15, 2017 · In general no, but if for example the stuck thread is holding a lock on an object (database, file, etc. I'm getting stuck threads intermittently in one of the managed servers. Stuck Thread Timer Interval: Enter the Dec 24, 2012 · Alternatively you can use something as simple as a http check on the port of the server to see if it responds. kernel. It cannot do so, the message is only a status info so that you (i. Take 4 or 5 sets of thread dumps at an interval of 5 seconds between each. answered Apr 21, 2015 at 21:32. The timer starts when the thread begins processing the request within the server. Mar 12, 2019 · In this post, we are going to share the Weblogic WLST Stuck Thread Monitoring Script which will trigger an email notification and create heap and thread dumps when the maximum Thread stuck count has reached the threshold. clientTimeout setting in order to avoid thread stuck situations when the target server is experiencing problems. at the the time we can login to the particular warning server. Can anyone please help me in understanding, why thread dump does not get generated from console and Why I am getting the message saying server must be running. One thread dump is not enough to analyze the problem. The long running task executed on timerExpired takes longer than the server wide StuckThreadMaxTime of 600 seconds. We are seeing server overloading (network IO), admin inaccessible errors. Feb 23, 2023 · Symptoms. jar in your path; Execute a command on UNIX session: java weblogic. Jul 20, 2017 · A stuck thread does not always mean that the thread is waiting for something. The thread dump will be generated in the defined server stdout. Is it possible to monitor web sessions and Stuck threads through the WebLogic Admin Console? Solution The execution time of SQL statement is too long. java:228) Maybe it is just the query which is taking more than StuckThreadMaxTime and WL issues a Warning. driver. Also, Windows XP is eol and you know it. In WL 8. As per my best knowledge you cannot do it without re-boot . The queue monitors throughput over time and based on history, determines whether to adjust the thread count. For example, if historical throughput statistics indicate that a higher thread count increased throughput, WebLogic increases the thread count. One socket reader thread is usually in the To run WebLogic Server in the 81-style-execute-queues mode, and be able to tune the thread count, an additional step is needed. If a request is taking longer than 10 minutes, the stuck thread is reported to Oracle WebLogic Server server_name. Mar 8, 2016 · But, when I try to generate thread dump using kill -3 <PID>, it gets generated. Cause. Third, Check the Performance and Thread of the Managed Server. This can be determined if you see the Nov 23, 2015 · WebLogic Console. Servers -> Some_Server -> Configuration -> Tuning -> Stuck Thread Max Time This will check on the Stuck Thread Timer Interval for any and all stuck threads and report it in the log file for the server like: 'WebLogic. Now you can go to Monitoring, Threads. OracleResultSetImpl. How to check Stuck Threads in logs – Doyensys Blog. Increased Max Stuck Thread. we can see the servers are in warning state. 2) Increase the number of threads to prevent thread blocking. Also you can request a dump of the Thread stacks. Is there any way to kill these threads which got stuck? I don't want to hinder the activities of the users in another managed servers. Default (self-tuning)' has been busy for "zzz" seconds working on the request "-----" , which is more than the configured time Oracle WebLogic Server is a powerful and scalable application server that can run various applications and services. next(OracleResultSetImpl. This post contains a sample script that sends an alert email whenever stuck threads exist, and in the comments section you will find some samples of Dec 14, 2019 · WebLogic Stuck Threads: Understanding Hogger and Stuck threads Stuck Threads are threads that are blocked, and can’t return to the threadpool for a certain amout of time. [solution]: It can be solved by the following methods: 1) modify the stuckthreadmaxtime parameter, change the default 600s to 1200s, or other suitable values. 0 Weblogic server tries to suppress current working thread and begins processing the new request. Also, depending on what the stuck thread is doing, it may use excessive resources (cpu, memory, disk, etc. 6 and later: Stuck Threads in WebLogic Server. Feb 12, 2016 · 1. I tried to get the stuck thread count of To use user-defined execute queues in a WebLogic Server 10. 2. then check for the blocked threads or any deadlock threads. For instance, the The first thing to check when there are Stuck Threads, is to generate a Thread Dump and analyze the threads that are being stuck. Otherwise, you might have to use your own custom-built OneAgent extension. I have recently configured obi 12c application ver. 6 Information in this document applies to any platform. but the 2nd one. WebLogic stuck thread protection. getState() print ‘Status of Jul 21, 2020 · When stuck thread increase in WebLogic Server alarm will be triggered. for key in result: print (key + " has " + str (result [key]) + " stuck threads. In between the refresh actions, identify the threads that are still executing the same request (slow running threads). serverRuntime = getMBean('/ServerRuntimes/server1') Get the thread pool MBean: Next, you can use the getThreadPoolRuntime() method to get the MBean for the thread pool. ) called by other requests, these may be affected too. WebLogic Server diagnoses a thread as stuck if it is continually working (not idle) for a set period of time. Apr 24, 2013 · You can access the thread pool health details using WLST (WebLogic Scripting Tools) scripts (if you are familiar with Python, it should not be a problem), or by accessing JMX counters using Java. – May 31, 2012 · Shut down the application if there are stuck threads in the application. Dec 8, 2017 · For better clarification: BEA-002936. You can try WLSDM. 0_80. sh plugin from the Nagios XI server to your WebLogic server. The thread dump you have provided seems to be just fine, the one thread marked stuck seems to one where you are trying to uninstall a composite, something uninstall does take time and cause 1 thread to be stuck, this has Dec 26, 2012 · STUCK: A stuck thread is identified by Weblogic when it is taking more time than the configured stuck thread time (default is 600 seconds). Login to weblogic admin console with admin credentials. we will get alerts. 0 and later Information in this document applies to any platform. One set of thread dumps alone wont be too helpful to get to the root cause. You can try this feature in case it is available for Weblogic. Symptoms. Mar 17, 2022 · It seems you are looking for the hung threads. Mark the server instance as failed and shut it down it down if there are stuck threads in the server. '19' for queue: 'weblogic By default, WebLogic Server considers a thread to be "stuck" after 600 seconds of continuous use. Michel. I've done some changes as per some blogs suggest: 1. The following commands executed on your WebLogic server as the root user will do this: Aug 20, 2012 · You could use the weblogic. ## Format for Nagios output etc. To resolve an out-of-memory issue, see Section 24. From a command line or shell, a thread dump can be generated via the following command (deprecated from WLS 9. In work load tab make sure pending requests are not too many, else you need to look at the number of work managers and there health and work done. com Aug 18, 2023 · Oracle WebLogic Server - Version 12. Jun 13, 2016 · Is the client on Weblogic or the web service hosted on Weblogic or both? Can you check netstat -an | grep . " Because a stuck thread cannot complete its current work or accept new work, the server logs a message each time it diagnoses a stuck thread. Also weblogic won't consider a thread assigned to a long running task as being stuck. 3 Stuck Threads. Apr 26, 2010 · Weblogic does NOT kill stuck threads after the StuckThreadMaxTime. i am trying to open weblogic and Em console and its keeps loading forever ( though it had opened few days before while May 17, 2017 · The weblogic console only shows threads that it (the weblogic server) created and therefore monitors. NOTE: Server is in running state. Firstly never troubleshoot stuck threads using only 1 dump, always take 3-4 dumps each spaced about 3-4 secs apart. Take several thread dumps to check what threads are doing. Cause: The number of threads assigned to this work manager has reached the configured value in the maximum threads constraint Feb 17, 2021 · redirect ('/dev/null','true') ## Print all server names and the number of stuck threads we counted per server. Oracle WebLogic Server - Version 10. Sep 25, 2023 · warning Symptom{STUCK_THREADS,HIGH, ThreadPoolRuntime, ThreadPool has stuck threads} Weblogic Logs: <[STUCK] ExecuteThread: '7' for queue: 'weblogic. ADF application is deployed in this cluster. Mar 11, 2016 · 3. If you have alarm from WLSDM stuck thread metric, follow the below instructions: · Quick health check of application server. My Oracle Support provides customers with access to over a million knowledge articles and a vibrant support community of peers and Oracle experts. This will put minimal load on the server. 5 to 10. 12. 4. Use the WebLogic Console to change this value (base_domain > Environment > Servers > Admin Server > Configuration/Tuning). Default (self-tuning)’ has been busy for “631” seconds working on the request “weblogic. 20201201-130524 . 0 and later: Frequent Stuck Threads In Imaging ExecuteThread: '31' for queue: 'weblogic. 問題: Account Reconciliation Managementが 「構成された時間 (StuckThreadMaxTime)」 を示すメッセージでタイムアウトします。. Default (self-tuning)' has been busy for "609" seconds working on the request "Http Request: /getSunPayment. Namrta. Gone is gone, end of story. template. So, you want the execute threads to be 4 + the number of stuck threads. Dec 16, 2023 · Click to get started! In this Document. 24. May 13, 2018 · An execute thread is classified as STUCK when the thread is stuck working on a request for more than the configured stuck thread maximum time (i. It can simply takes a long time to execute the request. Thread dumps show following stuck threads: Changes Cause . 1. grep -i stuck logname. Login into WebLogic admin console; Navigate to Server > Monitoring > Threads > Dump Thread Stacks; WLST Aug 15, 2011 · 8. 0 with two domains. If some thread doesn't return in 600 secs, It gets a flag 'stuck thread'. just drap the thread dump file into the samuria console . Stuck Thread. Click on Environment --> Servers --> Managed Server --> Monitoring --> Performance. WebLogic Server is in unresponsive state and health of the server is in warning state. import thread import time def checkHealth(serverName): while 1: slBean = getSLCRT(serverName) status = slBean. Stuck Thread Max Time: Enter the number of seconds, that a thread must be continually working before this server diagnoses the thread as being stuck. A thread might get stuck due to various reasons. You have two options: make processing faster. . Here you can look at threads and identify stuck and hogging threads. 4 Aug 7, 2023 · Oracle WebLogic Server - Version 12. TemplateRegistry. We're confident our code did not introduce this issue since, in the example above, it's clear we're using Apr 8, 2022 · I think this will cause weblogic to create a new thread for the task instead of taking a thread from the work manager thread pool. Once installed you will also need to copy the check_wlsagent. Default (self-tuning)' has been busy for "626" seconds working on the request "Workmanager: default, Version: 0, Scheduled=false, Started=true, Started time: 626850 ms", which is more than このエラーは、WebLogic Server で「Stuck Thread Max Time:」のデフォルト値 600 秒を超過したために発生します。 回避方法: コンフィギュレータが応答しない場合は、再起動します。WebLogic Server の「Stuck Thread Max Time」値をデフォルトの 600 秒から 1200 秒などもっと Mar 21, 2019 · 0. 21. Select a discussion category from the picklist. Solution. Aug 13, 2012 · The WebLogic Diagnostic Image includes a thread dump and can be automatically fired upon certain JMX metric thresholds being crossed by configuring a Watch and corresponding Notification in a WebLogic Diagnostic Framwork (WLDF) System Module that is targeted at your server. Related questions. start(), this does not use any weblogic API and are unknown to it. qu ep an vz gf rx cj kn zj ye