DS WhitePapers 3DEXPERIENCE Platform Online Monitoring and Maintenance [PDF]

Zitiervorschau

Best Practices

3DEXPERIENCE Platform

© Dassault Systèmes | Confidential Information| ref.: 3DS_Document_2020

Online Monitoring and Maintenance

Version 1.0 - 6/9/2022 Written by: Jérôme LESOURD, Rajendra NENE, Swapnil MASUTAGE Validated by: Jérôme LESOURD, Fabien RAYNAUD, Nilesh PATIL Edited by: Arati DESAI 3DEXPERIENCE Platform R2018x

Best Practices

Executive Summary This document covers monitoring and alert configuration for different services of 3DEXPERIENCE platform. It also provides database maintenance information. This best practice is applicable to 3DEXPERIENCE Platform R2018x and above releases.

© Dassault Systèmes | Confidential Information | ref.: 3DS_Document_2020

Target audience: Site IT, Software Architect, and Administrator who want to monitor 3DEXPERIENCE platform On-Premise Installation.

Confidential information. Copyright Dassault Systèmes [2018 - 2022]. This document is provided for information purpose only and may be used by authorized users only for their own internal use. Any other use without prior written authorization from Dassault Systèmes is strictly prohibited, except as may be permitted by law.

2

Best Practices

Table of contents Monitoring the 3DEXPERIENCE Platform ...................................................................... 5

1.1.

Prerequisites for Monitoring ........................................................................................ 5

1.1.1.

Activating the Monitoring Agent for Health Check ................................................... 5

1.1.2.

Preparing HTTP Layer Logs ................................................................................... 6

1.1.3.

Using Apache httpd Logs ........................................................................................ 6

1.1.4.

Apache Monitoring .................................................................................................. 8

1.1.5.

Apache Logs Rotation............................................................................................. 9

1.1.6.

Configuring Apache tomcat Access Logs .............................................................. 10

1.1.7.

Allowing JMX Connection to Retrieve the MBeans Objects................................... 13

1.1.8.

Capturing Long Running Java Threads using the Monitor Server Command ........ 13

1.1.9.

3DEXPERIENCE Service Logs Rotation............................................................... 14

1.2.

Machine Monitoring .................................................................................................. 15

1.2.1.

Windows Errors with the Event Logs ..................................................................... 15

1.2.2.

Windows Resource Monitoring.............................................................................. 16

1.2.3.

Linux Resource Monitoring ................................................................................... 19

1.3.

Configuring Alerts for the Operating System (O.S.) .................................................. 23

1.4.

HTTP Monitoring ...................................................................................................... 27

1.4.1.

Comparing Total Number of HTTP Errors with Total Number of Requests............ 27

1.4.2.

HTTP Concurrency Monitoring .............................................................................. 29

1.5.

Java Monitoring with AdoptOpenJDK ....................................................................... 30

1.5.1.

AdoptOpenJDK Open J9 Diagnostic Data and Tools ............................................ 30

1.5.2.

Generating Dump Using Eclipse Open J9 Option Builder...................................... 32

1.5.3.

Java Heap Memory Monitoring ............................................................................. 35

1.5.4.

Java Live Thread Monitoring ................................................................................. 36

1.6.

C++ (3DSpace Kernel) Monitoring ............................................................................ 37

1.7.

Database Monitoring ................................................................................................ 39

1.7.1.

Monitoring the Number of Connections Exceeded ................................................ 39

1.7.2.

Monitoring Number of SQL Parsed / Executed...................................................... 40

1.7.3.

Capturing TOP SQL Queries and Metrics ............................................................. 41

2.

3DEXPERIENCE Platform Applicative Monitoring........................................................ 47

2.1.

Monitoring DSLS Usage ........................................................................................... 47

2.1.1.

Getting Information About Licensed Feature Usage .............................................. 48

2.1.2.

Getting Information About Licensed Feature Usage Per User ............................... 49

Confidential information. Copyright Dassault Systèmes [2018 - 2022]. This document is provided for information purpose only and may be used by authorized users only for their own internal use. Any other use without prior written authorization from Dassault Systèmes is strictly prohibited, except as may be permitted by law.

3

© Dassault Systèmes | Confidential Information | ref.: 3DS_Document_2020

1.

Best Practices

2.1.3.

Tracking License Statistics.................................................................................... 50

2.1.4.

Monitoring the License Server .............................................................................. 51 Monitoring 3DPassport Services............................................................................... 52

2.2.1.

Monitoring User Login/Logout using 3DPassport Audit logs .................................. 52

2.3.

Monitoring 3DDashboard Service ............................................................................. 53

2.4.

Monitoring 3DSpace Service .................................................................................... 54

2.4.1.

Setting Dedicated 3DSpace Instance for Trace/Logs Collection............................ 54

2.4.2.

Monitoring Thread Dump and Process Dump ....................................................... 55

2.5.

Monitoring 3DSpace Exceptions Number ................................................................. 56

2.6.

Monitoring Logs ........................................................................................................ 57

2.7.

Monitoring 3DSearch Service ................................................................................... 58

2.8.

3DSpace Index ......................................................................................................... 59

2.8.1.

The CloudView Monitoring Console ...................................................................... 59

2.8.2.

Monitoring Performances At Runtime.................................................................... 60

2.8.3.

Build Your Monitoring ............................................................................................ 60

2.8.4.

MQL Indexation .................................................................................................... 61

2.8.5.

Display Index Statistics ......................................................................................... 61

2.8.6.

Accesslist Tracing for Long Partial Indexing Operation ......................................... 62

2.8.7.

Count and Check Missing Objects ........................................................................ 63

2.8.8.

Monitoring of Aggregation Activity ......................................................................... 64

2.8.9.

Monitoring Crash of the CloudView Process ......................................................... 64

2.8.10. 2.9.

Configuring Email Alerts .................................................................................... 65

Accessing 3DIndex Report ....................................................................................... 66

2.10.

Monitoring 3DComment Service ........................................................................... 68

2.11.

Monitoring 3DNotification Service ......................................................................... 68

2.12.

Monitoring 3DSwym Service ................................................................................. 69

3.

Maintenance ................................................................................................................ 69

3.1.

Database .................................................................................................................. 69

3.2.

Cleaning Logs .......................................................................................................... 71

4.

Document History ........................................................................................................ 72

Confidential information. Copyright Dassault Systèmes [2018 - 2022]. This document is provided for information purpose only and may be used by authorized users only for their own internal use. Any other use without prior written authorization from Dassault Systèmes is strictly prohibited, except as may be permitted by law.

4

© Dassault Systèmes | Confidential Information | ref.: 3DS_Document_2020

2.2.

Best Practices

1. Monitoring the 3DEXPERIENCE Platform

3DEXPERIENCE platform has a multitier architecture and consists of client, application, database layer, and so on. Monitoring and maintaining the 3DEXPERIENCE platform is very important to prevent the problems or identify them as soon as possible to minimize any impact on the users. 3DEXPERIENCE platform includes the monitoring agent tool and MBeans used to monitor it. For a 3DEXPERIENCE platform, monitoring consists of: Resources of the machines involved in the architecture captured by a third-party tool. Intelligent health check performed by the monitoring agent tool. Monitoring the different layers as listed below: o http o Java o C++ (3DSpace kernel) o SQL for the database.

1.1.

Prerequisites for Monitoring

This section gives information about steps for activating the tools and configuring the logs in order to monitor the 3DEXPERIENCE platform.

1.1.1. Activating the Monitoring Agent for Health Check

Monitoring agent delivers some probes to perform health checks for each service. The health check not only performs a ping but also contacts the database server. It helps to detect if there is any issue in connecting to the TomEE instance or in getting information from the database. The following packages are a part of the 3DEXPERIENCE platform media. For installation purpose, download them from 3ds.com. • •

MonitoringAgentAndProbes.Windows64_package MonitoringAgentAndProbes.Linux64_package

Dassault Systèmes Documentation reference: Installation and Setup | Install | 3DEXPERIENCE Platform | Installing 3DEXPERIENCE Platform Services for the First Time | Installing Services One-by-One | Monitoring Agent | Installing the Monitoring Agent

Confidential information. Copyright Dassault Systèmes [2018 - 2022]. This document is provided for information purpose only and may be used by authorized users only for their own internal use. Any other use without prior written authorization from Dassault Systèmes is strictly prohibited, except as may be permitted by law.

5

© Dassault Systèmes | Confidential Information | ref.: 3DS_Document_2020

• • •

Different Metrics are available for different 3DEXPERIENCE platform services which can be monitored using a monitoring agent. Dassault Systèmes Documentation reference: Installation and Setup | Administrate | 3DEXPERIENCE Platform | Monitoring and Troubleshooting | Monitoring the 3DEXPERIENCE Platform using the Monitoring Agent | Metrics

1.1.2. Preparing HTTP Layer Logs

To monitor an HTTP layer, use the logs delivered by different TomEE instances. The Reverse Proxy/Load Balancer which is an intermediate point for all 3DEXPERIENCE platform services also provides such access logs. Access logs deliver interesting information such as HTTP code to identify errors, timeouts, the HTTP duration for performances, and so on. Configure the captured logs properly to collect the right information about the HTTP layer without impacting the system performance.

1.1.3. Using Apache httpd Logs

The following two types of logs are available to configure using the monitoring tools: • •

Error logs Access logs

Confidential information. Copyright Dassault Systèmes [2018 - 2022]. This document is provided for information purpose only and may be used by authorized users only for their own internal use. Any other use without prior written authorization from Dassault Systèmes is strictly prohibited, except as may be permitted by law.

6

© Dassault Systèmes | Confidential Information | ref.: 3DS_Document_2020

Best Practices

Best Practices

Error logs: It is the most important log file where Apache httpd sends diagnostic information and records the errors that it encounters while processing the requests. An Error log is the first place to look for when a problem occurs during the server start up or operation of the server. This log file mostly contains information about what went wrong and how to fix it. Location of the Error log file is set by an ErrorLog directive set in the httpd.conf file, the default is logs/error.log. For example: ErrorLog "logs/error.log" Recommendation: On a 3DEXPERIENCE platform, define a log file with each file. This makes it easy to check the service-specific logs.

Access logs: The server access log files record all the requests processed by the server. The CustomLog directive controls the location and content of the access log. For example: CustomLog "logs/access.log" common The location to which the logs are written is based on the setting of a ServerRoot value in the httpd.conf file. For example: ServerRoot "c:/Apache24" So the location will be C:/Apache24/logs/acces.log. The default log format is: (httpd.conf) LogFormat "%h %l %u %t \"%r\" %>s %b" common Set the log format as shown below to get more details such as %T - The time taken to serve the request, value in seconds, JSESSIONID, and so on. Use JSESSION to link the HTTP layer and Java layer. LogFormat "[%{%d/%b/%Y:%H:%M:%S %z}t] | %>s | %{ms}T ms | %b B | %h | %{XForwarded-For}i | %{JSESSIONID}C | %{SERVERID}C | \"%r\""

Confidential information. Copyright Dassault Systèmes [2018 - 2022]. This document is provided for information purpose only and may be used by authorized users only for their own internal use. Any other use without prior written authorization from Dassault Systèmes is strictly prohibited, except as may be permitted by law.

7

© Dassault Systèmes | Confidential Information | ref.: 3DS_Document_2020

For example:

Best Practices

For example:

https://httpd.apache.org/docs/current/mod/mod_log_config.html Use Access logs to monitor Apache activity, performances, and errors.

1.1.4. Apache Monitoring

The Apache server-status page provides an instant status of the server. To access the page, modify the following configuration in the httpd.conf file:

SetHandler server-status Require all granted

For more details, refer to the following link on Apache: https://httpd.apache.org/docs/2.4/mod/mod_status.html Access the server-status page from the Apache server as follows:

Confidential information. Copyright Dassault Systèmes [2018 - 2022]. This document is provided for information purpose only and may be used by authorized users only for their own internal use. Any other use without prior written authorization from Dassault Systèmes is strictly prohibited, except as may be permitted by law.

8

© Dassault Systèmes | Confidential Information | ref.: 3DS_Document_2020

Note: For more details about the possible string formats, refer to the following document:

The server-status page is also useful to monitor the Apache server in association with some monitoring tool (such as “collectd”) and its apache plug-in, to deliver a live monitoring console.

Use the server-status page live to ensure smooth working on Apache side. Additionally, add an alert when the idl_workers threads are less than 20% of the allocated threads.

1.1.5. Apache Logs Rotation

In the Apache httpd rotate logs, the program used for rotation is based on a time interval or maximum size of the log.

Confidential information. Copyright Dassault Systèmes [2018 - 2022]. This document is provided for information purpose only and may be used by authorized users only for their own internal use. Any other use without prior written authorization from Dassault Systèmes is strictly prohibited, except as may be permitted by law.

9

© Dassault Systèmes | Confidential Information | ref.: 3DS_Document_2020

Best Practices

Best Practices

Syntax: rotatelogs [ -l ] [ -f ] logfile rotationtime|filesizeM [ offset ] For more info, refer to rotatelogs. For example: The following example will rotate the access.log file daily. CustomLog "|bin/rotatelogs.exe logs/access.log.%Y-%m-%d-%H_%M_%S" common

1.1.6. Configuring Apache tomcat Access Logs

Configure the Tomcat access logs on various attributes such as, start time, status code, time taken to process the request, session id, hostname, and so on. For more details, refer to the following documentation: https://tomcat.apache.org/tomcat-9.0-doc/config/valve.html#Extended_Access_Log_Valve

%{begin:dd/MMM/Y:HH:mm:ss.SSS Z}t %{xxx}t write timestamp at the end of the request formatted using the enhanced SimpleDateFormat pattern xxx, begin or prefix begin: Chooses the request start time %s HTTP status code of the response %D Time taken to process the request, in millis %b Bytes sent, excluding HTTP headers, or '-' if zero %h Remote host name (or IP address if enableLookups for the connector is false). %{X-Forwarded-For}i %{xxx}i write value of incoming header with name xxx %S User session ID %{SERVERID}c %{xxx}c write value of cookie with name xxx %r First line of the request (method and request URI) %{emxRoleList}s for 3DSpace only %{__userIdt}s for 3DPassport only %{xxx}s write value of HttpSession attribute with name xxx

Confidential information. Copyright Dassault Systèmes [2018 - 2022]. This document is provided for information purpose only and may be used by authorized users only for their own internal use. Any other use without prior written authorization from Dassault Systèmes is strictly prohibited, except as may be permitted by law.

10

© Dassault Systèmes | Confidential Information | ref.: 3DS_Document_2020

TomEE Valve

Best Practices

Example with 3DSpace: Add the following lines to the 3DSpace TomEE -> server.xml file to activate the traces:

The above change will create a link between the http requests and add calls with the session information. Use this session information while accessing the verbose logs (reference is JsessionID).

1. Modify TomEE’s CAS (interactif and batch) and NOCAS configurations: 2. For each TomEE’s instances and servers, modify the server.xml file. 3. Replace the following lines at the end of the server.xml file:

By:

4. To apply these modifications, restart each TomEE’s instances. [09/Mar/2018:14:41:11.214 +0530] | 200 | 37 ms | 23784 B | fe80:0:0:0:ecaf:5a8c:bd8e:a169%8 | 10.89.19.166 | 15555DD97CC2135CE0057C3FAA04DB99 | - | "GET /3DSpace/common/scripts/emxExtendedPageHeaderFreezePaneValidation.js HTTP/1.1" | [ctx::VPLMProjectAdministrator.Company Name.Default, ctx::VPLMSecuredCrossAccess.Company Name.Common Space, Service Administrator, ctx::VPLMProjectAdministrator.Company Name.Common Space, ctx::VPLMSecuredCrossAccess.Company Name.Default, ctx::VPLMAdmin.Company Name.Default, Grant.Company Name.admin_platform_PRJ, admin_platform] Example with 3DPassport: Modify 3DPassport TomEE’s configuration: 1. Modify the server.xml file for each TomEE’s 3DPassport instances and servers.

Confidential information. Copyright Dassault Systèmes [2018 - 2022]. This document is provided for information purpose only and may be used by authorized users only for their own internal use. Any other use without prior written authorization from Dassault Systèmes is strictly prohibited, except as may be permitted by law.

11

© Dassault Systèmes | Confidential Information | ref.: 3DS_Document_2020

For example:

Best Practices

2. Replace the following lines at the end of this file:

By:

[09/Mar/2018:14:58:11.717 +0530] | 200 | 55 ms | 105104 B | fe80:0:0:0:ecaf:5a8c:bd8e:a169%8 | 10.89.135.38 | 2C62D8234A5A4FDF6C1670D04C8E768B | - | "GET /3dpassport/resources171110021716/css/main-ifwe.css HTTP/1.1" | Example with 3DDashboard: 1. Replace the following lines at the end of the server.xml file:

By:

2. To apply these modifications, restart each TomEE’s instances. [09/Mar/2018:14:57:11.161 +0530] | 200 | 20 ms | 401 B | fe80:0:0:0:ecaf:5a8c:bd8e:a169%8 | 10.89.19.166 | EB71D6DB3AEB5853D6A90B67D5603BA2 | - | "GET /3ddashboard/resources/20171115T194511Z/en/webapps/WidgetPreview/WidgetPre view_en.js HTTP/1.1"

Confidential information. Copyright Dassault Systèmes [2018 - 2022]. This document is provided for information purpose only and may be used by authorized users only for their own internal use. Any other use without prior written authorization from Dassault Systèmes is strictly prohibited, except as may be permitted by law.

12

© Dassault Systèmes | Confidential Information | ref.: 3DS_Document_2020

3. To apply these modifications, restart each TomEE’s instances.

Best Practices

Example with Federated Search: 1. Replace the following lines at the end of the server.xml file:

By: