An Enhanced Workload Statistics Collector for large customer systems – Part II
This blog is about the Workload Statistics Collector for releases up to NW2004.
The enhanced Workload Statistics Collector described in my previous blog (An Enhanced Workload Statistics Collector for large customer systems – Part I) is able to handle amounts of statistical data a lot larger than the standard collector can do. However, there still are limits.
If the workload statistics get larger and larger, especially when creating the monthly statistics for the systemwide TOTAL, the enhanced Workload Statistics Collector may also fail. An indication is a short dump EXPORT_INIT_COMPRESS_FAILED in program RSTOTL30.
Analysis of the problem
The enhanced Workload Statistics Collector has to re-combine the profiles into one large cluster and write it to table MONI, otherwise ST03N won’t find the data to display it. The maximum amount of data that can be written into a cluster dataset is two gigabytes, so if the sum of the sizes of all profiles exceed that limit, the profiles cannot be written to table MONI anymore and the short dump will occur.
When analyzing the sizes of each profile, the result is always the same: The four types of RFC profiles (RFC Server profile, RFC Client profile, RFC Server Destination profile, RFC Client Destination profile) are really large, representing together more than 50 percent of the summarized size of all profiles. So the solution is quite clear: De-activate the RFC profiles. No, stop. That’s not what we want to do.
A closer look at the RFC profiles
There are four different RFC profile: RFC Server profile, RFC Client profile, RFC Server Destination profile and RFC Client Destination profile.
Let’s have a closer look at the structure of the RFC profiles:
They all have the same key fields (screenshots taken from release 4.6C)
and the same data fields
The RFC Server profiles additionally have one more key field
and the RFC … Destination profiles one additional data field
You can see that the number of key fields is definitely larger than the number of data fields. When aggregating the data, a change of the value in one key field creates a new line within the profile. RFC profiles are obviously stored with very fine granularity, a granularity that is not needed in most cases. At this point we can start the improvement to reduce the size of the RFC profiles: Omit one or more key fields during aggregation!
How can we omit key fields in the RFC profiles?
We must not change the structures in the data dictionary, so instead of removing the field itself, the value can be removed by CLEARing the contents before aggregating the data. Different values then have no effect anymore. Granularity and size of the RFC profiles are reduced, on the other hand, field values are lost.
The technical realization is a quite easy task, the more difficult task is to determine, which of the key fields can be “blanked” without losing valuable data.
Having a look at transaction ST03N, e.g. the RFC Client profile, you’ll see the following tabs in the right pane
Did you ever look at the Targets tab? Or at the Remote or Local Destinations tab? If not, these are candidates for key fields to be omitted.
What is the impact of omitting selected key fields?
To determine the reduction that can be gathered by omitting key fields, program RSTOTL45 is helpful. It offers a simulation mode that determines the reduction based on available RFC profiles in table SAPWLMONI. Additionally, to fix the short dump EXPORT_INIT_COMPRESS_FAILED in RSTOTL30, it can actually perform a re-aggregation of the existing RFC profiles using the settings found to be best while simulating.
Running it with transaction SE38 will give the following selection screen:
I checked “TARGET” and “DESTS” to demonstrate the effect. While “playing around”, leave the “SIMULATE” checkbox marked. In the lower box, the periodtype of the statistical data to be processed can be specified, “D” for daily statistics, “W” for weekly statistics, “M” for monthly statistics and blank for all of them. To avoid large amounts of data to be processed, it is a good idea to restrict to weekly statistics.
The result after running the program is
This excerpt shows the weekly statistics starting at Monday, June 18th, 2007 for the systemwide TOTAL instance.
Adapting the collector’s coding
Corresponding to the checkboxes in program RSTOTL45, the collector program itself has analog switches to modify its behaviour, they are located in program RSTOTL20.
As you can see, by default the switches are comments only, you can activate them by just commenting in the corresponding coding. To omit the Targets, Remote and Local Destination key fields, uncomment the lines with rfc_target and rfc_dests. The comments at the end of each line give the field names within the data dictionary structures that will be affected when activating that switch.
Why does SAP do not give a recommendation which key fields to omit?
I have been asked several times which combination of key fields is “the best”. There is no general answer. Every SAP system is used in a different way, leading to very different internal structure of the key fields values within the RFC profiles. Sometimes there are many users using only few function modules, sometimes few users use many different function modules, sometimes the SAP system is mostly an RFC server, sometimes it’s mostly a client. Some people do look at targets or destinations, others don’t. You got the point. It is essential to analyze the current situation in each case to get the best result, therefore a general answer cannot be given.
When leaving the SIMULATE checkbox marked, the program can be run with different settings many times without any impact on the data in the database. When unchecking “SIMULATE”, the RFC profiles will be modified according to the settings. Running the program again with a different setting will remove more data from the RFC profiles. Additionally, a simulation won’t give good results anymore, because some data is already removed.
It is a good idea to save the settings found to be best during simulation as variant (take care that you uncheck “SIMULATE” and clear “PERITYPE” before saving). Then run the re-aggregation in a background job using this variant. The amount of data being processed can be huge, each RFC profile has to be re-aggregated to apply the settings, this can take some time and consume some memory. Therefore a background job is suited better to perform this task than doing it in dialog.
SAP Notes to look at
0945279 Workload Collector terminates due to memory problems
0955618 Workload Collector: Increased aggregation of RFC profile
1030486 Workload Collector: Increased aggregation of RFC profile II
1036771 Workload Collector: Analyzing & compressing RFC profiles
ABAP programs used for the enhanced Workload Statistics Collector
RSTOTL00 This program is intended to be scheduled as periodic (daily) background job and executes RSTOTL10..40 in the appropriate order.
RSTOTL10 This program is splitting up the clusters from table MONI containing all profiles into single clusters in table SAPWLMONI containing only one profile each.
RSTOTL20 This program is doing the aggregation of all profiles into weekly and monthly statistics per instance and daily, weekly and monthly statistics for the systemwide TOTAL.
RSTOTL30 This program is re-combining the clusters from table SAPWLMONI containing one profile each into one large cluster in table MONI containing all profiles.
RSTOTL40 This program is cleaning up (deleting) profiles from table SAPWLMONI which are not needed for further aggregation work anymore.
RSTOTL05 This program can display the (verbose) log of programs RSTOTL10..40.
RSTOTL45 This program can analyze the existing RFC profiles in table SAPWLMONI to help with a decision which key fields should be omitted to reduce the size of the RFC profiles. It additionally can re-aggregate the existing RFC profiles in SAPWLMONI to actually reduce their size.