Fatal编程技术网
  • perl/
  • Perl的新功能-解析文件并用动态值替换模式

Perl的新功能-解析文件并用动态值替换模式

perl shell parsing csv

Perl的新功能-解析文件并用动态值替换模式,perl,shell,parsing,csv,Perl,Shell,Parsing,Csv,我对Perl非常陌生，目前正在尝试将bash脚本转换为Perl 我的脚本用于转换nmon文件（AIX/Linux perf monitoring tool），它获取目录中的nmon文件grep并将特定部分重定向到temp文件grep并将相关时间戳重定向到另一个文件 然后，它将数据解析为最终的csv文件，该文件将由第三个要利用的工具编制索引 示例NMON数据如下所示： TOP,%CPU Utilisation TOP,+PID,Time,%CPU,%Usr,%Sys,Threads,Siz

我对Perl非常陌生，目前正在尝试将bash脚本转换为Perl

我的脚本用于转换nmon文件（AIX/Linux perf monitoring tool），它获取目录中的nmon文件grep并将特定部分重定向到temp文件grep并将相关时间戳重定向到另一个文件

然后，它将数据解析为最终的csv文件，该文件将由第三个要利用的工具编制索引

示例NMON数据如下所示：

    TOP,%CPU Utilisation
TOP,+PID,Time,%CPU,%Usr,%Sys,Threads,Size,ResText,ResData,CharIO,%RAM,Paging,Command,WLMclass
TOP,5165226,T0002,10.93,9.98,0.95,1,54852,4232,51220,311014,0.755,1264,PatrolAgent,Unclassified
TOP,5365876,T0002,1.48,0.81,0.67,135,85032,132,84928,38165,1.159,0,db2sysc,Unclassified
TOP,5460056,T0002,0.32,0.27,0.05,1,5060,616,4704,1719,0.072,0,db2kmchan64.v9,Unclassified
字段“Time”（见T0002，在NMON中真正称为ZZZZ）是一个特定的NMON时间戳，该时间戳的实际值稍后（在专用部分）出现在NMON文件中，如下所示：

ZZZZ,T0001,00:09:55,01-JAN-2014
ZZZZ,T0002,00:13:55,01-JAN-2014
ZZZZ,T0003,00:17:55,01-JAN-2014
ZZZZ,T0004,00:21:55,01-JAN-2014
ZZZZ,T0005,00:25:55,01-JAN-2014
NMON格式非常具体，不经过解析就无法直接利用，时间戳必须与相应的值相关联。（NMON文件几乎就像是许多不同csv文件的串联，每个文件都有不同的格式、不同的文件等等。）

我编写了下面的bash脚本来解析我感兴趣的部分（“TOP”部分代表每个主机的顶级进程cpu、mem、io统计数据）

#/bin/bash
#集合x
################################################################
#信息
################################################################
#nmon2csv_TOP.sh
#将nmon文件的顶部部分转换为csv
#警告：此脚本预计将由主工作流启动
#$DST和DST_CONVERTED_TOP正在由它导出，否则此脚本将在启动时退出
################################################################
#瓦尔斯
################################################################
#NMON文件的位置
NMON_DIR=${DST}
#生成文件的位置
输出_DIR=${DST_转换的_TOP}
#临时文件
rawdatafile=/tmp/temp\u rawdata.$$.temp
timestampfile=/tmp/temp\u timestamp.$$.temp
#主输出文件
finalfile=${DST_CONVERTED_TOP}/NMON_TOP_processed_at_date_`date'+%F'`.csv
###########################
#开工
###########################
#验证导出的变量是否不为null
如果[-z${NMON_DIR}]；然后
echo-e“\n错误：Var NMON\u DIR为空！\n”&退出1
elif[-z${OUTPUT_DIR}]；然后
echo-e“\n错误：变量输出\u目录为空！\n”&退出1
fi
#检查临时文件和输出文件是否已存在
如果[-s${rawdatafile}]；然后
rm-f${rawdatafile}
elif[-s${timestampfile}]；然后
rm-f${timestampfile}
elif[-s${finalfile}]；然后
rm-f${finalfile}
fi
#获取当前位置
PWD=`PWD`
#转到NMON文件位置
cd${NMON_DIR}
#对于存在的每个NMON文件：
#仅限于产品环境：`ls*.nmon | grep-E-i'sp | gp | ge'`
对于'ls*.NMON | grep-E-i'sp | gp | ge'中的NMON|u文件；做
#设置主机名标识
serialnum=`grep'AAA，SerialNumber，'${NMON_FILE}}| awk-F，'{print$3}'OFS=，| tr[：lower:[：upper:]`
主机名=`grep'AAA，主机，'${NMON_FILE}| awk-F，'{print$3}'OFS=，| tr[：lower:[：upper:]`
#Grep和重定向顶部部分
grep'TOP'${NMON_FILE}| grep-v'AAA，版本，TOPAS-NMON'| grep-v'TOP%，CPU利用率'>${rawdatafile}
#Grep和重定向相关时间戳（ZZZZ）
grep'ZZZZ'${NMON_FILE}>${timestampfile}
#开工
当IFS=，read TOP PID Time Pct_CPU Pct_Usr Pct_Sys Threads Size ResText ResData CharIO Pct_RAM Paging Command WLMclass
做
timestamp=`grep${Time}${timestampfile}|awk-F'，{print$4”“$3}'OFS=,`
echo${serialnum}、${hostname}、${timestamp}、${Time}、${PID}、${Pct_CPU}、${Pct_Usr}、${Pct_Sys}、${Threads}、${Size}、${ResText}、${ResData}、${CharIO}、${Pct_RAM Pct}、${Paging}、${Command}、${WLMclass}\
|grep-v'+PID，%CPU，%Usr，%Sys，Threads，Size，ResText，ResData，CharIO，%RAM，Paging，Command，WLMclass'>>${finalfile}
完成<${rawdatafile}
echo-e“信息：为Serialnum:${Serialnum}主机名：${Hostname}完成”
完成
#返回初始位置
cd${PWD}
###########################
#工作结束
###########################
这可以根据需要工作，并生成一个主csv文件（您将在代码中看到，我没有在文件中保留csv头），它是所有解析主机的串联

但是，我每天要处理大量主机（大约3000台主机），使用当前的代码，在最坏的情况下，为一台主机生成数据可能需要几分钟，每台主机数乘以几分钟很容易变成几小时

因此，这段代码的执行能力确实不足以处理如此大量的数据

10台主机代表大约200.000行，最终代表大约20 MB的csv文件。 这并不多，但我认为shell脚本可能不是管理这样一个过程的更好选择

我想perl在这项任务中会更好（即使shell脚本可能会得到改进），但我（目前）对perl的了解非常差，这就是为什么我请求您的帮助。。。我认为用perl编写代码应该很简单，但我现在还不能让它正常工作

有一个人曾经开发一个perl脚本来管理NMON文件并将其转换为sql文件（将这些数据转储到数据库中），我将其用于使用其功能，并在一些shell脚本的帮助下管理sql文件以获得最终的csv文件

但是上面的部分没有集成到perl脚本中，如果不重新开发，就无法使用它

有关守则：

#!/usr/bin/perl
# Program name: nmon2mysql.pl
# Purpose - convert nmon.csv file(s) into mysql insert file
# Author - Bruce Spencer
# Disclaimer:  this provided "as is".  
# Date - March 2007
#
$nmon2mysql_ver="1.0. March 2007";

use Time::Local;


#################################################
##  Your Customizations Go Here            ##
#################################################

#  Source directory for nmon csv files
my $NMON_DIR=$ENV{DST_TMP};
my $OUTPUT_DIR=$ENV{DST_CONVERTED_CPU_ALL};


# End "Your Customizations Go Here".  
# You're on your own, if you change anything beyond this line :-)

####################################################################
#############       Main Program            ############
####################################################################

# Initialize common variables
&initialize;

# Process all "nmon" files located in the $NMON_DIR
# @nmon_files=`ls $NMON_DIR/*.nmon $NMON_DIR/*.csv`;
@nmon_files=`ls $NMON_DIR/*.nmon`;

if (@nmon_files eq 0 ) { die ("No \*.nmon or csv files found in $NMON_DIR\n"); }

@nmon_files=sort(@nmon_files);
chomp(@nmon_files);

foreach $FILENAME ( @nmon_files ) {

  @cols= split(/\//,$FILENAME);
  $BASEFILENAME= $cols[@cols-1];

  unless (open(INSERT, ">$OUTPUT_DIR/$BASEFILENAME.sql")) { 
    die("Can not open /$OUTPUT_DIR/$BASEFILENAME.sql\n"); 
  }
  print INSERT ("# nmon version: $NMONVER\n");
  print INSERT ("# AIX version: $AIXVER\n");
  print INSERT ("use nmon;\n");

  $start=time();
  @now=localtime($start);
  $now=join(":",@now[2,1,0]);
  print ("$now: Begin processing file = $FILENAME\n");

  # Parse nmon file, skip if unsuccessful
  if (( &get_nmon_data ) gt 0 ) { next; }
  $now=time();
  $now=$now-$start;
  print ("\t$now: Finished get_nmon_data\n");


  # Static variables (number of fields always the same)
  #@static_vars=("LPAR","CPU_ALL","FILE","MEM","PAGE","MEMNEW","MEMUSE","PROC");
  #@static_vars=("LPAR","CPU_ALL","FILE","MEM","PAGE","MEMNEW","MEMUSE");

  @static_vars=("CPU_ALL");

  foreach $key (@static_vars) {
     &mk_mysql_insert_static($key);;
     $now=time();
     $now=$now-$start;
     print ("\t$now: Finished $key\n");
  } # end foreach



  # Dynamic variables (variable number of fields)
  #@dynamic_vars=("DISKBSIZE","DISKBUSY","DISKREAD","DISKWRITE","DISKXFER","ESSREAD","ESSWRITE","ESSXFER","IOADAPT","NETERROR","NET","NETPACKET");

  @dynamic_vars=("");

  foreach $key (@dynamic_vars) {
    &mk_mysql_insert_variable($key);;
    $now=time();
    $now=$now-$start;
    print ("\t$now: Finished $key\n");
  }

  close(INSERT);
#  system("gzip","$FILENAME");

}
exit(0);


############################################
#############  Subroutines  ############
############################################

##################################################################
## Extract CPU_ALL data for Static fields
##################################################################
sub mk_mysql_insert_static {

my($nmon_var)=@_; 
my $table=lc($nmon_var);

my @rawdata;
my $x;
my @cols;
my $comma;
my $TS;
my $n;


  @rawdata=grep(/^$nmon_var,/, @nmon);

  if (@rawdata < 1) { return(1); }

  @rawdata=sort(@rawdata);

  @cols=split(/,/,$rawdata[0]);
  $x=join(",",@cols[2..@cols-1]);
  $x=~ s/\%/_PCT/g;
  $x=~ s/\(MB\)/_MB/g;
  $x=~ s/-/_/g;
  $x=~ s/ /_/g;
  $x=~ s/__/_/g;
  $x=~ s/,_/,/g;
  $x=~ s/_,/,/g;
  $x=~ s/^_//;
  $x=~ s/_$//;

  print INSERT (qq|insert into $table (serialnum,hostname,mode,nmonver,time,ZZZZ,$x) values\n| );

  $comma="";
  $n=@cols;
  $n=$n-1; # number of columns -1 

  for($i=1;$i<@rawdata;$i++){ 

    $TS=$UTC_START + $INTERVAL*($i);

    @cols=split(/,/,$rawdata[$i]);
    $x=join(",",@cols[2..$n]);
    $x=~ s/,,/,-1,/g; # replace missing data ",," with a ",-1,"

    print INSERT (qq|$comma("$SN","$HOSTNAME","$MODE","$NMONVER",$TS,"$DATETIME{@cols[1]}",$x)| );

    $comma=",\n";
  }
  print INSERT (qq|;\n\n|);

} # end mk_mysql_insert

##################################################################
## Extract CPU_ALL data for variable fields
##################################################################
sub mk_mysql_insert_variable {

my($nmon_var)=@_; 
my $table=lc($nmon_var);

my @rawdata;
my $x;
my $j;
my @cols;
my $comma;
my $TS;
my $n;
my @devices;


  @rawdata=grep(/^$nmon_var,/, @nmon);

  if ( @rawdata < 1) { return; }

  @rawdata=sort(@rawdata);

  $rawdata[0]=~ s/\%/_PCT/g;
  $rawdata[0]=~ s/\(/_/g;
  $rawdata[0]=~ s/\)/_/g;
  $rawdata[0]=~ s/ /_/g;
  $rawdata[0]=~ s/__/_/g;
  $rawdata[0]=~ s/,_/,/g;

  @devices=split(/,/,$rawdata[0]);
  print INSERT (qq|insert into $table (serialnum,hostname,time,ZZZZ,device,value) values\n| );

  $n=@rawdata;
  $n--; 
  for($i=1;$i<@rawdata;$i++){ 

    $TS=$UTC_START + $INTERVAL*($i);
    $rawdata[$i]=~ s/,$//;
    @cols=split(/,/,$rawdata[$i]);

      print INSERT (qq|\n("$SN","$HOSTNAME",$TS,"$DATETIME{$cols[1]}","$devices[2]",$cols[2])| );
    for($j=3;$j<@cols;$j++){
      print INSERT (qq|,\n("$SN","$HOSTNAME",$TS,"$DATETIME{$cols[1]}","$devices[$j]",$cols[$j])| );
    }
    if ($i < $n) { print INSERT (","); } 
  }
  print INSERT (qq|;\n\n|);

} # end mk_mysql_insert_variable

########################################################
### Get an nmon setting from csv file            ###
### finds first occurance of $search             ###
### Return the selected column...$return_col     ###
### Syntax:                                      ###
###     get_setting($search,$col_to_return,$separator)##
########################################################

sub get_setting {

my $i;
my $value="-1";
my ($search,$col,$separator)= @_;    # search text, $col, $separator

for ($i=0; $i<@nmon; $i++){

  if ($nmon[$i] =~ /$search/ ) {
    $value=(split(/$separator/,$nmon[$i]))[$col];
    $value =~ s/["']*//g;  #remove non alphanum characters
    return($value);
    } # end if

  } # end for

return($value);

} # end get_setting

#####################
##  Clean up       ##
#####################
sub clean_up_line {

    # remove characters not compatible with nmon variable
    # Max rrdtool variable length is 19 chars
    # Variable can not contain special characters (% - () )
    my ($x)=@_; 

    # print ("clean_up, before: $i\t$nmon[$i]\n");
    $x =~ s/\%/Pct/g;
    # $x =~ s/\W*//g;
    $x =~ s/\/s/ps/g;       # /s  - ps
    $x =~ s/\//s/g;     # / - s
    $x =~ s/\(/_/g;
    $x =~ s/\)/_/g;
    $x =~ s/ /_/g;
    $x =~ s/-/_/g;
    $x =~ s/_KBps//g;
    $x =~ s/_tps//g;
    $x =~ s/[:,]*\s*$//;
    $retval=$x; 

} # end clean up


##########################################
##  Extract headings from nmon csv file ##
##########################################
sub initialize {

%MONTH2NUMBER =  ("jan", 1, "feb",2, "mar",3, "apr",4, "may",5, "jun",6, "jul",7, "aug",8, "sep",9, "oct",10, "nov",11, "dec",12 );

@MONTH2ALPHA =  (   "junk","jan", "feb", "mar", "apr", "may", "jun", "jul", "aug", "sep", "oct", "nov", "dec" );

} # end initialize

# Get data from nmon file, extract specific data fields (hostname, date, ...)
sub get_nmon_data {

my $key;
my $x;
my $category;
my %toc;
my @cols;

# Read nmon file
unless (open(FILE, $FILENAME)) { return(1); }
@nmon=<FILE>;  # input entire file
close(FILE);
chomp(@nmon);

# Cleanup nmon data remove trainig commas and colons
for($i=0; $i<@nmon;$i++ ) {
    $nmon[$i] =~ s/[:,]*\s*$//;
}

# Get nmon/server settings (search string, return column, delimiter)
$AIXVER     =&get_setting("AIX",2,",");
$DATE       =&get_setting("date",2,",");
$HOSTNAME   =&get_setting("host",2,",");
$INTERVAL   =&get_setting("interval",2,","); # nmon sampling interval

$MEMORY     =&get_setting(qq|lsconf,"Good Memory Size:|,1,":");
$MODEL      =&get_setting("modelname",3,'\s+');
$NMONVER    =&get_setting("version",2,",");

$SNAPSHOTS  =&get_setting("snapshots",2,",");  # number of readings

$STARTTIME  =&get_setting("AAA,time",2,",");
($HR, $MIN)=split(/\:/,$STARTTIME);


if ($AIXVER eq "-1") {
    $SN=$HOSTNAME;  # Probably a Linux host
} else {
    $SN =&get_setting("systemid",4,",");
    $SN     =(split(/\s+/,$SN))[0]; # "systemid IBM,SN ..."
}

$TYPE       =&get_setting("^BBBP.*Type",3,",");
if ( $TYPE =~ /Shared/ ) { $TYPE="SPLPAR"; } else { $TYPE="Dedicated"; }

$MODE       =&get_setting("^BBBP.*Mode",3,",");
$MODE       =(split(/: /, $MODE))[1];
# $MODE     =~s/\"//g;


# Calculate UTC time (seconds since 1970)
# NMON V9  dd/mm/yy
# NMON V10+ dd-MMM-yyyy

if ( $DATE =~ /[a-zA-Z]/ ) {   # Alpha = assume dd-MMM-yyyy date format
    ($DAY, $MMM, $YR)=split(/\-/,$DATE);
    $MMM=lc($MMM);
    $MON=$MONTH2NUMBER{$MMM};
} else {
    ($DAY, $MON, $YR)=split(/\//,$DATE);
    $YR=$YR + 2000;
    $MMM=$MONTH2ALPHA[$MON];
} # end if

## Calculate UTC time (seconds since 1970).  Required format for the rrdtool.

##  timelocal format
##    day=1-31
##    month=0-11
##    year = x -1900  (time since 1900) (seems to work with either 2006 or 106)

$m=$MON - 1;  # jan=0, feb=2, ...

$UTC_START=timelocal(0,$MIN,$HR,$DAY,$m,$YR); 
$UTC_END=$UTC_START + $INTERVAL * $SNAPSHOTS;

@ZZZZ=grep(/^ZZZZ,/,@nmon);
for ($i=0;$i<@ZZZZ;$i++){

    @cols=split(/,/,$ZZZZ[$i]);
    ($DAY,$MON,$YR)=split(/-/,$cols[3]);
    $MON=lc($MON);
    $MON="00" . $MONTH2NUMBER{$MON};
    $MON=substr($MON,-2,2);
    $ZZZZ[$i]="$YR-$MON-$DAY $cols[2]";
    $DATETIME{$cols[1]}="$YR-$MON-$DAY $cols[2]";


} # end ZZZZ

return(0);
} # end get_nmon_data

#/usr/bin/perl
#程序名称：nmon2mysql.pl
#用途-将nmon.csv文件转换为mysql插入文件
#作者-布鲁斯·斯宾塞
#免责声明：这是“按原样”提供的。
#日期-2007年3月
#
$nmon2mysql\u ver=“1.0.2007年3月”；
使用时间：：本地；
#################################################
##你的定制在这里##
#################################################
#nmon csv文件的源目录
my$NMON_DIR=$ENV{DST_TMP}；
my$OUTPUT_DIR=$ENV{DST_CONVERTED_CPU_ALL}；
#结束“你的自订”

#!/usr/bin/perl
# Program name: nmon2mysql.pl
# Purpose - convert nmon.csv file(s) into mysql insert file
# Author - Bruce Spencer
# Disclaimer:  this provided "as is".  
# Date - March 2007
#
$nmon2mysql_ver="1.0. March 2007";

use Time::Local;


#################################################
##  Your Customizations Go Here            ##
#################################################

#  Source directory for nmon csv files
my $NMON_DIR=$ENV{DST_TMP};
my $OUTPUT_DIR=$ENV{DST_CONVERTED_CPU_ALL};


# End "Your Customizations Go Here".  
# You're on your own, if you change anything beyond this line :-)

####################################################################
#############       Main Program            ############
####################################################################

# Initialize common variables
&initialize;

# Process all "nmon" files located in the $NMON_DIR
# @nmon_files=`ls $NMON_DIR/*.nmon $NMON_DIR/*.csv`;
@nmon_files=`ls $NMON_DIR/*.nmon`;

if (@nmon_files eq 0 ) { die ("No \*.nmon or csv files found in $NMON_DIR\n"); }

@nmon_files=sort(@nmon_files);
chomp(@nmon_files);

foreach $FILENAME ( @nmon_files ) {

  @cols= split(/\//,$FILENAME);
  $BASEFILENAME= $cols[@cols-1];

  unless (open(INSERT, ">$OUTPUT_DIR/$BASEFILENAME.sql")) { 
    die("Can not open /$OUTPUT_DIR/$BASEFILENAME.sql\n"); 
  }
  print INSERT ("# nmon version: $NMONVER\n");
  print INSERT ("# AIX version: $AIXVER\n");
  print INSERT ("use nmon;\n");

  $start=time();
  @now=localtime($start);
  $now=join(":",@now[2,1,0]);
  print ("$now: Begin processing file = $FILENAME\n");

  # Parse nmon file, skip if unsuccessful
  if (( &get_nmon_data ) gt 0 ) { next; }
  $now=time();
  $now=$now-$start;
  print ("\t$now: Finished get_nmon_data\n");


  # Static variables (number of fields always the same)
  #@static_vars=("LPAR","CPU_ALL","FILE","MEM","PAGE","MEMNEW","MEMUSE","PROC");
  #@static_vars=("LPAR","CPU_ALL","FILE","MEM","PAGE","MEMNEW","MEMUSE");

  @static_vars=("CPU_ALL");

  foreach $key (@static_vars) {
     &mk_mysql_insert_static($key);;
     $now=time();
     $now=$now-$start;
     print ("\t$now: Finished $key\n");
  } # end foreach



  # Dynamic variables (variable number of fields)
  #@dynamic_vars=("DISKBSIZE","DISKBUSY","DISKREAD","DISKWRITE","DISKXFER","ESSREAD","ESSWRITE","ESSXFER","IOADAPT","NETERROR","NET","NETPACKET");

  @dynamic_vars=("");

  foreach $key (@dynamic_vars) {
    &mk_mysql_insert_variable($key);;
    $now=time();
    $now=$now-$start;
    print ("\t$now: Finished $key\n");
  }

  close(INSERT);
#  system("gzip","$FILENAME");

}
exit(0);


############################################
#############  Subroutines  ############
############################################

##################################################################
## Extract CPU_ALL data for Static fields
##################################################################
sub mk_mysql_insert_static {

my($nmon_var)=@_; 
my $table=lc($nmon_var);

my @rawdata;
my $x;
my @cols;
my $comma;
my $TS;
my $n;


  @rawdata=grep(/^$nmon_var,/, @nmon);

  if (@rawdata < 1) { return(1); }

  @rawdata=sort(@rawdata);

  @cols=split(/,/,$rawdata[0]);
  $x=join(",",@cols[2..@cols-1]);
  $x=~ s/\%/_PCT/g;
  $x=~ s/\(MB\)/_MB/g;
  $x=~ s/-/_/g;
  $x=~ s/ /_/g;
  $x=~ s/__/_/g;
  $x=~ s/,_/,/g;
  $x=~ s/_,/,/g;
  $x=~ s/^_//;
  $x=~ s/_$//;

  print INSERT (qq|insert into $table (serialnum,hostname,mode,nmonver,time,ZZZZ,$x) values\n| );

  $comma="";
  $n=@cols;
  $n=$n-1; # number of columns -1 

  for($i=1;$i<@rawdata;$i++){ 

    $TS=$UTC_START + $INTERVAL*($i);

    @cols=split(/,/,$rawdata[$i]);
    $x=join(",",@cols[2..$n]);
    $x=~ s/,,/,-1,/g; # replace missing data ",," with a ",-1,"

    print INSERT (qq|$comma("$SN","$HOSTNAME","$MODE","$NMONVER",$TS,"$DATETIME{@cols[1]}",$x)| );

    $comma=",\n";
  }
  print INSERT (qq|;\n\n|);

} # end mk_mysql_insert

##################################################################
## Extract CPU_ALL data for variable fields
##################################################################
sub mk_mysql_insert_variable {

my($nmon_var)=@_; 
my $table=lc($nmon_var);

my @rawdata;
my $x;
my $j;
my @cols;
my $comma;
my $TS;
my $n;
my @devices;


  @rawdata=grep(/^$nmon_var,/, @nmon);

  if ( @rawdata < 1) { return; }

  @rawdata=sort(@rawdata);

  $rawdata[0]=~ s/\%/_PCT/g;
  $rawdata[0]=~ s/\(/_/g;
  $rawdata[0]=~ s/\)/_/g;
  $rawdata[0]=~ s/ /_/g;
  $rawdata[0]=~ s/__/_/g;
  $rawdata[0]=~ s/,_/,/g;

  @devices=split(/,/,$rawdata[0]);
  print INSERT (qq|insert into $table (serialnum,hostname,time,ZZZZ,device,value) values\n| );

  $n=@rawdata;
  $n--; 
  for($i=1;$i<@rawdata;$i++){ 

    $TS=$UTC_START + $INTERVAL*($i);
    $rawdata[$i]=~ s/,$//;
    @cols=split(/,/,$rawdata[$i]);

      print INSERT (qq|\n("$SN","$HOSTNAME",$TS,"$DATETIME{$cols[1]}","$devices[2]",$cols[2])| );
    for($j=3;$j<@cols;$j++){
      print INSERT (qq|,\n("$SN","$HOSTNAME",$TS,"$DATETIME{$cols[1]}","$devices[$j]",$cols[$j])| );
    }
    if ($i < $n) { print INSERT (","); } 
  }
  print INSERT (qq|;\n\n|);

} # end mk_mysql_insert_variable

########################################################
### Get an nmon setting from csv file            ###
### finds first occurance of $search             ###
### Return the selected column...$return_col     ###
### Syntax:                                      ###
###     get_setting($search,$col_to_return,$separator)##
########################################################

sub get_setting {

my $i;
my $value="-1";
my ($search,$col,$separator)= @_;    # search text, $col, $separator

for ($i=0; $i<@nmon; $i++){

  if ($nmon[$i] =~ /$search/ ) {
    $value=(split(/$separator/,$nmon[$i]))[$col];
    $value =~ s/["']*//g;  #remove non alphanum characters
    return($value);
    } # end if

  } # end for

return($value);

} # end get_setting

#####################
##  Clean up       ##
#####################
sub clean_up_line {

    # remove characters not compatible with nmon variable
    # Max rrdtool variable length is 19 chars
    # Variable can not contain special characters (% - () )
    my ($x)=@_; 

    # print ("clean_up, before: $i\t$nmon[$i]\n");
    $x =~ s/\%/Pct/g;
    # $x =~ s/\W*//g;
    $x =~ s/\/s/ps/g;       # /s  - ps
    $x =~ s/\//s/g;     # / - s
    $x =~ s/\(/_/g;
    $x =~ s/\)/_/g;
    $x =~ s/ /_/g;
    $x =~ s/-/_/g;
    $x =~ s/_KBps//g;
    $x =~ s/_tps//g;
    $x =~ s/[:,]*\s*$//;
    $retval=$x; 

} # end clean up


##########################################
##  Extract headings from nmon csv file ##
##########################################
sub initialize {

%MONTH2NUMBER =  ("jan", 1, "feb",2, "mar",3, "apr",4, "may",5, "jun",6, "jul",7, "aug",8, "sep",9, "oct",10, "nov",11, "dec",12 );

@MONTH2ALPHA =  (   "junk","jan", "feb", "mar", "apr", "may", "jun", "jul", "aug", "sep", "oct", "nov", "dec" );

} # end initialize

# Get data from nmon file, extract specific data fields (hostname, date, ...)
sub get_nmon_data {

my $key;
my $x;
my $category;
my %toc;
my @cols;

# Read nmon file
unless (open(FILE, $FILENAME)) { return(1); }
@nmon=<FILE>;  # input entire file
close(FILE);
chomp(@nmon);

# Cleanup nmon data remove trainig commas and colons
for($i=0; $i<@nmon;$i++ ) {
    $nmon[$i] =~ s/[:,]*\s*$//;
}

# Get nmon/server settings (search string, return column, delimiter)
$AIXVER     =&get_setting("AIX",2,",");
$DATE       =&get_setting("date",2,",");
$HOSTNAME   =&get_setting("host",2,",");
$INTERVAL   =&get_setting("interval",2,","); # nmon sampling interval

$MEMORY     =&get_setting(qq|lsconf,"Good Memory Size:|,1,":");
$MODEL      =&get_setting("modelname",3,'\s+');
$NMONVER    =&get_setting("version",2,",");

$SNAPSHOTS  =&get_setting("snapshots",2,",");  # number of readings

$STARTTIME  =&get_setting("AAA,time",2,",");
($HR, $MIN)=split(/\:/,$STARTTIME);


if ($AIXVER eq "-1") {
    $SN=$HOSTNAME;  # Probably a Linux host
} else {
    $SN =&get_setting("systemid",4,",");
    $SN     =(split(/\s+/,$SN))[0]; # "systemid IBM,SN ..."
}

$TYPE       =&get_setting("^BBBP.*Type",3,",");
if ( $TYPE =~ /Shared/ ) { $TYPE="SPLPAR"; } else { $TYPE="Dedicated"; }

$MODE       =&get_setting("^BBBP.*Mode",3,",");
$MODE       =(split(/: /, $MODE))[1];
# $MODE     =~s/\"//g;


# Calculate UTC time (seconds since 1970)
# NMON V9  dd/mm/yy
# NMON V10+ dd-MMM-yyyy

if ( $DATE =~ /[a-zA-Z]/ ) {   # Alpha = assume dd-MMM-yyyy date format
    ($DAY, $MMM, $YR)=split(/\-/,$DATE);
    $MMM=lc($MMM);
    $MON=$MONTH2NUMBER{$MMM};
} else {
    ($DAY, $MON, $YR)=split(/\//,$DATE);
    $YR=$YR + 2000;
    $MMM=$MONTH2ALPHA[$MON];
} # end if

## Calculate UTC time (seconds since 1970).  Required format for the rrdtool.

##  timelocal format
##    day=1-31
##    month=0-11
##    year = x -1900  (time since 1900) (seems to work with either 2006 or 106)

$m=$MON - 1;  # jan=0, feb=2, ...

$UTC_START=timelocal(0,$MIN,$HR,$DAY,$m,$YR); 
$UTC_END=$UTC_START + $INTERVAL * $SNAPSHOTS;

@ZZZZ=grep(/^ZZZZ,/,@nmon);
for ($i=0;$i<@ZZZZ;$i++){

    @cols=split(/,/,$ZZZZ[$i]);
    ($DAY,$MON,$YR)=split(/-/,$cols[3]);
    $MON=lc($MON);
    $MON="00" . $MONTH2NUMBER{$MON};
    $MON=substr($MON,-2,2);
    $ZZZZ[$i]="$YR-$MON-$DAY $cols[2]";
    $DATETIME{$cols[1]}="$YR-$MON-$DAY $cols[2]";


} # end ZZZZ

return(0);
} # end get_nmon_data



use strict;
use warnings;

open INFILE, "<", "path/to/file.nmon";      # Open the file.

my @topLines;                               # Initialize variables.
my %timestamps;

while <INFILE>                              # This will walk over all the lines of the infile.
{                                           # Storing the current line in $_.
    chomp $_;                               # Remove newline at the end.
    if ($_ =~ m/^TOP/)                      # If the line starts with TOP...
    {
        push @topLines, $_;                 # ...store it in the array for later use.
    }
    elsif ($_ =~ m/^ZZZZ/)                  # If it is in the ZZZZ section...
    {
        my @fields = split ',', $_;         # ...split the line at commas...
        my $timestamp = join ",", $fields(2), $fields(3);   # ...join the timestamp into a string as you wish...
        $timestamps{$fields(1)} = $timestamp;               # ...and store it in the hash with the Twhatever thing as key.
    }

# This iteration could certainly be improved with more knowledge
# of how the file looks. For example the search could be cancelled
# after the ZZZZ section if the file is still long.
}

close INFILE;

open OUTFILE, ">", "path/to/output.csv";    # Open the file you want your output in.

foreach (@topLines)                         # Iterate through all elements of the array.
{                                           # Once again storing the current value in $_.
    my @fields = split ',', $_;             # Probably not necessary, depending on how output should be formated.
    my $outstring = join ',', $fields(0), $fields(1), $timestamps{$fields(2)};  # And whatever other fields you care for.
    print OUTFILE $outstring, "\n";         # Print.
}
close OUTFILE;
print "Done.\n";