Jun 9, 2012

Uploading flat files into Postgres Database for specific tables


#!/bin/sh
if [ ! $# -eq 2 ] ; then
echo "Invalid arguments! Usage : DATTFilesToDB.sh  inputdir(fullpath) tablelistfilename";
echo "ex:DATFilesToDB.sh  usr/1L_joblogs joblogtables.txt";
exit;
else
echo "Executing database  uploading from DAT flies ..." ;
exec<$2
echo "----------- FLAT FILES (.dat) to DATABSE Tables------------------">>/usr/FFtoDB.log
echo "---Start Date:"$(date)>>/usr/FFtoDB.log
echo "-------------------------------------------------">>/usr/FFtoDB.log
echo "FlatFiles Loc:$1">>/usr/FFtoDB.log
cd $1
value=0;
while read line
 do
   value=`expr $value + 1`;
   echo "Currently running File/Table:$line">>/usr/FFtoDB.log
   query="COPY "$line" FROM '"$1"/public_"$line".dat' WITH DELIMITER ' ';"
   echo $query
   psql -h localhost -d webacc -U postgres -c "$query">> /usr/FFtoDB.log 2>&1
 done
echo "****$value file got created..";
echo "****$value file got created..">>/usr/FFtoDB.log
echo "---End Date:"$(date)>>/usr/FFtoDB.log
fi
echo "---------------------------------End of Script-----------------------";


.txt file should contain the table names.Script will upload flat files data into selected tables from the input dir. Errors and return values will be saved in  /usr/FFtoDB.log file.



Jun 7, 2012

sed command to identify date pattern yyyy-mm-dd hh:mm:ss.mse and replace with current date

sed -e "s/[0-9][0-9]\([0-9][0-9]\)-\([0-9][0-9]\)-\([0-9][0-9]\) \([0-9][0-9]\):\([0-9][0-9]\):\([0-9][0-9]\).\([0-9][0-9][0-9]\)/$(date +"%F %T.")$(($(date +%N)/1000000))/g" inputfile > newfile

Creating postgres database tables into Flat files (.copy) using shell script


#!/bin/sh
if [ ! $# -eq 2 ] ; then
echo "Invalid arguments! Usage : creatflatfiles.sh  outpurdir tablelistfilename";
echo "ex:creatflatfiles.sh  1L_joblogs tableslist.txt";
exit;
else
echo "Executing database  downloading to faltflies ..." ;
exec<$2
/bin/mkdir -p $1
cd $1
value=0;
while read line
 do
   value=`expr $value + 1`;  
   touch $line.copy;
   chmod 666 $line.copy;
   query="COPY "$line" TO '/usr/"$1"/"$line".copy';"
   echo $query
  #psql -h localhost -d databasename -U username -c command
   psql -h localhost -d webacc -U postgres -c "$query" >> /usr/FFtoDB.log 2>&1
 done
echo "****$value file got created..";
fi
#------------------------------------------

tableslist.txt should contain the table names.Script will generate tables data into flat files (tablename.copy) in the outputdir. Errors and return values will be saved in  /usr/FFtoDB.log file.