Ive been looking at syslog tools tryed logzilla and logstash
yum -y install java-1.6.0-openjdk
cd /var/www/html
export http_proxy=http://rutherc:99999999@10.20.98.24:8080
wget http://logstash.objects.dreamhost.com/release/logstash-1.2.1-flatjar.jar
#export https_proxy=http://rutherc:99999999@10.20.98.24:8080
#wget http://github.com/elasticsearch/kibana/archive/master.zip --no-check-certificate -O master.zip
wget http://download.elasticsearch.org/kibana/kibana/kibana-latest.zip
unzip kibana-latest.zip
#perl -pi -e 's/DocumentRoot \"\/var\/www\/html\"/DocumentRoot \"\/var\/www\/html\/kibana-master\"/' /etc/httpd/conf/httpd.conf
chkconfig --level 2345 httpd on
service httpd start
perl -ni -e 'print; print "-A INPUT -m state --state NEW -p tcp --dport 9200 -j ACCEPT\n" if $. == 9' /etc/sysconfig/iptables
perl -ni -e 'print; print "-A INPUT -m state --state NEW -p tcp --dport 80 -j ACCEPT\n" if $. == 9' /etc/sysconfig/iptables
service iptables restart
cat << EOF > logstash.conf
input {
file {
type => "syslog"
path => [ "/var/log/*.log", "/var/log/messages", "/var/log/syslog" ]
}
}
output {
elasticsearch { embedded => true }
}
EOF
mv ./app/dashboards/default.json ./app/dashboards/default.json.org
cp ./app/dashboards/logstash.json ./app/dashboards/default.json
java -Xmx1024m -jar logstash-1.2.1-flatjar.jar agent -f logstash.conf -web
10.22.122.16 10.22.122.1 2675
10.20.122.16 10.20.122.1 2675
pvcreate /dev/sdb
vgextend vgRoot /dev/sdb
lvresize -l +90%FREE /dev/mapper/vgRoot-lvVar
resize2fs /dev/mapper/vgRoot-lvVar
Wednesday, November 22, 2017
Thursday, April 18, 2013
How to port the Linux Foremost data carving tool to IBMs AIX on POWER
http://foremost.sourceforge.net/pkg/foremost-1.5.3.tar.gz
[root:busen]/data/foremost-1.5.3$ make unix
gcc -Wall -O2 -DVERSION=\"1.5.3\" -D__UNIX -c main.c
gcc -Wall -O2 -DVERSION=\"1.5.3\" -D__UNIX -c state.c
gcc -Wall -O2 -DVERSION=\"1.5.3\" -D__UNIX -c helpers.c
gcc -Wall -O2 -DVERSION=\"1.5.3\" -D__UNIX -c config.c
config.c: In function 'translate':
config.c:27: warning: value computed is not used
config.c:32: warning: value computed is not used
config.c:37: warning: value computed is not used
config.c:42: warning: value computed is not used
config.c:47: warning: value computed is not used
config.c:52: warning: value computed is not used
config.c:57: warning: value computed is not used
gcc -Wall -O2 -DVERSION=\"1.5.3\" -D__UNIX -c cli.c
gcc -Wall -O2 -DVERSION=\"1.5.3\" -D__UNIX -c engine.c
gcc -Wall -O2 -DVERSION=\"1.5.3\" -D__UNIX -c dir.c
gcc -Wall -O2 -DVERSION=\"1.5.3\" -D__UNIX -c extract.c
gcc -Wall -O2 -DVERSION=\"1.5.3\" -D__UNIX -c api.c
api.c: In function 'get_dir_info':
api.c:107: warning: comparison is always false due to limited range of data type
gcc -Wall -O2 -DVERSION=\"1.5.3\" -D__UNIX main.o state.o
helpers.o config.o cli.o engine.o dir.o extract.o api.o -o foremost
Target "unix" is up to date.
[root:busen]/data/foremost-1.5.3$ make install
install -m 755 foremost /usr/local/bin
install: 0653-233 File 755 was not found.
make: 1254-004 The error code from the last command is 2.
Stop.
AIX has installbsd command so edit makefile.....
=================
#---------------------------------------------------------------------
# INSTALLATION AND REMOVAL
#---------------------------------------------------------------------
install: goals
installbsd -m 755 $(NAME) $(BIN)
installbsd -m 444 $(MAN_PAGES) $(MAN)
installbsd -m 444 foremost.conf $(CONF)
macinstall: BIN = /usr/local/bin/
macinstall: MAN = /usr/share/man/man1/
macinstall: CONF = /usr/local/etc/
macinstall: mac install
uninstall:
rm -f -- $(BIN)/{$(RM_GOALS)}
rm -f -- $(MAN)/{$(RM_DOCS)}
macuninstall: BIN = /usr/bin
macuninstall: MAN = /usr/share/man/man1
macuninstall: uninstall
#---------------------------------------------------------------------
# CLEAN UP
"Makefile" 193 lines, 5188 characters
=======================
[root:busen]/data/foremost-1.5.3$ make install
installbsd -m 755 foremost /usr/local/bin
installbsd -m 444 foremost.1 /usr/local/man/man1
installbsd -m 444 foremost.conf /usr/local/etc
WORKS! (I also tested it on some images)
[root:busen]/$ foremost -V
1.5.3
This program is a work of the US Government. In accordance with 17 USC 105,
copyright protection is not available for any work of the US Government.
This is free software; see the source for copying conditions. There is NO
warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.
[root:busen]/$ gcc -v
Using built-in specs.
Target: powerpc-ibm-aix5.3.0.0
Configured with: ../gcc-4.1.1/configure --disable-nls
Thread model: aix
gcc version 4.1.1
[root:busen]/$ ldd $(which foremost)
/usr/local/bin/foremost needs:
/usr/lib/libc.a(shr.o)
/unix
/usr/lib/libcrypt.a(shr.o)
Thursday, November 15, 2012
POWER7/PowerVM vs x86/VMware/OracleVM for oracle DB hosting
A Colleague ( http://twitter.com/ZoranGagic ) sent me this great breakdown of x86 vs Power7 for hosting oracle...nice to have it all in one spot (below) also this http://virtualgeek.typepad.com/virtual_geek/2011/07/even-more-reasons-to-run-oracle-on-vmware.html makes a similar case and now that VMWare have softened up on the vSphere 5 memory tax there is not as much need to go 4.1 as per below
HP c7000 with 16 x BL460 G8
|
IBM 770
| |
CPU cores
|
16 x 16 = 256
|
26
|
Memory
|
16 x 144GB = 2304
|
512GB
|
Performance, SPECintrate2006
|
16 x 662 = 10,592
|
942
|
Price
|
est $200K
|
$452K (approx)
|
COD
|
None
|
18 cores/512GB, cost: $10K for 1 core/16GB
|
Price difference
|
2.3 x more expensive
| |
Performance difference
|
11.2 x faster
| |
Price /performance difference
|
25.7 x better price performance
|
This does not include virtualisation or OS costs, but numbers are overwhelming….
General purpose CPU - Intel Xeon E5-2600:
This will be in every major rackmount, blade server, standard chipset support 24 DIMM sockets (24 x 32GB = 768GB), 10 GbE etc….
(Sub $10K for 16 core server with over 144GB memory)
Xeon E5 (2.3b transistors, more CPU cores (8), less heat/power due to 22nm process, bigger L3 cache (20MB)) :
16 cores, 2 chips, 8 cores/chip, 2 threads/core
SPECint_rate_base2006 = 662 (41.3/c)
Power 770 with Power 7:
48 cores, 8 chips, 6 cores/chip, 4 threads/core
SPECint_rate_base2006 =1740 (36.2/c)
Xeon 5670 in HP BL460:
12 cores, 2 chips, 6 cores/chip, 2 threads/core
SPECint_rate_base2006 = 318 (26.5/c)
Enterprise Edition Per-core licensing
Multi-core processors are priced as (number of cores)*(multi-core factor) processors, where the multi-core factor is:
IBM Power 7 CPU Pool with 8 cores = 8 x $47,500 = $380,000
Intel E5 with 16 cores = 16 x $23,750 = $380,000
Better to buy a low cost Intel E5 based server with maximum memory available (768GB) and choose VMware ESX 4.1 (not huge memory costs with ESX 5) and RHEL 6.2 on a farm of blade servers such as:
HP BL460 G8 is available with Intel E5 CPU:
(VMware ESX 4.1 Enterprise Plus - Unlimited memory
VMware vSphere 4.1 Enterprise Plus for 1 processor (Max 12 cores per processor) + Production (24x7 for Severity 1 issues) 3 Year Support
|
$5,723.70
|
2-sockets with unlimited virtual guests
Standard Subscription (1 year) $1,999
Premium Subscription (1 year) $3,249
Oracle Prod on physical database farm with RAC (or one node RAC to keep it very simple)….many instance on one OS image:
- Large memory (Intel memory is cheap), separate SGAs for different instance – no memory contention
- Separate LUNs so IO is well separated between instances, if there are ever any “problem instances” can easily move to another server
- Huge CPU as hardware and Oracle costs are significantly cheaper
Oracle on VMware (other better supported solution is Oracle VM):
Monday, September 17, 2012
Sunday, April 22, 2012
3 Networky AIX 6/7 gotchas....
A colleague came across this tricky gotcha in AIX7 - his scp sessions were stalling on AIX 7.1 (VIO client) LPARs.
There is not much on the net about this one yet
By disabling tcp timestamp randomisation feature on both source and target VIO clients the randomised timestamp value is not used for setting the retransmission timer
Fix
#no -o tcp_rand_timestamp=1
Here is the APAR http://www-01.ibm.com/support/docview.wss?uid=isg1IV13121 IV13121: TCP RETRANSMIT PROCESSING IS VERY SLOW.
One possible cause of an SEA on VIOS flip flopping from primary to backup is processor unfolding delays http://www-01.ibm.com/support/docview.wss?uid=isg3T1012941
Fix - stop folding
# schedo -p -o vpm_fold_policy=4
There is a fair bit on the net about this one...
Another one I'll put here (AIX 6/7 and Oracle 11g network stalling/delays)
With Oracle 11g came IPv6 support and even though a hostname may be resolved in IPv4 land a second lookup into /etc/hosts is done for the IPv6 address/hostname.
If the IPv6 address is not matched in /etc/hosts then it goes out to the DNS servers for IPv6 name resolution because of the (default) name resolution order in net service (/etc/netsvc.conf) was local then bind
Fix
Force the IPv4 ie in /etc/netsvc.conf change local, bind to local4, bind4
There is a LOT on the net about this one....
There is not much on the net about this one yet
By disabling tcp timestamp randomisation feature on both source and target VIO clients the randomised timestamp value is not used for setting the retransmission timer
Fix
#no -o tcp_rand_timestamp=1
Here is the APAR http://www-01.ibm.com/support/docview.wss?uid=isg1IV13121 IV13121: TCP RETRANSMIT PROCESSING IS VERY SLOW.
One possible cause of an SEA on VIOS flip flopping from primary to backup is processor unfolding delays http://www-01.ibm.com/support/docview.wss?uid=isg3T1012941
Fix - stop folding
# schedo -p -o vpm_fold_policy=4
There is a fair bit on the net about this one...
Another one I'll put here (AIX 6/7 and Oracle 11g network stalling/delays)
With Oracle 11g came IPv6 support and even though a hostname may be resolved in IPv4 land a second lookup into /etc/hosts is done for the IPv6 address/hostname.
If the IPv6 address is not matched in /etc/hosts then it goes out to the DNS servers for IPv6 name resolution because of the (default) name resolution order in net service (/etc/netsvc.conf) was local then bind
Fix
Force the IPv4 ie in /etc/netsvc.conf change local, bind to local4, bind4
There is a LOT on the net about this one....
Monday, March 26, 2012
Graphical representation of machine generated data
In this plot Im running a (ruby) gltail of apache access log while I rip the site with a recursive wget
Often with machine generated data (eg a syslog stream) there is too much to read every item and its coming in real time from multiple sources so you cant see the forest for the trees.
By graphically showing the data the eye can see if something looks strange or different than normal.
The human eye has a data bandwidth of ~10 Mbps and couple that with the brain and you have a kick ass data miner!
Wednesday, March 7, 2012
Ask For Forgiveness Programming
I recon THIS is how the brain mitigates Amdahl's law and why our brains are 'slow and inaccurate' when compared to a computer at adding numbers.
ACID/locking/mutex/semaphore etc just wont scale to billions of cores (neurons)
However a computer is 'slow and inaccurate' at identifying real word objects or doing stand-up comedy.
ACID/locking/mutex/semaphore etc just wont scale to billions of cores (neurons)
However a computer is 'slow and inaccurate' at identifying real word objects or doing stand-up comedy.
Subscribe to:
Posts (Atom)