Big Data

Best Of Amazon AWS Big Data Blog https://aws.amazon.com/blogs/big-data/orchestrate-multiple-etl-jobs-using-aws-step-functions-and-aws-lambda/ https://github.com/aws-samples/aws-etl-orchestrator https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/best-practices.html#organizingstacks https://aws.amazon.com/blogs/big-data/build-a-data-lake-foundation-with-aws-glue-and-amazon-s3/ https://aws.amazon.com/blogs/big-data/orchestrate-apache-spark-applications-using-aws-step-functions-and-apache-livy/

# you can install this Scala anywhere – but I install it with sudo / root wget https://www.scala-lang.org/files/archive/scala-2.13.0-M5.tgz sudo tar xzvf scala-2.13.0-M5.tgz sudo mv scala-2.13.0-M5.tgz scala213 alternatives –install /usr/bin/scala scala /opt/hadoop/scala213 1 # just verify where it is and that there are no other versions – at least that alternatives can see alternatives –config scala  There is …

installing scala on RHEL 7 or clones and do we really need this with Spark? No! Read More »

TCPH – http://www.tpc.org/tpch/ Here is where you get the kit to generate data – https://github.com/gregrahn/tpch-kit # make a dir to keep the data – # make sure you have space these are big files # export the DSS_PATH which is where dbgen puts the data # run dbgen with params mkdir redshiftdata export DSS_PATH=$HOME/redshiftdata cd …

tcph-kit to generate data Read More »

While this post is a bit backwards (the installation is at the bottom) – it has served me well to get an initial install of hadoop up and running on RHEL 7 (and pre-requisites), and running a very basic map-reduce job and some basic spark stuff – at least for single node install.  It is …

installing and using hadoop and spark from scratch singlenode and cluster Read More »