Upserts, Deletes And Incremental Processing on Big Data.

Template for Spark Projects

type-class based data cleansing library for Apache Spark SQL

Connect to SQL Server using Apache Spark

Delta Lake Examples

Repository for Lab “Distributed Big Data Analytics” (MA-INF 4223), University of Bonn

Apache Spark with Kafka via JDBC !!!

Trigger spark-submit in Golang. A Go implementation of famous SparkLauncher.java.

Microservices for Spark application

Examples usages for cleanframes library

Link Prediction is about predicting the future connections in a graph. In this project, Link Prediction is about predicting whether two authors will be collaborating for their future paper or not given the graph of authors who collaborated for atleast one paper together.

Use this project to join data from multiple csv files. Currently in this project we support one to one and one to many join. Along with this you can find how to use kafka producer efficiently with spark.

This repository contains all the projects and labs I worked on while pursuing professional certificate programs, specializations, and bootcamp. [Areas: Deep Learning, Machine Learning, Applied Data Science].

Working with Apache Spark, Creating some small tutorials and at last implemeting a small project

You will find here the demo codes for my Data+AI 2020 talk about customizing Apache Spark state store.

Source code for the work "dSpark: Deadline-Based Resource Allocation for Big Data Applications in Apache Spark" published in IEEE e-Science 2017

Implementation of GraphFrames using PySpark in Eclipse IDE

This is a Jupyter Notebook to practice Apache Spark in Google Colab, especially for the exam CCA Spark and Hadoop Developer Exam (CCA175).

The rapid pace of innovation in Artificial Intelligence (AI) is creating enormous opportunity for transforming entire industries and our very existence. After competing this comprehensive 6 course Professional Certificate, you will get a practical understanding of Machine Learning and Deep Learning. You will master fundamental concepts of Machine Learning and Deep Learning, including supervised and unsupervised learning. You will utilize popular Machine Learning and Deep Learning libraries such as SciPy, ScikitLearn, Keras, PyTorch, and Tensorflow applied to industry problems involving object recognition and Computer Vision, image and video processing, text analytics, Natural Language Processing, recommender systems, and other types of classifiers. You will be able to scale Machine Learning on Big Data using Apache Spark. You will build, train, and deploy different types of Deep Architectures, including Convolutional Networks, Recurrent Networks, and Autoencoders. By the end of this Professional Certificate, you will have completed several projects showcasing your proficiency in Machine Learning and Deep Learning, and become armed with skills for a career as an AI Engineer.

As a coursera certified specialization completer you will have a proven deep understanding on massive parallel data processing, data exploration and visualization, and advanced machine learning & deep learning. You'll understand the mathematical foundations behind all machine learning & deep learning algorithms. You can apply knowledge in practical use cases, justify architectural decisions, understand the characteristics of different algorithms, frameworks & technologies & how they impact model performance & scalability. If you choose to take this specialization and earn the Coursera specialization certificate, you will also earn an IBM digital badge. To find out more about IBM digital badges follow the link ibm.biz/badging.

Hadoop,MachineLearningAlgos,Spark,Pig,Hive

A published paper in PEARC18: Combining HPC and Big Data Infrastructures in Large-Scale Post-Processing of SimulaBon Data: A Case Study

In diesem Tutorial werden verschiedene Punkte aus dem Arbeitsalltag mit Apache Spark beleuchtet. Es wird nicht 100% Vollständigkeit angestrebt. Wichtig ist zu verstehen, was wie gemacht wurde.