river의 기술 로그

[Spark SQL] Date and Time Functions

riverruns — Sat, 25 Sep 2021 06:21:10 +0900

(Subset of) Standard Functions for Date and Time

current_date	Gives current date as a date column
current_timestamp
date_format
to_date	Converts column to date type (with an optional date format)
to_timestamp	Converts column to timestamp type (with an optional timestamp format)
unix_timestamp	Converts current or specified time to Unix timestamp (in seconds)
window	Generates time windows (i.e. tumbling, sliding and delayed windows)

current_date function gives the current date as a date column.

Internally, date_format creates a Column with DateFormatClass binary expression. DateFormatClass takes the expression from dateExpr column and format.

current_timestamp is also now function in SQL.

Converting Current or Specified Time to Unix Timestamp — unix_timestamp Function

Gives current timestamp (in seconds)
Converts time string in format yyyy-MM-dd HH:mm:ss to Unix timestamp (in seconds)unix_timestamp converts the current or specified time in the specified format to a Unix timestamp (in seconds).
unix_timestamp supports a column of type Date, Timestamp or String.
unix_timestamp returns null if conversion fails.

[SQL] collect_list() 함수

riverruns — Sun, 16 May 2021 15:45:07 +0900

collect_list()

: 한 개의 id 기준으로 여러가지 값을 가질 때, 그 여러가지 값들을 배열 형식으로 묶어주는 함수 (*SparkSQL도 가능)

SELECT id, COLLECT_LIST(ITEM) AS item_list 
FROM item_info 
GROUP BY id

[PySpark] DataFrame 의 show()

riverruns — Sat, 15 May 2021 21:08:43 +0900

pyspark의 DataFrame이라고 해서 pandas DataFrame이랑 같은 것이 아니었다.

show()

이것은 파이썬의 matplotlib 라이브러리를 활용해 시각화 할때 많이 보던 거라 익숙하지만 말이다. 말 그대로 보여달라는 것.

여기서는 plt.show()가 아니니 당연히 DataFrame을 보여달라는 것이다.

익히 아는 깔끔한 모습

이때, 사실 show()에는 두 개의 인자를 넣을 수 있다.

df.show(3, false)

e.g.

show(3, false) 라고 쓴다면

3개의 데이터셋을 보여주고
요약하지 말고 모든 스트링을 출력

한다는 뜻이다.

엉망진창이지만 다 보여준다

필요에 따라 목적에 맞게 쓰면 될 것이다.

[PySpark] csv, parquet 파일 읽어오기

riverruns — Sat, 15 May 2021 19:28:38 +0900

csv 파일 읽기

# csv 파일 읽기
df = spark.read.csv("...")
df.printSchema()  # 컬럼의 데이터타입 확인
df.show()

# parquet으로 저장된 파일 읽기
df_2 = spark.read.parquet("...")
df_2.printSchema()  # 컬럼의 데이터타입 확인
df_2.show()

csv와 parquet 사이에 별 차이가 없다.

csv 쓰는 자리에 parquet을 쓰면 된다. 간-단!

.....이 아님

저렇게만 하고 불러오면 다음과 같은 결과가 나온다.

그냥 다 기본 false로 읽나보다

헤더로 쓰일 컬럼 이름이 모두 행으로 (레코드로) 내려간 것을 볼 수 있다.

또한 정수여야 하는 Count의 컬럼도 String으로 처리된 것을 알 수 있다.

이를 방지할려면 불러 올 때 옵션을 넣어줘야한다.

스칼라는 공백을 무시했지만, pyspark는 줄 바꾸는 공백에 대해 표시를 해줘야했다. 안 그러면 에러 남.

이때 \ 를 넣으면 된다.

df = spark.read.option("header", True) \
               .option("inferSchema", True) \
               .csv("...")
               
df.printSchema()
df.show()

그러면 다음과 같이 깔끔한 결과가 나온다.

헤더도 정상, 컬럼의 데이터타입도 정상

굳

[PySpark] 빈 데이터프레임 empty DataFrame 생성

riverruns — Sat, 15 May 2021 13:46:57 +0900

빈 데이터프레임 생성하기 (Create empty Dataframe in PySpark)

schema = StructType([
    StructField("v1", LongType(), True), StructField("v2", StringType(), False), StructField("v3", StringType(), False)
])
df = sqlContext.createDataFrame([],schema)

# StructField() 안의 세 번째 인자는 nullable의 여부

그런데 이때 빼먹으면 안되는 게, StructType, LongType() 등을 import 해야한다는 것

안 하면 에러가 뜬다

그러니 아래와 같은 import를 꼭 해주고, 빈 데이터프레임을 생성하자

pyspark.sql.types import StructType, StructField, LongType(), StringType()
# 내가 사용할 것들만 import 했음

Getting to know PySpark

riverruns — Sun, 9 May 2021 17:33:57 +0900

Learn how Spark manages data and how can you read and write tables from Python.

What is Spark, anyway?

Spark is a platform for cluster computing. Spark lets you spread data and computations over clusters with multiple nodes (think of each node as a separate computer). Splitting up your data makes it easier to work with very large datasets because each node only works with a small amount of data.

As each node works on its own subset of the total data, it also carries out a part of the total calculations required, so that both data processing and computation are performed in parallel over the nodes in the cluster. It is a fact that parallel computation can make certain types of programming tasks much faster.

However, with greater computing power comes greater complexity.

Deciding whether or not Spark is the best solution for your problem takes some experience, but you can consider questions like:

Is my data too big to work with on a single machine?
Can my calculations be easily parallelized?

Using Spark in Python

The first step in using Spark is connecting to a cluster.

In practice, the cluster will be hosted on a remote machine that's connected to all other nodes. There will be one computer, called the master that manages splitting up the data and the computations. The master is connected to the rest of the computers in the cluster, which are called worker. The master sends the workers data and calculations to run, and they send their results back to the master.

When you're just getting started with Spark it's simpler to just run a cluster locally. Thus, for this course, instead of connecting to another computer, all computations will be run on DataCamp's servers in a simulated cluster.

Creating the connection is as simple as creating an instance of the SparkContext class. The class constructor takes a few optional arguments that allow you to specify the attributes of the cluster you're connecting to.

An object holding all these attributes can be created with the SparkConf() constructor. Take a look at the documentation for all the details!

For the rest of this course you'll have a SparkContext called sc already available in your workspace.

Q. How do you connect to a Spark cluster from PySpark?

- Create an instance of the SparkContext class.

Examining The SparkContext

In this exercise you'll get familiar with the SparkContext.

You'll probably notice that code takes longer to run than you might expect. This is because Spark is some serious software. It takes more time to start up than you might be used to. You may also find that running simpler computations might take longer than expected. That's because all the optimizations that Spark has under its hood are designed for complicated operations with big data sets. That means that for simple or small problems Spark may actually perform worse than some other solutions!

Get to know the SparkContext.

Call print() on sc to verify there's a SparkContext in your environment.
print() sc.version to see what version of Spark is running on your cluster.

# Verify SparkContext
print(sc)

# Print Spark version
print(sc.version)

Using DataFrames

Spark's core data structure is the Resilient Distributed Dataset (RDD). This is a low level object that lets Spark work its magic by splitting data across multiple nodes in the cluster. However, RDDs are hard to work with directly, so in this course you'll be using the Spark DataFrame abstraction built on top of RDDs.

The Spark DataFrame was designed to behave a lot like a SQL table (a table with variables in the columns and observations in the rows). Not only are they easier to understand, DataFrames are also more optimized for complicated operations than RDDs.

When you start modifying and combining columns and rows of data, there are many ways to arrive at the same result, but some often take much longer than others. When using RDDs, it's up to the data scientist to figure out the right way to optimize the query, but the DataFrame implementation has much of this optimization built in!

To start working with Spark DataFrames, you first have to create a SparkSession object from your SparkContext. You can think of the SparkContext as your connection to the cluster and the SparkSession as your interface with that connection.

emember, for the rest of this course you'll have a SparkSession called spark available in your workspace!

Q. Which of the following is an advantage of Spark DataFrames over RDDs?

- Operations using DataFrames are automatically optimized.

Creating a SparkSession

We've already created a SparkSession for you called spark, but what if you're not sure there already is one? Creating multiple SparkSessions and SparkContexts can cause issues, so it's best practice to use the SparkSession.builder.getOrCreate() method. This returns an existing SparkSession if there's already one in the environment, or creates a new one if necessary!

Import SparkSession from pyspark.sql.
Make a new SparkSession called my_spark using SparkSession.builder.getOrCreate().
Print my_spark to the console to verify it's a SparkSession.

# Import SparkSession from pyspark.sql
from pyspark.sql import SparkSession

# Create my_spark
my_spark = SparkSession.builder.getOrCreate()

# Print my_spark
print(my_spark)

Viewing tables

Once you've created a SparkSession, you can start poking around to see what data is in your cluster!

Your SparkSession has an attribute called catalog which lists all the data inside the cluster. This attribute has a few methods for extracting different pieces of information.

One of the most useful is the .listTables() method, which returns the names of all the tables in your cluster as a list.

See what tables are in your cluster by calling spark.catalog.listTables() and printing the result!

# Print the tables in the catalog
print(spark.catalog.listTables())

--> 결과

# Print the tables in the catalog print(spark.catalog.listTables())

[Table(name='flights', database=None, description=None, tableType='TEMPORARY', isTemporary=True)]

Are you query-ious?

One of the advantages of the DataFrame interface is that you can run SQL queries on the tables in your Spark cluster. If you don't have any experience with SQL, don't worry, we'll provide you with queries! (To learn more SQL, start with our Introduction to SQL course.)

As you saw in the last exercise, one of the tables in your cluster is the flights table. This table contains a row for every flight that left Portland International Airport (PDX) or Seattle-Tacoma International Airport (SEA) in 2014 and 2015.

Running a query on this table is as easy as using the .sql() method on your SparkSession. This method takes a string containing the query and returns a DataFrame with the results!

If you look closely, you'll notice that the table flights is only mentioned in the query, not as an argument to any of the methods. This is because there isn't a local object in your environment that holds that data, so it wouldn't make sense to pass the table as an argument.

Remember, we've already created a SparkSession called spark in your workspace. (It's no longer called my_spark because we created it for you!)

Use the .sql() method to get the first 10 rows of the flights table and save the result to flights10. The variable query contains the appropriate SQL query.
Use the DataFrame method .show() to print flights10.

# Wrong Answer

# Don't change this query
query = "FROM flights SELECT * LIMIT 10"

# Get the first 10 rows of flights
flights10 = spark.sql("SELECT * FROM flights LIMIT 10")

# Show the results
flights10.show()

Remember you can run a query by doing spark.sql(a_query)!

# Solution

# Don't change this query
query = "FROM flights SELECT * LIMIT 10"

# Get the first 10 rows of flights
flights10 = spark.sql(query)

# Show the results
flights10.show()

Pandafy a Spark DataFrame

Suppose you've run a query on your huge dataset and aggregated it down to something a little more manageable.

Sometimes it makes sense to then take that table and work with it locally using a tool like pandas. Spark DataFrames make that easy with the .toPandas() method. Calling this method on a Spark DataFrame returns the corresponding pandas DataFrame. It's as simple as that!

This time the query counts the number of flights to each airport from SEA and PDX.

Remember, there's already a SparkSession called spark in your workspace!

Run the query using the .sql() method. Save the result in flight_counts.
Use the .toPandas() method on flight_counts to create a pandas DataFrame called pd_counts.
Print the .head() of pd_counts to the console.

# Don't change this query
query = "SELECT origin, dest, COUNT(*) as N FROM flights GROUP BY origin, dest"

# Run the query
flight_counts = spark.sql("SELECT origin, dest, COUNT(*) as N FROM flights GROUP BY origin, dest")

# Convert the results to a pandas DataFrame
pd_counts = flight_counts.toPandas()

# Print the head of pd_counts
print(pd_counts.head())

Put some Spark in your data

In the last exercise, you saw how to move data from Spark to pandas. However, maybe you want to go the other direction, and put a pandas DataFrame into a Spark cluster! The SparkSession class has a method for this as well.

The .createDataFrame() method takes a pandas DataFrame and returns a Spark DataFrame.

The output of this method is stored locally, not in the SparkSession catalog. This means that you can use all the Spark DataFrame methods on it, but you can't access the data in other contexts.

For example, a SQL query (using the .sql() method) that references your DataFrame will throw an error. To access the data in this way, you have to save it as a temporary table.

You can do this using the .createTempView() Spark DataFrame method, which takes as its only argument the name of the temporary table you'd like to register. This method registers the DataFrame as a table in the catalog, but as this table is temporary, it can only be accessed from the specific SparkSession used to create the Spark DataFrame.

There is also the method .createOrReplaceTempView(). This safely creates a new temporary table if nothing was there before, or updates an existing table if one was already defined. You'll use this method to avoid running into problems with duplicate tables.

Check out the diagram to see all the different ways your Spark data structures interact with each other.

There's already a SparkSession called spark in your workspace, numpy has been imported as np, and pandas as pd.

The code to create a pandas DataFrame of random numbers has already been provided and saved under pd_temp.
Create a Spark DataFrame called spark_temp by calling the .createDataFrame() method with pd_temp as the argument.
Examine the list of tables in your Spark cluster and verify that the new DataFrame is not present. Remember you can use spark.catalog.listTables() to do so.
Register spark_temp as a temporary table named "temp" using the .createOrReplaceTempView() method. Remember that the table name is set including it as the only argument!
Examine the list of tables again!

# Wrong Answer

# Create pd_temp
pd_temp = pd.DataFrame(np.random.random(10))

# Create spark_temp from pd_temp
spark_temp = pd_temp.createDataFrame()

# Examine the tables in the catalog
print(spark_temp)

# Add spark_temp to the catalog
spark_temp.spark.catalog.listTables()

# Examine the tables in the catalog again
print(spark_temp)

# Solution

# Create pd_temp
pd_temp = pd.DataFrame(np.random.random(10))

# Create spark_temp from pd_temp
spark_temp = spark.createDataFrame(pd_temp)

# Examine the tables in the catalog
print(spark.catalog.listTables())

# Add spark_temp to the catalog
spark_temp.createOrReplaceTempView("temp")

# Examine the tables in the catalog again
print(spark.catalog.listTables())

Dropping the middle man

Now you know how to put data into Spark via pandas, but you're probably wondering why deal with pandas at all? Wouldn't it be easier to just read a text file straight into Spark? Of course it would!

Luckily, your SparkSession has a .read attribute which has several methods for reading different data sources into Spark DataFrames. Using these you can create a DataFrame from a .csv file just like with regular pandas DataFrames!

The variable file_path is a string with the path to the file airports.csv. This file contains information about different airports all over the world.

A SparkSession named spark is available in your workspace.

Use the .read.csv() method to create a Spark DataFrame called airports
- The first argument is file_path
- Pass the argument header=True so that Spark knows to take the column names from the first line of the file.
Print out this DataFrame by calling .show().

# Don't change this file path
file_path = "/usr/local/share/datasets/airports.csv"

# Read in the airports data
airports = spark.read.csv("/usr/local/share/datasets/airports.csv",header=True)

# Show the data
airports.show()

[SQL] SELECT DISTINCT | DISTINCT한 로우들을 COUNT한 것들의 COUNT

riverruns — Fri, 9 Apr 2021 23:56:39 +0900

groupBy를 하든 group by를 하든 이렇게 중복 없이 묶인 DISTINCT한 로우들을 COUNT 하는 방법

SELECT Count(*) AS 이름맘대로 
FROM (SELECT DISTINCT 컬럼명 FROM 테이블명)

예를 들어 우리집에 사놓은 라면이 열라면, 진라면, 신라면이 각각 뭐 2개 3개 3개 이렇다고 하면

총 라면은 8개가 있지만 라면의 종류는 3종류이다. 이런 종류, 분류, 항목, 카테고리 기타 등등을 세는 방법이다

SELECT Count(*) 우리집이먹는라면종류
FROM (SELECT DISTINCT 라면이담긴컬럼 FROM 테이블명)