Index of /apache/spark/docs/3.0.0-preview2
Name Last modified Size Description
![[PARENTDIR]](/icons/back.gif)
Parent Directory -
![[DIR]](/icons/folder.gif)
api/ 2024-08-16 12:37 -
css/ 2024-08-16 12:37 -
img/ 2024-08-16 12:37 -
js/ 2024-08-16 12:37 -
![[TXT]](/icons/text.gif)
README.html 2024-08-16 12:27 8.0K
api.html 2024-08-16 12:27 9.1K Spark API Documentation - Spark 3.0.0-preview2 Documentation
building-spark.html 2024-08-16 12:27 30K Building Spark - Spark 3.0.0-preview2 Documentation
building-with-maven.html 2024-08-16 12:27 425 Redirecting…
cloud-integration.html 2024-08-16 12:27 24K Integration with Cloud Infrastructures - Spark 3.0.0-preview2 Documentation
cluster-overview.html 2024-08-16 12:27 16K Cluster Mode Overview - Spark 3.0.0-preview2 Documentation
configuration.html 2024-08-16 12:27 128K Configuration - Spark 3.0.0-preview2 Documentation
contributing-to-spark.html 2024-08-16 12:27 8.9K Contributing to Spark - Spark 3.0.0-preview2 Documentation
core-migration-guide.html 2024-08-16 12:27 12K Migration Guide: Spark Core - Spark 3.0.0-preview2 Documentation
graphx-programming-guide.html 2024-08-16 12:27 163K GraphX - Spark 3.0.0-preview2 Documentation
hadoop-provided.html 2024-08-16 12:27 12K Using Spark's "Hadoop Free" Build - Spark 3.0.0-preview2 Documentation
hardware-provisioning.html 2024-08-16 12:27 13K Hardware Provisioning - Spark 3.0.0-preview2 Documentation
index.html 2024-08-16 12:27 20K Overview - Spark 3.0.0-preview2 Documentation
job-scheduling.html 2024-08-16 12:27 33K Job Scheduling - Spark 3.0.0-preview2 Documentation
migration-guide.html 2024-08-16 12:27 10K Migration Guide - Spark 3.0.0-preview2 Documentation
ml-advanced.html 2024-08-16 12:27 20K Advanced topics - Spark 3.0.0-preview2 Documentation
ml-ann.html 2024-08-16 12:27 12K Multilayer perceptron classifier - Spark 3.0.0-preview2 Documentation
ml-classification-regression.html 2024-08-16 12:27 415K Classification and regression - Spark 3.0.0-preview2 Documentation
ml-clustering.html 2024-08-16 12:27 89K Clustering - Spark 3.0.0-preview2 Documentation
ml-collaborative-filtering.html 2024-08-16 12:27 61K Collaborative Filtering - Spark 3.0.0-preview2 Documentation
ml-datasource.html 2024-08-16 12:27 45K Data sources - Spark 3.0.0-preview2 Documentation
ml-decision-tree.html 2024-08-16 12:27 12K Decision trees - Spark 3.0.0-preview2 Documentation
ml-ensembles.html 2024-08-16 12:27 12K Tree ensemble methods - Spark 3.0.0-preview2 Documentation
ml-features.html 2024-08-16 12:27 572K Extracting, transforming and selecting features - Spark 3.0.0-preview2 Documentation
ml-frequent-pattern-mining.html 2024-08-16 12:27 47K Frequent Pattern Mining - Spark 3.0.0-preview2 Documentation
ml-guide.html 2024-08-16 12:27 20K MLlib: Main Guide - Spark 3.0.0-preview2 Documentation
ml-linear-methods.html 2024-08-16 12:27 12K Linear methods - Spark 3.0.0-preview2 Documentation
ml-migration-guide.html 2024-08-16 12:27 52K Migration Guide: MLlib (Machine Learning) - Spark 3.0.0-preview2 Documentation
ml-pipeline.html 2024-08-16 12:27 98K ML Pipelines - Spark 3.0.0-preview2 Documentation
ml-statistics.html 2024-08-16 12:27 62K Basic Statistics - Spark 3.0.0-preview2 Documentation
ml-survival-regression.html 2024-08-16 12:27 12K Survival Regression - Spark 3.0.0-preview2 Documentation
ml-tuning.html 2024-08-16 12:27 80K ML Tuning - Spark 3.0.0-preview2 Documentation
mllib-classification-regression.html 2024-08-16 12:27 15K Classification and Regression - RDD-based API - Spark 3.0.0-preview2 Documentation
mllib-clustering.html 2024-08-16 12:27 124K Clustering - RDD-based API - Spark 3.0.0-preview2 Documentation
mllib-collaborative-filtering.html 2024-08-16 12:27 43K Collaborative Filtering - RDD-based API - Spark 3.0.0-preview2 Documentation
mllib-data-types.html 2024-08-16 12:27 87K Data Types - RDD-based API - Spark 3.0.0-preview2 Documentation
mllib-decision-tree.html 2024-08-16 12:27 71K Decision Trees - RDD-based API - Spark 3.0.0-preview2 Documentation
mllib-dimensionality-reduction.html 2024-08-16 12:27 46K Dimensionality Reduction - RDD-based API - Spark 3.0.0-preview2 Documentation
mllib-ensembles.html 2024-08-16 12:27 118K Ensembles - RDD-based API - Spark 3.0.0-preview2 Documentation
mllib-evaluation-metrics.html 2024-08-16 12:27 156K Evaluation Metrics - RDD-based API - Spark 3.0.0-preview2 Documentation
mllib-feature-extraction.html 2024-08-16 12:27 75K Feature Extraction and Transformation - RDD-based API - Spark 3.0.0-preview2 Documentation
mllib-frequent-pattern-mining.html 2024-08-16 12:27 48K Frequent Pattern Mining - RDD-based API - Spark 3.0.0-preview2 Documentation
mllib-guide.html 2024-08-16 12:27 16K MLlib: RDD-based API - Spark 3.0.0-preview2 Documentation
mllib-isotonic-regression.html 2024-08-16 12:27 36K Isotonic regression - RDD-based API - Spark 3.0.0-preview2 Documentation
mllib-linear-methods.html 2024-08-16 12:27 77K Linear Methods - RDD-based API - Spark 3.0.0-preview2 Documentation
mllib-naive-bayes.html 2024-08-16 12:27 32K Naive Bayes - RDD-based API - Spark 3.0.0-preview2 Documentation
mllib-optimization.html 2024-08-16 12:27 48K Optimization - RDD-based API - Spark 3.0.0-preview2 Documentation
mllib-pmml-model-export.html 2024-08-16 12:27 18K PMML model export - RDD-based API - Spark 3.0.0-preview2 Documentation
mllib-statistics.html 2024-08-16 12:27 112K Basic Statistics - RDD-based API - Spark 3.0.0-preview2 Documentation
monitoring.html 2024-08-16 12:27 64K Monitoring and Instrumentation - Spark 3.0.0-preview2 Documentation
programming-guide.html 2024-08-16 12:27 8.9K Spark Programming Guide - Spark 3.0.0-preview2 Documentation
pyspark-migration-guide.html 2024-08-16 12:27 19K Migration Guide: PySpark (Python on Spark) - Spark 3.0.0-preview2 Documentation
quick-start.html 2024-08-16 12:27 53K Quick Start - Spark 3.0.0-preview2 Documentation
rdd-programming-guide.html 2024-08-16 12:27 154K RDD Programming Guide - Spark 3.0.0-preview2 Documentation
![[ ]](/icons/unknown.gif)
redirects.json 2024-08-16 12:27 52
running-on-kubernetes.html 2024-08-16 12:27 87K Running Spark on Kubernetes - Spark 3.0.0-preview2 Documentation
running-on-mesos.html 2024-08-16 12:27 53K Running Spark on Mesos - Spark 3.0.0-preview2 Documentation
running-on-yarn.html 2024-08-16 12:27 51K Running Spark on YARN - Spark 3.0.0-preview2 Documentation
security.html 2024-08-16 12:27 54K Security - Spark 3.0.0-preview2 Documentation
spark-standalone.html 2024-08-16 12:27 42K Spark Standalone Mode - Spark 3.0.0-preview2 Documentation
sparkr-migration-guide.html 2024-08-16 12:27 20K Migration Guide: SparkR (R on Spark) - Spark 3.0.0-preview2 Documentation
sparkr.html 2024-08-16 12:27 85K SparkR (R on Spark) - Spark 3.0.0-preview2 Documentation
sql-data-sources-avro.html 2024-08-16 12:27 42K Apache Avro Data Source Guide - Spark 3.0.0-preview2 Documentation
sql-data-sources-binaryFile.html 2024-08-16 12:27 15K Binary File Data Source - Spark 3.0.0-preview2 Documentation
sql-data-sources-hive-tables.html 2024-08-16 12:27 52K Hive Tables - Spark 3.0.0-preview2 Documentation
sql-data-sources-jdbc.html 2024-08-16 12:27 43K JDBC To Other Databases - Spark 3.0.0-preview2 Documentation
sql-data-sources-json.html 2024-08-16 12:27 29K JSON Files - Spark 3.0.0-preview2 Documentation
sql-data-sources-load-save-functions.html 2024-08-16 12:27 56K Generic Load/Save Functions - Spark 3.0.0-preview2 Documentation
sql-data-sources-orc.html 2024-08-16 12:27 13K ORC Files - Spark 3.0.0-preview2 Documentation
sql-data-sources-parquet.html 2024-08-16 12:27 61K Parquet Files - Spark 3.0.0-preview2 Documentation
sql-data-sources-troubleshooting.html 2024-08-16 12:27 12K Troubleshooting - Spark 3.0.0-preview2 Documentation
sql-data-sources.html 2024-08-16 12:27 15K Data Sources - Spark 3.0.0-preview2 Documentation
sql-distributed-sql-engine.html 2024-08-16 12:27 16K Distributed SQL Engine - Spark 3.0.0-preview2 Documentation
sql-getting-started.html 2024-08-16 12:27 158K Getting Started - Spark 3.0.0-preview2 Documentation
sql-keywords.html 2024-08-16 12:27 34K Spark SQL Keywords - Spark 3.0.0-preview2 Documentation
sql-migration-guide.html 2024-08-16 12:27 103K Migration Guide: SQL, Datasets and DataFrame - Spark 3.0.0-preview2 Documentation
sql-migration-old.html 2024-08-16 12:27 10K Migration Guide - Spark 3.0.0-preview2 Documentation
sql-performance-tuning.html 2024-08-16 12:27 21K Performance Tuning - Spark 3.0.0-preview2 Documentation
sql-programming-guide.html 2024-08-16 12:27 14K Spark SQL and DataFrames - Spark 3.0.0-preview2 Documentation
sql-pyspark-pandas-with-arrow.html 2024-08-16 12:27 65K PySpark Usage Guide for Pandas with Apache Arrow - Spark 3.0.0-preview2 Documentation
sql-ref-arithmetic-ops.html 2024-08-16 12:27 11K Arithmetic Operations - Spark 3.0.0-preview2 Documentation
sql-ref-datatypes.html 2024-08-16 12:27 27K Data Types - Spark 3.0.0-preview2 Documentation
sql-ref-functions-builtin-aggregate.html 2024-08-16 12:27 12K Builtin Aggregate Functions - Spark 3.0.0-preview2 Documentation
sql-ref-functions-builtin-scalar.html 2024-08-16 12:27 12K Builtin Scalar Functions - Spark 3.0.0-preview2 Documentation
sql-ref-functions-builtin.html 2024-08-16 12:27 12K Reference - Spark 3.0.0-preview2 Documentation
sql-ref-functions-udf-aggregate.html 2024-08-16 12:27 12K User defined Aggregate Functions (UDAF) - Spark 3.0.0-preview2 Documentation
sql-ref-functions-udf-scalar.html 2024-08-16 12:27 12K User defined Scalar Functions (UDF) - Spark 3.0.0-preview2 Documentation
sql-ref-functions-udf.html 2024-08-16 12:27 12K Reference - Spark 3.0.0-preview2 Documentation
sql-ref-functions.html 2024-08-16 12:27 11K Reference - Spark 3.0.0-preview2 Documentation
sql-ref-nan-semantics.html 2024-08-16 12:27 11K Nan Semantics - Spark 3.0.0-preview2 Documentation
sql-ref-null-semantics.html 2024-08-16 12:27 69K NULL Semantics - Spark 3.0.0-preview2 Documentation
sql-ref-syntax-aux-analyze-table.html 2024-08-16 12:27 18K ANALYZE TABLE - Spark 3.0.0-preview2 Documentation
sql-ref-syntax-aux-analyze.html 2024-08-16 12:27 13K ANALYZE - Spark 3.0.0-preview2 Documentation
sql-ref-syntax-aux-cache-cache-table.html 2024-08-16 12:27 17K CACHE TABLE - Spark 3.0.0-preview2 Documentation
sql-ref-syntax-aux-cache-clear-cache.html 2024-08-16 12:27 14K CLEAR CACHE - Spark 3.0.0-preview2 Documentation
sql-ref-syntax-aux-cache-uncache-table.html 2024-08-16 12:27 15K UNCACHE TABLE - Spark 3.0.0-preview2 Documentation
sql-ref-syntax-aux-cache.html 2024-08-16 12:27 13K Cache - Spark 3.0.0-preview2 Documentation
sql-ref-syntax-aux-conf-mgmt-reset.html 2024-08-16 12:27 14K RESET - Spark 3.0.0-preview2 Documentation
sql-ref-syntax-aux-conf-mgmt-set.html 2024-08-16 12:27 16K SET - Spark 3.0.0-preview2 Documentation
sql-ref-syntax-aux-conf-mgmt.html 2024-08-16 12:27 13K Reference - Spark 3.0.0-preview2 Documentation
sql-ref-syntax-aux-describe-database.html 2024-08-16 12:27 21K DESCRIBE DATABASE - Spark 3.0.0-preview2 Documentation
sql-ref-syntax-aux-describe-function.html 2024-08-16 12:27 25K DESCRIBE FUNCTION - Spark 3.0.0-preview2 Documentation
sql-ref-syntax-aux-describe-query.html 2024-08-16 12:27 22K DESCRIBE QUERY - Spark 3.0.0-preview2 Documentation
sql-ref-syntax-aux-describe-table.html 2024-08-16 12:27 36K DESCRIBE TABLE - Spark 3.0.0-preview2 Documentation
sql-ref-syntax-aux-describe.html 2024-08-16 12:27 13K Reference - Spark 3.0.0-preview2 Documentation
sql-ref-syntax-aux-refresh-table.html 2024-08-16 12:27 15K REFRESH TABLE - Spark 3.0.0-preview2 Documentation
sql-ref-syntax-aux-resource-mgmt-add-file.html 2024-08-16 12:27 14K ADD FILE - Spark 3.0.0-preview2 Documentation
sql-ref-syntax-aux-resource-mgmt-add-jar.html 2024-08-16 12:27 14K ADD JAR - Spark 3.0.0-preview2 Documentation
sql-ref-syntax-aux-resource-mgmt-list-file.html 2024-08-16 12:27 14K LIST FILE - Spark 3.0.0-preview2 Documentation
sql-ref-syntax-aux-resource-mgmt-list-jar.html 2024-08-16 12:27 15K LIST JAR - Spark 3.0.0-preview2 Documentation
sql-ref-syntax-aux-resource-mgmt.html 2024-08-16 12:27 13K Reference - Spark 3.0.0-preview2 Documentation
sql-ref-syntax-aux-show-columns.html 2024-08-16 12:27 18K SHOW COLUMNS - Spark 3.0.0-preview2 Documentation
sql-ref-syntax-aux-show-create-table.html 2024-08-16 12:27 17K SHOW CREATE TABLE - Spark 3.0.0-preview2 Documentation
sql-ref-syntax-aux-show-databases.html 2024-08-16 12:27 18K SHOW DATABASES - Spark 3.0.0-preview2 Documentation
sql-ref-syntax-aux-show-functions.html 2024-08-16 12:27 21K SHOW FUNCTIONS - Spark 3.0.0-preview2 Documentation
sql-ref-syntax-aux-show-partitions.html 2024-08-16 12:27 23K SHOW PARTITIONS - Spark 3.0.0-preview2 Documentation
sql-ref-syntax-aux-show-table.html 2024-08-16 12:27 40K SHOW TABLE EXTENDED - Spark 3.0.0-preview2 Documentation
sql-ref-syntax-aux-show-tables.html 2024-08-16 12:27 22K SHOW TABLES - Spark 3.0.0-preview2 Documentation
sql-ref-syntax-aux-show-tblproperties.html 2024-08-16 12:27 21K SHOW TBLPROPERTIES - Spark 3.0.0-preview2 Documentation
sql-ref-syntax-aux-show.html 2024-08-16 12:27 14K Reference - Spark 3.0.0-preview2 Documentation
sql-ref-syntax-aux.html 2024-08-16 12:27 13K Reference - Spark 3.0.0-preview2 Documentation
sql-ref-syntax-ddl-alter-database.html 2024-08-16 12:27 18K ALTER DATABASE - Spark 3.0.0-preview2 Documentation
sql-ref-syntax-ddl-alter-table.html 2024-08-16 12:27 35K ALTER TABLE - Spark 3.0.0-preview2 Documentation
sql-ref-syntax-ddl-alter-view.html 2024-08-16 12:27 33K ALTER VIEW - Spark 3.0.0-preview2 Documentation
sql-ref-syntax-ddl-create-database.html 2024-08-16 12:27 19K CREATE DATABASE - Spark 3.0.0-preview2 Documentation
sql-ref-syntax-ddl-create-function.html 2024-08-16 12:27 24K CREATE FUNCTION - Spark 3.0.0-preview2 Documentation
sql-ref-syntax-ddl-create-table.html 2024-08-16 12:27 14K CREATE TABLE - Spark 3.0.0-preview2 Documentation
sql-ref-syntax-ddl-create-view.html 2024-08-16 12:27 18K CREATE VIEW - Spark 3.0.0-preview2 Documentation
sql-ref-syntax-ddl-drop-database.html 2024-08-16 12:27 17K DROP DATABASE - Spark 3.0.0-preview2 Documentation
sql-ref-syntax-ddl-drop-function.html 2024-08-16 12:27 20K DROP FUNCTION - Spark 3.0.0-preview2 Documentation
sql-ref-syntax-ddl-drop-table.html 2024-08-16 12:27 18K DROP TABLE - Spark 3.0.0-preview2 Documentation
sql-ref-syntax-ddl-drop-view.html 2024-08-16 12:27 17K DROP VIEW - Spark 3.0.0-preview2 Documentation
sql-ref-syntax-ddl-repair-table.html 2024-08-16 12:27 17K REPAIR TABLE - Spark 3.0.0-preview2 Documentation
sql-ref-syntax-ddl-truncate-table.html 2024-08-16 12:27 19K TRUNCATE TABLE - Spark 3.0.0-preview2 Documentation
sql-ref-syntax-ddl.html 2024-08-16 12:27 14K Reference - Spark 3.0.0-preview2 Documentation
sql-ref-syntax-dml-insert-into.html 2024-08-16 12:27 38K INSERT INTO - Spark 3.0.0-preview2 Documentation
sql-ref-syntax-dml-insert-overwrite-directory-hive.html 2024-08-16 12:27 17K INSERT OVERWRITE DIRECTORY with Hive format - Spark 3.0.0-preview2 Documentation
sql-ref-syntax-dml-insert-overwrite-directory.html 2024-08-16 12:27 17K INSERT OVERWRITE DIRECTORY - Spark 3.0.0-preview2 Documentation
sql-ref-syntax-dml-insert-overwrite-table.html 2024-08-16 12:27 34K INSERT OVERWRITE - Spark 3.0.0-preview2 Documentation
sql-ref-syntax-dml-insert.html 2024-08-16 12:27 12K INSERT - Spark 3.0.0-preview2 Documentation
sql-ref-syntax-dml-load.html 2024-08-16 12:27 23K LOAD DATA - Spark 3.0.0-preview2 Documentation
sql-ref-syntax-dml.html 2024-08-16 12:27 12K Data Manipulation Statements - Spark 3.0.0-preview2 Documentation
sql-ref-syntax-qry-aggregation.html 2024-08-16 12:27 12K Aggregation (CUBE/ROLLUP/GROUPING) - Spark 3.0.0-preview2 Documentation
sql-ref-syntax-qry-explain.html 2024-08-16 12:27 12K EXPLAIN - Spark 3.0.0-preview2 Documentation
sql-ref-syntax-qry-sampling.html 2024-08-16 12:27 12K Sampling Queries - Spark 3.0.0-preview2 Documentation
sql-ref-syntax-qry-select-cte.html 2024-08-16 12:27 14K Common Table Expression (CTE) - Spark 3.0.0-preview2 Documentation
sql-ref-syntax-qry-select-distinct.html 2024-08-16 12:27 14K Distinct operator - Spark 3.0.0-preview2 Documentation
sql-ref-syntax-qry-select-groupby.html 2024-08-16 12:27 14K GROUPBY Clause - Spark 3.0.0-preview2 Documentation
sql-ref-syntax-qry-select-having.html 2024-08-16 12:27 14K HAVING Clause - Spark 3.0.0-preview2 Documentation
sql-ref-syntax-qry-select-hints.html 2024-08-16 12:27 14K Query Hints - Spark 3.0.0-preview2 Documentation
sql-ref-syntax-qry-select-join.html 2024-08-16 12:27 14K JOIN - Spark 3.0.0-preview2 Documentation
sql-ref-syntax-qry-select-limit.html 2024-08-16 12:27 14K LIMIT operator - Spark 3.0.0-preview2 Documentation
sql-ref-syntax-qry-select-orderby.html 2024-08-16 12:27 14K ORDER BY Clause - Spark 3.0.0-preview2 Documentation
sql-ref-syntax-qry-select-setops.html 2024-08-16 12:27 14K Set Operations - Spark 3.0.0-preview2 Documentation
sql-ref-syntax-qry-select-subqueries.html 2024-08-16 12:27 14K Subqueries - Spark 3.0.0-preview2 Documentation
sql-ref-syntax-qry-select-usedb.html 2024-08-16 12:27 16K USE Database - Spark 3.0.0-preview2 Documentation
sql-ref-syntax-qry-select.html 2024-08-16 12:27 14K SELECT - Spark 3.0.0-preview2 Documentation
sql-ref-syntax-qry-window.html 2024-08-16 12:27 12K Windowing Analytic Functions - Spark 3.0.0-preview2 Documentation
sql-ref-syntax-qry.html 2024-08-16 12:27 13K Reference - Spark 3.0.0-preview2 Documentation
sql-ref-syntax.html 2024-08-16 12:27 12K Reference - Spark 3.0.0-preview2 Documentation
sql-ref.html 2024-08-16 12:27 11K Reference - Spark 3.0.0-preview2 Documentation
ss-migration-guide.html 2024-08-16 12:27 11K Migration Guide: Structured Streaming - Spark 3.0.0-preview2 Documentation
storage-openstack-swift.html 2024-08-16 12:27 15K Accessing OpenStack Swift from Spark - Spark 3.0.0-preview2 Documentation
streaming-custom-receivers.html 2024-08-16 12:27 30K Spark Streaming Custom Receivers - Spark 3.0.0-preview2 Documentation
streaming-kafka-0-10-integration.html 2024-08-16 12:27 47K Spark Streaming + Kafka Integration Guide (Kafka broker version 0.10.0 or higher) - Spark 3.0.0-preview2 Documentation
streaming-kafka-integration.html 2024-08-16 12:27 9.2K Spark Streaming + Kafka Integration Guide - Spark 3.0.0-preview2 Documentation
streaming-kinesis-integration.html 2024-08-16 12:27 30K Spark Streaming + Kinesis Integration - Spark 3.0.0-preview2 Documentation
streaming-programming-guide.html 2024-08-16 12:27 213K Spark Streaming - Spark 3.0.0-preview2 Documentation
structured-streaming-kafka-integration.html 2024-08-16 12:27 79K Structured Streaming + Kafka Integration Guide (Kafka broker version 0.10.0 or higher) - Spark 3.0.0-preview2 Documentation
structured-streaming-programming-guide.html 2024-08-16 12:27 261K Structured Streaming Programming Guide - Spark 3.0.0-preview2 Documentation
submitting-applications.html 2024-08-16 12:27 26K Submitting Applications - Spark 3.0.0-preview2 Documentation
tuning.html 2024-08-16 12:27 35K Tuning - Spark 3.0.0-preview2 Documentation
web-ui.html 2024-08-16 12:27 38K Web UI - Spark 3.0.0-preview2 Documentation
Welcome to the Spark documentation!
This readme will walk you through navigating and building the Spark documentation, which is included
here with the Spark source code. You can also find documentation specific to release versions of
Spark at https://spark.apache.org/documentation.html.
Read on to learn more about viewing documentation in plain text (i.e., markdown) or building the
documentation yourself. Why build it yourself? So that you have the docs that correspond to
whichever version of Spark you currently have checked out of revision control.
Prerequisites
The Spark documentation build uses a number of tools to build HTML docs and API docs in Scala, Java,
Python, R and SQL.
You need to have Ruby and
Python
installed. Also install the following libraries:
$ sudo gem install jekyll jekyll-redirect-from rouge
# Following is needed only for generating API docs
$ sudo pip install sphinx pypandoc mkdocs
$ sudo Rscript -e 'install.packages(c("knitr", "devtools", "rmarkdown"), repos="https://cloud.r-project.org/")'
$ sudo Rscript -e 'devtools::install_version("roxygen2", version = "5.0.1", repos="https://cloud.r-project.org/")'
$ sudo Rscript -e 'devtools::install_version("testthat", version = "1.0.2", repos="https://cloud.r-project.org/")'
Note: If you are on a system with both Ruby 1.9 and Ruby 2.0 you may need to replace gem with gem2.0.
Note: Other versions of roxygen2 might work in SparkR documentation generation but RoxygenNote field in $SPARK_HOME/R/pkg/DESCRIPTION is 5.0.1, which is updated if the version is mismatched.
Generating the Documentation HTML
We include the Spark documentation as part of the source (as opposed to using a hosted wiki, such as
the github wiki, as the definitive documentation) to enable the documentation to evolve along with
the source code and be captured by revision control (currently git). This way the code automatically
includes the version of the documentation that is relevant regardless of which version or release
you have checked out or downloaded.
In this directory you will find text files formatted using Markdown, with an “.md” suffix. You can
read those text files directly if you want. Start with index.md.
Execute jekyll build from the docs/ directory to compile the site. Compiling the site with
Jekyll will create a directory called _site containing index.html as well as the rest of the
compiled files.
You can modify the default Jekyll build as follows:
# Skip generating API docs (which takes a while)
$ SKIP_API=1 jekyll build
# Serve content locally on port 4000
$ jekyll serve --watch
# Build the site with extra features used on the live page
$ PRODUCTION=1 jekyll build
API Docs (Scaladoc, Javadoc, Sphinx, roxygen2, MkDocs)
You can build just the Spark scaladoc and javadoc by running ./build/sbt unidoc from the $SPARK_HOME directory.
Similarly, you can build just the PySpark docs by running make html from the
$SPARK_HOME/python/docs directory. Documentation is only generated for classes that are listed as
public in __init__.py. The SparkR docs can be built by running $SPARK_HOME/R/create-docs.sh, and
the SQL docs can be built by running $SPARK_HOME/sql/create-docs.sh
after building Spark first.
When you run jekyll build in the docs directory, it will also copy over the scaladoc and javadoc for the various
Spark subprojects into the docs directory (and then also into the _site directory). We use a
jekyll plugin to run ./build/sbt unidoc before building the site so if you haven’t run it (recently) it
may take some time as it generates all of the scaladoc and javadoc using Unidoc.
The jekyll plugin also generates the PySpark docs using Sphinx, SparkR docs
using roxygen2 and SQL docs
using MkDocs.
NOTE: To skip the step of building and copying over the Scala, Java, Python, R and SQL API docs, run SKIP_API=1
jekyll build. In addition, SKIP_SCALADOC=1, SKIP_PYTHONDOC=1, SKIP_RDOC=1 and SKIP_SQLDOC=1 can be used
to skip a single step of the corresponding language. SKIP_SCALADOC indicates skipping both the Scala and Java docs.
Automatically Rebuilding API Docs
jekyll serve --watch will only watch what’s in docs/, and it won’t follow symlinks. That means it won’t monitor your API docs under python/docs or elsewhere.
To work around this limitation for Python, install entr and run the following in a separate shell:
cd "$SPARK_HOME/python/docs"
find .. -type f -name '*.py' \
| entr -s 'make html && cp -r _build/html/. ../../docs/api/python'
Whenever there is a change to your Python code, entr will automatically rebuild the Python API docs and copy them to docs/, thus triggering a Jekyll update.