Index of /apache/spark/docs/3.0.0-preview
Name Last modified Size Description
![[PARENTDIR]](/icons/back.gif)
Parent Directory -
![[DIR]](/icons/folder.gif)
api/ 2024-08-16 12:37 -
css/ 2024-08-16 12:37 -
img/ 2024-08-16 12:37 -
js/ 2024-08-16 12:37 -
![[TXT]](/icons/text.gif)
README.html 2024-08-16 12:27 5.2K
api.html 2024-08-16 12:27 9.1K Spark API Documentation - Spark 3.0.0-preview Documentation
building-spark.html 2024-08-16 12:27 27K Building Spark - Spark 3.0.0-preview Documentation
building-with-maven.html 2024-08-16 12:27 425 Redirecting…
cloud-integration.html 2024-08-16 12:27 23K Integration with Cloud Infrastructures - Spark 3.0.0-preview Documentation
cluster-overview.html 2024-08-16 12:27 15K Cluster Mode Overview - Spark 3.0.0-preview Documentation
configuration.html 2024-08-16 12:27 122K Configuration - Spark 3.0.0-preview Documentation
contributing-to-spark.html 2024-08-16 12:27 8.9K Contributing to Spark - Spark 3.0.0-preview Documentation
core-migration-guide.html 2024-08-16 12:27 11K Migration Guide: Spark Core - Spark 3.0.0-preview Documentation
graphx-programming-guide.html 2024-08-16 12:27 159K GraphX - Spark 3.0.0-preview Documentation
hadoop-provided.html 2024-08-16 12:27 10K Using Spark's "Hadoop Free" Build - Spark 3.0.0-preview Documentation
hardware-provisioning.html 2024-08-16 12:27 13K Hardware Provisioning - Spark 3.0.0-preview Documentation
index.html 2024-08-16 12:27 19K Overview - Spark 3.0.0-preview Documentation
job-scheduling.html 2024-08-16 12:27 30K Job Scheduling - Spark 3.0.0-preview Documentation
migration-guide.html 2024-08-16 12:27 10K Migration Guide - Spark 3.0.0-preview Documentation
ml-advanced.html 2024-08-16 12:27 20K Advanced topics - Spark 3.0.0-preview Documentation
ml-ann.html 2024-08-16 12:27 12K Multilayer perceptron classifier - Spark 3.0.0-preview Documentation
ml-classification-regression.html 2024-08-16 12:27 390K Classification and regression - Spark 3.0.0-preview Documentation
ml-clustering.html 2024-08-16 12:27 82K Clustering - Spark 3.0.0-preview Documentation
ml-collaborative-filtering.html 2024-08-16 12:27 58K Collaborative Filtering - Spark 3.0.0-preview Documentation
ml-datasource.html 2024-08-16 12:27 43K Data sources - Spark 3.0.0-preview Documentation
ml-decision-tree.html 2024-08-16 12:27 12K Decision trees - Spark 3.0.0-preview Documentation
ml-ensembles.html 2024-08-16 12:27 12K Tree ensemble methods - Spark 3.0.0-preview Documentation
ml-features.html 2024-08-16 12:27 553K Extracting, transforming and selecting features - Spark 3.0.0-preview Documentation
ml-frequent-pattern-mining.html 2024-08-16 12:27 44K Frequent Pattern Mining - Spark 3.0.0-preview Documentation
ml-guide.html 2024-08-16 12:27 19K MLlib: Main Guide - Spark 3.0.0-preview Documentation
ml-linear-methods.html 2024-08-16 12:27 12K Linear methods - Spark 3.0.0-preview Documentation
ml-migration-guide.html 2024-08-16 12:27 45K Migration Guide: MLlib (Machine Learning) - Spark 3.0.0-preview Documentation
ml-pipeline.html 2024-08-16 12:27 92K ML Pipelines - Spark 3.0.0-preview Documentation
ml-statistics.html 2024-08-16 12:27 62K Basic Statistics - Spark 3.0.0-preview Documentation
ml-survival-regression.html 2024-08-16 12:27 12K Survival Regression - Spark 3.0.0-preview Documentation
ml-tuning.html 2024-08-16 12:27 75K ML Tuning - Spark 3.0.0-preview Documentation
mllib-classification-regression.html 2024-08-16 12:27 15K Classification and Regression - RDD-based API - Spark 3.0.0-preview Documentation
mllib-clustering.html 2024-08-16 12:27 120K Clustering - RDD-based API - Spark 3.0.0-preview Documentation
mllib-collaborative-filtering.html 2024-08-16 12:27 42K Collaborative Filtering - RDD-based API - Spark 3.0.0-preview Documentation
mllib-data-types.html 2024-08-16 12:27 83K Data Types - RDD-based API - Spark 3.0.0-preview Documentation
mllib-decision-tree.html 2024-08-16 12:27 69K Decision Trees - RDD-based API - Spark 3.0.0-preview Documentation
mllib-dimensionality-reduction.html 2024-08-16 12:27 45K Dimensionality Reduction - RDD-based API - Spark 3.0.0-preview Documentation
mllib-ensembles.html 2024-08-16 12:27 114K Ensembles - RDD-based API - Spark 3.0.0-preview Documentation
mllib-evaluation-metrics.html 2024-08-16 12:27 157K Evaluation Metrics - RDD-based API - Spark 3.0.0-preview Documentation
mllib-feature-extraction.html 2024-08-16 12:27 71K Feature Extraction and Transformation - RDD-based API - Spark 3.0.0-preview Documentation
mllib-frequent-pattern-mining.html 2024-08-16 12:27 49K Frequent Pattern Mining - RDD-based API - Spark 3.0.0-preview Documentation
mllib-guide.html 2024-08-16 12:27 16K MLlib: RDD-based API - Spark 3.0.0-preview Documentation
mllib-isotonic-regression.html 2024-08-16 12:27 35K Isotonic regression - RDD-based API - Spark 3.0.0-preview Documentation
mllib-linear-methods.html 2024-08-16 12:27 75K Linear Methods - RDD-based API - Spark 3.0.0-preview Documentation
mllib-naive-bayes.html 2024-08-16 12:27 32K Naive Bayes - RDD-based API - Spark 3.0.0-preview Documentation
mllib-optimization.html 2024-08-16 12:27 46K Optimization - RDD-based API - Spark 3.0.0-preview Documentation
mllib-pmml-model-export.html 2024-08-16 12:27 18K PMML model export - RDD-based API - Spark 3.0.0-preview Documentation
mllib-statistics.html 2024-08-16 12:27 106K Basic Statistics - RDD-based API - Spark 3.0.0-preview Documentation
monitoring.html 2024-08-16 12:27 58K Monitoring and Instrumentation - Spark 3.0.0-preview Documentation
programming-guide.html 2024-08-16 12:27 8.9K Spark Programming Guide - Spark 3.0.0-preview Documentation
pyspark-migration-guide.html 2024-08-16 12:27 17K Migration Guide: PySpark (Python on Spark) - Spark 3.0.0-preview Documentation
quick-start.html 2024-08-16 12:27 52K Quick Start - Spark 3.0.0-preview Documentation
rdd-programming-guide.html 2024-08-16 12:27 146K RDD Programming Guide - Spark 3.0.0-preview Documentation
![[ ]](/icons/unknown.gif)
redirects.json 2024-08-16 12:27 52
running-on-kubernetes.html 2024-08-16 12:27 81K Running Spark on Kubernetes - Spark 3.0.0-preview Documentation
running-on-mesos.html 2024-08-16 12:27 51K Running Spark on Mesos - Spark 3.0.0-preview Documentation
running-on-yarn.html 2024-08-16 12:27 47K Running Spark on YARN - Spark 3.0.0-preview Documentation
security.html 2024-08-16 12:27 50K Security - Spark 3.0.0-preview Documentation
spark-standalone.html 2024-08-16 12:27 41K Spark Standalone Mode - Spark 3.0.0-preview Documentation
sparkr-migration-guide.html 2024-08-16 12:27 17K Migration Guide: SparkR (R on Spark) - Spark 3.0.0-preview Documentation
sparkr.html 2024-08-16 12:27 66K SparkR (R on Spark) - Spark 3.0.0-preview Documentation
sql-data-sources-avro.html 2024-08-16 12:27 40K Apache Avro Data Source Guide - Spark 3.0.0-preview Documentation
sql-data-sources-binaryFile.html 2024-08-16 12:27 15K Binary File Data Source - Spark 3.0.0-preview Documentation
sql-data-sources-hive-tables.html 2024-08-16 12:27 49K Hive Tables - Spark 3.0.0-preview Documentation
sql-data-sources-jdbc.html 2024-08-16 12:27 41K JDBC To Other Databases - Spark 3.0.0-preview Documentation
sql-data-sources-json.html 2024-08-16 12:27 28K JSON Files - Spark 3.0.0-preview Documentation
sql-data-sources-load-save-functions.html 2024-08-16 12:27 54K Generic Load/Save Functions - Spark 3.0.0-preview Documentation
sql-data-sources-orc.html 2024-08-16 12:27 13K ORC Files - Spark 3.0.0-preview Documentation
sql-data-sources-parquet.html 2024-08-16 12:27 54K Parquet Files - Spark 3.0.0-preview Documentation
sql-data-sources-troubleshooting.html 2024-08-16 12:27 12K Troubleshooting - Spark 3.0.0-preview Documentation
sql-data-sources.html 2024-08-16 12:27 15K Data Sources - Spark 3.0.0-preview Documentation
sql-distributed-sql-engine.html 2024-08-16 12:27 15K Distributed SQL Engine - Spark 3.0.0-preview Documentation
sql-getting-started.html 2024-08-16 12:27 142K Getting Started - Spark 3.0.0-preview Documentation
sql-keywords.html 2024-08-16 12:27 34K Spark SQL Keywords - Spark 3.0.0-preview Documentation
sql-migration-guide.html 2024-08-16 12:27 86K Migration Guide: SQL, Datasets and DataFrame - Spark 3.0.0-preview Documentation
sql-migration-old.html 2024-08-16 12:27 10K Migration Guide - Spark 3.0.0-preview Documentation
sql-performance-tuning.html 2024-08-16 12:27 20K Performance Tuning - Spark 3.0.0-preview Documentation
sql-programming-guide.html 2024-08-16 12:27 14K Spark SQL and DataFrames - Spark 3.0.0-preview Documentation
sql-pyspark-pandas-with-arrow.html 2024-08-16 12:27 48K PySpark Usage Guide for Pandas with Apache Arrow - Spark 3.0.0-preview Documentation
sql-ref-arithmetic-ops.html 2024-08-16 12:27 11K Arithmetic Operations - Spark 3.0.0-preview Documentation
sql-ref-datatypes.html 2024-08-16 12:27 26K Data Types - Spark 3.0.0-preview Documentation
sql-ref-functions-builtin-aggregate.html 2024-08-16 12:27 12K Builtin Aggregate Functions - Spark 3.0.0-preview Documentation
sql-ref-functions-builtin-scalar.html 2024-08-16 12:27 12K Builtin Scalar Functions - Spark 3.0.0-preview Documentation
sql-ref-functions-builtin.html 2024-08-16 12:27 12K Reference - Spark 3.0.0-preview Documentation
sql-ref-functions-udf-aggregate.html 2024-08-16 12:27 12K User defined Aggregate Functions (UDAF) - Spark 3.0.0-preview Documentation
sql-ref-functions-udf-scalar.html 2024-08-16 12:27 12K User defined Scalar Functions (UDF) - Spark 3.0.0-preview Documentation
sql-ref-functions-udf.html 2024-08-16 12:27 12K Reference - Spark 3.0.0-preview Documentation
sql-ref-functions.html 2024-08-16 12:27 11K Reference - Spark 3.0.0-preview Documentation
sql-ref-nan-semantics.html 2024-08-16 12:27 11K Nan Semantics - Spark 3.0.0-preview Documentation
sql-ref-null-semantics.html 2024-08-16 12:27 67K NULL Semantics - Spark 3.0.0-preview Documentation
sql-ref-syntax-aux-analyze-table.html 2024-08-16 12:27 18K ANALYZE TABLE - Spark 3.0.0-preview Documentation
sql-ref-syntax-aux-analyze.html 2024-08-16 12:27 13K ANALYZE - Spark 3.0.0-preview Documentation
sql-ref-syntax-aux-cache-cache-table.html 2024-08-16 12:27 16K CACHE TABLE - Spark 3.0.0-preview Documentation
sql-ref-syntax-aux-cache-clear-cache.html 2024-08-16 12:27 14K CLEAR CACHE - Spark 3.0.0-preview Documentation
sql-ref-syntax-aux-cache-uncache-table.html 2024-08-16 12:27 14K UNCACHE TABLE - Spark 3.0.0-preview Documentation
sql-ref-syntax-aux-cache.html 2024-08-16 12:27 13K Cache - Spark 3.0.0-preview Documentation
sql-ref-syntax-aux-conf-mgmt-reset.html 2024-08-16 12:27 14K RESET - Spark 3.0.0-preview Documentation
sql-ref-syntax-aux-conf-mgmt-set.html 2024-08-16 12:27 16K SET - Spark 3.0.0-preview Documentation
sql-ref-syntax-aux-conf-mgmt.html 2024-08-16 12:27 13K Reference - Spark 3.0.0-preview Documentation
sql-ref-syntax-aux-describe-database.html 2024-08-16 12:27 21K DESCRIBE DATABASE - Spark 3.0.0-preview Documentation
sql-ref-syntax-aux-describe-function.html 2024-08-16 12:27 26K DESCRIBE FUNCTION - Spark 3.0.0-preview Documentation
sql-ref-syntax-aux-describe-query.html 2024-08-16 12:27 22K DESCRIBE QUERY - Spark 3.0.0-preview Documentation
sql-ref-syntax-aux-describe-table.html 2024-08-16 12:27 36K DESCRIBE TABLE - Spark 3.0.0-preview Documentation
sql-ref-syntax-aux-describe.html 2024-08-16 12:27 13K Reference - Spark 3.0.0-preview Documentation
sql-ref-syntax-aux-refresh-table.html 2024-08-16 12:27 15K REFRESH TABLE - Spark 3.0.0-preview Documentation
sql-ref-syntax-aux-resource-mgmt-add-file.html 2024-08-16 12:27 14K ADD FILE - Spark 3.0.0-preview Documentation
sql-ref-syntax-aux-resource-mgmt-add-jar.html 2024-08-16 12:27 14K ADD JAR - Spark 3.0.0-preview Documentation
sql-ref-syntax-aux-resource-mgmt-list-file.html 2024-08-16 12:27 14K LIST FILE - Spark 3.0.0-preview Documentation
sql-ref-syntax-aux-resource-mgmt-list-jar.html 2024-08-16 12:27 15K LIST JAR - Spark 3.0.0-preview Documentation
sql-ref-syntax-aux-resource-mgmt.html 2024-08-16 12:27 13K Reference - Spark 3.0.0-preview Documentation
sql-ref-syntax-aux-show-columns.html 2024-08-16 12:27 18K SHOW COLUMNS - Spark 3.0.0-preview Documentation
sql-ref-syntax-aux-show-create-table.html 2024-08-16 12:27 17K SHOW CREATE TABLE - Spark 3.0.0-preview Documentation
sql-ref-syntax-aux-show-databases.html 2024-08-16 12:27 18K SHOW DATABASES - Spark 3.0.0-preview Documentation
sql-ref-syntax-aux-show-functions.html 2024-08-16 12:27 21K SHOW FUNCTIONS - Spark 3.0.0-preview Documentation
sql-ref-syntax-aux-show-partitions.html 2024-08-16 12:27 14K SHOW PARTITIONS - Spark 3.0.0-preview Documentation
sql-ref-syntax-aux-show-table.html 2024-08-16 12:27 14K SHOW TABLE - Spark 3.0.0-preview Documentation
sql-ref-syntax-aux-show-tables.html 2024-08-16 12:27 22K SHOW TABLES - Spark 3.0.0-preview Documentation
sql-ref-syntax-aux-show-tblproperties.html 2024-08-16 12:27 21K SHOW TBLPROPERTIES - Spark 3.0.0-preview Documentation
sql-ref-syntax-aux-show.html 2024-08-16 12:27 14K Reference - Spark 3.0.0-preview Documentation
sql-ref-syntax-aux.html 2024-08-16 12:27 13K Reference - Spark 3.0.0-preview Documentation
sql-ref-syntax-ddl-alter-database.html 2024-08-16 12:27 18K ALTER DATABASE - Spark 3.0.0-preview Documentation
sql-ref-syntax-ddl-alter-table.html 2024-08-16 12:27 34K ALTER TABLE - Spark 3.0.0-preview Documentation
sql-ref-syntax-ddl-alter-view.html 2024-08-16 12:27 14K ALTER VIEW - Spark 3.0.0-preview Documentation
sql-ref-syntax-ddl-create-database.html 2024-08-16 12:27 19K CREATE DATABASE - Spark 3.0.0-preview Documentation
sql-ref-syntax-ddl-create-function.html 2024-08-16 12:27 24K CREATE FUNCTION - Spark 3.0.0-preview Documentation
sql-ref-syntax-ddl-create-table.html 2024-08-16 12:27 14K CREATE TABLE - Spark 3.0.0-preview Documentation
sql-ref-syntax-ddl-create-view.html 2024-08-16 12:27 18K CREATE VIEW - Spark 3.0.0-preview Documentation
sql-ref-syntax-ddl-drop-database.html 2024-08-16 12:27 17K DROP DATABASE - Spark 3.0.0-preview Documentation
sql-ref-syntax-ddl-drop-function.html 2024-08-16 12:27 20K DROP FUNCTION - Spark 3.0.0-preview Documentation
sql-ref-syntax-ddl-drop-table.html 2024-08-16 12:27 14K DROP TABLE - Spark 3.0.0-preview Documentation
sql-ref-syntax-ddl-drop-view.html 2024-08-16 12:27 14K DROP VIEW - Spark 3.0.0-preview Documentation
sql-ref-syntax-ddl-repair-table.html 2024-08-16 12:27 17K REPAIR TABLE - Spark 3.0.0-preview Documentation
sql-ref-syntax-ddl-truncate-table.html 2024-08-16 12:27 19K TRUNCATE TABLE - Spark 3.0.0-preview Documentation
sql-ref-syntax-ddl.html 2024-08-16 12:27 14K Reference - Spark 3.0.0-preview Documentation
sql-ref-syntax-dml-insert-into.html 2024-08-16 12:27 38K INSERT INTO - Spark 3.0.0-preview Documentation
sql-ref-syntax-dml-insert-overwrite-directory-hive.html 2024-08-16 12:27 17K INSERT OVERWRITE DIRECTORY with Hive format - Spark 3.0.0-preview Documentation
sql-ref-syntax-dml-insert-overwrite-directory.html 2024-08-16 12:27 17K INSERT OVERWRITE DIRECTORY - Spark 3.0.0-preview Documentation
sql-ref-syntax-dml-insert-overwrite-table.html 2024-08-16 12:27 34K INSERT OVERWRITE - Spark 3.0.0-preview Documentation
sql-ref-syntax-dml-insert.html 2024-08-16 12:27 12K INSERT - Spark 3.0.0-preview Documentation
sql-ref-syntax-dml-load.html 2024-08-16 12:27 23K LOAD DATA - Spark 3.0.0-preview Documentation
sql-ref-syntax-dml.html 2024-08-16 12:27 12K Data Manipulation Statements - Spark 3.0.0-preview Documentation
sql-ref-syntax-qry-aggregation.html 2024-08-16 12:27 12K Aggregation (CUBE/ROLLUP/GROUPING) - Spark 3.0.0-preview Documentation
sql-ref-syntax-qry-explain.html 2024-08-16 12:27 12K EXPLAIN - Spark 3.0.0-preview Documentation
sql-ref-syntax-qry-sampling.html 2024-08-16 12:27 12K Sampling Queries - Spark 3.0.0-preview Documentation
sql-ref-syntax-qry-select-cte.html 2024-08-16 12:27 14K Common Table Expression (CTE) - Spark 3.0.0-preview Documentation
sql-ref-syntax-qry-select-distinct.html 2024-08-16 12:27 14K Distinct operator - Spark 3.0.0-preview Documentation
sql-ref-syntax-qry-select-groupby.html 2024-08-16 12:27 14K GROUPBY Clause - Spark 3.0.0-preview Documentation
sql-ref-syntax-qry-select-having.html 2024-08-16 12:27 14K HAVING Clause - Spark 3.0.0-preview Documentation
sql-ref-syntax-qry-select-hints.html 2024-08-16 12:27 14K Query Hints - Spark 3.0.0-preview Documentation
sql-ref-syntax-qry-select-join.html 2024-08-16 12:27 14K JOIN - Spark 3.0.0-preview Documentation
sql-ref-syntax-qry-select-limit.html 2024-08-16 12:27 14K LIMIT operator - Spark 3.0.0-preview Documentation
sql-ref-syntax-qry-select-orderby.html 2024-08-16 12:27 14K ORDER BY Clause - Spark 3.0.0-preview Documentation
sql-ref-syntax-qry-select-setops.html 2024-08-16 12:27 14K Set Operations - Spark 3.0.0-preview Documentation
sql-ref-syntax-qry-select-subqueries.html 2024-08-16 12:27 14K Subqueries - Spark 3.0.0-preview Documentation
sql-ref-syntax-qry-select-usedb.html 2024-08-16 12:27 16K USE Database - Spark 3.0.0-preview Documentation
sql-ref-syntax-qry-select.html 2024-08-16 12:27 14K SELECT - Spark 3.0.0-preview Documentation
sql-ref-syntax-qry-window.html 2024-08-16 12:27 12K Windowing Analytic Functions - Spark 3.0.0-preview Documentation
sql-ref-syntax-qry.html 2024-08-16 12:27 13K Reference - Spark 3.0.0-preview Documentation
sql-ref-syntax.html 2024-08-16 12:27 12K Reference - Spark 3.0.0-preview Documentation
sql-ref.html 2024-08-16 12:27 11K Reference - Spark 3.0.0-preview Documentation
ss-migration-guide.html 2024-08-16 12:27 11K Migration Guide: Structured Streaming - Spark 3.0.0-preview Documentation
storage-openstack-swift.html 2024-08-16 12:27 15K Accessing OpenStack Swift from Spark - Spark 3.0.0-preview Documentation
streaming-custom-receivers.html 2024-08-16 12:27 29K Spark Streaming Custom Receivers - Spark 3.0.0-preview Documentation
streaming-kafka-0-10-integration.html 2024-08-16 12:27 47K Spark Streaming + Kafka Integration Guide (Kafka broker version 0.10.0 or higher) - Spark 3.0.0-preview Documentation
streaming-kafka-integration.html 2024-08-16 12:27 9.2K Spark Streaming + Kafka Integration Guide - Spark 3.0.0-preview Documentation
streaming-kinesis-integration.html 2024-08-16 12:27 28K Spark Streaming + Kinesis Integration - Spark 3.0.0-preview Documentation
streaming-programming-guide.html 2024-08-16 12:27 210K Spark Streaming - Spark 3.0.0-preview Documentation
structured-streaming-kafka-integration.html 2024-08-16 12:27 82K Structured Streaming + Kafka Integration Guide (Kafka broker version 0.10.0 or higher) - Spark 3.0.0-preview Documentation
structured-streaming-programming-guide.html 2024-08-16 12:27 257K Structured Streaming Programming Guide - Spark 3.0.0-preview Documentation
submitting-applications.html 2024-08-16 12:27 24K Submitting Applications - Spark 3.0.0-preview Documentation
tuning.html 2024-08-16 12:27 33K Tuning - Spark 3.0.0-preview Documentation
web-ui.html 2024-08-16 12:27 38K Web UI - Spark 3.0.0-preview Documentation
Welcome to the Spark documentation!
This readme will walk you through navigating and building the Spark documentation, which is included
here with the Spark source code. You can also find documentation specific to release versions of
Spark at https://spark.apache.org/documentation.html.
Read on to learn more about viewing documentation in plain text (i.e., markdown) or building the
documentation yourself. Why build it yourself? So that you have the docs that correspond to
whichever version of Spark you currently have checked out of revision control.
Prerequisites
The Spark documentation build uses a number of tools to build HTML docs and API docs in Scala, Java,
Python, R and SQL.
You need to have Ruby and
Python
installed. Also install the following libraries:
$ sudo gem install jekyll jekyll-redirect-from pygments.rb
$ sudo pip install Pygments
# Following is needed only for generating API docs
$ sudo pip install sphinx pypandoc mkdocs
$ sudo Rscript -e 'install.packages(c("knitr", "devtools", "rmarkdown"), repos="https://cloud.r-project.org/")'
$ sudo Rscript -e 'devtools::install_version("roxygen2", version = "5.0.1", repos="https://cloud.r-project.org/")'
$ sudo Rscript -e 'devtools::install_version("testthat", version = "1.0.2", repos="https://cloud.r-project.org/")'
Note: If you are on a system with both Ruby 1.9 and Ruby 2.0 you may need to replace gem with gem2.0.
Note: Other versions of roxygen2 might work in SparkR documentation generation but RoxygenNote field in $SPARK_HOME/R/pkg/DESCRIPTION is 5.0.1, which is updated if the version is mismatched.
Generating the Documentation HTML
We include the Spark documentation as part of the source (as opposed to using a hosted wiki, such as
the github wiki, as the definitive documentation) to enable the documentation to evolve along with
the source code and be captured by revision control (currently git). This way the code automatically
includes the version of the documentation that is relevant regardless of which version or release
you have checked out or downloaded.
In this directory you will find text files formatted using Markdown, with an “.md” suffix. You can
read those text files directly if you want. Start with index.md.
Execute jekyll build from the docs/ directory to compile the site. Compiling the site with
Jekyll will create a directory called _site containing index.html as well as the rest of the
compiled files.
$ cd docs
$ jekyll build
You can modify the default Jekyll build as follows:
# Skip generating API docs (which takes a while)
$ SKIP_API=1 jekyll build
# Serve content locally on port 4000
$ jekyll serve --watch
# Build the site with extra features used on the live page
$ PRODUCTION=1 jekyll build
API Docs (Scaladoc, Javadoc, Sphinx, roxygen2, MkDocs)
You can build just the Spark scaladoc and javadoc by running ./build/sbt unidoc from the $SPARK_HOME directory.
Similarly, you can build just the PySpark docs by running make html from the
$SPARK_HOME/python/docs directory. Documentation is only generated for classes that are listed as
public in __init__.py. The SparkR docs can be built by running $SPARK_HOME/R/create-docs.sh, and
the SQL docs can be built by running $SPARK_HOME/sql/create-docs.sh
after building Spark first.
When you run jekyll build in the docs directory, it will also copy over the scaladoc and javadoc for the various
Spark subprojects into the docs directory (and then also into the _site directory). We use a
jekyll plugin to run ./build/sbt unidoc before building the site so if you haven’t run it (recently) it
may take some time as it generates all of the scaladoc and javadoc using Unidoc.
The jekyll plugin also generates the PySpark docs using Sphinx, SparkR docs
using roxygen2 and SQL docs
using MkDocs.
NOTE: To skip the step of building and copying over the Scala, Java, Python, R and SQL API docs, run SKIP_API=1
jekyll build. In addition, SKIP_SCALADOC=1, SKIP_PYTHONDOC=1, SKIP_RDOC=1 and SKIP_SQLDOC=1 can be used
to skip a single step of the corresponding language. SKIP_SCALADOC indicates skipping both the Scala and Java docs.