PySpark SparkFiles

by Online Tutorials Library July 14, 2022

PySpark SparkFiles

PySpark provides the facility to upload your files using sc.addFile. We can also get the path of working directory using SparkFiles.get. Moreover, to resolve the path to the files added through SparkContext.addFile(), the following types of class method are available in SparkFiles, such as:

get(filename)
getrootdirectory()

Note: SparkFiles only contain the class method that can be used as required. Users should not create SparkFiles instances.

Let’s learn about the class method in detail.

Class Methods of PySpark SparkFiles

get(filename)

The get(filename) specifies the path of the file which is added through SparkContext.addFile().

 import os class SparkFiles(object):      “””      Resolves paths to files added through      L{SparkContext.addFile()<pyspark.context.SparkContext.addFile>}.      SparkFiles consist of only class methods; users should not create SparkFiles      instances.      “””       root_directory = None       is_running_on_worker = False       sc = None       def __init__(self):          raise NotImplementedError(“Do not construct SparkFiles objects”)      @classmethod        def get(cls, filename):          “””          Get the absolute path of a file added through C{SparkContext.addFile()}.          “””          path = os.path.join(SparkFiles.getRootDirectory(), filename)          return os.path.abspath(path)         @classmethod         def getRootDirectory(cls): 

getrootdirectory()

This class method specifies the path to the root directory. Basically, it contains the whole file which is added through the SparkContext.addFile().

 import os   class SparkFiles(object):      “””      Resolves paths to files added through      L{SparkContext.addFile()<pyspark.context.SparkContext.addFile>}.      SparkFiles contains only classmethods; users should not create SparkFiles      instances.      “””       root_directory = None       is_running_on_worker = False       sc = None       def __init__(self):          raise NotImplementedError(“Do not construct SparkFiles objects”)      @classmethod       def get(cls, filename):         …      @classmethod       def getRootDirectory(cls):          “””          Than Get the root directory which contains files added through          C{SparkContext.addFile()}.          “””  if cls._is_running_on_worker:      return cls._root_directory  else:      # This will have to change if we support multiple SparkContexts:      return cls._sc._jvm.spark.SparkFiles.getRootDirectory() 

Next TopicPySpark StorageLevel

PySpark SparkFiles

PySpark SparkFiles

Note: SparkFiles only contain the class method that can be used as required. Users should not create SparkFiles instances.

Class Methods of PySpark SparkFiles

Arithmetic in Prolog

Automate Instagram Messages using Python

You may also like