Reference

dsg_lib.common_functions.file_functions

file_functions.py

This module provides a function to delete a file with a specified name from a specified directory.

Functions:

Name

Description

delete_file

str) -> str: Deletes a file with the specified file name from the directory specified by the directory_to_files variable. The file type is determined by the file extension, and the file is deleted from the subdirectory corresponding to the file type. Args: file_name (str): The name of the file to be deleted. Returns: str: A string indicating that the file has been deleted. Raises: TypeError: If the file name is not a string. ValueError: If the file name contains a forward slash or backslash, or if the file type is not supported. FileNotFoundError: If the file does not exist.

Example:

from dsg_lib.common_functions import file_functions

file_functions.delete_file("test.csv")

# Outputs: 'complete'

Author: Mike Ryan Date: 2024/05/16 License: MIT

create_sample_files(file_name, sample_size)

Create sample CSV and JSON files with random data.

Parameters:

Name	Type	Description	Default
file_name	str	The base name for the sample files (without extension).	required
sample_size	int	The number of rows to generate for the sample files.	required

Returns:

Type	Description
None	None

Raises:

Type	Description
Exception	If an error occurs while creating the sample files.

Example: ```python from dsg_lib.common_functions import file_functions

file_functions.create_sample_files("test", 100)

Creates 'test.csv' and 'test.json' each with 100 rows of random data ```

Source code in dsg_lib/common_functions/file_functions.py

def create_sample_files(file_name: str, sample_size: int) -> None:
    """
    Create sample CSV and JSON files with random data.

    Args:
        file_name (str): The base name for the sample files (without extension).
        sample_size (int): The number of rows to generate for the sample files.

    Returns:
        None

    Raises:
        Exception: If an error occurs while creating the sample files.

    Example:
    ```python
    from dsg_lib.common_functions import file_functions

    file_functions.create_sample_files("test", 100)

    # Creates 'test.csv' and 'test.json' each with 100 rows of random data ```
    """
    logger.debug(f"Creating sample files for {file_name} with {sample_size} rows.")

    try:
        # Generate the CSV data
        csv_header = ["name", "birth_date", "number"]
        csv_data: List[List[str]] = [csv_header]

        # Generate rows for CSV data
        for i in range(1, sample_size + 1):
            r_int: int = random.randint(0, len(first_name) - 1)
            name = first_name[r_int]
            row: List[str] = [name, generate_random_date(), str(i)]
            csv_data.append(row)

        # Save the CSV file
        csv_file = f"{file_name}.csv"
        save_csv(csv_file, csv_data)

        # Generate the JSON data
        json_data: List[dict] = []

        # Generate rows for JSON data
        for _ in range(1, sample_size + 1):
            r_int: int = random.randint(0, len(first_name) - 1)
            name = first_name[r_int]
            sample_dict: dict = {
                "name": name,
                "birthday_date": generate_random_date(),
            }
            json_data.append(sample_dict)

        # Save the JSON file
        json_file: str = f"{file_name}.json"
        save_json(json_file, json_data)

        # Log the data
        logger.debug(f"CSV Data: {csv_data}")
        logger.debug(f"JSON Data: {json_data}")

    except Exception as e:  # pragma: no cover
        logger.exception(
            f"Error occurred while creating sample files: {e}"
        )  # pragma: no cover
        raise  # pragma: no cover

delete_file(file_name)

Deletes a file with the specified file name from the specified directory. The file type is determined by the file extension.

Parameters:

Name	Type	Description	Default
directory_to_files	str	The directory where the file is located.	required
file_name	str	The name of the file to be deleted.	required

Returns:

Name	Type	Description
str	str	A message indicating whether the file has been deleted successfully
	str	or an error occurred.

Raises:

Type	Description
TypeError	If the directory or file name is not a string. ValueError: If
is not supported. FileNotFoundError	If the file does not exist.

Example:

from dsg_lib.common_functions import file_functions

file_functions.delete_file("test.csv")

# Outputs: 'File deleted successfully'

Source code in dsg_lib/common_functions/file_functions.py

def delete_file(file_name: str) -> str:
    """
    Deletes a file with the specified file name from the specified directory.
    The file type is determined by the file extension.

    Args:
        directory_to_files (str): The directory where the file is located.
        file_name (str): The name of the file to be deleted.

    Returns:
        str: A message indicating whether the file has been deleted successfully
        or an error occurred.

    Raises:
        TypeError: If the directory or file name is not a string. ValueError: If
        the file name contains a forward slash or backslash, or if the file type
        is not supported. FileNotFoundError: If the file does not exist.

    Example:
    ```python
    from dsg_lib.common_functions import file_functions

    file_functions.delete_file("test.csv")

    # Outputs: 'File deleted successfully'
    ```
    """
    logger.info(f"Deleting file: {file_name}")

    # Check that the file name is a string
    if not isinstance(file_name, str):
        raise TypeError(f"{file_name} is not a valid string")

    # Split the file name into its name and extension components
    file_name, file_ext = os.path.splitext(file_name)

    # Check that the file name does not contain a forward slash or backslash
    if os.path.sep in file_name:
        raise ValueError(f"{file_name} cannot contain {os.path.sep}")

    # Check that the file type is supported
    if file_ext not in directory_map:
        raise ValueError(
            f"unsupported file type: {file_ext}. Supported file types are: {', '.join(directory_map.keys())}"
        )

    # Construct the full file path
    file_directory = Path.cwd() / directory_to_files / directory_map[file_ext]
    file_path = file_directory / f"{file_name}{file_ext}"

    # Check that the file exists
    if not file_path.is_file():
        raise FileNotFoundError(f"file not found: {file_name}{file_ext}")

    # Delete the file
    os.remove(file_path)
    logger.info(f"File {file_name}{file_ext} deleted from file path: {file_path}")

    # Return a string indicating that the file has been deleted
    return "complete"

generate_random_date()

Generate a random datetime string in the format yyyy-mm-dd hh:mm:ss.ffffff.

Returns:

Name	Type	Description
str	str	A randomly generated datetime string.

Example:

from dsg_lib.common_functions import file_functions
random_date = file_functions.generate_random_date()
# Returns: '1992-03-15 10:30:45.123456'

Source code in dsg_lib/common_functions/file_functions.py

def generate_random_date() -> str:
    """
    Generate a random datetime string in the format yyyy-mm-dd hh:mm:ss.ffffff.

    Returns:
        str: A randomly generated datetime string.

    Example:
    ```python
    from dsg_lib.common_functions import file_functions
    random_date = file_functions.generate_random_date()
    # Returns: '1992-03-15 10:30:45.123456'
    ```
    """
    # Define the minimum and maximum years for the date range
    min_year: int = 1905
    max_year: int = datetime.now().year

    # Generate random values for the year, month, day, hour, minute, and second
    year: int = random.randrange(min_year, max_year + 1)
    month: int = random.randint(1, 12)
    day: int = random.randint(1, 28)
    hour: int = random.randint(0, 12)
    minute: int = random.randint(0, 59)
    second: int = random.randint(0, 59)

    # Create a datetime object with the random values
    date_value: datetime = datetime(year, month, day, hour, minute, second)

    # Format the datetime string and return it
    return f"{date_value:%Y-%m-%d %H:%M:%S.%f}"

open_csv(file_name, delimiter=',', quote_level='minimal', skip_initial_space=True)

Opens a CSV file with the specified file name and returns its contents as a list of dictionaries.

Parameters:

Name	Type	Description	Default
file_name	str	The name of the file to open. Should include the '.csv'	required
extension.	delimiter (str	The character used to separate	required
fields	in the CSV file. Defaults to ','. quote_level (str		required
optional)		Whether to skip initial whitespace in the CSV file. Defaults	required

Returns:

Name	Type	Description
list	list	The contents of the CSV file as a list of dictionaries. Each
	list	dictionary represents a row in the CSV file, where the keys are column
	list	names and the values are the data for those columns.

Raises:

Type	Description
TypeError	If file_name is not a string. ValueError: If quote_level
is not a valid level. FileNotFoundError	If the file does not exist.

Example:

from dsg_lib.common_functions import file_functions data =
file_functions.open_csv("test.csv", delimiter=";", quote_level="all",
skip_initial_space=False)  # Returns: [{'column1': 'value1', 'column2':
'value2'}]

Source code in dsg_lib/common_functions/file_functions.py

def open_csv(
    file_name: str,
    delimiter: str = ",",
    quote_level: str = "minimal",
    skip_initial_space: bool = True,
) -> list:
    """
    Opens a CSV file with the specified file name and returns its contents as a
    list of dictionaries.

    Args:
        file_name (str): The name of the file to open. Should include the '.csv'
        extension. delimiter (str, optional): The character used to separate
        fields in the CSV file. Defaults to ','. quote_level (str, optional):
        The quoting level used in the CSV file. Valid levels are "none",
        "minimal", and "all". Defaults to "minimal". skip_initial_space (bool,
        optional): Whether to skip initial whitespace in the CSV file. Defaults
        to True.

    Returns:
        list: The contents of the CSV file as a list of dictionaries. Each
        dictionary represents a row in the CSV file, where the keys are column
        names and the values are the data for those columns.

    Raises:
        TypeError: If `file_name` is not a string. ValueError: If `quote_level`
        is not a valid level. FileNotFoundError: If the file does not exist.

    Example:
    ```python
    from dsg_lib.common_functions import file_functions data =
    file_functions.open_csv("test.csv", delimiter=";", quote_level="all",
    skip_initial_space=False)  # Returns: [{'column1': 'value1', 'column2':
    'value2'}]
    ```
    """
    # A dictionary that maps quote levels to csv quoting constants
    quote_levels = {
        "none": csv.QUOTE_NONE,
        "minimal": csv.QUOTE_MINIMAL,
        "all": csv.QUOTE_ALL,
    }

    # Check that file name is a string
    if not isinstance(file_name, str):
        error = f"{file_name} is not a valid string"
        logger.error(error)
        raise TypeError(error)

    # Check that quote level is valid
    quote_level = quote_level.lower()
    if quote_level not in quote_levels:
        error = f"Invalid quote level: {quote_level}. Valid levels are: {', '.join(quote_levels)}"
        logger.error(error)
        raise ValueError(error)
    quoting = quote_levels[quote_level]

    # Add extension to file name and create file path
    file_name = f"{file_name}.csv"
    file_directory = Path.cwd().joinpath(directory_to_files).joinpath("csv")
    file_path = file_directory.joinpath(file_name)

    # Check that file exists
    if not file_path.is_file():
        error = f"File not found: {file_path}"
        logger.error(error)
        raise FileNotFoundError(error)

    # Read CSV file
    data = []
    with file_path.open(encoding="utf-8") as f:
        reader = csv.DictReader(
            f,
            delimiter=delimiter,
            quoting=quoting,
            skipinitialspace=skip_initial_space,
        )
        for row in reader:
            data.append(dict(row))

    logger.info(f"File opened: {file_name}")
    return data

open_json(file_name)

Open a JSON file and load its contents into a dictionary.

Parameters:

Name	Type	Description	Default
file_name	str	The name of the JSON file to open.	required

Returns:

Name	Type	Description
dict	dict	The contents of the JSON file as a dictionary.

Raises:

Type	Description
TypeError	If the file name is not a string. FileNotFoundError: If the

Source code in dsg_lib/common_functions/file_functions.py

def open_json(file_name: str) -> dict:
    """
    Open a JSON file and load its contents into a dictionary.

    Args:
        file_name (str): The name of the JSON file to open.

    Returns:
        dict: The contents of the JSON file as a dictionary.

    Raises:
        TypeError: If the file name is not a string. FileNotFoundError: If the
        file does not exist.
    """
    # Check if file name is a string
    if not isinstance(file_name, str):
        error = f"{file_name} is not a valid string"
        logger.error(error)
        raise TypeError(error)

    file_directory = Path(directory_to_files) / directory_map[".json"]
    file_save = file_directory / file_name

    # Check if path correct
    if not file_save.is_file():
        error = f"file not found error: {file_save}"
        logger.exception(error)
        raise FileNotFoundError(error)

    # open file
    with open(file_save) as read_file:
        # load file into data variable
        result = json.load(read_file)

    logger.info(f"File Opened: {file_name}")
    return result

open_text(file_name)

Opens a text file with the specified file name and returns its contents as a string.

Parameters:

Name	Type	Description	Default
file_name	str	The name of the file to open. Should include the '.txt'	required

Returns:

Name	Type	Description
str	str	The contents of the text file as a string.

Raises:

Type	Description
TypeError	If the file_name parameter is not a string or contains a
forward slash. FileNotFoundError	If the file does not exist.

Example:

from dsg_lib.common_functions import file_functions
data = file_functions.open_text("test.txt")
# Returns: 'This is a test text file.'

Source code in dsg_lib/common_functions/file_functions.py

def open_text(file_name: str) -> str:
    """
    Opens a text file with the specified file name and returns its contents as a
    string.

    Args:
        file_name (str): The name of the file to open. Should include the '.txt'
        extension.

    Returns:
        str: The contents of the text file as a string.

    Raises:
        TypeError: If the `file_name` parameter is not a string or contains a
        forward slash. FileNotFoundError: If the file does not exist.

    Example:
    ```python
    from dsg_lib.common_functions import file_functions
    data = file_functions.open_text("test.txt")
    # Returns: 'This is a test text file.'
    ```
    """
    # Replace backslashes with forward slashes in the file name
    if "\\" in file_name:  # pragma: no cover
        file_name = file_name.replace("\\", "/")  # pragma: no cover

    # Check that file_name does not contain invalid characters
    if "/" in file_name:
        logger.error(f"{file_name} cannot contain /")
        raise TypeError(f"{file_name} cannot contain /")

    # Get the path to the text directory and the file path
    file_directory = os.path.join(directory_to_files, "text")
    file_path = Path.cwd().joinpath(file_directory, file_name)

    # Check if the file exists
    if not file_path.is_file():
        raise FileNotFoundError(f"file not found error: {file_path}")

    # Open the file and read the data
    with open(file_path, "r", encoding="utf-8") as file:
        data = file.read()

    logger.info(f"File opened: {file_path}")
    return data

save_csv(file_name, data, root_folder=None, delimiter=',', quotechar='"')

Saves a list of dictionaries as a CSV file with the specified file name in the specified directory. Each dictionary in the list should represent a row in the CSV file.

Parameters:

Name	Type	Description	Default
file_name	str	The name of the file to save the data in. Should	required
include	the '.csv' extension. data (list	The data to be saved. Each	required
optional)		The root directory where the file will be saved. If None, the	required
(str,	optional	The character used to separate fields in the CSV file.	required
Defaults	to ','. quotechar (str	The character used to quote	required

Returns:

Name	Type	Description
str	str	A message indicating whether the file has been saved successfully
	str	or an error occurred.

Raises:

Type	Description
TypeError	If the data is not a list, or the file name, delimiter, or
quotechar is not a string. ValueError	If the file name does not end

Example: ```python from dsg_lib.common_functions import file_functions

data = [{"column1": "value1", "column2": "value2"}]

file_functions.save_csv("test.csv", data, "/path/to/directory", delimiter=";", quotechar="'")

Saves data to '/path/to/directory/test.csv'

```

Source code in dsg_lib/common_functions/file_functions.py

def save_csv(
    file_name: str,
    data: list,
    root_folder: str = None,
    delimiter: str = ",",
    quotechar: str = '"',
) -> str:
    """
    Saves a list of dictionaries as a CSV file with the specified file name in
    the specified directory. Each dictionary in the list should represent a row
    in the CSV file.

    Args:
        file_name (str): The name of the file to save the data in. Should
        include the '.csv' extension. data (list): The data to be saved. Each
        element of the list should be a dictionary where the keys are column
        names and the values are the data for those columns. root_folder (str,
        optional): The root directory where the file will be saved. If None, the
        file will be saved in the current directory. Defaults to None. delimiter
        (str, optional): The character used to separate fields in the CSV file.
        Defaults to ','. quotechar (str, optional): The character used to quote
        fields in the CSV file. Defaults to '"'.

    Returns:
        str: A message indicating whether the file has been saved successfully
        or an error occurred.

    Raises:
        TypeError: If the data is not a list, or the file name, delimiter, or
        quotechar is not a string. ValueError: If the file name does not end
        with '.csv'.

    Example: ```python
    from dsg_lib.common_functions import file_functions

    data = [{"column1": "value1", "column2": "value2"}]

    file_functions.save_csv("test.csv", data, "/path/to/directory", delimiter=";", quotechar="'")

    # Saves data to '/path/to/directory/test.csv'
    ```
    """
    # Set the root folder to directory_to_files if None
    if root_folder is None:
        root_folder = directory_to_files

    # Create the csv directory if it does not exist
    csv_directory = Path(root_folder) / "csv"
    csv_directory.mkdir(parents=True, exist_ok=True)

    # Check that delimiter and quotechar are single characters
    if len(delimiter) != 1:
        raise TypeError(f"{delimiter} can only be a single character")
    if len(quotechar) != 1:
        raise TypeError(f"{quotechar} can only be a single character")

    # Check that data is a list
    if not isinstance(data, list):
        raise TypeError(f"{data} is not a valid list")

    # Check that file_name is a string and does not contain invalid characters
    if not isinstance(file_name, str) or "/" in file_name or "\\" in file_name:
        raise TypeError(f"{file_name} is not a valid file name")

    # Add extension to file_name if needed
    if not file_name.endswith(".csv"):
        file_name += ".csv"

    # Create the file path
    file_path = csv_directory / file_name

    # Write data to file
    with open(file_path, "w", encoding="utf-8", newline="") as csv_file:
        csv_writer = csv.writer(csv_file, delimiter=delimiter, quotechar=quotechar)
        csv_writer.writerows(data)

    logger.info(f"File Create: {file_name}")
    return "complete"

save_json(file_name, data, root_folder=None)

Saves a dictionary or a list as a JSON file with the specified file name in the specified directory.

Parameters:

Name	Type	Description	Default
file_name	str	The name of the file to save the data in. Should	required
include	the '.json' extension. data (list or dict	The data to be	required
saved.	root_folder (str	The root directory where the file	required

Returns:

Name	Type	Description
str	str	A message indicating whether the file has been saved successfully
	str	or an error occurred.

Raises:

Type	Description
TypeError	If the data is not a list or a dictionary, or the file name
or directory is not a string. ValueError	If the file name contains a

Example:

from dsg_lib.common_functions import file_functions

data = {"key": "value"}

file_functions.save_json("test.json", data, "/path/to/directory")

# Saves data to '/path/to/directory/test.json'

Source code in dsg_lib/common_functions/file_functions.py

def save_json(file_name: str, data, root_folder: str = None) -> str:
    """
    Saves a dictionary or a list as a JSON file with the specified file name in
    the specified directory.

    Args:
        file_name (str): The name of the file to save the data in. Should
        include the '.json' extension. data (list or dict): The data to be
        saved. root_folder (str, optional): The root directory where the file
        will be saved. Defaults to None, which means the file will be saved in
        the 'data' directory.

    Returns:
        str: A message indicating whether the file has been saved successfully
        or an error occurred.

    Raises:
        TypeError: If the data is not a list or a dictionary, or the file name
        or directory is not a string. ValueError: If the file name contains a
        forward slash or backslash, or if the file name does not end with
        '.json'.

    Example:
    ```python
    from dsg_lib.common_functions import file_functions

    data = {"key": "value"}

    file_functions.save_json("test.json", data, "/path/to/directory")

    # Saves data to '/path/to/directory/test.json'
    ```
    """
    try:
        # Validate inputs
        if not isinstance(data, (list, dict)):
            raise TypeError(
                f"data must be a list or a dictionary instead of type {type(data)}"
            )
        if "/" in file_name or "\\" in file_name:
            raise ValueError(f"{file_name} cannot contain / or \\")

        # Add extension if not present in file_name
        if not file_name.endswith(".json"):  # pragma: no cover
            file_name += ".json"  # pragma: no cover

        if root_folder is None:
            root_folder = directory_to_files

        # Determine directory
        json_directory = Path(root_folder) / "json"

        # Construct file path
        file_path = json_directory / file_name

        # Create the json directory if it does not exist
        json_directory.mkdir(parents=True, exist_ok=True)

        # Write data to file
        with open(file_path, "w") as write_file:
            json.dump(data, write_file)

        # Log success message
        logger.info(f"File created: {file_path}")

        return "File saved successfully"

    except (TypeError, ValueError) as e:
        logger.error(f"Error creating file {file_name}: {e}")
        raise

save_text(file_name, data, root_folder=None)

Saves a string of text to a file with the specified file name in the specified directory.

Parameters:

Name	Type	Description	Default
file_name	str	The name of the file to save the data in. Should not	required
include	the '.txt' extension. data (str	The text data to be saved.	required
root_folder	str	The root directory where the file will be	None

Returns:

Name	Type	Description
str	str	A message indicating whether the file has been saved successfully
	str	or an error occurred.

Raises:

Type	Description
TypeError	If the data parameter is not a string, or the file_name
contains a forward slash or backslash. FileNotFoundError	If the

Example:

from dsg_lib.common_functions import file_functions

file_functions.save_text("test", "This is a test text file.", "/path/to/directory")

# Saves data to '/path/to/directory/test.txt'

Source code in dsg_lib/common_functions/file_functions.py

def save_text(file_name: str, data: str, root_folder: str = None) -> str:
    """
    Saves a string of text to a file with the specified file name in the
    specified directory.

    Args:
        file_name (str): The name of the file to save the data in. Should not
        include the '.txt' extension. data (str): The text data to be saved.
        root_folder (str, optional): The root directory where the file will be
        saved. If None, the file will be saved in the current directory.
        Defaults to None.

    Returns:
        str: A message indicating whether the file has been saved successfully
        or an error occurred.

    Raises:
        TypeError: If the `data` parameter is not a string, or the `file_name`
        contains a forward slash or backslash. FileNotFoundError: If the
        directory does not exist.

    Example:
    ```python
    from dsg_lib.common_functions import file_functions

    file_functions.save_text("test", "This is a test text file.", "/path/to/directory")

    # Saves data to '/path/to/directory/test.txt'
    ```
    """
    # If no root folder is provided, use the default directory
    if root_folder is None:  # pragma: no cover
        root_folder = directory_to_files  # pragma: no cover

    # Determine the directory for text files
    text_directory = Path(root_folder) / "text"

    # Construct the file path for text files
    file_path = text_directory / file_name

    # Create the text directory if it does not exist
    text_directory.mkdir(parents=True, exist_ok=True)

    # Check that data is a string and that file_name does not contain invalid
    # characters
    if not isinstance(data, str):
        logger.error(f"{file_name} is not a valid string")
        raise TypeError(f"{file_name} is not a valid string")
    elif "/" in file_name or "\\" in file_name:
        logger.error(f"{file_name} cannot contain \\ or /")
        raise ValueError(f"{file_name} cannot contain \\ or /")

    # Add extension to file_name if needed
    if not file_name.endswith(".txt"):
        file_name += ".txt"
    # Open or create the file and write the data
    with open(file_path, "w+", encoding="utf-8") as file:
        file.write(data)

    logger.info(f"File created: {file_path}")
    return "complete"