import_files.py

Source: sunholo/llamaindex/import_files.py

Functions

llamaindex_chunker_check(message_data, metadata, vector_name)

No docstring available.

check_llamaindex_in_memory(vector_name)

No docstring available.

do_llamaindex(message_data, metadata, vector_name)

Configures and manages the corpus for a VertexAI project using the specified vector name by importing message data from Google Cloud Storage or Google Drive URLs.

This function loads configuration from a YAML file, initializes a Vertex AI environment, and either fetches an existing corpus or creates a new one if it doesn't exist. It supports importing files directly from cloud storage links.

Parameters: message_data (str): The URL to the data on Google Cloud Storage or Google Drive that needs to be imported to the corpus. metadata (dict): Additional metadata not explicitly used in this function but might be needed for extended functionality. vector_name (str): The name of the vector (and corpus) which will be used to locate and configure the specific settings from the configuration files.

Raises: ValueError: If the necessary configurations for GCP or project ID are not found, or if the corpus could not be established. NotImplementedError: If the data is not from supported sources (Google Cloud Storage or Google Drive).

Example:

message_data = "gs://bucket_name/path_to_file.txt"
metadata = {"user": "admin"}
vector_name = "example_vector"
response = do_llamaindex(message_data, metadata, vector_name)
print(response)
# Imported file to corpus: {'status': 'success'}