Skip to main content

upsertFromHuggingFace

Inserts or updates vectors from a HuggingFace dataset.

upsertFromHuggingFace(
    datasetsName: string,
    options?: {
      embeddingColumn?: string
      metadata?: Record<string, any>
    }
): Promise<string[]>

Reference

import { myVectorStore } from "#elements";
export default async function () {
  const count = await myVectorStore.upsertFromHuggingFace(
    "fka/awesome-chatgpt-prompts",
    { embeddingColumn: "prompt" }
  );
  console.log(`${count} vectors upserted`);
}

Parameters

datasetsName: The name of the Hugging Face dataset to extract content from.
options: Optional configuration parameters, including:
- embeddingColumn: (optional) The column in the dataset to use for embeddings. If not provided, the method will attempt to convert the entire dataset.
- metadata: (optional) The metadata to associate with the vectors.

Returns

Promise of an array of IDs of the upserted vectors.

Caveats

This method will insert a new vector if the datasetsName does not exist, or update the existing vector if the datasetsName exists.
You can query all the results by filtering the metadata field source-by-babel to datasetsName.

Reference