! pip install pymilvus ollama
复制

! wget https://github.com/milvus-io/milvus-docs/releases/download/v2.4.6-preview/milvus_docs_2.4.x_en.zip
! unzip -q milvus_docs_2.4.x_en.zip -d milvus_docs


复制

from glob import glob


text_lines = []


for file_path in glob("milvus_docs/en/faq/*.md", recursive=True):
    with open(file_path, "r") as file:
        file_text = file.read()


    text_lines += file_text.split("# ")
复制

! ollama pull mxbai-embed-large
复制

! ollama pull qwq
复制

import ollama
from ollama import Client


ollama_client = Client(host="http://localhost:11434")
def emb_text(text):
    response = ollama_client.embeddings(model="mxbai-embed-large", prompt=text)
    return response["embedding"]
#test
test_embedding = emb_text("This is a test")
embedding_dim = len(test_embedding)
print(embedding_dim)
print(test_embedding[:10])
复制

1024
[0.23217937350273132, 0.42540550231933594, 0.19742339849472046, 0.4618139863014221, -0.46017369627952576, -0.14087969064712524, -0.18214142322540283, -0.07724273949861526, 0.40015509724617004, 0.8331164121627808]


复制

from pymilvus import MilvusClient


milvus_client = MilvusClient(uri="./milvus_demo.db")


collection_name = "my_rag_collection"
复制

if milvus_client.has_collection(collection_name):
    milvus_client.drop_collection(collection_name)
复制

milvus_client.create_collection(
    collection_name=collection_name,
    dimension=embedding_dim,
    metric_type="IP",  # Inner product distance
    consistency_level="Strong",  # Strong consistency level
)
复制

from tqdm import tqdm


data = []


for i, line in enumerate(tqdm(text_lines, desc="Creating embeddings")):
    data.append({"id": i, "vector": emb_text(line), "text": line})


milvus_client.insert(collection_name=collection_name, data=data)


复制

Creating embeddings: 100%|████████████████████████████████████████████████████████████████████████████████████████████████████████| 72/72 [00:06<00:00, 11.86it/s]


{'insert_count': 72, 'ids': [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47, 48, 49, 50, 51, 52, 53, 54, 55, 56, 57, 58, 59, 60, 61, 62, 63, 64, 65, 66, 67, 68, 69, 70, 71], 'cost': 0}




复制

question = "How is data stored in milvus?"
复制

search_res = milvus_client.search(
    collection_name=collection_name,
    data=[
        emb_text(question)
    ],  # Use the `emb_text` function to convert the question to an embedding vector
    limit=3,  # Return top 3 results
    search_params={"metric_type": "IP", "params": {}},  # Inner product distance
    output_fields=["text"],  # Return the text field
)
复制

import json


retrieved_lines_with_distances = [
    (res["entity"]["text"], res["distance"]) for res in search_res[0]
]
print(json.dumps(retrieved_lines_with_distances, indent=4))
复制

[
    [
        " Where does Milvus store data?\n\nMilvus deals with two types of data, inserted data and metadata. \n\nInserted data, including vector data, scalar data, and collection-specific schema, are stored in persistent storage as incremental log. Milvus supports multiple object storage backends, including [MinIO](https://min.io/), [AWS S3](https://aws.amazon.com/s3/?nc1=h_ls), [Google Cloud Storage](https://cloud.google.com/storage?hl=en#object-storage-for-companies-of-all-sizes) (GCS), [Azure Blob Storage](https://azure.microsoft.com/en-us/products/storage/blobs), [Alibaba Cloud OSS](https://www.alibabacloud.com/product/object-storage-service), and [Tencent Cloud Object Storage](https://www.tencentcloud.com/products/cos) (COS).\n\nMetadata are generated within Milvus. Each Milvus module has its own metadata that are stored in etcd.\n\n###",
        231.9922637939453
    ],
    [
        "How does Milvus flush data?\n\nMilvus returns success when inserted data are loaded to the message queue. However, the data are not yet flushed to the disk. Then Milvus' data node writes the data in the message queue to persistent storage as incremental logs. If `flush()` is called, the data node is forced to write all data in the message queue to persistent storage immediately.\n\n###",
        226.54090881347656
    ],
    [
        "What is the maximum dataset size Milvus can handle?\n\n  \nTheoretically, the maximum dataset size Milvus can handle is determined by the hardware it is run on, specifically system memory and storage:\n\n- Milvus loads all specified collections and partitions into memory before running queries. Therefore, memory size determines the maximum amount of data Milvus can query.\n- When new entities and and collection-related schema (currently only MinIO is supported for data persistence) are added to Milvus, system storage determines the maximum allowable size of inserted data.\n\n###",
        210.63682556152344
    ]
]
复制

context = "\n".join(
    [line_with_distance[0] for line_with_distance in retrieved_lines_with_distances]
)
复制

SYSTEM_PROMPT = """Human: You are an AI assistant. You are able to find answers to the questions from the contextual passage snippets provided."""
USER_PROMPT = 
f"""
Use the following pieces of information enclosed in <context> tags to provide an answer to the question enclosed in <question> tags.
<context>
{context}
</context>
<question>
{question}
</question>
"""
复制

from ollama import chat
from ollama import ChatResponse


response: ChatResponse = ollama_client.chat(
    model="qwq",
    messages=[
        {"role": "system", "content": SYSTEM_PROMPT},
        {"role": "user", "content": USER_PROMPT},
    ],
)
print(response["message"]["content"])
复制

<think>
Okay, I need to answer the question "How is data stored in Milvus?" using the provided context. Let me start by reading through the context carefully.


First, looking at the first context section titled "Where does Milvus store data?". It mentions that Milvus handles two types of data: inserted data and metadata. 


Inserted data includes vectors, scalar data, and collection schema, stored as incremental logs in persistent storage. The supported backends are listed like MinIO, S3, GCS, etc. So I should note that inserted data goes to these object storages via incremental logs.


Then metadata is stored in etcd for each Milvus module. That's straightforward—metadata in etcd.


The second context about flushing data explains that data first goes to the message queue and then to persistent storage when flushed. But this might be more about the process rather than where it's stored, so maybe mention that insertion is into the message queue first before being written out.


Third section on maximum dataset size mentions hardware constraints: memory for queryable data and storage for persistence. This could relate to how data is managed in terms of memory vs disk. The user asked "how is data stored," which might include both where (like object stores, etcd) and the process (message queue then persistent logs).


Putting it all together, the answer should mention:


- Two types: inserted data and metadata.


For inserted data:
- Stored as incremental logs in persistent storage using backends like MinIO, S3 etc.
- Initially loaded into message queues, then written to disk. The flush() forces immediate writing if needed.


Metadata:
- Stored in etcd for each module.


Also, the hardware part says that memory limits queryable data size and storage (like the object stores) limit total dataset size. So maybe mention these factors as part of how data is stored, considering capacity constraints.


I should structure this clearly, perhaps bullet points or sections on inserted vs metadata, then processes like flushing, and finally limitations based on hardware.
</think>


Data in Milvus is stored in two main ways:  


1. **Inserted Data**:  
   - Includes vector data, scalar data, and collection schema.  
   - Stored as **incremental logs** in **persistent storage**, supported by object storage backends such as MinIO, AWS S3, Google Cloud Storage (GCS), Azure Blob Storage, Alibaba Cloud OSS, and Tencent Cloud Object Storage (COS).  
   - Initially loaded into a **message queue** when inserted. Milvus returns success upon insertion into the queue but does not immediately write to disk. A background process later writes this data to persistent storage. The `flush()` method can force immediate writing of all queued data to storage.  


2. **Metadata**:  
   - Generated internally by Milvus modules (e.g., collection configurations, partitions).  
   - Stored in **etcd**, a distributed key-value store.  


**Hardware Considerations**:  
- **Memory**: The amount of data Milvus can query is limited by system memory since it loads specified collections/partitions into memory for queries.  
- **Storage Capacity**: The maximum dataset size is constrained by the underlying storage backend (e.g., object storage), which stores all inserted data and schema incrementally.


复制

作者介绍

王舒虹

Zilliz Social Media Advocate