# What is Curio? What is Curio and how is it different from Lotus-Miner? ## Overview Curio is the new implementation of Filecoin storage protocol. It aims to simplify the setup and operation of storage providers. {% hint style="danger" %} Please note that Curio cluster cannot be shared across different networks. Example: A single Curio cluster cannot host miners IDs coming from Mainnet and Calibnet together. {% endhint %} ## Key Features #### High Availability Curio is designed for high availability. You can run multiple instances of Curio nodes to handle similar type of tasks. The distributed scheduler and greedy worker design will ensure that tasks are completed on time despite most partial outages. You can safely update one of your Curio machines without disrupting the operation of the others. #### Node Heartbeat Each Curio node in a cluster must post a heartbeat message every 10 minutes in HarmonyDB updating its status. If a heartbeat is missed, the node is considered lost and all tasks can now be scheduled on remaining nodes. #### Task Retry Each task in Curio has a limit on how many times it should be tried before being declared lost. This ensures that Curio does not keep retrying bad tasks indefinitely. This safeguards against lost computation time and storage. #### Polling Curio avoids overloading nodes with a polling system. Nodes check for tasks they can handle, prioritizing idle nodes for even workload distribution. #### Simple Configuration Management The configuration is stored in the database in the forms of layers. These layers can be stacked on top of each other create a final configuration. Users can reuse these layers to control the behaviour of multiple machines without needing to maintain the configuration of each node. Start the binary with the appropriate flags to connect with YugabyteDB and specify which configuration layers to use to get desired behaviour. #### Running Curio with Multiple GPUs Curio can handle multiple GPUs simultaneously without needing to run multiple instances of the Curio process. Therefore, Curio can be managed as a single systemd service without concerns about GPU allocations. ## Curio vs Lotus Miner | Feature | Curio | Lotus-Miner | | ------------------------------------ | -------------------------------------------------------------- | --------------------------------------------------- | | Scheduling | Collaborative (Prioritized Greedy) | Single point of failure | | High Availability | Available | Single control process | | Redundant Post | Available | Not Available | | Task Retry Control | Task retry with a cutoff limit (per task) | Unlimited retry leading to resource exhaustion | | Multiple Miner IDs | Curio cluster can support multiple Miner IDs | Single Miner ID per Lotus-Miner | | Shared Task nodes | Curio nodes can handle task for multiple Miner IDS | Attached workers handle tasks for a single Miner ID | | Distributed Configuration Management | Configuration stored in the highly-available Yugabyte Database | All configuration in a single File | ## Future of Curio The long-term vision for Curio is to eventually replace the current lotus-miner and lotus-worker processes. This is part of an ongoing effort to simplify and streamline the setup and operation of storage providers.
# Filecoin Storage Provider This pages covers some basic concepts useful to be a Filecoin storage providers ## 1. Overview of Filecoin Storage Providers A Storage Provider in the Filecoin network: * **Stores client data** in exchange for fees and potential block rewards. * Must **continuously prove** that stored data is available (via cryptographic proofs). * Must **pledge collateral** in the form of FIL tokens to ensure honest behavior. * Is subject to **penalties** if data is lost or if proofs are missed. Becoming a Storage Provider blends **technical** and **economic** considerations. It requires: 1. **Sufficient hardware** to store, seal, and prove data reliably. 2. **Continuous on-chain proofs** to show you are storing data as promised. 3. **Economic planning** to handle locked collateral, fees, and eventual block rewards. *** ## 2. Key Concepts in Filecoin Below are the foundational ideas you need to understand thoroughly before running a Storage Provider. #### 2.1 Sectors * **Definition**: The basic unit of storage in Filecoin. You commit user data to the network by packaging it into *sectors*. * **Sizes**: Commonly 32 GiB (the most popular) or 64 GiB. Once you choose a sector size for your provider setup, you typically **cannot change** it without a new identity. * **Committed Capacity (CC)**: You can commit “empty” sectors to the network to build storage power without actual client data initially. These sectors can be upgraded with real data later via a feature called **SnapDeals**. **Sealed vs. Unsealed Sectors** 1. **Unsealed sector**: Contains the original (plaintext) data. Clients sometimes request that you keep an unsealed copy for faster retrieval. 2. **Sealed sector**: Cryptographically processed data ready for continuous proofs (Proof-of-Spacetime). Sealed data cannot be directly read until it’s unsealed (if needed for retrieval). #### 2.2 Proof-of-Spacetime (PoSt) Two key proof mechanisms ensure a provider’s data is verifiably present: 1. **WindowPoSt**: Happens in \~24-hour windows; you must prove that each sector is still stored. If you miss this proof, you incur faults and penalties. 2. **WinningPoSt**: A small subset of providers is elected every epoch (\~30 seconds) to produce a new block. If chosen, you submit a short proof showing you hold your pledged data. In return, you can earn a **block reward**. #### 2.3 Deals and Retrieval * **Storage Deals**: Contracts made with clients who pay you to store their data for a specified duration. * **Retrieval Deals**: When clients pull their data, they pay you retrieval fees, typically via payment channels. * **Verified Clients**: Some clients store “verified” data. Providers who store verified data get higher *quality-adjusted power* (and thus a higher chance for block rewards). #### 2.4 Epoch Filecoin time is measured in **epochs** (\~30-second intervals). * Each epoch triggers new tasks like verifying proofs, adding blocks, distributing block rewards, and so on. *** ## 3. Economics & Rewards #### 3.1 Rewards 1. **Storage fees**: Paid over time by clients whose data you store. 2. **Block rewards**: Awarded by the network when you produce a block. You typically need **at least 10 TiB** of raw power to be eligible for these. A portion of each block reward is locked and vests linearly over \~180 days. #### 3.2 Collateral and Locked Funds 1. **PreCommit Deposits**: You lock up some FIL when you “pre-commit” a sector. If the sector never proceeds to full commitment, the deposit is lost. 2. **Initial Pledge**: More FIL is locked when you fully commit a sector (prove-commit). This collateral is held to ensure you keep data online. 3. **Locked Rewards**: 75% of any block reward vests over \~180 days; 25% is immediately available. #### 3.3 Penalties & Slashing * **Fault Fees**: If you fail to prove a sector on time (WindowPoSt), you pay a daily fault fee until you fix or terminate the sector. * **Sector Penalties**: If you do not declare a fault before a scheduled proof check, you pay an immediate penalty. * **Termination Fees**: For sectors that are terminated early (voluntarily or involuntarily). * **Consensus Fault Slashing**: Severe penalty for malicious consensus-level actions. *** ## 4. Theory of Sector Lifecycle Although exact commands differ, every Filecoin SP software follows a similar lifecycle for sector creation and maintenance: 1. **Sector Allocation** * The provider decides to add capacity, either in the form of empty (CC) or client-filled sectors. 2. **PreCommit** * You produce a sector with partial sealing steps (encoding the data) and record this “precommit” on-chain with the required deposit. 3. **ProveCommit** * Final sealing steps generate a proof of replication (PoRep). * You submit this proof on-chain to finalize the commitment, locking your initial pledge collateral. 4. **Continuous Proof (WindowPoSt)** * Regularly (every \~24 hours), you must prove that each committed sector remains intact. * Missing this proof for a sector causes it to become **faulty**. 5. **Maintenance** * You can recover faulty sectors by re-running proofs. * You can terminate sectors early if needed, but you pay a termination penalty. 6. **Expiration or Upgrades** * Deals eventually expire. You can extend deals or reuse the sector for new deals (SnapDeals). * Once the entire sector’s lifetime ends, you can choose to remove it from the network or keep it sealed if you plan to extend. *** ## 5. Daily Operations & Tasks (General) #### 5.1 Monitoring Proofs * **WindowPoSt** schedules are strict. Know your deadlines and ensure your system has enough CPU/GPU resources to generate proofs on time. * Watch for I/O bottlenecks that could slow proof generation. #### 5.2 Accepting and Managing Deals * **Negotiation**: Clients discover your provider (through a market or index) and propose deals. * **Data Transfer**: You receive client data. This data is prepared for sealing. * **Sealing**: The data is integrated into new or partially filled sectors. * **Payments**: Storage fees accumulate over time, typically locked and then released to your available balance after each deal cycle. #### 5.3 Retrievals * **Unsealing on Demand**: If you do not maintain an unsealed copy of the data, you need to unseal it when a client requests retrieval (this can be slow). * **Incremental Payments**: Retrieval fees are often paid via payment channels, chunk by chunk. #### 5.4 Maintenance * **Fault Recovery**: If a sector goes faulty, you can declare it in advance (to reduce penalties) and later recover it. * **Hardware Upgrades**: As you add more capacity or want to speed up sealing, you may add more RAM, GPUs, or worker machines. * **Software Updates**: Regularly update your chosen node/miner software to get protocol improvements and security fixes. *** ## 6. SnapDeals and Batching (Advanced Concepts) #### 6.1 SnapDeals * **Upgrade CC Sectors**: SnapDeals lets you add real client data to a sector that was previously committed as empty capacity. * **Advantages**: * Saves time and resources: no need to run a full seal again. * Increases your effective power if the data is from verified clients (quality-adjusted power boost). #### 6.2 Batching * **On-Chain Efficiency**: Sending many single-sector proofs can be expensive in network fees (gas). Batching helps you combine multiple proofs into a single message. * **Collateral Savings**: Batching can also help group sector pledges, reducing overhead. *** ## 7. Handling Balances & Collateral 1. **Ensure Adequate FIL** in your provider or worker wallets for day-to-day operations. 2. **Monitor** locked vs. unlocked balances. Collateral must be readily available to avoid interruptions in sealing and deal-making. 3. **Reward Vesting**: Remember that block rewards vest linearly over \~180 days, so you can’t immediately withdraw 100% of your earnings. *** ## 8. Penalties and How to Avoid Them 1. **Declare Faults in Time**: If you know a sector or machine is failing, declare a fault early. This is cheaper than letting a proof window pass without submission. 2. **Maintain Redundancy**: Keep backups or have spares for crucial components (power supplies, GPUs, or entire machines). 3. **Stay Online**: A stable network connection is key. If your system is offline at your WindowPoSt deadline, you’ll incur faults. *** ## 9. Scaling Your Operation As you gain experience, you may want to scale: * **Multiple Storage Locations**: Distribute sealed data across multiple racks or data centers. * **Parallel Sealing**: Invest in more CPU cores, more RAM, and multiple GPUs to process many sectors at once. * **Distributed Workers**: In some implementations, you can run separate “worker” processes on different machines to handle sealing tasks more efficiently. *** ## 10. Best Practices and Tips 1. **Monitor Everything**: Use dashboards and logs to watch for unusual CPU, RAM, or disk usage. Keep track of system health. 2. **Time Synchronization**: Keep your system clock accurate (via NTP or Chrony). Mismatched clocks can cause missed proofs. 3. **Hardware Stress Testing**: Sealing is CPU/GPU heavy. Make sure your hardware can handle sustained loads. 4. **Diversify Deals**: Seek out verified clients if possible, since that can boost your block reward chances (due to higher quality-adjusted power). 5. **Community Interaction**: Join Filecoin Slack or other community channels to stay updated on protocol changes and to troubleshoot issues with other providers. *** ## 11. Summary Becoming a successful Filecoin Storage Provider requires: 1. **Solid Understanding** of Filecoin’s protocol and concepts (sectors, PoSt, deals, collateral, etc.). 2. **Robust Hardware** sized for your planned sealing rate and total storage. 3. **Continuous Maintenance** to ensure you never miss your proofs or run out of collateral. 4. **Economic Planning** to manage rewards, locked funds, and the cost of potential faults. 5. **Attention to Detail** with advanced features like SnapDeals, batching, and retrieval market optimizations. *** ## Next Steps * **Deep-Dive Documentation**: Explore the [official Filecoin Specification](https://spec.filecoin.io/) for in-depth details of each proof, state machine, and economic mechanism. * **Pilot Deployment**: Start small to gain hands-on experience with sealing, proof submission, and deal flow. * **Expand** after building confidence in your setup, hardware reliability, and economic strategy. By following these principles, you can build and manage a reliable Filecoin storage operation, confidently handle client deals, avoid excessive faults, and steadily grow your provider business over time. # Design This page provides a detailed overview of the core concepts and components that make up Curio, including HarmonyDB, HarmonyTask, and more. ## Design ### Curio Cluster The core internal components of Curio are HarmonyDB, HarmonyTask, ChainScheduler and a database abstraction of configuration & today’s storage definitions.
Curio Node

Curio nodes

A Curio cluster is a cluster of multiple Curio nodes connected to a YugabyteDB cluster and market nodes. A single Curio cluster can serve multiple miner ID and share the computation resources between them as required.
Curio cluster

Curio cluster

### HarmonyDB HarmonyDB is a simple SQL database abstraction layer used by HarmonyTask and other components of the Curio stack to store and retrieve information from YugabyteDB. #### Key Features: * **Resilience:** Automatically fails over to secondary databases if the primary connection fails. * **Security:** Protects against SQL injection vulnerabilities. * **Convenience:** Offers helper functions for common Go + SQL operations. * **Monitoring:** Provides insights into database behavior through Prometheus stats and error logging. See [Prometheus Metrics](/configuration/prometheus-metrics) for setup and available metrics. #### Basic Database Details * The Postgres DB schema is called “curio” and all the harmony DB tables reside under this schema. * Table `harmony_task` stores a list of pending tasks. * Table `harmony_task_history` stores completed tasks, retried tasks exceeding limits, and serves as input for triggering follower tasks (potentially on different machines). * Table `harmony_task_machines` is managed by lib/harmony/resources. This table references registered machines for task distribution. Registration does not imply obligation, but facilitates discovery. ### HarmonyTask The HarmonyTask is pure (no task logic) distributed task manager. #### Design Overview * Task-Centric: HarmonyTask focuses on managing tasks as small units of work, relieving developers from scheduling and management concerns. * Distributed: Tasks are distributed across machines for efficient execution. * Greedy Workers: Workers actively claim tasks they can handle. * Round Robin Assignment: After a Curio node claims a task, HarmonyDB attempts to distribute remaining work among other machines.
Curio Tasks

Harmony tasks

#### Model * **Blocked Tasks:** Tasks can be blocked due to: * Configuration under ‘subsystems’ disabled on the running node * Reaching specified maximum task limits * Resource exhaustion * CanAccept() function (task-specific) rejecting the task * **Task Initiation:** Tasks can be initiated through: * Periodic database reads (every 3 seconds) * Addition to the database by the current process * **Task Addition Methods:** * Asynchronous listener tasks (e.g., for blockchains) * Follower tasks triggered by task completion (sealing pipeline) * **Duplicate Task Prevention:** * The mechanism for avoiding duplicate tasks is left to the task definition, most likely using a unique key. ### Distributed Scheduling Curio implements a distributed scheduling mechanism co-ordinated via the HarmonyDB. The tasks are picked by the Curio nodes based on what they can handle (type and resources). Nodes are not greedy after taking a task, even if they have enough resources. Other nodes get a turn to claim a task. On 3 second intervals, if resources are available then an additional task will be taken. This ensures a more even scheduling of the tasks. ### Chain Scheduler The `CurioChainSched` or the chain scheduler trigger some call back functions when a new TipSet is applied or removed. This is equivalent to getting the heaviest TipSet on each epoch. These callback function in turn add the new tasks for each type that depends on the changes in the chain. These task types are WindowPost, WinningPost and MessageWatcher. ### Poller Poller is a simple loop that fetches pending tasks periodically according to predefined durations (100ms), or until a graceful exit is initiated by the context. Once the pending tasks are fetched from the database, it attempts to schedule all the tasks on the Curio node. This attempt will result in one of the following results: * Task is accepted * Task is not scheduled as machine is busy * Task is not accepted as the node’s CanAccept (defined by the task) elects not to handle the specified Task If the task is accepted during a polling cycle, the wait time before the next cycle is equal to 100ms. But if the task is not scheduled for any reason the poller will retry after 3 seconds. ### Task Decision Logic For each task type a machine can handle, it first checks if the machine has enough capacity to execute the said task. Then it queries the database for tasks with no `owner_id` and the same name as the task type. If such tasks are present, it attempts to accept their work. It returns true if any work has been accepted and false otherwise. The decision-making logic to accept each task is following: 1. Checks if there are any tasks to do. If there are none, returns true. 2. Checks if the maximum number of tasks of this type is reached. If the number of running tasks meets or exceeds the maximum limit, log a message and returns false. 3. Checks if the machine has enough resources to handle the task. This includes checking the CPU, RAM, GPU capacity, and available storage. If the machine does not have enough resources, log a message and return false. 4. Checks if the task can be accepted by calling the `CanAccept` method. If it cannot be accepted, log a message and return false. 5. If the task requires storage space, the machine attempts to claim it. If the claim fails, log a message and releases the claimed storage space, then return false. 6. If the task source is `recover` i.e. the machine was performing this task before shutdown then increase the task count by one and begin processing the task in a separate goroutine. 7. If the task source is `poller` i.e. new pending task, attempt to claim the task for the current hostname. If unsuccessful, release the claimed storage and attempt to consider the next task. 8. If successful, increase the task count by one and begin processing the task in a separate goroutine. 9. This goroutine also updates the task status in the task history and depending on whether the task was successful or not, either deletes the task or updates the task in the tasks table. 10. Returns true, indicating that the work was accepted and will be processed. ### GPU Management in Curio #### **Historical Issues with Lotus-Miner Scheduler** Historically, the Lotus-Miner scheduler has encountered difficulties efficiently utilizing GPUs when more than one GPU is available to the lotus-worker process. These issues primarily arise from the underlying proofs library, which handles all GPU-related tasks and manages GPU assignments. This has led to problems such as: * A single task being assigned to multiple GPUs. * Multiple tasks being assigned to a single GPU. #### **Solution with Curio: The GPU Picker Library "ffiselect"** To address these issues in Curio, we have implemented a GPU picker library called "ffiselect". This library ensures that each task requiring a GPU is assigned one individually. The process works as follows: 1. **Task Assignment**: Each GPU-requiring task is assigned a specific GPU. 2. **Subprocess Creation**: A new subprocess is spawned for each task, with the dedicated GPU allocated to it. 3. **Proofs Library Call**: The subprocess calls the Proofs library with a single GPU and the specific task.

Curio FFISelect in action

This approach ensures efficient and conflict-free GPU usage, with each task being handled by a dedicated GPU, thus resolving the historical issues observed with the `lotus-miner` scheduler. ## Security Boundary This is what Curio expects an SP to secure in order to have a safe experience. Curio is cluster software which coordinates directly and through the database. It also communicates to the public through chain providers (Lotus) and the market node. To secure this properly, ensure that only trusted people & services have access to: * logs: (these include inputs to failing processes) * physical machines, * virtual machine access (ssh) for Curio, Lotus, or Yugabyte * Curio or Lotus' or Yugabyte's open ports (with exceptions noted by Lotus, and the Curio market node) -- This includes the admin web ui for Curio which exposes numerous capabilities beyond viewing. Safe to share with untrusted parties: (will not receive private information) * Prometheus output * alerts can be sent to untrusted receivers * CuView (at your own risk) has modes for light investigation. Curio team recommends a network (VPN) containing all the pieces to have limited access. Logs are mostly clean except for errors which try to be as specific as possible, so partial redaction may be best here if sharing with untrusted parties. # Sealing This page explains how the sealing pipeline functions in Curio ## Sealing Pipeline Curio’s sealing process is powered by HarmonyTasks. Each stage involved in sealing a sector is divided into smaller, independent tasks. These individual tasks are then picked up by different machines within the Curio cluster. This ensures tasks are distributed effectively and resources are used efficiently across the entire system.
Overview of Curio sealing pipeline

Curio sealing pipeline

## SealPoller The SealPoller struct is designed to track the progress of sealing operations. Each possible state in the sealing workflow is represented by a pollTask struct. This struct tracks each step a sealing operation might be in by setting boolean flags and saving task ids in individual columns of the `sectors_sdr_pipeline` table within the harmony database. ``` type pollTask struct { SpID int64 `db:"sp_id"` SectorNumber int64 `db:"sector_number"` TaskSDR *int64 `db:"task_id_sdr"` AfterSDR bool `db:"after_sdr"` TaskTreeD *int64 `db:"task_id_tree_d"` AfterTreeD bool `db:"after_tree_d"` TaskTreeC *int64 `db:"task_id_tree_c"` AfterTreeC bool `db:"after_tree_c"` TaskTreeR *int64 `db:"task_id_tree_r"` AfterTreeR bool `db:"after_tree_r"` TaskPrecommitMsg *int64 `db:"task_id_precommit_msg"` AfterPrecommitMsg bool `db:"after_precommit_msg"` AfterPrecommitMsgSuccess bool `db:"after_precommit_msg_success"` SeedEpoch *int64 `db:"seed_epoch"` TaskPoRep *int64 `db:"task_id_porep"` PoRepProof []byte `db:"porep_proof"` AfterPoRep bool `db:"after_porep"` TaskFinalize *int64 `db:"task_id_finalize"` AfterFinalize bool `db:"after_finalize"` TaskMoveStorage *int64 `db:"task_id_move_storage"` AfterMoveStorage bool `db:"after_move_storage"` TaskCommitMsg *int64 `db:"task_id_commit_msg"` AfterCommitMsg bool `db:"after_commit_msg"` AfterCommitMsgSuccess bool `db:"after_commit_msg_success"` Failed bool `db:"failed"` FailedReason string `db:"failed_reason"` } ``` The SealPoller retrieves all the `pollTasks` from the database, for which `after_commit_msg_success` or `after_move_storage` is not true, and tries to advance their state if possible. A `pollTask` is advanced when its dependencies, indicated by the “after\_” fields, are completed and the task itself is not yet queued (its task id is nil) or completed (its “After” field is false). Each pollTask’s advancement will trigger a database transaction attempting to update the task id with the new task id received from the HarmonyDB. The transaction makes sure that the task hasn’t been queued by others between reading the state and updating the task id. This polling process happens sequentially with different conditions for each stage, ensuring that all previous conditions are fulfilled before proceeding. If a task cannot proceed due to its previous dependencies not being completed, the poller will come back in the next round. Mostly, errors occurring during the poller operation are logged and don’t cause the poller to stop. But if something serious happens during a database transaction, it will be rolled back, with an error message giving details. By organizing work in this way, SealPoller ensures that each step in the sealing procedure occurs in the correct order, and that progress is made whenever it is possible to do so. It allows sectors to be sealed as efficiently as possible given the constraints of other tasks in progress.
Sealing task execution

Curio harmony task execution

## Piece Park Traditionally, data needs to be available before it can be sealed for storage. However, this can lead to inefficiencies. Curio addresses this by introducing a “Piece Park.” Curio’s sealing pipeline does not require the data to be readily available upfront. This allows us to initiate the sealing process even before the data is downloaded. While the sealing process progresses, the data is “parked” in a designated directory called “piece” within the storage location. This avoids keeping market connections open for extended periods. In essence, the local piece park acts as a temporary holding area for data, streamlining the sealing process and optimizing resource usage. Curio utilizes two tasks: ParkPiece: This task handles downloading the data and placing it in the “piece” directory. DropPiece: Once the data is no longer needed, this task takes care of cleaning up the parked data. In the future, this local storage can also allow Curio to reseal data in a new sector if the original sector gets lost during the sealing process. ## LMRPCProvider The LMRPCProvider provides a set of methods to interact with various data related to sectors and pieces. These methods are required by market implementation (Boost) to track the sealing progress of a deal. ``` ActorAddress: This method returns the actor's address associated with the LMRPCProvider. In other words, it returns the miner's address. WorkerJobs: This function returns a map of worker jobs, indexed by UUID. SectorsStatus: This method returns the status of a sector given the sector identifier sid. This function includes detailed information about the sector such as the state of sealing, deal ids, log, pledge, and expiration etc. SectorsList: This function provides a list of sector numbers currently stored. SectorsSummary: This function gives a summary of the sectors, categorized by their state. It returns a map that maps each sector state to its count. SectorsListInStates: This method returns a list of sector numbers that are in a given set of states. ComputeDataCid: This function is used to compute the CID of the data. AuthNew: This function creates a new authorization token (JWT) for the given permissions. ``` ## Piece Ingester The Piece Ingestor allocates a piece to a sector for a given miner address. It checks if the piece size matches the sector size, determines the preferred seal proof type, retrieves the miner ID, allocates a sector number, inserts the piece and sector pipeline entries into the database, and returns the sector and offset of the allocated piece.
# Harmony Tasks This guide explains the different HarmonyTasks available in Curio Curio uses HarmonyTask as a generic task container which can be scheduled by the poller in regular basis for the execution. To perform the different aspects of sealing and proving, Curio implements the following task types. ### SDR The SDR task is the first phase of the Proof-of-Replication process and is where encoding and replication of the data takes place. The SDR task is predominantly using a single CPU core, and is heavily utilizing the SHA256 instruction set. Using a CPU that has the SHA256 instruction set is therefore recommended. All 11 layers of calculation, layer by layer, are calculated sequentially. Each layer is 32GiB in size. When the SDR process is finished you will have generated data to the amount of 384GiB (A 32GiB unsealed sector + (11 layers x 32GiB)). SDR task requires commD as one of the input parameters. The commD calculation requires piece size and CID for all the pieces that will be part of the sector. The pieces themselves (data) is not required at this stage of the pipeline. ### SDRTrees The SDRTrees task can be further divided into 3 parts which are completed sequentially. #### TreeD Building the TreeD requires the access to the data to be sealed into the sector. It builds a Merkle tree using the data and writes it to the specified output path ending with “tree-d.dat”. It also returns the root CID of the generated tree. #### TreeRC In the TreeRC task, a column hash computation based on the 11 layers generated in PreCommit 1 is calculated, and a merkle tree gets constructed. This is same as PreCommit 2 on Lotus-miner. These tasks generates unsealed CID and sealed CID. The unsealed CID should match the root CID of the TreeD output. During this phase an additional 64GiB file (32GiB sectors) that represents the merkle tree is stored, in addition to the sealed 32GiB sector. Bringing the total amount of storage needed to approximately 500 GiB for one sector. ### PreCommitSubmit Through the `PreCommitSector` message a storage provider submits a deposit for a given sector’s sealed data, often referred to as the SealedCID, or commitment to replica (commR). After the message is included on-chain, the sector is registered to the storage provider and the sector enters the WaitSeed state, which is a security wait requirement by the network. This message type can also be batched to include multiple PreCommitSector messages in a single message to save gas fees paid to the network. These batched messages are called `PreCommitSectorBatch`. The message is not sent by the PreCommitSubmit task itself but is handed over to the queue of `SendMessage` task. ### PoRep The PoRep task combines the Commit1 and Commit2 parts of the Lotus-Miner sealing pipeline. The randomness acquired at the end of the wait seed state is used in the Commit 1 phase to select a random subset of leaf nodes from the merkle tree generated in the PreCommit 2 phase. From the subset of leaf nodes it checks, it generates a much smaller file than the full merkle tree. That file is approximately 16MiB in size. In the Commit 2 phase, the file from the Commit 1 gets compressed into a much smaller proof using zk-SNARKs. The proof generated at the end of Commit 2 can be verified that is correct very fast, and is small enough to be suitable for a blockchain. The final size of the proof is approximately 2kib, and gets published on the blockchain. ### Finalize The Finalize task performs the following operations: 1. It truncates the output of TreeD file to the sector size and them moves it to the unsealed file location of the sector. User should not that unsealed sector copy will not exist till point in the sealing pipeline. The unsealed copy is created on if “KeepUnsealed” is true for the deal. 2. The cache files for the sector are cleaned up at this stage. 3. Delete the local copy of the pieces that have been added to the sector. ### MoveStorage The MoveStorage task moves the data from sealing storage to permanent storage. ### CommitSubmit In CommitSubmit task, we create the `ProveCommitSector` message for the sector and hand it over to the queue of `SendMessage` task. Through the `ProveCommitSector` message the storage provider provides a Proof of Replication (PoRep) for the sector committed in the `PreCommitSector` message. This proof must be submitted AFTER the security wait requirement by the network (WaitSeed), and before the PreCommit expiration of the sector. This message type can also be aggregated to include multiple ProveCommitSector messages in a single message. These aggregated messages are called `ProveCommitAggregate`. ### WindowPost The WindowPost allow storage providers to verifiably prove they have the data they have committed to the network on disk to create a verifiable, and public record attesting to the storage providers continued commitment of storing the data, or for the network to reward storage providers for their contributions. The overall WindowPost process has been broken into 3 independent tasks in Curio. Each of these tasks are triggered by the `CurioChainScheduler` when the TipSet changes. #### WdPost WindowPost task is responsible for generating the proof for an individual partition in the current deadline. Curio runs multiple such tasks in parallel to speed up the calculation time for each deadline. #### WdPostRecover The WdPostRecover task is also executed on per partition basis for each deadline. We check all the previously faulty sectors and determine which sectors have now recovered since then. it creates the recovery message for each partition in the current deadline and submits these messages to the queue of `SendMessage` task. #### WdPostSubmit WdPostSubmit creates the WindowPost messages for each partition in the current deadline and submits these messages to the queue of `SendMessage` task. ### WinPost Winning Proof-of-SpaceTime (WinningPoSt) is the mechanism by which storage providers are rewarded by the Filecoin network for their contributions to it. As a requirement for doing so, each storage provider is tasked with submitting a compressed Proof-of-Spacetime for a specified sector. Each elected storage provider who successfully creates a block is granted FIL, as well as the opportunity to charge other Filecoin participants fees to include messages in the block. Storage providers who fail to do this in the necessary window will forfeit their opportunity to mine a block. The WinPost task is triggered on each epoch change and if Miner address wins the election then a new block is created and submitted to the chain. ### SendMessage The SendMessage task implements a message queue where message can be added by any other task. These messages are then processed by the `SendMessage` and processed individually. It abstracts away highly-available message sending with coordination through HarmonyDB. It makes sure that Nonce are assigned in transactional manner, and that messages are correctly broadcast to the network. It ensures that messages are sent serially, and that failures to send doesn’t cause a nonce gap. ### ParkPiece Curio has implemented a new file location within the storage subsystem called “piece”. This directory is used to temporarily park the pieces while they are being sealed. The `parked_pieces` also contains the URL and headers to download the data. The ParkPiece task scans the `parked_pieces` table in HarmonyDB every 15 seconds. If any pieces are found, a corresponding file is created in under “piece” directory of the storage and data is downloaded to the file from the URL. When `SectorAddPieceToAny` method is called by an external market node, it creates a ParkPiece tasks. ### DropPiece The DropPiece tasks are responsible for removing a piece from `Piece Park` and ensuring all the files and reference related to the piece are cleaned up. This task is triggered by the Finalize task of a sector is sealing pipeline. ### UpdateEncode SnapDeal sealing tasks are a special type of sealing tasks which allows a storage provider to takes already committed sealed sectors and place deal data into them. The UpdateEncode task encodes the incoming unsealed data (deal data), into an existing sealed sector. Once the encoding is complete, vanilla proofs are generated and verified to check and confirm that the data has been encoded correctly in the sealed sector file. ### UpdateProve In the UpdateProve phase, the output from the UpdateEncode task gets compressed into a smaller proof using zk-SNARKs. The zk-SNARK generated after the UpdateProve can verify that the new data is encoded in the new sealed sector, and is small enough to be suitable for a blockchain. The generation of the zk-SNARK can be done by the CPU or accelerated by using a GPU. ### Resource requirements for each Task type in Curio By default, the number of tasks allowed for each type are not limited on any Curio node. The distributed scheduler ensures that no Curio node over-commits the resources. | Task Name | CPU | RAM(GiB) | GPU | Retry | | --------------- | --- | -------- | --- | ----- | | SDR | 4 | 64 | 0 | 2 | | SDRTreeD | 1 | 1 | 0 | 3 | | SDRTreeRC | 1 | 8 | 1 | 3 | | SyntheticProofs | 1 | 8 | 0 | 5 | | PreCommitSubmit | 0 | 1 | 0 | 16 | | PoRep | 1 | 50 | 1 | 5 | | Finalize | 1 | 0.1 | 0 | 10 | | MoveStorage | 1 | 0.128 | 0 | 10 | | CommitSubmit | 0 | 0.001 | 0 | 16 | | WdPostSubmit | 0 | 0.010 | 0 | 10 | | WdPostRecover | 1 | 0.128 | 0 | 10 | | WdPost | 1 | 32 | 1 | 3 | | WinPost | 1 | 1 | 1 | 3 | | SendMessage | 0 | 0.001 | 0 | 1000 | | UpdateEncode | 1 | 1 | 1 | 3 | | UpdateProve | 1 | 50 | 1 | 3 | # Getting Started This is a step by step guide for new users to get onboarded with Curio ## Curio Database and Distributed Architecture ### Familiarizing Yourself with Curio Before diving into the setup and configuration of Curio, we highly recommend becoming familiar with [Curio's design and fundamental principles](/design). This foundational knowledge will greatly assist in effective administration and troubleshooting. ### **HarmonyDB with YugabyteDB** Curio utilizes YugabyteDB to create an abstraction layer known as HarmonyDB. This HarmonyDB serves two primary purposes: 1. **Metadata Storage**: It stores all Curio-related metadata. 2. **Consensus Layer**: It establishes a consensus layer for the distributed architecture of a Curio cluster. {% hint style="danger" %} We recommend using at least 3 node YugabyteDB cluster for HA and scalability. Loss of the DB will render Curio dead. YugabyteDB should also be backed up regularly. {% endhint %} ### Key Features of HarmonyDB * **High Availability**: Ensures that the metadata and consensus information is always available, even in the event of node failures. * **Scalability**: Capable of handling increasing amounts of data and expanding as the Curio cluster grows. * **Consistency**: Maintains data consistency across the distributed nodes of the Curio cluster. ### Benefits of Using YugabyteDB for HarmonyDB * **Distributed SQL**: Combines the benefits of SQL with the resilience and scalability of a distributed database. * **Fault Tolerance**: Provides strong fault tolerance, ensuring the reliability of the Curio cluster. * **Multi-Region Deployment**: Supports deployment across multiple regions for improved performance and redundancy. ## Chain Node Curio requires access to at least one Filecoin chain node like [Lotus](https://lotus.filecoin.io/lotus/get-started/what-is-lotus/) or [Forest](https://docs.forest.chainsafe.io/) (integration in progress). This chain node is used by Curio to get the current chain state and send messages to the chain. Curio support using multiple chain nodes. ## Network Following port must be opened on each Curio node for API and GUI access | Port | Details | | ----- | ---------------------------------------------------------------- | | 12300 | Default API port | | 4701 | Default GUI port. Not all Curio nodes are required to enable GUI | | 12310 | HTTP server port | ## Boost Compatibility (Deprecated) Boost is no longer compatible with latest Curio releases. Boost adapter is no longer shipped with our main branch and we recommend users to migrate to Curio markets. ## Installing Curio and creating a Curio cluster With an understanding of Curio's internal mechanisms, you can now proceed to [install the Curio binaries](/installation). We recommend using [Debian packages](/installation#debian-package-installation) for the installation, as they facilitate easy installation, upgrades, and process management. After installing your first Curio binary, you can move on to [setting up Curio](/setup), whether you are [migrating from lotus-miner](/setup#migrating-from-lotus-miner-to-curio) or [initializing a new minerID](/setup#initiating-a-new-curio-cluster). ## Best Practices We have compiled [a list of best practices](/best-practices) for deploying and maintaining a Curio cluster. All users are encouraged to follow these recommendations to avoid potential issues. New users should also familiarize themselves with [both binaries shipped with Curio](/curio-cli) and the [GUI pages](/curio-gui). # Versions This is the compatibility matrix for the latest free Curio releases. | Curio Version | Lotus Version | Net | Boost | Yugabyte | Forest | | ------------------------------------------------------------ | ------------- | ------- | ---------- | ------------------- | ---------------- | | 1.22.1 / Automatic | v1.27.X | MainNet | v2.3.0-rc2 | 2.20.X / Automatic | 0.19 / Automatic | | 1.23.0 | >v1.28.1 | MainNet | v2.3.0 | 2.20.X / Automatic | 0.19 / Automatic | | v1.23.1 | >v1.28.1 | MainNet | v2.3.0 | 2.20.x / Automatic | 0.19 / Automatic | | v1.24.0 | v1.30.0-rcX | MainNet | v2.4.0-rc1 | 2.20.x / Automatic | 0.21 / Automatic | | v1.24.1 | v1.30.0-rcX | MainNet | v2.4.0-rc1 | 2.20.x / Automatic | 0.21 / Automatic | | v1.24.2 | v1.30.0 | MainNet | v2.4.0 | 2.20.x / Automatic | 0.21 / Automatic | | v1.24.3 | v1.32.0-rcX | MainNet | v2.4.1 | 2.20.x / Automatic | 0.21 / Automatic | | v1.24.4 | v1.32.0-rcX | MainNet | v2.4.1 | 2.20.x / Automatic | 0.23 / Automatic | | v1.24.5 | v1.32.0-rcX | Mainnet | v2.4.1 | v2024.2 / Automatic | 0.23 / Automatic | | v1.25.0 | v1.32.2 | Mainnet | NA | v2024.2 / Automatic | 0.25 / Automatic | | v1.25.1 | v1.33.0 | Mainnet | NA | v2024.2 / Automatic | 0.26 / Automatic | | v1.26.0 | v1.33.1 | Mainnet | NA | v2024.2 / Automatic | 0.26 / Automatic | | v1.27.0 | v1.34.0 | Mainnet | NA | v2025.1 / Automatic | 0.30 / Automatic | | v1.27.1 | v1.34.1 | Mainnet | NA | v2025.1 / Automatic | 0.30 / Automatic | | v1.27.2 | v1.34.1 | Mainnet | NA | v2025.1 / Automatic | 0.30 / Automatic | | v1.27.3 | v1.34.1 | Mainnet | NA | v2025.1 / Automatic | 0.30 / Automatic | | v1.27.4 | v1.35.1 | Mainnet | NA | v2025.1 / Automatic | 0.33 / Automatic | | v1.28.0 | v1.36.0 | Mainnet | NA | v2025.1 / Automatic | 0.33 / Automatic | | v1.28.1 | v1.36.0 | Mainnet | NA | v2025.1 / Automatic | 0.33 / Automatic | | v1.28.2 | v1.36.1 | Mainnet | NA | v2025.1 / Automatic | 0.34 / Automatic | | v1.28.3 | v1.36.2 | Mainnet | NA | v2025.1 / Automatic | 0.35 / Automatic | {% hint style="danger" %} Releases in red color are **not recommended**. Please proceed with the next stable release. {% endhint %} No preference is denoted by "X". Configurations and the number of machines needed: A: Lotus, Curio (numerous), YugabyteDB (1 or 3), (optional Boost) B: Forest, Curio (numerous), Yugabyte (1 or 3) ## Automatic Updates * Docker has Watchtower which offers automatic updates which work for YugabyteDB and Forest. * Curio can automatically be updated on MainNet through the Debian update process on Ubuntu. * Today, only Lotus & Boost lacks automatic updates and must be built and deployed. * Curio's DEBs include curio-cuda (for Nvidia) and curio-opencl (others like ATI). * These can be mixed in a Curio cluster as they only relate to the hardware on the box. ## Database Schema Versions * When the latest Curio starts-up, it applies any upgrades & migrations to Yugabyte's schema. * This may cause errors on other nodes in your cluster that run the old version (low likelihood), which has the simple solution of completing the upgrade. * If, however, the upgrade has a serious bug and you need to downgrade, "curio toolbox downgrade --last\_good\_date=20250515" ## Notes * Forest (0.19+ & Docker Watchtower) is a light alternative to Lotus Client. It meets Curio's needs, but Boost compatibility is in development. ## Building for CalibrationNet * Required for CalibrationNet participation * Use the Go version specified in curio/GO\_VERSION\_MIN * The available Curio branches are named as release/vVERSION like: release/v1.23.4 * CalibrationNet may be a network-version ahead of MainNet. * DEBs are only for MainNet releases and will be available early so MainNet upgrades cause no interruption. # Installation This guide will show how to build, install and update Curio binaries ## Debian package installation Curio packages are available to be installed directly on Ubuntu / Debian systems. {% hint style="danger" %} Debian packages are only available for mainnet right now. For any other network like calibration network or devnet, binaries must be built from source. {% endhint %} 1. Install prerequisites ```shell sudo apt install mesa-opencl-icd ocl-icd-opencl-dev gcc git jq pkg-config curl clang build-essential hwloc libhwloc-dev wget libarchive-dev libgmp-dev libconfig++-dev protobuf-compiler -y && sudo apt upgrade -y ``` 2. Enable Curio package repo ```bash sudo wget -O /usr/share/keyrings/curiostorage-archive-keyring.gpg https://filecoin-project.github.io/apt/KEY.gpg echo "deb [signed-by=/usr/share/keyrings/curiostorage-archive-keyring.gpg] https://filecoin-project.github.io/apt stable main" | sudo tee /etc/apt/sources.list.d/curiostorage.list sudo apt update ``` 3. Install Curio binaries based on your GPU. For NVIDIA GPUs: ```bash sudo apt install curio-cuda ``` For OpenCL GPUs: ```bash sudo apt install curio-opencl ``` ## Linux Build from source You can build the Curio executables from source by following these steps. ### Software dependencies You will need the following software installed to install and run Curio. #### System-specific Building Curio requires some system dependencies, usually provided by your distribution. {% hint style="warning" %} **Note (batch sealing now builds by default on Linux):** Curio's Linux build now compiles `extern/supraseal` as part of the standard `make build` flow (needed for SnapDeals fast TreeR and batch sealing). This adds the following build requirements: * **CUDA Toolkit 12.x or later** (`nvcc` must be in PATH) * **GCC 12 or 13** — pick the version your CUDA toolkit supports: * CUDA 12.0–12.5 → `gcc-12`/`g++-12` * CUDA 12.6+ or 13+ → `gcc-13`/`g++-13` * Python venv tooling (`python3-venv`) and build tools (`autoconf`, `automake`, `libtool`, `nasm`, `xxd`) To skip the supraseal build (e.g. if you don't have CUDA), use: `make build FFI_USE_OPENCL=1` or `make build DISABLE_SUPRASEAL=1` {% endhint %} Arch: ```shell sudo pacman -Syu opencl-icd-loader gcc git jq pkg-config opencl-headers hwloc libarchive nasm xxd python python-pip python-virtualenv aria2 time protobuf # For batch sealing builds (SnapDeals fast TreeR / batch sealing toolchain): sudo pacman -Syu cuda # GCC 12 or 13 required — pick based on your CUDA version (see note above). ``` Ubuntu/Debian: ```shell sudo apt install -y \ mesa-opencl-icd ocl-icd-opencl-dev \ git jq pkg-config curl clang build-essential hwloc libhwloc-dev wget \ python3 python3-dev python3-pip python3-venv \ autoconf automake libtool \ xxd nasm \ libarchive-dev libssl-dev uuid-dev libfuse3-dev \ libnuma-dev libaio-dev libkeyutils-dev libncurses-dev \ libgmp-dev libconfig++-dev \ protobuf-compiler \ aria2 time \ && sudo apt upgrade -y # GCC 12 or 13 — pick based on your CUDA version: # CUDA 12.0–12.5: sudo apt install gcc-12 g++-12 # CUDA 12.6+/13+: sudo apt install gcc-13 g++-13 # On older Ubuntu (e.g. 20.04) you may need the toolchain PPA: # sudo add-apt-repository ppa:ubuntu-toolchain-r/test -y && sudo apt update # CUDA Toolkit (needed for GPU proving and supraseal build; nvcc must be in PATH) # Install via NVIDIA's CUDA repository for your distro. ``` Fedora: ```shell sudo dnf -y install gcc make git jq pkgconfig mesa-libOpenCL mesa-libOpenCL-devel opencl-headers ocl-icd ocl-icd-devel clang llvm wget hwloc hwloc-devel libarchive-devel protobuf-compiler aria2 time ``` OpenSUSE: ```shell sudo zypper in gcc git jq make libOpenCL1 opencl-headers ocl-icd-devel clang llvm hwloc libarchive-devel protobuf-devel aria2 time && sudo ln -s /usr/lib64/libOpenCL.so.1 /usr/lib64/libOpenCL.so ``` Amazon Linux 2: ```shell sudo yum install -y https://dl.fedoraproject.org/pub/epel/epel-release-latest-7.noarch.rpm; sudo yum install -y git gcc jq pkgconfig clang llvm mesa-libGL-devel opencl-headers ocl-icd ocl-icd-devel hwloc-devel libarchive-devel protobuf-compiler aria2 time ``` ### Rustup Curio needs [rustup](https://rustup.rs/). The easiest way to install it is: ```shell curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh ``` ### Go To build Curio, you need a working installation of [Go](https://golang.org/dl/): It needs to be at-least the version specified in go.mod. Example (match `go.mod`; current repo min is **1.26.2**): ```shell wget -c https://go.dev/dl/go1.26.2.linux-amd64.tar.gz -O - | sudo tar -xz -C /usr/local ``` {% hint style="info" %} You'll need to add `/usr/local/go/bin` to your path. For most Linux distributions you can run something like: ```shell echo 'export PATH=$PATH:/usr/local/go/bin' >> ~/.bashrc && source ~/.bashrc ``` See the [official Golang installation instructions](https://golang.org/doc/install) if you get stuck. {% endhint %} ### System Configuration Before you proceed with the installation, you should increase the UDP buffer. You can do this by running the following commands: ```shell sudo sysctl -w net.core.rmem_max=2097152 sudo sysctl -w net.core.rmem_default=2097152 ``` To persist the UDP buffer size across the reboot, update the `/etc/sysctl.conf` file. ```bash echo 'net.core.rmem_max=2097152' | sudo tee -a /etc/sysctl.conf echo 'net.core.rmem_default=2097152' | sudo tee -a /etc/sysctl.conf ``` ### Build and install Curio Once all the dependencies are installed, you can build and install Curio. 1. Clone the repository:\\ ```bash git clone https://github.com/filecoin-project/curio.git cd curio/ ``` 2. Switch to the latest stable release branch:\\ ```bash git checkout ``` 3. Enable the use of SHA extensions by adding these two environment variables: 
export RUSTFLAGS="-C target-cpu=native -g"
   export FFI_BUILD_FROM_SOURCE=1

   echo 'export RUSTFLAGS="-C target-cpu=native -g"' >> ~/.bashrc
   echo 'export FFI_BUILD_FROM_SOURCE=1' >> ~/.bashrc
   source ~/.bashrc
4. If you are using a **Nvidia GPU**, please set the below environment variables.\\ ```bash export FFI_USE_CUDA=1 export FFI_USE_CUDA_SUPRASEAL=1 echo 'export FFI_USE_CUDA=1' >> ~/.bashrc echo 'export FFI_USE_CUDA_SUPRASEAL=1' >> ~/.bashrc source ~/.bashrc ```

On Linux, the Curio build requires CUDA by default and will fail if nvcc is not found in your PATH. This ensures you don't accidentally build without proper GPU support.

If you want to use OpenCL instead of CUDA (e.g., for AMD GPUs or systems without CUDA), build with:

FFI_USE_OPENCL=1 make clean build
5. Curio is compiled to operate on a single network. Choose the network you want to join, then run the corresponding command to build the Curio node:\\ ```shell # For Mainnet: make clean build # For Calibration Testnet: make clean calibnet ``` 6. Install Curio. This will put `curio` in `/usr/local/bin`. `curio` will use the `$HOME/.curio` folder by default. ```shell sudo make install ``` 7. Run `curio --version` ```md curio version 1.24.5+mainnet+git_214226e7_2025-02-19T17:02:54+04:00 # or curio version 1.24.5+calibnet+git_214226e7_2025-02-19T17:02:54+04:00 ``` You should now have Curio installed. You can now [finish setting up the Curio node](/setup). ## MacOS Build from source You can build the Curio executables from source by following these steps. ### Software dependencies You must have XCode and Homebrew installed to build Curio from source. #### **XCode Command Line Tools** Curio requires that X-Code CLI tools be installed before building the Curio binaries. Check if you already have the XCode Command Line Tools installed via the CLI, run: ```shell xcode-select -p ``` This should output something like: ```plaintext /Library/Developer/CommandLineTools ``` If this command returns a path, then you have Xcode already installed! You can [move on to installing dependencies with Homebrew](#homebrew). If the above command doesn't return a path, install Xcode: ```shell xcode-select --install ``` Next up is installing Curio's dependencies using Homebrew. ### **Homebrew** We recommend that macOS users use [Homebrew](https://brew.sh/) to install each of the necessary packages. Use the command `brew install` to install the following packages: ```shell brew install jq pkg-config hwloc coreutils brew install go@1.24 ``` Next up is cloning the Lotus repository and building the executables. ### **Rust** Rustup is an installer for the systems programming language Rust. Run the installer and follow the onscreen prompts. The default installation option should be chosen unless you are familiar with customisation: ```shell curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh ``` ### Build and install Curio The installation instructions are different depending on which CPU is in your Mac: * [ARM-based CPUs (M1, M2, M3)](#arm-based-cpus) * [Intel CPUs](#intel-cpus) #### **Arm based CPUs** 1. Clone the repository: ```shell git clone https://github.com/filecoin-project/curio.git cd curio/ ``` 2. Switch to the latest stable release branch: ```shell git checkout ``` 3. Create the necessary environment variables to allow Curio to run on Arm architecture: ```shell export LIBRARY_PATH=/opt/homebrew/lib export FFI_BUILD_FROM_SOURCE=1 export PATH="$(brew --prefix coreutils)/libexec/gnubin:/usr/local/bin:$PATH" ``` 4. Build the `curio` binary: ```shell make clean curio ``` 5. Run the final `make` command to move this `curio` executable to `/usr/local/bin`. This allows you to run `curio` from any directory. ```shell sudo make install ``` 6. Run `curio --version` ```md curio version 1.24.5+mainnet+git_214226e7_2025-02-19T17:02:54+04:00 # or curio version 1.24.5+calibnet+git_214226e7_2025-02-19T17:02:54+04:00 ``` 7. You should now have Curio installed. You can now [finish setting up the Curio node](/setup). #### **Intel CPUs** ❗These instructions are for installing Curio on an Intel Mac. If you have an Arm-based CPU, use the [Arm-based CPU instructions ↑](#arm-based-cpus) 1. Clone the repository: ```shell git clone https://github.com/filecoin-project/curio.git cd curio/ ``` 2. Switch to the latest stable release branch: ```shell git checkout ``` 3. Build and install Curio: ```shell make clean curio sudo make install ``` 4. Run `curio --version` ```md curio version 1.23.0+mainnet+git_ae625a5_2024-08-21T15:21:45+04:00 # or curio version 1.23.0+calibnet+git_ae625a5_2024-08-21T15:21:45+04:00 ``` You can now [finish setting up the Curio node](/setup). # Setup This guide will show you setup a new Curio cluster or migrate to Curio from Lotus-Miner ## Setup YugabyteDB {% hint style="warning" %} If you have already set up a YugabyteDB for Boost then you can reuse the same YugabyteDB instance for Curio. You must ensure that YugabyteDB is multiple node cluster for HA. You can skip directly to [migrating from Lotus-Miner to Curio](#migrating-from-lotus-miner-to-curio) or [Initializing new Curio Miner.](#initiating-a-new-curio-cluster) {% endhint %} For this guide, we’re setting up a single node YugaByteDB. However, you must set up multiple YugaByteDB instances in a cluster to enable high availability. {% hint style="danger" %} **Note:** We do **not recommend** using a single-node YugabyteDB setup in production. The instructions below are intended **only for testing and development purposes**. For production deployments, please refer to the official [YugabyteDB documentation](https://docs.yugabyte.com/preview/deploy/manual-deployment/) for setting up a highly available (HA) cluster. {% endhint %} Ensure that you have the following available before we install and set up YugabyteDB: {% hint style="danger" %} **Do not use ZFS as the backing drive for YugabyteDB because of advanced filesystem commands so-far is unavailable.** {% endhint %} 1. One of the following operating systems: * CentOS 7 or later * Ubuntu 16.04 or later For other operating systems, Docker or Kubernetes. Please check out the [YugabyteDB documentation](https://docs.yugabyte.com/preview/quick-start/). 2. **Python 3.** To check the version, execute the following command:
```shell python --version ``` ```undefined Python 3.7.3 ``` \ If you encounter a `Command 'python' not found` error, you might not have an unversioned system-wide python command. * Starting from Ubuntu 20.04, python is no longer available. To fix this, run `sudo apt install python-is-python3`. * For CentOS 8, set `python3` as the alternative for python by running `sudo alternatives --set python /usr/bin/python3` Once these dependencies have been installed, we can run the install script:
```shell # YugabyteDB (single-node dev/test example) # NOTE: For production/HA, follow YugabyteDB’s official deployment docs. wget https://software.yugabyte.com/releases/2.25.1.0/yugabyte-2.25.1.0-b381-linux-x86_64.tar.gz tar xvfz yugabyte-2.25.1.0-b381-linux-x86_64.tar.gz && cd yugabyte-2.25.1.0/ ./bin/post_install.sh ./bin/yugabyted start --advertise_address 127.0.0.1 --master_flags rpc_bind_addresses=127.0.0.1 --tserver_flags rpc_bind_addresses=127.0.0.1 ``` ```yaml +----------------------------------------------------------------------------------------------------------+ | yugabyted | +----------------------------------------------------------------------------------------------------------+ | Status : | | Replication Factor : None | | YugabyteDB UI : http://127.0.0.1:15433 | | JDBC : jdbc:postgresql://127.0.0.1:5433/yugabyte?user=yugabyte&password=yugabyte | | YSQL : bin/ysqlsh -U yugabyte -d yugabyte | | YCQL : bin/ycqlsh -u cassandra | | Data Dir : /root/var/data | | Log Dir : /root/var/logs | | Universe UUID : 411422ee-4c17-4f33-996e-ced847d10f5c | +----------------------------------------------------------------------------------------------------------+ ``` You can adjust the `--advertise_address`, `--rpc_bind_addresses` and `--tserver_flags` according to your own configuration and needs. ## Migrating from Lotus-miner to Curio Curio provides a utility to users onboard quickly. Please run the below command on your `lotus-miner` node and follow the on-screen instructions. It communicates in English (en), Chinese (zh), and Korean (ko). ```shell curio guided-setup ``` Once the migration is complete, you can shut down all of your workers and miner processes. You can start `curio` process to replace them with correct [configuration layer](/configuration#configuration-layers). If you entered non-default in step 3 then please export the relevant details in your `~/.bashrc` file as Curio command needs access to the Database. This step is different from the
| Env Variable | UseCase | | ------------------- | ------------------------- | | CURIO\_DB\_HOST | YugabyteDB SQL IP | | CURIO\_DB\_NAME | YugabyteDB Name | | CURIO\_DB\_USER | DB user for connection | | CURIO\_DB\_PASSWORD | User’s password | | CURIO\_DB\_PORT | YugabyteDB’s SQL port | | CURIO\_REPO\_PATH | Curio’s default repo path | Please proceed to [curio service configuration](/curio-service) for the first node. Once you have completed the service and storage configuration, you can come back to this page and proceed with testing the setup. ### Testing the setup You can confirm that the `curio` process is able to schedule and compute WindowPoSt by running a WindowPoSt test computation: ```shell curio test window-post task ``` From the output we can confirm that a WindowPoSt gets inserted to the database, and is being picked up by the Curio process running with the *wdpost* configuration layer. ## Initiating a new Curio cluster To create a new Curio cluster, a [Lotus daemon node](https://bafybeib7hujkpoqohpby6dqabdea2t6ehcysics3ejoh4jrgtuke4rmolu.on.fleek.co/lotus/install/prerequisites/) is required. {% hint style="warning" %} The Lotus daemon node must be part of the same network as Curio being setup. Example: A `calibration` network daemon cannot be used with `mainnet` Curio cluster. {% endhint %} ### Wallet setup Initiating a new miner ID on the Filecoin network requires an owner, worker and sender address. These address can be same or different depending on the user’s choice. Users must create these wallet on Lotus node before running the Curio commands. ```shell lotus wallet new bls lotus wallet new bls ``` Once new wallet are created, we must send some funds to them. ```shell lotus send 5 lotus send 5 ``` ### Creating new miner ID Curio provides a utility for users to onboard quickly. Please run the below command on your new Curio node, choose `Create a new miner` option and follow the on-screen instructions. It communicates in English (en), Chinese (zh), and Korean (ko). 1. Start the guided setup.
```shell curio guided-setup ``` 2. Choose “Create a new miner” option.
``` Defaulting to English. Please reach out to the Curio team if you would like to have additional language support. Use the arrow keys to navigate: ↓ ↑ → ← ? I want to:: Migrate from existing Lotus-Miner ▸ Create a new miner ``` 3. Enter your YugabyteDB details.
``` This process is partially idempotent. Once a new miner actor has been created and subsequent steps fail, the user need to run 'curio config new-cluster < miner ID >' to finish the configuration. Use the arrow keys to navigate: ↓ ↑ → ← ? Enter the info to connect to your Yugabyte database installation (https://download.yugabyte.com/): ▸ Host: 127.0.0.1 Port: 5433 Username: yugabyte Password: yugabyte Database: yugabyte Continue to connect and update schema. ✔ Step Complete: Pre-initialization steps complete ``` 4. Enter the wallet details be used for “create miner” message.
``` Initializing a new miner actor. Use the arrow keys to navigate: ↓ ↑ → ← ? Enter the info to create a new miner: ▸ Owner Address: <------ Enter wallet 1 here Worker Address: <------ Enter wallet 2 here Sender Address: <------ Enter wallet 1 here Sector Size: 0 <--------------- Sector Size (32 G/GiB/GB) Confidence epochs: 0 Continue to verify the addresses and create a new miner actor. ``` ``` Initializing a new miner actor. ✔ Owner Address: Enter the owner address: t3weiymrx3iyivzeuub5l232gb62ocu7zbjtztudiipm6wkkmsehdydrdddm6cdrflxir26cmrz4xui6t5gruq ✔ Worker Address: Enter worker address: t3xhmgfxurecrusgubzdgme4t2ecxbiyny5uanfzvcrrihzhia654f6gp2ynugpiyp5xe7ibg6fqly76kowfva ✔ Sender Address: Enter sender address: t3weiymrx3iyivzeuub5l232gb62ocu7zbjtztudiipm6wkkmsehdydrdddm6cdrflxir26cmrz4xui6t5gruq ✔ Sector Size: 0 Enter the sector size: 8 MiB ✔ Confidence epochs: 0 Confidence epochs: 0 Pushed CreateMiner message: bafy2bzacebu3mhaj6chnz5frjo2sbxduebnh4e7e37fwm3jd7xhvhla7t6ylu Waiting for confirmation ``` 5. Wait for new miner actor to get created.
``` New miners address is: t01004 (t2cmgqvicpcil5zlp6bqsffmjjfz7ix66k4zaojay) ✔ Step Complete: Miner t01004 created successfully ✔ Step Complete: Configuration 'base' was updated to include this miner's address ``` 6. We request you to please share the basic data about your miner with us to help us improve Curio.
``` The Curio team wants to improve the software you use. Tell the team you're using `curio`. Use the arrow keys to navigate: ↓ ↑ → ← ? Select what you want to share with the Curio team.: ▸ Individual Data: Miner ID, Curio version, chain (mainnet or calibration). Signed. Aggregate-Anonymous: version, chain, and Miner power (bucketed). Hint: I am someone running Curio on whichever chain. Nothing. ``` 7. Finish the initialisation.
``` ✔ Step Complete: New Miner initialization complete. Try the web interface with curio run --layers=gui for further guided improvements. ``` 1. If you entered non-default in step 3 then please export the relevant details via your `~/.bashrc` file before running the Curio command.
| Env Variable | UseCase | | ------------------- | ------------------------- | | CURIO\_DB\_HOST | YugabyteDB SQL IP | | CURIO\_DB\_NAME | YugabyteDB Name | | CURIO\_DB\_USER | DB user for connection | | CURIO\_DB\_PASSWORD | User’s password | | CURIO\_DB\_PORT | YugabyteDB’s SQL port | | CURIO\_REPO\_PATH | Curio’s default repo path | 8. Try running Curio with only `GUI` first.
```shell curio run --layers gui ``` 9. If the `curio` process starts successfully, please proceed with GUI and verify that you can access all the pages. Once, verified, please proceed to [curio service configuration](/curio-service). # Troubleshooting Troubleshooting and operational playbooks for common Curio issues. This section is built from real operator issues seen in `#fil-curio-help`, with notes tied back to the actual Curio code paths. Start here if you are: * stuck during setup/migration, * seeing task failures in the UI, * seeing repeated errors/warnings in logs, * unsure what information to collect before asking for help. ## A simple mental model (helps you debug faster) Most Curio incidents fall into one of these buckets: 1. Chain / Lotus connectivity * Curio can’t talk to the full node, or the full node is unhealthy/out of sync. 2. Control plane (DB + scheduler) * Yugabyte is slow/unhealthy, schema upgrades failed, or tasks are not being scheduled/claimed. 3. Data plane (storage + retrieval) * the bytes aren’t where the system thinks they are (paths, indexes, URLs, permissions). If you don’t know where to start: check **DB health** first. A sick DB creates “random” symptoms everywhere. ## Quick links * [Collect debug info](/troubleshooting/collect-debug-info) (copy/paste checklist) * [Common errors](/troubleshooting/common-errors) (error string → meaning → next step) * [Market / indexing troubleshooting](/curio-market/troubleshooting) * [YugabyteDB troubleshooting](/administration/yugabyte-troubleshooting) # Collect debug info What to collect before reporting a Curio issue. When asking for help, posting the right info up front saves *hours*. Copy/paste this checklist and fill it in. ## Environment * Curio version: ```bash curio --version ``` * Deployment type: * systemd / docker / k8s / other * Enabled layers (and where you set them): * `/etc/curio.env` (`CURIO_LAYERS=...`) or docker env / service unit ## Config snapshot If you use layered config, include the layer list and the relevant sections. * If available in your build: ```bash curio config view --layers ``` * Otherwise: paste the relevant layer TOML sections (Subsystems/Fees/Market/Ingest/HTTP/etc.). ## Logs (include context) * systemd: ```bash journalctl -u curio -n 2000 --no-pager ``` * docker: ```bash docker logs --tail 2000 ``` Include: * the **exact error line** * \~30–100 lines before/after ## DB connectivity and health Confirm the Curio host can reach Yugabyte (YSQL): ```bash ysqlsh -h "$CURIO_DB_HOST" -p "${CURIO_DB_PORT:-5433}" -U "$CURIO_DB_USER" -d "${CURIO_DB_NAME:-yugabyte}" -c "select 1;" ``` If you have multiple DB hosts, include the host list and whether load-balancing is enabled. ## UI / task context * Screenshot of failing task(s) * The task ID(s) from the UI ### Sealing issues Include: * `sp_id` (miner id) * sector number(s) * which stage (SDR/TreeD/PC1/PC2/C2/PreCommit/Commit/WdPoSt) ### Market / deal ingestion issues Include: * deal UUID * piece CID * whether offline/online * the URL you provided (if any) and headers (redact secrets) * indexing status (IPNI enabled? CheckIndex errors?) ### PDP issues Include: * PDP endpoint URL you are testing * the command used (e.g. `pdptool ping ...`) * whether you are testing locally or from outside the network ## Redaction Before posting publicly: * redact API tokens, passwords, and private IPs (if needed) * keep message CIDs / actor IDs (they’re useful for debugging) # Common errors Common Curio errors seen in support, with meaning and next steps. This page maps frequent error strings to: * what they usually mean (in plain language), * whether they are benign vs urgent, * what to do next. If your error isn’t listed, start with [Collect debug info](/troubleshooting/collect-debug-info) and include logs + context. Tip: many errors are symptoms, not causes. If the same task keeps failing, scroll **up** to find the first error in the chain. *** ### API / CLI #### `ERROR: could not get API info for Curio: could not determine API endpoint ... Try setting environment variable: CURIO_API_INFO` What it means: * Your CLI environment can’t find the Curio API endpoint/token. What to do: * If running via systemd, env vars may exist in the service unit but not in your shell. * Export `CURIO_API_INFO` in the shell where you run `curio cli ...`, or run the command on the node/container where Curio is running. *** ### Chain / Lotus connectivity #### `Not able to establish connection to node with addr: ws://.../rpc/v1` What it means: * Curio can’t reach the configured full node endpoint. What to do: * Verify connectivity from the Curio host (DNS, firewall, port). * Confirm Lotus is synced and serving RPC. * Check the configured endpoint in your layers and restart Curio after changes. *** ### Sealing / PoSt #### `invalid partIdx 0 (deadline has 0 partitions)` Plain-English meaning: * Curio asked Lotus for “deadline partitions” (groups of sectors) and got **zero** partitions back. Then it tried to run a proof against partition index `0`, which doesn’t exist. When it’s benign: * Fresh miner / devnet / early migration: you simply don’t have proving sectors yet, so there are no partitions. When it’s not: * You *expect* proving sectors, but Lotus reports none (possible chain sync issue, wrong miner address, wrong network, or a migration/config mismatch). What to do: 1. Confirm your miner actually has proving sectors (via Lotus commands / UI). 2. Confirm Curio is pointing at the right Lotus node and the right network. 3. If this is a `curio test window-post` run, remember it is a *diagnostic*; it will fail if there’s nothing meaningful to prove. Code reference (for maintainers): `tasks/window/compute_do.go`. #### `Error computing WindowPoSt ... no sectors to prove` What it means: * There are no sectors in that deadline/partition that require proofs. Next steps: * Confirm there are proving sectors and deadlines populated. * If you expect proofs: look earlier in the pipeline for commit/precommit failures. *** ### Storage / paths #### `sector {..} redeclared in ... paths/db_index.go` What it suggests: * The same sector appears declared in multiple storage locations, or your local store index contains conflicting entries. What to do: * Verify storage attach/mount configuration. * Confirm there aren’t duplicated mount paths or reused repo paths between nodes. * If you recently moved storage, ensure the old path is detached/removed. *** ### Market / indexing #### `duplicate key value violates unique constraint ... (SQLSTATE 23505)` during indexing / CheckIndex What it means: * Two workers attempted to insert the same identity row (often a retry/concurrency artifact). What to do: * Determine if the pipeline is *progressing* despite the error (harmless noise) or if tasks are stuck. * Collect: deal UUID, piece CID, task ID, and the full log span. * If stuck: see [curio-market troubleshooting](/curio-market/troubleshooting). #### `no suitable data URL found for piece_id ` (ParkPiece) Plain-English meaning: * Curio has a “parked piece” that it needs to read, but it can’t find any usable location for the bytes. * In most setups, that location is an HTTP(S) **data URL** (and optional headers) stored in the DB. Where it happens: * The ParkPiece workflow reads `parked_piece_refs` rows for the piece. Common causes: * The data URL was never added. * The data URL exists but is malformed (bad scheme) or requires headers that weren’t stored. * The task is running on a node that is *not* actually responsible for ingestion (layers mismatch). What to do (operator workflow): 1. Find how the deal was supposed to be ingested: * online (URL) vs offline (local file / piece locator) 2. Verify you actually added a data URL for the deal (see Storage Market docs). 3. Confirm the node running ParkPiece has the **market/ingest** layers enabled. 4. Collect: deal UUID, piece CID, and the ParkPiece task ID. Docs: * See `curio-market/storage-market.md` (“Add data URL for offline deals”). Code reference (for maintainers): `tasks/piece/task_park_piece.go`. *** ### Pipeline #### Sectors stuck mid-pipeline after cordoning a node What it means: * Cordoning blocks all new task scheduling, including location-bound pipeline stages (TreeD, TreeRC, SyntheticProofs, Finalize). If sector data lives on the cordoned node's local storage, no other node can pick up those tasks. What to do: * Uncordon the node to let pipelines resume, or attach storage to another node. * See: [Node Maintenance & Cordoning](/administration/node-maintenance) for recommended workflows. *** ### Batch sealing #### `panic: SupraSealInit: supraseal build tag not enabled` What it means: * The binary you’re running does not include batch sealing support. What to do: * Prefer official release binaries. * If building from source, follow the batch sealing build instructions and ensure required build flags/features are enabled. ### Batch commit failures: `all-or-nothing ... Batch successes 67/68 ... idx=N` What it means: * Batch submission/claiming is atomic: one failing item can cause the batch to fail. What to do: * Identify the failing sector/task corresponding to `idx=N` (UI + logs). * Check wallet balance / message errors. * Consider temporarily reducing batch size/concurrency to isolate the failing entry. (Expanded workflow will live in the Batch Sealing page’s troubleshooting section.) *** ### YugabyteDB #### `Could not upgrade! ...` / `Rewriting of YB table is not yet implemented (SQLSTATE 0A000)` What it means: * You hit a schema migration or table rewrite Yugabyte can’t perform in-place for your version. What to do: * Verify Yugabyte version compatibility with your Curio release. * Always back up before upgrades/downgrades. * Collect the migration filename and the error. See: [YugabyteDB troubleshooting](/administration/yugabyte-troubleshooting) # Curio Service This page explains how to setup a systemd service for Curio Curio can handle multiple GPUs simultaneously without needing to run multiple instances of the Curio process. Therefore, Curio can be managed as a single systemd service without concerns about GPU allocations. ## Systemd Service Configuration The service file for Curio is included in the Debian package and is named `curio.service`. If you have built Curio from source, you can create the service file manually as described below. ### **Service File for Curio** To create the `curio.service` file manually, use the following content: ```ini [Unit] Description=Curio After=network.target [Service] ExecStart=/usr/local/bin/curio run Environment=GOLOG_FILE="/var/log/curio/curio.log" Environment=GOLOG_LOG_FMT="json" LimitNOFILE=1000000 Restart=always RestartSec=10 EnvironmentFile=/etc/curio.env RestartForceExitStatus=100 [Install] WantedBy=multi-user.target ``` ### Environment Variables Configuration The service file requires an `/etc/curio.env` file to be present. This file contains all the necessary environment variables to connect to the database. The `env` file should be created automatically during the Debian package installation. If you are running Curio built from source, you can create the `env` file manually with the following content: **/etc/curio.env File** ```sh CURIO_LAYERS=gui,post CURIO_ALL_REMAINING_FIELDS_ARE_OPTIONAL=true CURIO_DB_HOST=yugabyte1,yugabyte2,yugabyte3 CURIO_DB_USER=yugabyte CURIO_DB_PASSWORD=yugabyte CURIO_DB_PORT=5433 CURIO_DB_NAME=yugabyte CURIO_DB_CASSANDRA_PORT=9042 CURIO_REPO_PATH=~/.curio CURIO_NODE_NAME=ChangeMe FIL_PROOFS_USE_MULTICORE_SDR=1 ``` | Variable | Description | Example Value | | ----------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------- | | `CURIO_LAYERS` | Config layers to be stacked and used for this Curio node | `gui,post` | | `CURIO_ALL_REMAINING_FIELDS_ARE_OPTIONAL` | Allows optional fields to use defaults. This ensures that missing configurations are not treated as errors, instead, they will be assigned defaults | `true` | | `CURIO_DB_HOST` | Database hosts (comma-separated) for the YugabyteDB cluster | `yugabyte1,yugabyte2,yugabyte3` | | `CURIO_DB_USER` | Username for authenticating with YugabyteDB's Postgres Database defined by `CURIO_DB_NAME` | `yugabyte` | | `CURIO_DB_PASSWORD` | Password for the specified CURIO\_DB\_USER | `yugabyte` | | `CURIO_DB_PORT` | Port for YugabyteDB Postgres connection | `5433` | | `CURIO_DB_NAME` | Name of the PostgreSQL database in YugabyteDB | `yugabyte` | | `CURIO_DB_CASSANDRA_PORT` | Port for YugabyteDB Cassandra connection | `9042` | | `CURIO_REPO_PATH` | Directory for Curio storage configuration `json` file | `~/.curio` | | `CURIO_NODE_NAME` | Name of the Curio node | `ChangeMe` | | `FIL_PROOFS_USE_MULTICORE_SDR` | Enables multi-core SDR in Filecoin | `1` | | `FIL_PROOFS_MULTICORE_SDR_PRODUCERS` | Optional. Controls the number of multicore SDR producer threads. Valid values are `1` to `3`; SDR CPU cost is derived from this value when set. | `3` | Ensure all variables are correctly set according to your environment. Additionally, you can also export the following variable for cache location. ```sh FIL_PROOFS_PARAMETER_CACHE=/path/to/folder/in/fast/disk FIL_PROOFS_PARENT_CACHE=/path/to/folder/in/fast/disk2 ``` ## Starting the Curio Service Once all the variables are correctly updated, create the log directory: ```sh mkdir -p /var/log/curio ``` Now, you can start the systemd service with the following command: ```sh sudo systemctl start curio.service ``` Verify that process started successfully by monitoring `systemctl status curio.service` Once Curio service is running, you can proceed to [attaching storage for sealing or permanent storage to the Curio node](/storage-configuration) or [setting up next Curio node in the cluster](/scaling-curio-cluster). # Storage Configuration This guide describes how to attach and configure sealing and permanent storage for Curio nodes Each Curio node keeps track of defined storage locations in `~/.curio/storage.json` (or `$CURIO_REPO_PATH/storage.json`) and uses `~/.curio` path as default. Upon initialization of a storage location, a `/sectorstorage.json` file is created that contains the UUID assigned to this location, along with whether it can be used for sealing or storing. ## Adding sealing storage location Before adding your sealing storage location you will need to consider where the sealing tasks are going to be performed. This command must be run locally from the Curio node where you want to attach the storage. ```sh curio cli storage attach --init --seal OR curio cli --machine storage attach --init --seal ``` {% hint style="info" %} The `--machine` flag accepts an `` input, specifying the machine on which the command should be executed. The **IP** refers to the address of the machine where the Curio process is running, and the **Port** indicates the port on which the Curio process is listening. By default, Curio operates on **port 12300**. This flag enables Curio commands to be executed on remote nodes, facilitating centralized administration from a single node. {% endhint %} ## Adding long-term storage location **Custom location for storing:** After the *sealing* process is completed, sealed sectors are moved to the *store* location, which can be specified as follows: ```sh curio cli storage attach --init --store OR curio cli --machine storage attach --init --store ``` This command must be run locally from the Curio node where you want to attach the storage. This location can be made of large capacity, albeit slower, spinning-disks. ## Attach existing storage to Curio The storage location used by `lotus-miner` or `lotus-worker` can be reused by the Curio cluster. It can be attached once the migrated `lotus-miner` or `lotus-worker` is already running a Curio service. ## Common errors (from support) ### `permission denied` when attaching storage Example error: * `mkdir '/key': permission denied` Plain-English meaning: * Curio writes small metadata into the storage path when attaching/initializing it (for example `sectorstorage.json` and a `key/` directory). If Curio can’t write there, attach will fail. Fix checklist: 1. Confirm the filesystem is mounted **read-write**. 2. Ensure the Curio service user owns (or can write to) the directory. 3. If you use `curio cli --machine ... storage attach ...`, the operation is executed on the *remote machine*, so permissions must be correct there. ### Stale/ghost storage endpoints in `storage list` Symptom: * `curio cli storage list` shows IDs/paths with errors like “all endpoints failed for remote storage ”. Meaning: * Curio still remembers a storage entry, but the endpoint is no longer reachable (retired node, IP change, firewall). Safe cleanup options: * Preferred: run `curio cli storage detach --really-do-it ` on the machine where that path was attached. * If the path is gone and detach can’t work: last resort is editing `~/.curio/storage.json` (or `$CURIO_REPO_PATH/storage.json`). Warning: * Removing the wrong entry can make sectors look “missing” until you re-attach the correct storage path. ```sh curio cli storage attach OR curio cli --machine storage attach ``` ## Filter sector types You can filter for what sectors types are allowed in each sealing path by adjusting the configuration file in: `/sectorstorage.json`. ```json { "ID": "1626519a-5e05-493b-aa7a-0af71612010b", "Weight": 10, "CanSeal": false, "CanStore": true, "MaxStorage": 0, "Groups": [], "AllowTo": [], "AllowTypes": null, "DenyTypes": null } ``` Valid values for `AllowTypes` and `DenyTypes` are: ```javascript "unsealed" "sealed" "cache" "update" "update-cache" ``` These values must be put in an array to be valid (e.g `"AllowTypes": ["unsealed", "update-cache"]`), any other values will generate an error on startup of the `Curio`. A restart of the `Curio` node where this storage is attached is also needed for changes to take effect. ## Separate sealed and unsealed A very basic setup where you want to separate unsealed and sealed sectors could be achieved by: * Add `"DenyTypes": ["unsealed"]` to long-term storage path(s) where you want to store the sealed sectors. * Add `"AllowTypes": ["unsealed"]` to long-term storage path(s) where you want to store the unsealed sectors. Setting only `unsealed` for `AllowTypes` will still allow `cache` and `update-cache` files to be placed in this storage path. If you want to completely deny all other types of sectors in this path, you can add additional valid values to the `"DenyTypes"` field. {% hint style="info" %} If there are existing files with disallowed types in a storage path, those files will remain readable for PoSt/Retrieval. So the worst that can happen in case of misconfiguration in the storage path is that sealing tasks will get stuck waiting for storage to become available. {% endhint %} ## Segregating long-term storage per miner Users can allocate long-term storage to specific miner IDs by specifying miner address strings in the `sectorstore.json` file. This configuration allows precise control over which miners can use the storage. * To allow specific miners, include their addresses in the AllowMiners array: ``` "AllowMiners": ["t01000", "t01002"] ``` This configuration permits only the listed miners (t01000 and t01002) to access the storage. * Similarly, to deny specific miners access to the storage, include their addresses in the DenyMiners array: ``` "DenyMiners": ["t01003", "t01004"] ``` In this example, miners with addresses t01003 and t01004 are explicitly denied access to the storage. This dual configuration approach allows for flexible and secure management of storage access based on miner IDs.
# Configuration How to edit and manage configuration for a Curio cluster ## Configuration The configuration for Curio is stored in the HarmonyDB in a table called `harmony_config`. When a Curio node is started, one or more layer names are supplied to get the desired configuration for the node. ### Configuration Layers Configuration layers provide a set of configuration parameters that determine how a system will operate. These layers can be defined at different levels, meaning that higher layer can override the lower layer, and the system will behave based on the final stacked output. Configuration layers can be arranged in a hierarchy, often from `base`(most general) to most specific. The `base` layer defines default configuration values. More specific layers override these defaults with more targeted configurations. For example, in a simple two-layer configuration system, layers could be organized in the following order: Base Layer - This is the most general layer. It is always included, so add any modifications to defaults here. If you include all your miner IDs here (defined in the addresses section) all hardware will be used for all miner needs. Task Layer - This layer enables SDR tasks. Consider the included layers (below). If a Curio node is started with above 2 layers then it will perform SDR tasks for all miner IDs and will use default values of any other configuration parameter. ``` Example Use of Layers: curio run --layers=post ``` {% hint style="warning" %} `base` layer is always applied by default when a Curio node is started. {% endhint %} #### Advantages of Configuration Layers Flexibility: Configuration layers allow different parts of the system or different users to behave differently according to predefined settings. Scalability: By separating concerns and allowing for specific configuration, the systems become easier to manage as they scale. Maintainability: Changes to configuration can be made on an appropriate layer without affecting the entire system. #### Layer Stacking The configuration layers are stack in the supplied order. The `base` layer is always applied by default so you don't need to specify it. For example, if a Curio node is started with the following layers: ``` --layers miner1,sdr,wdPost,pricing ``` These layer will be stacked on top of each other to create the final configuration for the node. The order of stacking will base > miner1 > sdr > wdPost > pricing. If a configuration parameter is defined in multiple layers then the final layer value will be used. #### Working with layers Curio allows you to manage node configurations using layers. Each layer can be applied or modified independently, with the ‘base’ layer being essential at startup. **Print default configuration** The default configuration is used in base layer by default. ```shell curio config default ``` **Adding a New Layer** To add a new configuration layer or update an existing one, you can provide a filename or input directly via stdin. ```shell curio config set --title ``` **List all layers** List all configuration layers present in the database. ```shell curio config ls ``` **Editing a Layer** Directly edit a configuration layer. * Edit with `vim` editor\\ ```shell curio config edit --editor vim ``` * Edit with a different editor like nano\\ ```shell curio config edit --editor nano ``` **Interpreting Stacked Layers** Interpret and view the combined effect of all applied configuration layers, including system-generated comments. ```shell curio config view --layers [layer1,layers2] ``` **Removing a Layer** Remove a specific configuration layer by name. ```shell curio config rm ``` #### Pre-built Layers When the first Curio miner is initialized or when the first Lotus-Miner is migrated to Curio, the process creates some layers by default for the users. These layers mostly define if a particular task should be picked by the machine or not. **post** ```toml [Subsystems] EnableWindowPost = true EnableWinningPost = true ``` **sdr** ```toml [Subsystems] EnableSealSDR = true ``` **seal** ```toml [Subsystems] EnableSealSDR = true EnableSealSDRTrees = true EnableSendPrecommitMsg = true EnablePoRepProof = true EnableSendCommitMsg = true EnableMoveStorage = true ``` **seal-gpu** ```toml [Subsystems] EnableSealSDRTrees = true EnableSendPrecommitMsg = true ``` **seal-snark** ```toml [Subsystems] EnablePoRepProof = true EnableSendCommitMsg = true ``` **gui** ```toml [Subsystems] EnableWebGui = true ``` ### Configuration management in UI The Curio GUI provides a user-friendly interface for managing configurations. To access this feature, navigate to the “Configurations” page from the UI menu. On this page, all available layers in the database are listed. Users can edit each layer by clicking on it.
Configurations

Curio GUI configuration page

To update a configuration field, users must first enable it by checking the corresponding box. After enabling, the field value can be populated. To comment out the field, simply uncheck the box.
Configuration edit

Curio GUI configuration editor

### Dynamic Configuration Updates Historically, **all configuration changes required a full server restart**, impacting uptime and SLA.\ This is no longer the case — **a growing number of settings can now be updated on the fly**, without restarting any node. More dynamic options will be added over time. **Identifying Hot-Reloadable Settings** In the UI and documentation, these settings are clearly marked with: > **“Updates will affect running instances”** When updated (via UI, CLI, or direct SQL), they are applied automatically within **\~30 seconds**. #### **Behaviour & Caveats** * **Invalid values apply immediately** (e.g., unparsable or out-of-range values) and may cause system-wide disruption.\ They can be corrected by pushing a valid update, but there is **no safety delay or rollback**. * A few configuration changes still **require a restart** — typically structural or first-time operations\ (e.g., adding the first miner address while Market 2 is active).\ In these cases the system will **clearly return an error stating that a restart is required**. #### **Quick Reference** | Type of Change | Restart Needed? | Notes | | ------------------------------------------------------- | ---------------- | ------------------------------------------- | | Marked with **“Updates will affect running instances”** | ❌ No | Takes effect in \~30s | | Bad or invalid value | ❌ No, but unsafe | Applies instantly, may break services | | Structural / unsupported change (rare) | ✅ Yes | System explicitly errors and refuses update | This gradual shift to dynamic configuration reduces downtime, shortens maintenance windows, and improves operational flexibility. # Listen Address How to update default listen address for Curio service By default, all Curio nodes bind to the address "0.0.0.0" on port "12300," ensuring that the Curio API listens on all interfaces. You can change this behavior by specifying an explicit IP address and a different port for Curio to use. ```bash echo "CURIO_LISTEN=x.x.x.x:12301" >> /etc/curio.env ``` and restart the Curio service ```bash systemctl restart curio.service ``` # Prometheus Metrics How to configure Prometheus to scrape metrics from Curio Curio exposes a comprehensive set of metrics via a built-in Prometheus endpoint. These metrics provide visibility into database performance, task execution, resource utilization, and more. ## Metrics Endpoint Curio exposes metrics at the `/debug/metrics` endpoint on the RPC server: ``` http:///debug/metrics ``` The endpoint is automatically enabled on all Curio nodes—no configuration is required to expose basic metrics. All metrics are prefixed with the `curio_` namespace. ## Prometheus Configuration To scrape metrics from Curio, add a job to your Prometheus configuration: ```yaml scrape_configs: - job_name: 'curio' static_configs: - targets: - 'curio-node-1:12300' - 'curio-node-2:12300' - 'curio-node-3:12300' # Optional: adjust scrape interval scrape_interval: 15s ``` Replace the targets with your actual Curio node addresses and ports. ### Service Discovery Curio also provides a service discovery endpoint at `/debug/service-discovery` which can be used with Prometheus file-based service discovery for dynamic node discovery. ## Available Metrics For a complete list of all exported metrics, see the [**Metrics Reference**](/configuration/metrics-reference). Key metric categories include: * **Database (HarmonyDB)**: Query latency, connection pool, errors, failover tracking * **Tasks (HarmonyTask)**: Task counts, durations, success/failure rates, resource usage * **Proof Service**: Proofshare queue, durations, retry counts * **Storage**: NVMe health, cache stats, slot management * **HTTP/Retrieval**: Request counts, bytes served, response codes ## Wallet Exporter (Optional) The Wallet Exporter provides additional metrics about wallet balances, miner power, and message activity. {% hint style="warning" %} **Enable the Wallet Exporter on exactly ONE Curio node in the cluster.** Enabling it on multiple nodes causes duplicate metrics and incorrect aggregations. {% endhint %} ### Enabling the Wallet Exporter Add to a layer active on exactly one node: ```toml [CurioSubsystems] EnableWalletExporter = true ``` Restart the node. Metrics appear within \~30 seconds. See [Wallet Exporter](/experimental-features/wallet-exporter) for detailed usage and the full list of wallet metrics. ## Example Prometheus Queries ### Task Performance ```promql # Average task duration over last hour rate(curio_harmonytask_task_duration_seconds_sum[1h]) / rate(curio_harmonytask_task_duration_seconds_count[1h]) # Task success rate sum(rate(curio_harmonytask_tasks_completed[5m])) / sum(rate(curio_harmonytask_tasks_started[5m])) # Active tasks by type curio_harmonytask_active_tasks ``` ### Database Health ```promql # Average query latency in milliseconds rate(curio_db_total_wait[5m]) / rate(curio_db_hits[5m]) # Database error rate rate(curio_db_errors[5m]) # Connection pool usage curio_db_open_connections ``` ### Resource Utilization ```promql # Node resource usage curio_harmonytask_cpu_usage curio_harmonytask_gpu_usage curio_harmonytask_ram_usage # Node uptime curio_harmonytask_uptime ``` ## Grafana Integration These Prometheus metrics can be visualized using Grafana. Create dashboards to monitor: * **Cluster Overview**: Node uptime, task throughput, error rates * **Task Performance**: Duration histograms, success/failure rates by task type * **Resource Utilization**: CPU, GPU, RAM usage across nodes * **Database Health**: Query latency, connection pool, failover events * **Wallet Activity**: Balances, gas usage, message throughput (with Wallet Exporter) ## AlertManager Integration Curio can send alerts to Prometheus AlertManager. See [Alert Manager](/configuration/alert-manager) for configuration details. ## Troubleshooting ### Metrics not appearing 1. Verify the Curio node is running and accessible 2. Check that you can reach the metrics endpoint: `curl http://:/debug/metrics` 3. Verify your Prometheus configuration targets are correct ### Duplicate metrics If you see duplicate metric series, ensure the Wallet Exporter is enabled on only one node. ### Missing wallet metrics Wallet Exporter metrics require: * `EnableWalletExporter = true` in configuration * Wallets registered in the `wallet_names` table (via the Wallets page in UI) * At least one miner ID configured on the node # Metrics Reference This document lists Prometheus metrics exported by Curio using the exact metric names exposed on the scrape endpoint. > **Note**: This file is auto-generated from source code. Run `make docsgen-metrics` to update. ## Node Metrics | Metric | Type | Labels | Description | | ----------------- | ----- | ----------------- | --------------------------------------------------- | | `curio_node_info` | gauge | `node`, `version` | Curio node identity and version. Value is always 1. | ## Task Metrics (HarmonyTask) | Metric | Type | Labels | Description | | ----------------------------------------------- | --------- | --------------------- | ------------------------------------------------------------------------------------------------------ | | `curio_harmonytask_active_tasks` | gauge | `task_name` | Current number of active tasks. | | `curio_harmonytask_added_tasks` | counter | `task_name` | Total number of tasks added. | | `curio_harmonytask_cpu_usage` | gauge | — | Percentage of CPU in use. | | `curio_harmonytask_gpu_usage` | gauge | — | Percentage of GPU in use. | | `curio_harmonytask_poller_iterations` | counter | — | Total number of poller iterations. | | `curio_harmonytask_ram_usage` | gauge | — | Percentage of RAM in use. | | `curio_harmonytask_task_duration_seconds` | histogram | `task_name` | The histogram of task durations in seconds. | | `curio_harmonytask_task_scheduled_wait_seconds` | histogram | `task_name` | The histogram of task wait times from posting or previous attempt completion to work start in seconds. | | `curio_harmonytask_tasks_completed` | counter | `task_name` | Total number of tasks completed successfully. | | `curio_harmonytask_tasks_failed` | counter | `task_name` | Total number of tasks that failed. | | `curio_harmonytask_tasks_started` | counter | `task_name`, `source` | Total number of tasks started. | | `curio_harmonytask_uptime` | gauge | `version` | Total uptime of the node in seconds. | ## Wallet Exporter Metrics (Optional) | Metric | Type | Labels | Description | | -------------------------------------------- | --------- | -------------------------------------------------------------------------- | --------------------------------------------------- | | `curio_wallet_balance_nfil` | gauge | `address`, `type`, `name` | Balance in NanoFIL | | `curio_wallet_gas_paid_nfil` | counter | `from`, `from_name`, `to`, `to_name`, `method`, `send_reason`, `exit_code` | Gas paid NanoFIL | | `curio_wallet_gas_units_requested` | counter | `from`, `from_name`, `to`, `to_name`, `method`, `send_reason` | Gas units requested | | `curio_wallet_gas_units_used` | counter | `from`, `from_name`, `to`, `to_name`, `method`, `send_reason`, `exit_code` | Gas units used | | `curio_wallet_message_land_duration_seconds` | histogram | — | The histogram of message land durations in seconds. | | `curio_wallet_message_landed` | counter | `from`, `from_name`, `to`, `to_name`, `method`, `send_reason`, `exit_code` | Message landed | | `curio_wallet_message_sent` | counter | `from`, `from_name`, `to`, `to_name`, `method`, `send_reason` | Message sent | | `curio_wallet_power` | gauge | `address`, `type` | Power in Bytes | | `curio_wallet_sent_nfil` | counter | `from`, `from_name`, `to`, `to_name`, `method`, `send_reason`, `exit_code` | Sent NanoFIL | ## Proof Share Metrics | Metric | Type | Labels | Description | | -------------------------------------------------- | --------- | ------ | ------------------------------------------------------------ | | `curio_psvc_proofshare_adder_commits_total` | counter | — | Total number of successful task additions scheduled by Adder | | `curio_psvc_proofshare_adder_hold_decisions_total` | counter | `hold` | Total number of hold decisions made by Adder | | `curio_psvc_proofshare_create_asks_seconds` | histogram | — | Duration of create asks inner loop | | `curio_psvc_proofshare_duration_seconds` | histogram | `call` | Duration of proofshare client\_common operations | | `curio_psvc_proofshare_need_asks` | gauge | — | Number of asks still needed in current Do loop iteration | | `curio_psvc_proofshare_newly_added_total` | counter | — | Total number of new work requests inserted locally | | `curio_psvc_proofshare_queue_count` | gauge | — | Current proofshare request queue count | | `curio_psvc_proofshare_retry_count` | histogram | `call` | Retry count per call in proofshare client\_common operations | | `curio_psvc_proofshare_to_request_remaining` | gauge | — | Remaining requests to fulfill for high-water mark | ## Proof Service Metrics | Metric | Type | Labels | Description | | --------------------------------------- | --------- | ------ | ------------------------------------------------ | | `curio_psvc_clientctl_duration_seconds` | histogram | `call` | Duration of proofsvc clientctl operations | | `curio_psvc_l1ops_duration_seconds` | histogram | `call` | Duration of L1 operations | | `curio_psvc_provictl_duration_seconds` | histogram | `call` | Duration of proofsvc provider control operations | ## Sealing Metrics | Metric | Type | Labels | Description | | ---------------------------------------- | ------- | ------- | ---------------------------------- | | `curio_seal_commit_submitted_total` | counter | `miner` | Commit messages submitted. | | `curio_seal_finalize_completed_total` | counter | `miner` | Finalizations completed. | | `curio_seal_movestorage_completed_total` | counter | `miner` | Move storage operations completed. | | `curio_seal_porep_completed_total` | counter | `miner` | PoRep computations completed. | | `curio_seal_precommit_submitted_total` | counter | `miner` | Precommit messages submitted. | | `curio_seal_sdr_completed_total` | counter | `miner` | SDR computations completed. | | `curio_seal_synth_completed_total` | counter | `miner` | Synthetic proofs completed. | | `curio_seal_treed_completed_total` | counter | `miner` | Tree D computations completed. | | `curio_seal_treerc_completed_total` | counter | `miner` | Tree R/C computations completed. | ## Snap Pipeline Metrics | Metric | Type | Labels | Description | | ---------------------------------------- | ------- | ------- | --------------------------------------- | | `curio_snap_encode_completed_total` | counter | `miner` | Snap encodes completed. | | `curio_snap_movestorage_completed_total` | counter | `miner` | Snap move storage operations completed. | | `curio_snap_prove_completed_total` | counter | `miner` | Snap proves completed. | | `curio_snap_submit_completed_total` | counter | `miner` | Snap submissions completed. | ## Mining Metrics | Metric | Type | Labels | Description | | ------------------------------------- | --------- | ------- | --------------------------------------------------- | | `curio_mining_blocks_included_total` | counter | `miner` | Blocks included in the chain. | | `curio_mining_blocks_submitted_total` | counter | `miner` | Blocks submitted to the network. | | `curio_mining_compute_time_seconds` | histogram | — | Histogram of winning post compute times in seconds. | | `curio_mining_wins_total` | counter | `miner` | Blocks won (election success). | ## Storage Metrics | Metric | Type | Labels | Description | | ---------------------------------------------------------------- | --------- | --------------------------------- | ------------------------------------------------------ | | `curio_stor_available_bytes` | gauge | `id`, `can_seal`, `can_store` | Available storage capacity in bytes | | `curio_stor_capacity_bytes` | gauge | `id`, `can_seal`, `can_store` | Total storage capacity in bytes | | `curio_stor_find_sector_cache_hits` | counter | — | Number of findSectorCache hits | | `curio_stor_find_sector_cache_misses` | counter | — | Number of findSectorCache misses | | `curio_stor_find_sector_uncached` | counter | — | Number of findSector uncached calls | | `curio_stor_generate_single_vanilla_proof_calls` | counter | `update`, `cache_id`, `sealed_id` | Number of calls to GenerateSingleVanillaProof | | `curio_stor_generate_single_vanilla_proof_duration_milliseconds` | histogram | `update`, `cache_id`, `sealed_id` | Duration of GenerateSingleVanillaProof in milliseconds | | `curio_stor_generate_single_vanilla_proof_errors` | counter | `update`, `cache_id`, `sealed_id` | Number of errors in GenerateSingleVanillaProof | | `curio_stor_used_bytes` | gauge | `id`, `can_seal`, `can_store` | Used storage capacity in bytes | ## Cache Metrics | Metric | Type | Labels | Description | | ------------------------------------- | ------- | ---------------------- | -------------------------------------------------- | | `curio_cachedreader_cache_evictions` | counter | `cache_type`, `reason` | Number of cache evictions. | | `curio_cachedreader_cache_hits` | counter | `cache_type` | Number of cache hits. | | `curio_cachedreader_cache_misses` | counter | `cache_type` | Number of cache misses. | | `curio_cachedreader_cache_refs` | gauge | — | Number of active references to cached readers. | | `curio_cachedreader_cache_size` | gauge | `cache_type` | Current number of entries in cache. | | `curio_cachedreader_reader_errors` | counter | `reason` | Total number of piece reader errors. | | `curio_cachedreader_reader_successes` | counter | — | Total number of successful piece reader creations. | ## Network Metrics | Metric | Type | Labels | Description | | ----------------------------------- | ------- | ------ | -------------------------------------------------------- | | `curio_dealdata_data_read` | gauge | `kind` | Number of bytes read from data URLs | | `curio_robusthttp_active_transfers` | gauge | — | Current number of active robusthttp transfers | | `curio_robusthttp_bytes_read` | counter | — | Total bytes delivered by robusthttp readers | | `curio_robusthttp_read_errors` | counter | — | Total number of robusthttp read/request errors (non-EOF) | | `curio_robusthttp_read_failures` | counter | — | Number of robusthttp requests that failed after retries | | `curio_robusthttp_requests_started` | counter | — | Number of robusthttp logical requests started | | `curio_robusthttp_retries` | counter | — | Total number of robusthttp retries across requests | ## FFI Metrics | Metric | Type | Labels | Description | | ----------------------------- | ----- | ------- | ----------------------------- | | `curio_cuffi_snap_enc_active` | gauge | `phase` | Number of tasks in each phase | ## HTTP/Retrieval Metrics | Metric | Type | Labels | Description | | ------------------------------------------------- | --------- | ------------------------------- | ------------------------------------------------------------------------------------------------------ | | `curio_dagstore_pr_at_cache_fill_count` | counter | `network` | PieceReader ReadAt full cache fill count | | `curio_dagstore_pr_at_hit_bytes` | counter | `network` | PieceReader ReadAt bytes from cache | | `curio_dagstore_pr_at_hit_count` | counter | `network` | PieceReader ReadAt from cache hits | | `curio_dagstore_pr_at_read_bytes` | counter | `pr_size`, `network` | PieceReader ReadAt bytes read from source | | `curio_dagstore_pr_at_read_count` | counter | `pr_size`, `network` | PieceReader ReadAt reads from source | | `curio_dagstore_pr_discard_count` | counter | `network` | PieceReader discard count | | `curio_dagstore_pr_discarded_bytes` | counter | `network` | PieceReader discarded bytes | | `curio_dagstore_pr_init_count` | counter | `network` | PieceReader init count | | `curio_dagstore_pr_requested_bytes` | counter | `pr_type`, `network` | PieceReader requested bytes | | `curio_dagstore_pr_seek_back_bytes` | counter | `network` | PieceReader seek back bytes | | `curio_dagstore_pr_seek_back_count` | counter | `network` | PieceReader seek back count | | `curio_dagstore_pr_seek_forward_bytes` | counter | `network` | PieceReader seek forward bytes | | `curio_dagstore_pr_seek_forward_count` | counter | `network` | PieceReader seek forward count | | `curio_http_active_requests` | gauge | `path`, `method` | Number of active/in-flight HTTP requests | | `curio_http_blockstore_cache_hits` | counter | — | Counter of blockstore cache hits | | `curio_http_blockstore_cache_misses` | counter | — | Counter of blockstore cache misses | | `curio_http_piece_by_cid_200_response_count` | counter | — | Counter of /piece/ 200 responses | | `curio_http_piece_by_cid_400_response_count` | counter | — | Counter of /piece/ 400 responses | | `curio_http_piece_by_cid_404_response_count` | counter | — | Counter of /piece/ 404 responses | | `curio_http_piece_by_cid_500_response_count` | counter | — | Counter of /piece/ 500 responses | | `curio_http_piece_by_cid_request_count` | counter | — | Counter of /piece/ requests | | `curio_http_piece_by_cid_request_duration_ms` | histogram | — | Time spent retrieving a piece by cid | | `curio_http_rbls_bytes_sent_count` | counter | — | Counter of the number of bytes sent by bitswap since startup | | `curio_http_rbls_get_fail_response_count` | counter | — | Counter of failed RemoteBlockstore Get responses | | `curio_http_rbls_get_request_count` | counter | — | Counter of RemoteBlockstore Get requests | | `curio_http_rbls_get_success_response_count` | counter | — | Counter of successful RemoteBlockstore Get responses | | `curio_http_rbls_getsize_fail_response_count` | counter | — | Counter of failed RemoteBlockstore GetSize responses | | `curio_http_rbls_getsize_request_count` | counter | — | Counter of RemoteBlockstore GetSize requests | | `curio_http_rbls_getsize_success_response_count` | counter | — | Counter of successful RemoteBlockstore GetSize responses | | `curio_http_rbls_has_fail_response_count` | counter | — | Counter of failed RemoteBlockstore Has responses | | `curio_http_rbls_has_request_count` | counter | — | Counter of RemoteBlockstore Has requests | | `curio_http_rbls_has_success_response_count` | counter | — | Counter of successful RemoteBlockstore Has responses | | `curio_http_request_count` | counter | `path`, `method` | Counter of HTTP requests | | `curio_http_response_bytes_count` | counter | `status_code`, `path` | Sum of HTTP response content-length | | `curio_http_response_status_count` | counter | `status_code`, `path`, `method` | Counter of HTTP response status codes | | `curio_ipni_announce_attempts_total` | counter | `provider`, `result` | Total number of IPNI direct announce attempts. | | `curio_ipni_announce_http_roundtrip_milliseconds` | histogram | `provider`, `status` | Duration of outbound IPNI announce HTTP round trips in milliseconds. | | `curio_ipni_entry_cache_hit_wait_milliseconds` | histogram | `request`, `origin`, `state` | Duration callers wait after hitting the IPNI entry cache. | | `curio_ipni_entry_cache_hits_total` | counter | `request`, `origin`, `state` | Total number of IPNI entry cache hits by request type, cache origin, and readiness state. | | `curio_ipni_entry_cache_lookups_total` | counter | `request`, `result` | Total number of IPNI entry cache lookups. | | `curio_ipni_entry_reconstruction_milliseconds` | histogram | `request`, `source`, `result` | Duration of IPNI entry reconstruction after cache miss. | | `curio_ipni_entry_requests_total` | counter | `request` | Total number of IPNI entry requests handled by the serve chunker. | | `curio_ipni_entry_speculative_unused_total` | counter | — | Total number of speculative IPNI entry cache fills evicted without being consumed by a demand request. | | `curio_ipni_provider_http_request_milliseconds` | histogram | `provider`, `content`, `status` | Duration of inbound IPNI provider HTTP requests in milliseconds. | | `curio_ipni_provider_http_requests_total` | counter | `provider`, `content`, `status` | Total number of inbound IPNI provider HTTP requests. | | `curio_pdp_piece_by_cid_200_response_count` | counter | — | Counter of /piece/ 200 responses for PDP | | `curio_pdp_piece_by_cid_request_count` | counter | — | Counter of /piece/ requests for PDP | | `curio_pdp_piece_by_cid_request_duration_ms` | histogram | — | Time spent retrieving a piece by cid for PDP | | `curio_pdp_piece_bytes_served_count` | counter | — | Counter of the number of bytes served by PDP since startup | ## GC Metrics | Metric | Type | Labels | Description | | ------------------------------- | ------- | ------------------- | ---------------------- | | `curio_gc_sectors_marked_total` | counter | `miner`, `filetype` | Sectors marked for GC. | ## Batching Metrics | Metric | Type | Labels | Description | | ------------------------------------------ | ------- | ------------- | ------------------------------------------------------------ | | `curio_sealsupra_nvme_available_spare` | gauge | `nvme_device` | NVMe Available Spare | | `curio_sealsupra_nvme_bytes_read` | counter | `nvme_device` | NVMe Bytes Read | | `curio_sealsupra_nvme_bytes_written` | counter | `nvme_device` | NVMe Bytes Written | | `curio_sealsupra_nvme_critical_warning` | gauge | `nvme_device` | NVMe Critical Warning Flags | | `curio_sealsupra_nvme_error_log_entries` | gauge | `nvme_device` | NVMe Error Log Entries | | `curio_sealsupra_nvme_media_errors` | gauge | `nvme_device` | NVMe Media Errors | | `curio_sealsupra_nvme_percentage_used` | gauge | `nvme_device` | NVMe Percentage Used | | `curio_sealsupra_nvme_power_cycles` | gauge | `nvme_device` | NVMe Power Cycles | | `curio_sealsupra_nvme_power_on_hours` | gauge | `nvme_device` | NVMe Power On Hours | | `curio_sealsupra_nvme_read_io` | counter | `nvme_device` | NVMe Read IOs | | `curio_sealsupra_nvme_temperature_celsius` | gauge | `nvme_device` | NVMe Temperature in Celsius | | `curio_sealsupra_nvme_unsafe_shutdowns` | gauge | `nvme_device` | NVMe Unsafe Shutdowns | | `curio_sealsupra_nvme_write_io` | counter | `nvme_device` | NVMe Write IOs | | `curio_sealsupra_phase_avg_duration` | gauge | `phase` | Average duration of each phase in seconds | | `curio_sealsupra_phase_duration_so_far` | gauge | `phase` | Duration of the phase so far in seconds | | `curio_sealsupra_phase_lock_count` | gauge | `phase` | Number of active locks in each phase | | `curio_sealsupra_phase_waiting_count` | gauge | `phase` | Number of goroutines waiting for a phase lock | | `curio_slotmgr_slot_errors` | counter | — | Total number of slot errors (e.g., failed to put). | | `curio_slotmgr_slot_in_use` | gauge | `slot_offset` | Slot actively in use (batch sealing). 1=in use, 0=not in use | | `curio_slotmgr_slot_sector_count` | gauge | `slot_offset` | Number of sectors in the slot | | `curio_slotmgr_slots_acquired` | counter | — | Total number of slots acquired. | | `curio_slotmgr_slots_available` | gauge | — | Number of available slots. | | `curio_slotmgr_slots_released` | counter | — | Total number of slots released. | *** *Generated from source files. See* [*Prometheus Metrics*](/configuration/prometheus-metrics) *for setup instructions.* # Alert Manager Curio alert manager setup and configuration Curio has an AlertManager task which runs every 1 hour and allows Curio cluster to alert users about certain issues in the cluster. Currently, Curio supports the alert for following issues: 1. Wallet balance is below 5 Fil. 2. If not WindowPost task is run for a deadline. 3. If an orphan block is found or if no WinningPost task is not created for any epoch. 4. If permanent storage does not have enough space to accommodate the sectors currently being sealed. The AlertManager is a plugin based module and allows integration with any plugin. As of now, Curio has 2 plugins available. Alerts generated can be send to multiple plugins at the same time to allow for a more robust notification mechanism. {% hint style="info" %} Contributions to new critical alerts or integrations with other alerting systems are welcome. {% endhint %} ### PagerDuty Plugin Curio comes with a default integration with [PagerDuty.com](https://www.pagerduty.com/), allowing the sending of critical alerts to storage providers. To configure your Curio cluster to send alerts, you must set up a PagerDuty account. {% hint style="danger" %} Nobody associated with the development of this software has any business relationship with PagerDuty. This integration is provided as a convenient gateway to the storage provider’s alert system of choice. {% endhint %} 1. Sign up for a free PagerDuty account [here](https://www.pagerduty.com/sign-up-free/?type=free). 2. Create a new service that will handle the alerts from the Curio cluster. 3. During the service creation, on the “Integration” page, choose “Events API V2”. 4. Once the service creation is complete, copy the “Integration Key” from the service and paste it in the “base” layer configuration for “PagerDutyIntegrationKey”. 5. Enable the plugin in config layer. 6. Restart one of the nodes, and it will now generate critical alerts at the top of every hour. ``` [Alerting] # MinimumWalletBalance is the minimum balance all active wallets. If the balance is below this value, an # alerts will be triggered for the wallet # # type: types.FIL #MinimumWalletBalance = "5 FIL" [Alerting.PagerDuty] # Enable is a flag to enable or disable the PagerDuty integration. # # type: bool Enable = true # PagerDutyEventURL is URL for PagerDuty.com Events API v2 URL. Events sent to this API URL are ultimately # routed to a PagerDuty.com service and processed. # The default is sufficient for integration with the stock commercial PagerDuty.com company's service. # # type: string PagerDutyEventURL = "https://events.pagerduty.com/v2/enqueue" # PageDutyIntegrationKey is the integration key for a PagerDuty.com service. You can find this unique service # identifier in the integration page for the service. # # type: string PageDutyIntegrationKey = "" ``` ### Prometheus AlertManager 1. Setup a [Prometheus AlertManager](https://prometheus.io/docs/alerting/latest/alertmanager/) instance. 2. Enable the plugin but setting the `Enabled` to True. 3. Paste the `AlertManagerURL` in the configguration. 4. Restart the node with the updated config layer. ``` [Alerting.PrometheusAlertManager] # Enable is a flag to enable or disable the Prometheus AlertManager integration. # # type: bool Enable = true # AlertManagerURL is the URL for the Prometheus AlertManager API v2 URL. # # type: string AlertManagerURL = "http://localhost:9093/api/v2/alerts" ``` ### Slack Webhook Integration This setup ensures that alerts generated by Curio can be forwarded to a designated Slack channel for real-time notifications. 1. Set up a Slack Webhook integration for alert notifications. 2. Enable the plugin by setting `Enable` to `true`, and provide the **WebHookURL** in the configuration. This URL should point to the Slack Webhook endpoint where alerts will be sent. 3. Once configured, restart the node to apply the updated configuration. ``` [Alerting.SlackWebhook] # Enable is a flag to enable or disable the Prometheus AlertManager integration. # # type: bool #Enable = false # WebHookURL is the URL for the URL for slack Webhook. # Example: https://hooks.slack.com/services/T00000000/B00000000/XXXXXXXXXXXXXXXXXXXXXXXX # # type: string #WebHookURL = "" ``` # Balance Manager How to configure balance manager in Curio cluster ## Overview The **Balance Manager** is a subsystem within Curio responsible for **automatically managing balances** for certain actors. Currently, it is used to **top up deal collateral** when the available balance falls below a defined threshold. In future iterations, the Balance Manager will be extended to **handle control addresses and other balance-related operations.** ## Functionality The Balance Manager operates on a periodic task cycle and ensures that a miner's **market balance** remains within an acceptable range by: 1. **Monitoring market balance**: Checks the miner's market escrow balance at regular intervals. 2. **Comparing against thresholds**: * If the balance drops **below** `CollateralLowThreshold`, additional funds are added. * The balance is topped **up to** `CollateralHighThreshold`. 3. **Sending transactions**: Issues an `AddBalance` message to the **market actor (`f05`)** from the **configured wallet**. 4. **Ensuring sufficient funds**: Verifies that the deal collateral wallet has enough balance before sending funds. 5. In the future, this functionality will be expanded to include maintaining control address and other operational balances. The Balance Manager runs every **5 minutes** by default (`BalanceCheckInterval`). ## Configuration The **Balance Manager configuration** should be defined at the **base layer** within Curio. It is part of the **`BalanceManagerConfig`** section inside the `CurioConfig`. ### **Enabling Balance Manager** To enable the Balance Manager, modify the **`EnableBalanceManager`** setting under `Subsystems`: ```toml [Subsystems] EnableBalanceManager = true # Set to true to activate Balance Manager ``` ### **Balance Manager Settings** The Balance Manager's configuration is located under `Addresses`: ```toml [Addresses.BalanceManager.MK12Collateral] DealCollateralWallet = "t3xyz..." # The wallet used to fund market balance CollateralLowThreshold = "5 FIL" # If market balance drops below this, a top-up is triggered CollateralHighThreshold = "20 FIL" # The target balance after a top-up ``` #### Configuration Parameters | Parameter | Description | | ------------------------- | ----------------------------------------------------------------------------------- | | `DealCollateralWallet` | The wallet used to top up the miner's market balance. | | `CollateralLowThreshold` | When the balance falls below this threshold, the Balance Manager triggers a top-up. | | `CollateralHighThreshold` | The target balance after a top-up operation is performed. | ### Example Configuration ```toml [Subsystems] EnableBalanceManager = true [Addresses] [Addresses.BalanceManager] [Addresses.BalanceManager.MK12Collateral] DealCollateralWallet = "t3xyz..." CollateralLowThreshold = "5 FIL" CollateralHighThreshold = "20 FIL" ``` ## Future Enhancements Currently, the Balance Manager is only used for deal collateral top-ups, but it will be extended to: * Manage control addresses balances. * Monitor and maintain additional operation-specific balances. By enabling the Balance Manager, miners can **automate their collateral management**, reducing the risk of **deal failures due to insufficient balance** while optimizing operational efficiency. # Default Curio Configuration The default curio configuration ```toml # Subsystems defines configuration settings for various subsystems within the Curio node. # # type: CurioSubsystemsConfig [Subsystems] # EnableWindowPost enables window post to be executed on this curio instance. Each machine in the cluster # with WindowPoSt enabled will also participate in the window post scheduler. It is possible to have multiple # machines with WindowPoSt enabled which will provide redundancy, and in case of multiple partitions per deadline, # will allow for parallel processing of partitions. # # It is possible to have instances handling both WindowPoSt and WinningPoSt, which can provide redundancy without # the need for additional machines. In setups like this it is generally recommended to run # partitionsPerDeadline+1 machines. (Default: false) # # type: bool #EnableWindowPost = false # The maximum amount of WindowPostMaxTasks tasks that can run simultaneously. Note that the maximum number of tasks will # also be bounded by resources available on the machine. We do not recommend setting this value and let system resources determine # the maximum tasks (Default: 0 - unlimited) # # type: int #WindowPostMaxTasks = 0 # EnableWinningPost enables winning post to be executed on this curio instance. # Each machine in the cluster with WinningPoSt enabled will also participate in the winning post scheduler. # It is possible to mix machines with WindowPoSt and WinningPoSt enabled, for details see the EnableWindowPost # documentation. (Default: false) # # type: bool #EnableWinningPost = false # The maximum amount of WinningPostMaxTasks tasks that can run simultaneously. Note that the maximum number of tasks will # also be bounded by resources available on the machine. We do not recommend setting this value and let system resources determine # the maximum tasks (Default: 0 - unlimited) # # type: int #WinningPostMaxTasks = 0 # EnableParkPiece enables the "piece parking" task to run on this node. This task is responsible for fetching # pieces from the network and storing them in the storage subsystem until sectors are sealed. This task is # only applicable when integrating with boost, and should be enabled on nodes which will hold deal data # from boost until sectors containing the related pieces have the TreeD/TreeR constructed. # Note that future Curio implementations will have a separate task type for fetching pieces from the internet. (Default: false) # # type: bool #EnableParkPiece = false # The maximum amount of ParkPieceMaxTasks tasks that can run simultaneously. Note that the maximum number of tasks will # also be bounded by resources available on the machine (Default: 0 - unlimited) # # type: int #ParkPieceMaxTasks = 0 # The maximum number of pieces that should be in storage + active tasks writing to storage on this node (Default: 0 - unlimited) # # type: int #ParkPieceMaxInPark = 0 # The minimum free storage percentage required for the ParkPiece task to run. (Default: 5) # # type: float64 #ParkPieceMinFreeStoragePercent = 5.0 # EnableSealSDR enables SDR tasks to run. SDR is the long sequential computation # creating 11 layer files in sector cache directory. # # SDR is the first task in the sealing pipeline. It's inputs are just the hash of the # unsealed data (CommD), sector number, miner id, and the seal proof type. # It's outputs are the 11 layer files in the sector cache directory. # # In lotus-miner this was run as part of PreCommit1. (Default: false) # # type: bool #EnableSealSDR = false # The maximum amount of SDR tasks that can run simultaneously. Note that the maximum number of tasks will # also be bounded by resources available on the machine. (Default: 0 - unlimited) # # type: int #SealSDRMaxTasks = 0 # The maximum amount of SDR tasks that need to be queued before the system will start accepting new tasks. # The main purpose of this setting is to allow for enough tasks to accumulate for batch sealing. When batch sealing # nodes are present in the cluster, this value should be set to batch_size+1 to allow for the batch sealing node to # fill up the batch. # This setting can also be used to give priority to other nodes in the cluster by setting this value to a higher # value on the nodes which should have less priority. (Default: 0 - unlimited) # # type: int #SealSDRMinTasks = 0 # EnableSealSDRTrees enables the SDR pipeline tree-building task to run. # This task handles encoding of unsealed data into last sdr layer and building # of TreeR, TreeC and TreeD. # # This task runs after SDR # TreeD is first computed with optional input of unsealed data # TreeR is computed from replica, which is first computed as field # addition of the last SDR layer and the bottom layer of TreeD (which is the unsealed data) # TreeC is computed from the 11 SDR layers # The 3 trees will later be used to compute the PoRep proof. # # In case of SyntheticPoRep challenges for PoRep will be pre-generated at this step, and trees and layers # will be dropped. SyntheticPoRep works by pre-generating a very large set of challenges (~30GiB on disk) # then using a small subset of them for the actual PoRep computation. This allows for significant scratch space # saving between PreCommit and PoRep generation at the expense of more computation (generating challenges in this step) # # In lotus-miner this was run as part of PreCommit2 (TreeD was run in PreCommit1). # Note that nodes with SDRTrees enabled will also answer to Finalize tasks, # which just remove unneeded tree data after PoRep is computed. (Default: false) # # type: bool #EnableSealSDRTrees = false # The maximum amount of SealSDRTrees tasks that can run simultaneously. Note that the maximum number of tasks will # also be bounded by resources available on the machine. (Default: 0 - unlimited) # # type: int #SealSDRTreesMaxTasks = 0 # FinalizeMaxTasks is the maximum amount of finalize tasks that can run simultaneously. # The finalize task is enabled on all machines which also handle SDRTrees tasks. Finalize ALWAYS runs on whichever # machine holds sector cache files, as it removes unneeded tree data after PoRep is computed. # Finalize will run in parallel with the SubmitCommitMsg task. (Default: 0 - unlimited) # # type: int #FinalizeMaxTasks = 0 # EnableSendPrecommitMsg enables the sending of precommit messages to the chain # from this curio instance. # This runs after SDRTrees and uses the output CommD / CommR (roots of TreeD / TreeR) for the message (Default: false) # # type: bool #EnableSendPrecommitMsg = false # EnablePoRepProof enables the computation of the porep proof # # This task runs after interactive-porep seed becomes available, which happens 150 epochs (75min) after the # precommit message lands on chain. This task should run on a machine with a GPU. Vanilla PoRep proofs are # requested from the machine which holds sector cache files which most likely is the machine which ran the SDRTrees # task. # # In lotus-miner this was Commit1 / Commit2 (Default: false) # # type: bool #EnablePoRepProof = false # The maximum amount of PoRepProof tasks that can run simultaneously. Note that the maximum number of tasks will # also be bounded by resources available on the machine. (Default: 0 - unlimited) # # type: int #PoRepProofMaxTasks = 0 # EnableSendCommitMsg enables the sending of commit messages to the chain # from this curio instance. (Default: false) # # type: bool #EnableSendCommitMsg = false # Whether to abort if any sector activation in a batch fails (newly sealed sectors, only with ProveCommitSectors3). (Default: true) # # type: bool #RequireActivationSuccess = true # Whether to abort if any sector activation in a batch fails (updating sectors, only with ProveReplicaUpdates3). (Default: true) # # type: bool #RequireNotificationSuccess = true # EnableMoveStorage enables the move-into-long-term-storage task to run on this curio instance. # This tasks should only be enabled on nodes with long-term storage. # # The MoveStorage task is the last task in the sealing pipeline. It moves the sealed sector data from the # SDRTrees machine into long-term storage. This task runs after the Finalize task. (Default: false) # # type: bool #EnableMoveStorage = false # NoUnsealedDecode disables the decoding sector data on this node. Normally data encoding is enabled by default on # storage nodes with the MoveStorage task enabled. Setting this option to true means that unsealed data for sectors # will not be stored on this node (Default: false) # # type: bool #NoUnsealedDecode = false # The maximum amount of MoveStorage tasks that can run simultaneously. Note that the maximum number of tasks will # also be bounded by resources available on the machine. It is recommended that this value is set to a number which # uses all available network (or disk) bandwidth on the machine without causing bottlenecks. NOTE: unlike most other # tasks, when this value is set the maximum number of concurrent tasks will not be bounded by CPU core count (Default: 0 - unlimited) # # type: int #MoveStorageMaxTasks = 0 # EnableUpdateEncode enables the encoding step of the SnapDeal process on this curio instance. # This step involves encoding the data into the sector and computing updated TreeR (uses gpu). (Default: false) # # type: bool #EnableUpdateEncode = false # EnableUpdateProve enables the proving step of the SnapDeal process on this curio instance. # This step generates the snark proof for the updated sector. (Default: false) # # type: bool #EnableUpdateProve = false # EnableUpdateSubmit enables the submission of SnapDeal proofs to the blockchain from this curio instance. # This step submits the generated proofs to the chain. (Default: false) # # type: bool #EnableUpdateSubmit = false # UpdateEncodeMaxTasks sets the maximum number of concurrent SnapDeal encoding tasks that can run on this instance. (Default: 0 - unlimited) # # type: int #UpdateEncodeMaxTasks = 0 # BindEncodeToData forces the Encode task to be executed on the same node where the data was parked. # Please ensure that ParkPiece task is enabled and relevant resources are available before enabling this option. # (Default: false) # # type: bool #BindEncodeToData = false # AllowEncodeGPUOverprovision allows the Encode task to run on regardress of declared GPU usage. (Default: false) # NOTE: This definitely is not safe on PoSt nodes. # # type: bool #AllowEncodeGPUOverprovision = false # UpdateProveMaxTasks sets the maximum number of concurrent SnapDeal proving tasks that can run on this instance. (Default: 0 - unlimited) # # type: int #UpdateProveMaxTasks = 0 # EnableWebGui enables the web GUI on this curio instance. The UI has minimal local overhead, but it should # only need to be run on a single machine in the cluster. (Default: false) # # type: bool #EnableWebGui = false # The address that should listen for Web GUI requests. It should be in form "x.x.x.x:1234" (Default: 0.0.0.0:4701) # # type: string #GuiAddress = "0.0.0.0:4701" # UseSyntheticPoRep enables the synthetic PoRep for all new sectors. When set to true, will reduce the amount of # cache data held on disk after the completion of TreeRC task to 11GiB. (Default: false) # # type: bool #UseSyntheticPoRep = false # The maximum amount of SyntheticPoRep tasks that can run simultaneously. Note that the maximum number of tasks will # also be bounded by resources available on the machine. (Default: 0 - unlimited) # # type: int #SyntheticPoRepMaxTasks = 0 # EnableBatchSeal enabled SupraSeal batch sealing on the node. (Default: false) # # type: bool #EnableBatchSeal = false # EnableDealMarket enabled the deal market on the node. This would also enable libp2p on the node, if configured. (Default: false) # # type: bool #EnableDealMarket = false # Enable handling for PDP (proof-of-data possession) deals / proving on this node. # PDP deals allow the node to directly store and prove unsealed data with "PDP Services" like Storacha. # This feature is BETA and should only be enabled on nodes which are part of a PDP network. # # type: bool #EnablePDP = false # DataPath is the root directory Curio-PDP scans for writable storage locations. # The node treats this directory and every subdirectory as a candidate store path. # Overridden by the DATA_STORAGE env var and the --data CLI flag. (Default: /data) # # type: string #DataPath = "" # PDPPullPieceMaxTasks is the maximum number of PDPv0 pull-piece download tasks that can run simultaneously. # Set 0 for unlimited. (Default: 20) # # type: int #PDPPullPieceMaxTasks = 20 # PDPUnclaimedUploadKeepHours is how many hours to keep unclaimed PDP piece uploads (orphaned pdp_piecerefs # with data_set_refcount = 0) before PieceGC deletes them. Must be >= 1. (Default: 2) # Updates will affect running instances. # # type: int #PDPUnclaimedUploadKeepHours = 2 # EnableCommP enables the commP task on te node. CommP is calculated before sending PublishDealMessage for a Mk12 deal # Must have EnableDealMarket = True (Default: false) # # type: bool #EnableCommP = false # The maximum amount of CommP tasks that can run simultaneously. Note that the maximum number of tasks will # also be bounded by resources available on the machine. (Default: 0 - unlimited) # # type: int #CommPMaxTasks = 0 # BindCommPToData forces the CommP task to be executed on the same node where the data was parked. # Please ensure that ParkPiece task is enabled and relevant resources are available before enabling this option. # (Default: false) # # type: bool #BindCommPToData = false # The maximum amount of indexing and IPNI tasks that can run simultaneously. Note that the maximum number of tasks will # also be bounded by resources available on the machine. (Default: 8) # # type: int #IndexingMaxTasks = 8 # BindSDRTreeToNode forces the TreeD and TreeRC tasks to be executed on the same node where SDR task was executed # for the sector. Please ensure that TreeD and TreeRC task are enabled and relevant resources are available before # enabling this option. (Default: false) # # type: bool #BindSDRTreeToNode = false # EnableProofShare enables the ProofShare tasks on the node. This subsystem will request proof work from a marketplace # whenever local machine can take on more Snark work. ProofShare tasks have priority over local snark tasks, but new # ProofShare work will only be requested if there is no local work to do. # # This feature is currently experimental and may change in the future. (Default: false) # # type: bool #EnableProofShare = false # The maximum amount of ProofShare tasks that can run simultaneously. Note that the maximum number of tasks will # also be bounded by resources available on the machine. (Default: 0 - unlimited) # # type: int #ProofShareMaxTasks = 0 # EnableRemoteProofs enables the remote proof tasks on the node. Local snark tasks will be transformed into remote # proving tasks when this option is enabled. Details on which SP IDs are allowed to request remote proofs are managed # via Client Settings on the Proofshare webui page. Buy delay can also be set in the Client Settings page. (Default: false) # # type: bool #EnableRemoteProofs = false # The maximum number of remote proofs that can be uploaded simultaneously by each node. (Default: 15) # # type: int #RemoteProofMaxUploads = 15 # EnableWalletExporter enables the wallet exporter on the node. This will export wallet stats to prometheus. # NOTE: THIS MUST BE ENABLED ONLY ON A SINGLE NODE IN THE CLUSTER TO BE USEFUL (Default: false) # # type: bool #EnableWalletExporter = false # Fees holds the fee-related configuration parameters for various operations in the Curio node. # # type: CurioFees [Fees] # WindowPoSt is a high-value operation, so the default fee should be high. # Accepts a decimal string (e.g., "123.45") with optional "fil" or "attofil" suffix. (Default: "5 fil") # # type: types.FIL #MaxWindowPoStGasFee = "5 FIL" # Whether to use available miner balance for sector collateral instead of sending it with each message (Default: false) # # type: bool #CollateralFromMinerBalance = false # Don't send collateral with messages even if there is no available balance in the miner actor (Default: false) # # type: bool #DisableCollateralFallback = false # MaximizeFeeCap makes the sender set maximum allowed FeeCap on all sent messages. # This generally doesn't increase message cost, but in highly congested network messages # are much less likely to get stuck in mempool. (Default: true) # # type: bool #MaximizeFeeCap = true # maxBatchFee = maxBase + maxPerSector * nSectors # (Default: #Base = "0 FIL" and #PerSector = "0.02 FIL") # # type: BatchFeeConfig [Fees.MaxPreCommitBatchGasFee] # Accepts a decimal string (e.g., "123.45") with optional "fil" or "attofil" suffix. # # type: types.FIL #Base = "0 FIL" # Accepts a decimal string (e.g., "123.45") with optional "fil" or "attofil" suffix. # # type: types.FIL #PerSector = "0.02 FIL" # maxBatchFee = maxBase + maxPerSector * nSectors # (Default: #Base = "0 FIL" and #PerSector = "0.03 FIL") # # type: BatchFeeConfig [Fees.MaxCommitBatchGasFee] # Accepts a decimal string (e.g., "123.45") with optional "fil" or "attofil" suffix. # # type: types.FIL #Base = "0 FIL" # Accepts a decimal string (e.g., "123.45") with optional "fil" or "attofil" suffix. # # type: types.FIL #PerSector = "0.03 FIL" # Accepts a decimal string (e.g., "123.45") with optional "fil" or "attofil" suffix. # (Default: #Base = "0 FIL" and #PerSector = "0.03 FIL") # # type: BatchFeeConfig [Fees.MaxUpdateBatchGasFee] # Accepts a decimal string (e.g., "123.45") with optional "fil" or "attofil" suffix. # # type: types.FIL #Base = "0 FIL" # Accepts a decimal string (e.g., "123.45") with optional "fil" or "attofil" suffix. # # type: types.FIL #PerSector = "0.03 FIL" # Addresses specifies the list of miner addresses and their related wallet addresses. # Updates will affect running instances. # # type: []CurioAddresses [[Addresses]] # PreCommitControl is an array of Addresses to send PreCommit messages from # # type: []string #PreCommitControl = [] # CommitControl is an array of Addresses to send Commit messages from # # type: []string #CommitControl = [] # DealPublishControl is an array of Address to send the deal collateral from with PublishStorageDeal Message # # type: []string #DealPublishControl = [] # TerminateControl is a list of addresses used to send Terminate messages. # # type: []string #TerminateControl = [] # DisableOwnerFallback disables usage of the owner address for messages # sent automatically # # type: bool #DisableOwnerFallback = false # DisableWorkerFallback disables usage of the worker address for messages # sent automatically, if control addresses are configured. # A control address that doesn't have enough funds will still be chosen # over the worker address if this flag is set. # # type: bool #DisableWorkerFallback = false # MinerAddresses are the addresses of the miner actors # # type: []string #MinerAddresses = [] # BalanceManagerConfig specifies the configuration parameters for managing wallet balances and actor-related funds, # including collateral and other operational resources. # # type: BalanceManagerConfig [Addresses.BalanceManager] # MK12Collateral defines the configuration for managing collateral and related balance thresholds in the miner's market. # # type: MK12CollateralConfig [Addresses.BalanceManager.MK12Collateral] # DealCollateralWallet is the wallet used to add balance to Miner's market balance. This balance is # utilized for deal collateral in market (f05) deals. # # type: string #DealCollateralWallet = "" # CollateralLowThreshold is the balance below which more balance will be added to miner's market balance # Accepts a decimal string (e.g., "123.45" or "123 fil") with optional "fil" or "attofil" suffix. (Default: "5 FIL") # # type: types.FIL #CollateralLowThreshold = "5 FIL" # CollateralHighThreshold is the target balance to which the miner's market balance will be topped up # when it drops below CollateralLowThreshold. # Accepts a decimal string (e.g., "123.45" or "123 fil") with optional "fil" or "attofil" suffix. (Default: "20 FIL") # # type: types.FIL #CollateralHighThreshold = "20 FIL" # Proving defines the configuration settings related to proving functionality within the Curio node. # # type: CurioProvingConfig [Proving] # Maximum number of sector checks to run in parallel. (0 = unlimited) # # WARNING: Setting this value too high may make the node crash by running out of stack # WARNING: Setting this value too low may make sector challenge reading much slower, resulting in failed PoSt due # to late submission. # # After changing this option, confirm that the new value works in your setup by invoking # 'curio test wd task 0' (Default: 32) # # type: int #ParallelCheckLimit = 32 # Maximum amount of time a proving pre-check can take for a sector. If the check times out the sector will be skipped # # WARNING: Setting this value too low risks in sectors being skipped even though they are accessible, just reading the # test challenge took longer than this timeout # WARNING: Setting this value too high risks missing PoSt deadline in case IO operations related to this sector are # blocked (e.g. in case of disconnected NFS mount) # Time duration string (e.g., "1h2m3s") in TOML format. (Default: "10m0s") # # type: time.Duration #SingleCheckTimeout = "10m0s" # Maximum amount of time a proving pre-check can take for an entire partition. If the check times out, sectors in # the partition which didn't get checked on time will be skipped # # WARNING: Setting this value too low risks in sectors being skipped even though they are accessible, just reading the # test challenge took longer than this timeout # WARNING: Setting this value too high risks missing PoSt deadline in case IO operations related to this partition are # blocked or slow. Time duration string (e.g., "1h2m3s") in TOML format. (Default: "20m0s") # # type: time.Duration #PartitionCheckTimeout = "20m0s" # HTTP represents the configuration for the HTTP server settings in the Curio node. # # type: HTTPConfig [HTTP] # Enable the HTTP server on the node # # type: bool #Enable = false # DomainName specifies the domain name that the server uses to serve HTTP requests. DomainName cannot be empty and cannot be # an IP address # # type: string #DomainName = "" # ListenAddress is the address that the server listens for HTTP requests. It should be in form "x.x.x.x:1234" (Default: 0.0.0.0:12310) # # type: string #ListenAddress = "0.0.0.0:12310" # DelegateTLS allows the server to delegate TLS to a reverse proxy. When enabled the listen address will serve # HTTP and the reverse proxy will handle TLS termination. # # type: bool #DelegateTLS = false # ReadTimeout is the maximum duration for reading the entire or next request, including body, from the client. # Time duration string (e.g., "1h2m3s") in TOML format. (Default: "30m0s") # # type: time.Duration #ReadTimeout = "30m0s" # IdleTimeout is the maximum duration of an idle session. If set, idle connections are closed after this duration. # Time duration string (e.g., "1h2m3s") in TOML format. (Default: "30m0s") # # type: time.Duration #IdleTimeout = "30m0s" # ReadHeaderTimeout is amount of time allowed to read request headers # Time duration string (e.g., "1h2m3s") in TOML format. (Default: "0m5s") # # type: time.Duration #ReadHeaderTimeout = "5s" # CORSOrigins specifies the allowed origins for CORS requests to the Curio admin UI. If empty, CORS is disabled. # If not empty, only the specified origins will be allowed for CORS requests. # This is required for third-party UI servers. # "*" allows everyone, it's best to specify the UI servers' hostname. # # type: []string #CORSOrigins = [] # CSP sets the Content Security Policy for content served via the /piece/ retrieval endpoint. # Valid values: "off", "self", "inline" (Default: "inline") # # Since storage providers serve user-uploaded content on their domain, CSP helps control # what these files can do when rendered in browsers. Choose based on your use case: # # - "off": No CSP headers. Content can load any external resources and execute any scripts. # Use only if you fully trust all stored content or need maximum compatibility. # # - "self": Restricts content to only load resources from your domain. Prevents external # resource loading but allows stored HTML/JS/CSS to interact with each other. # Good for semi-trusted content that needs internal functionality. # # - "inline": (Default) Allows inline scripts/styles and same-origin resources. Provides # basic protection while maintaining compatibility with most web content. # Suitable for general-purpose content hosting. # # Note: Stricter policies may prevent some HTML content from displaying as intended. # Consider the trust level of your users and whether you need to support interactive content. # # type: string #CSP = "inline" # DenylistServers is a list of URLs pointing to denylist.json files. # Each URL should serve a JSON array of objects with an "anchor" field containing a SHA256 hash. # Denylisted CIDs will be rejected with HTTP 451. Requests arriving before denylists are loaded # will receive HTTP 503. (Default: ["https://badbits.dwebops.pub/denylist.json"]) # Updates will affect running instances. # # type: []string #DenylistServers = ["https://badbits.dwebops.pub/denylist.json"] # CompressionLevels hold the compression level for various compression methods supported by the server # # type: CompressionConfig [HTTP.CompressionLevels] # type: int #GzipLevel = 6 # type: int #BrotliLevel = 4 # type: int #DeflateLevel = 6 # Market specifies configuration options for the Market subsystem within the Curio node. # # type: MarketConfig [Market] # StorageMarketConfig houses all the deal related market configuration # # type: StorageMarketConfig [Market.StorageMarketConfig] # PieceLocator is a list of HTTP url and headers combination to query for a piece for offline deals # User can run a remote file server which can host all the pieces over the HTTP and supply a reader when requested. # The server must support "HEAD" request and "GET" request. # 1. ?id=pieceCID with "HEAD" request responds with 200 if found or 404 if not. Must send header "Content-Length" with file size as value # 2. ?id=pieceCID must provide a reader for the requested piece along with header "Content-Length" with file size as value # # type: []PieceLocatorConfig #PieceLocator = [] # MK12 encompasses all configuration related to deal protocol mk1.2.0 and mk1.2.1 (i.e. Boost deals) # # type: MK12Config [Market.StorageMarketConfig.MK12] # When a deal is ready to publish, the amount of time to wait for more # deals to be ready to publish before publishing them all as a batch # Time duration string (e.g., "1h2m3s") in TOML format. (Default: "5m0s") # # type: time.Duration #PublishMsgPeriod = "5m0s" # The maximum number of deals to include in a single PublishStorageDeals # message (Default: 8) # # type: uint64 #MaxDealsPerPublishMsg = 8 # The maximum fee to pay per deal when sending the PublishStorageDeals message # Accepts a decimal string (e.g., "123.45" or "123 fil") with optional "fil" or "attofil" suffix. (Default: "0.5 FIL") # # type: types.FIL #MaxPublishDealFee = "0.5 FIL" # ExpectedPoRepSealDuration is the expected time it would take to seal the deal sector # This will be used to fail the deals which cannot be sealed on time. # Time duration string (e.g., "1h2m3s") in TOML format. (Default: "8h0m0s") # # type: time.Duration #ExpectedPoRepSealDuration = "8h0m0s" # ExpectedSnapSealDuration is the expected time it would take to snap the deal sector # This will be used to fail the deals which cannot be sealed on time. # Time duration string (e.g., "1h2m3s") in TOML format. (Default: "2h0m0s") # # type: time.Duration #ExpectedSnapSealDuration = "2h0m0s" # SkipCommP can be used to skip doing a commP check before PublishDealMessage is sent on chain # Warning: If this check is skipped and there is a commP mismatch, all deals in the # sector will need to be sent again (Default: false) # # type: bool #SkipCommP = false # MaxConcurrentDealSizeGiB is a sum of all size of all deals which are waiting to be added to a sector # When the cumulative size of all deals in process reaches this number, new deals will be rejected. # (Default: 0 = unlimited) # # type: int64 #MaxConcurrentDealSizeGiB = 0 # DenyUnknownClients determines the default behaviour for the deal of clients which are not in allow/deny list # If True then all deals coming from unknown clients will be rejected. (Default: false) # # type: bool #DenyUnknownClients = false # DenyOnlineDeals determines if the storage provider will accept online deals (Default: false) # # type: bool #DenyOnlineDeals = false # DenyOfflineDeals determines if the storage provider will accept offline deals (Default: false) # # type: bool #DenyOfflineDeals = false # CIDGravityTokens is the list of authorization token to use for CIDGravity filters. These should be in format # "minerID1:Token1", "minerID2:Token2". If a token for a minerID within the cluster is not provided, # CIDGravity filters will not be applied to deals associated with that miner ID. # # type: []string #CIDGravityTokens = [] # DefaultCIDGravityAccept when set to true till accept deals when CIDGravity service is not available. # Default behaviors is to reject the deals (Default: false) # # type: bool #DefaultCIDGravityAccept = false # MK20 encompasses all configuration related to deal protocol mk2.0 i.e. market 2.0 # # type: MK20Config [Market.StorageMarketConfig.MK20] # ExpectedPoRepSealDuration is the expected time it would take to seal the deal sector # This will be used to fail the deals which cannot be sealed on time. # Time duration string (e.g., "1h2m3s") in TOML format. (Default: "8h0m0s") # # type: time.Duration #ExpectedPoRepSealDuration = "8h0m0s" # ExpectedSnapSealDuration is the expected time it would take to snap the deal sector # This will be used to fail the deals which cannot be sealed on time. # Time duration string (e.g., "1h2m3s") in TOML format. (Default: "2h0m0s") # # type: time.Duration #ExpectedSnapSealDuration = "2h0m0s" # SkipCommP can be used to skip doing a commP check before PublishDealMessage is sent on chain # Warning: If this check is skipped and there is a commP mismatch, all deals in the # sector will need to be sent again (Default: false) # # type: bool #SkipCommP = false # MaxConcurrentDealSizeGiB is a sum of all size of all deals which are waiting to be added to a sector # When the cumulative size of all deals in process reaches this number, new deals will be rejected. # (Default: 0 = unlimited) # # type: int64 #MaxConcurrentDealSizeGiB = 0 # DenyUnknownClients determines the default behaviour for the deal of clients which are not in allow/deny list # If True then all deals coming from unknown clients will be rejected. (Default: false) # # type: bool #DenyUnknownClients = false # MaxParallelChunkUploads defines the maximum number of upload operations that can run in parallel. (Default: 512) # # type: int #MaxParallelChunkUploads = 512 # MinimumChunkSize defines the smallest size of a chunk allowed for processing, expressed in bytes. Must be a power of 2. (Default: 16 MiB) # # type: int64 #MinimumChunkSize = 16777216 # MaximumChunkSize defines the maximum size of a chunk allowed for processing, expressed in bytes. Must be a power of 2. (Default: 256 MiB) # # type: int64 #MaximumChunkSize = 268435456 # IPNI configuration for ipni-provider # # type: IPNIConfig [Market.StorageMarketConfig.IPNI] # Disable set whether to disable indexing announcement to the network and expose endpoints that # allow indexer nodes to process announcements. Default: False # # type: bool #Disable = false # The network indexer web UI URL for viewing published announcements # # type: []string #ServiceURL = ["https://cid.contact", "https://filecoinpin.contact"] # The list of URLs of indexing nodes to announce to. This is a list of hosts we talk to tell them about new # heads. # # type: []string #DirectAnnounceURLs = ["https://cid.contact/ingest/announce", "https://filecoinpin.contact/announce"] # Indexing configuration for deal indexing # # type: IndexingConfig [Market.StorageMarketConfig.Indexing] # Number of records per insert batch # # type: int #InsertBatchSize = 1000 # Number of concurrent inserts to split AddIndex calls to # # type: int #InsertConcurrency = 10 # Ingest defines configuration parameters for handling and limiting deal ingestion pipelines within the Curio node. # # type: CurioIngestConfig [Ingest] # MaxMarketRunningPipelines is the maximum number of market pipelines that can be actively running tasks. # A "running" pipeline is one that has at least one task currently assigned to a machine (owner_id is not null). # If this limit is exceeded, the system will apply backpressure to delay processing of new deals. # 0 means unlimited. (Default: 64) # Updates will affect running instances. # # type: int #MaxMarketRunningPipelines = 64 # MaxQueueDownload is the maximum number of pipelines that can be queued at the downloading stage, # waiting for a machine to pick up their task (owner_id is null). # If this limit is exceeded, the system will apply backpressure to slow the ingestion of new deals. # 0 means unlimited. (Default: 8) # Updates will affect running instances. # # type: int #MaxQueueDownload = 8 # MaxQueueCommP is the maximum number of pipelines that can be queued at the CommP (verify) stage, # waiting for a machine to pick up their verification task (owner_id is null). # If this limit is exceeded, the system will apply backpressure, delaying new deal processing. # 0 means unlimited. (Default: 8) # Updates will affect running instances. # # type: int #MaxQueueCommP = 8 # Maximum number of sectors that can be queued waiting for deals to start processing. # 0 = unlimited # Note: This mechanism will delay taking deal data from markets, providing backpressure to the market subsystem. # The DealSector queue includes deals that are ready to enter the sealing pipeline but are not yet part of it. # DealSector queue is the first queue in the sealing pipeline, making it the primary backpressure mechanism. (Default: 8) # Updates will affect running instances. # # type: int #MaxQueueDealSector = 8 # Maximum number of sectors that can be queued waiting for SDR to start processing. # 0 = unlimited # Note: This mechanism will delay taking deal data from markets, providing backpressure to the market subsystem. # The SDR queue includes deals which are in the process of entering the sealing pipeline. In case of the SDR tasks it is # possible that this queue grows more than this limit(CC sectors), the backpressure is only applied to sectors # entering the pipeline. # Only applies to PoRep pipeline (DoSnap = false) (Default: 8) # Updates will affect running instances. # # type: int #MaxQueueSDR = 8 # Maximum number of sectors that can be queued waiting for SDRTrees to start processing. # 0 = unlimited # Note: This mechanism will delay taking deal data from markets, providing backpressure to the market subsystem. # In case of the trees tasks it is possible that this queue grows more than this limit, the backpressure is only # applied to sectors entering the pipeline. # Only applies to PoRep pipeline (DoSnap = false) (Default: 0) # Updates will affect running instances. # # type: int #MaxQueueTrees = 0 # Maximum number of sectors that can be queued waiting for PoRep to start processing. # 0 = unlimited # Note: This mechanism will delay taking deal data from markets, providing backpressure to the market subsystem. # Like with the trees tasks, it is possible that this queue grows more than this limit, the backpressure is only # applied to sectors entering the pipeline. # Only applies to PoRep pipeline (DoSnap = false) (Default: 0) # Updates will affect running instances. # # type: int #MaxQueuePoRep = 0 # MaxQueueSnapEncode is the maximum number of sectors that can be queued waiting for UpdateEncode tasks to start. # 0 means unlimited. # This applies backpressure to the market subsystem by delaying the ingestion of deal data. # Only applies to the Snap Deals pipeline (DoSnap = true). (Default: 16) # Updates will affect running instances. # # type: int #MaxQueueSnapEncode = 16 # MaxQueueSnapProve is the maximum number of sectors that can be queued waiting for UpdateProve to start processing. # 0 means unlimited. # This applies backpressure in the Snap Deals pipeline (DoSnap = true) by delaying new deal ingestion. (Default: 0) # Updates will affect running instances. # # type: int #MaxQueueSnapProve = 0 # Maximum time an open deal sector should wait for more deals before it starts sealing. # This ensures that sectors don't remain open indefinitely, consuming resources. # Time duration string (e.g., "1h2m3s") in TOML format. (Default: "1h0m0s") # Updates will affect running instances. # # type: time.Duration #MaxDealWaitTime = "1h0m0s" # DoSnap, when set to true, enables snap deal processing for deals ingested by this instance. # Unlike lotus-miner, there is no fallback to PoRep when no snap sectors are available. # When enabled, all deals will be processed as snap deals. (Default: false) # # type: bool #DoSnap = false # DisableSSRFProtection disables all SSRF (Server-Side Request Forgery) protection # on deal data URL fetching. When true, URLs pointing to private IPs, loopback # addresses, and local hostnames are allowed. URL structure validation (scheme, # control characters) is still enforced. # WARNING: Only enable in development or testing environments. (Default: false) # Updates will affect running instances. # # type: bool #DisableSSRFProtection = false # SSRFAllowedHosts is a list of hosts or host:port pairs that are allowed through # SSRF protection. Matched hosts bypass IP and hostname restrictions while other # safety checks (URL scheme, headers) remain active. # Entries without a port match any port for that host. # Example: ["192.168.1.100:8080", "my-dev-server.local"] # (Default: []) # Updates will affect running instances. # # type: []string #SSRFAllowedHosts = [] # Seal defines the configuration related to the sealing process in Curio. # # type: CurioSealConfig [Seal] # BatchSealSectorSize Allows setting the sector size supported by the batch seal task. # Can be any value as long as it is "32GiB". (Default: "32GiB") # # type: string #BatchSealSectorSize = "32GiB" # Number of sectors in a seal batch. Depends on hardware and supraseal configuration. (Default: 32) # # type: int #BatchSealBatchSize = 32 # Number of parallel pipelines. Can be 1 or 2. Depends on available raw block storage (Default: 2) # # type: int #BatchSealPipelines = 2 # SingleHasherPerThread is a compatibility flag for older CPUs. Zen3 and later supports two sectors per thread. # Set to false for older CPUs (Zen 2 and before). (Default: false) # # type: bool #SingleHasherPerThread = false # Apis defines the configuration for API-related settings in the Curio system. # # type: ApisConfig [Apis] # API auth secret for the Curio nodes to use. This value should only be set on the bade layer. # # type: string #StorageRPCSecret = "" # Alerting specifies configuration settings for alerting mechanisms, including thresholds and external integrations. # # type: CurioAlertingConfig [Alerting] # ClusterName identifies the Curio cluster in external alerts. When empty, the hostname of the node sending the alert is used. # # type: string #ClusterName = "" # MinimumWalletBalance is the minimum balance all active wallets. If the balance is below this value, an # alerts will be triggered for the wallet # Accepts a decimal string (e.g., "123.45" or "123 fil") with optional "fil" or "attofil" suffix. (Default: "5 FIL") # # type: types.FIL #MinimumWalletBalance = "5 FIL" # PagerDutyConfig is the configuration for the PagerDuty alerting integration. # # type: PagerDutyConfig [Alerting.PagerDuty] # Enable is a flag to enable or disable the PagerDuty integration. # # type: bool #Enable = false # PagerDutyEventURL is URL for PagerDuty.com Events API v2 URL. Events sent to this API URL are ultimately # routed to a PagerDuty.com service and processed. # The default is sufficient for integration with the stock commercial PagerDuty.com company's service. # # type: string #PagerDutyEventURL = "https://events.pagerduty.com/v2/enqueue" # PageDutyIntegrationKey is the integration key for a PagerDuty.com service. You can find this unique service # identifier in the integration page for the service. # # type: string #PageDutyIntegrationKey = "" # PrometheusAlertManagerConfig is the configuration for the Prometheus AlertManager alerting integration. # # type: PrometheusAlertManagerConfig [Alerting.PrometheusAlertManager] # Enable is a flag to enable or disable the Prometheus AlertManager integration. # # type: bool #Enable = false # AlertManagerURL is the URL for the Prometheus AlertManager API v2 URL. # # type: string #AlertManagerURL = "http://localhost:9093/api/v2/alerts" # SlackWebhookConfig is a configuration type for Slack webhook integration. # # type: SlackWebhookConfig [Alerting.SlackWebhook] # Enable is a flag to enable or disable the Prometheus AlertManager integration. # # type: bool #Enable = false # WebHookURL is the URL for the URL for slack Webhook. # Example: https://hooks.slack.com/services/T00000000/B00000000/XXXXXXXXXXXXXXXXXXXXXXXX # # type: string #WebHookURL = "" # AppriseConfig is the configuration for the Apprise (https://github.com/caronc/apprise-api) integration. # # type: AppriseConfig [Alerting.Apprise] # URL is the notify endpoint of a running Apprise API server (https://github.com/caronc/apprise-api). # Either its stateless endpoint (e.g. "http://127.0.0.1:8000/notify", use with NotifyURLs) or a # stateful, pre-configured endpoint (e.g. "http://127.0.0.1:8000/notify/curio", leave NotifyURLs empty). # Leave empty to disable the Apprise integration. # # type: string #URL = "" # Tag restricts delivery to Apprise URLs carrying this tag. Only applies to stateful configs. OPTIONAL. # # type: string #Tag = "" # Batching represents the batching configuration for pre-commit, commit, and update operations. # # type: CurioBatchingConfig [Batching] # Precommit Batching configuration # # type: PreCommitBatchingConfig [Batching.PreCommit] # Base fee value below which we should try to send Precommit messages immediately # Accepts a decimal string (e.g., "123.45" or "123 fil") with optional "fil" or "attofil" suffix. (Default: "0.005 FIL") # # type: types.FIL #BaseFeeThreshold = "0.005 FIL" # Maximum amount of time any given sector in the batch can wait for the batch to accumulate # Time duration string (e.g., "1h2m3s") in TOML format. (Default: "4h0m0s") # # type: time.Duration #Timeout = "4h0m0s" # Time buffer for forceful batch submission before sectors/deal in batch would start expiring # Time duration string (e.g., "1h2m3s") in TOML format. (Default: "6h0m0s") # # type: time.Duration #Slack = "6h0m0s" # Maximum number of sectors per precommit batch message. The batch will be submitted # immediately when this many sectors are ready, without waiting for the timeout. # 0 = use the protocol maximum. (Default: 0) # # type: int #MaxBatch = 0 # Commit batching configuration # # type: CommitBatchingConfig [Batching.Commit] # Base fee value below which we should try to send Commit messages immediately # Accepts a decimal string (e.g., "123.45" or "123 fil") with optional "fil" or "attofil" suffix. (Default: "0.005 FIL") # # type: types.FIL #BaseFeeThreshold = "0.005 FIL" # Maximum amount of time any given sector in the batch can wait for the batch to accumulate # Time duration string (e.g., "1h2m3s") in TOML format. (Default: "1h0m0s") # # type: time.Duration #Timeout = "1h0m0s" # Time buffer for forceful batch submission before sectors/deals in batch would start expiring # Time duration string (e.g., "1h2m3s") in TOML format. (Default: "1h0m0s") # # type: time.Duration #Slack = "1h0m0s" # Maximum number of sectors per commit batch message. The batch will be submitted # immediately when this many sectors are ready, without waiting for the timeout. # 0 = use the protocol maximum. (Default: 0) # # type: int #MaxBatch = 0 # Snap Deals batching configuration # # type: UpdateBatchingConfig [Batching.Update] # Base fee value below which we should try to send Commit messages immediately # Accepts a decimal string (e.g., "123.45" or "123 fil") with optional "fil" or "attofil" suffix. (Default: "0.005 FIL") # # type: types.FIL #BaseFeeThreshold = "0.005 FIL" # Maximum amount of time any given sector in the batch can wait for the batch to accumulate # Time duration string (e.g., "1h2m3s") in TOML format. (Default: "1h0m0s") # # type: time.Duration #Timeout = "1h0m0s" # Time buffer for forceful batch submission before sectors/deals in batch would start expiring # Time duration string (e.g., "1h2m3s") in TOML format. (Default: "1h0m0s") # # type: time.Duration #Slack = "1h0m0s" # Cuzk configures integration with the cuzk proving daemon. # When enabled, SNARK proving tasks (PoRep C2, SnapDeals prove, and PSProve) are delegated # to an external cuzk daemon over gRPC instead of using local GPU resources. # # type: CuzkConfig [Cuzk] # Address of the cuzk daemon gRPC endpoint. # Supports unix socket (e.g., "unix:///run/curio/cuzk.sock") or TCP (e.g., "127.0.0.1:9820"). # Empty string disables cuzk integration. (Default: "") # # type: string #Address = "" # MaxPending is the maximum number of proof jobs that may be pending in the cuzk daemon queue # before Curio stops accepting new proving tasks (backpressure). When the daemon's pending # queue reaches this level, CanAccept will reject new tasks until capacity frees up. # (Default: 10) # # type: int #MaxPending = 10 # ProveTimeout is the maximum time to wait for a proof result from the cuzk daemon. # If the proof is not completed within this duration, the task will be retried. # Time duration string (e.g., "30m", "1h"). (Default: "30m") # # type: time.Duration #ProveTimeout = "30m0s" ``` # Curio Market An overview of the Curio market module The Curio Storage Market is a comprehensive framework designed to manage Filecoin storage deals using a flexible and scalable architecture that supports multiple protocols, currently including MK1.2, with future protocols planned for seamless integration. The market handles deal-making through both online and offline processes, providing efficient tools for each step of the deal lifecycle. * **Protocols:** The market is designed to support multiple deal protocols. Currently, MK1.2 is implemented, but the system can easily expand to accommodate future protocols without major reconfiguration. * **Deal Flow:** The market manages both online and offline deals, ensuring smooth operations for various storage needs. Offline deals can utilize a PieceLocator configuration with HTTP servers to retrieve pieces when requested. * **libp2p and Networking:** Curio’s market integrates with libp2p for peer-to-peer communication, allowing for decentralized and secure data transmission between storage clients and providers. * **HTTP Server:** The HTTP server plays a vital role in managing tasks like offline deal data retrieval through URLs and handling client requests for deal processing. * **Retrieval and Indexing:** Deals are efficiently indexed and can be retrieved for future use, ensuring that stored data remains accessible and manageable throughout its lifecycle. The Curio Storage Market provides a highly configurable and extensible system for deal management, optimized for current and future decentralized storage protocols. ## Curio vs Boost How is Curio market different from Boost? ### Deal Processing In Curio’s deal processing, the task-based approach introduces modularity and flexibility in handling different stages of a deal's lifecycle. Each step, from data preparation to sealing, is managed as an individual task. This approach breaks down the complexity of the process by distributing specific responsibilities, such as data validation, publishing deals, and commP calculation, across multiple independent tasks. Each task is monitored, retried on failure, and orchestrated through the Harmony task system, ensuring better resource utilization and scalability. This structure allows for parallel processing of deals and ensures that failures in one part do not impact the entire deal flow. Additionally, it provides flexibility for incorporating new protocols as they are introduced, seamlessly integrating them into the task orchestration. This method ensures that Curio’s deal processing remains future-proof, adaptable to different protocols, and scalable across large volumes of data and interactions. ### Offline Deals In Boost, offline deals were initiated by manually importing data on the storage provider's side. However, Curio simplifies this process by allowing users to [add a URL for offline deal data directly into the database](/curio-market/storage-market#add-data-url-for-offline-deals) or use the [`PieceLocator` configuration](/curio-market/storage-market#piecelocator-configuration) to point to a remote server that can serve the deal’s piece. Since Curio operates as a cluster rather than a single node, data might be needed on different nodes depending on task scheduling and execution. The remote read design in Curio provides flexibility by enabling any node to fetch the required data dynamically during task execution, ensuring efficient processing across the cluster. ### Optional CommP In Boost, the CommP (Commitment to Piece) calculation was a mandatory step in the deal processing workflow. However, in Curio, CommP is optional and can be skipped if desired. This flexibility allows users to streamline deal processing by bypassing the CommP check when necessary, while still ensuring that deals can progress smoothly without compromising efficiency in specific use cases. ### HTTP only retrievals Curio’s retrieval mechanism is HTTP-only, allowing serving deal data from multiple nodes, each running an HTTP server. These nodes can operate under different domain names, providing flexibility and scalability. This distributed retrieval design ensures that data can be fetched from any node within the cluster, based on availability and proximity, optimizing retrieval speed. By supporting multiple nodes and domains for HTTP-based retrieval, Curio enhances resilience and performance, allowing clients to access deal data efficiently without relying on a single centralized source, all within the HTTP protocol. ### IPNI Sync Curio creates dedicated peerID for each miner ID which are used to identify the provider on IPNI. This peer ID is different from the peer ID on chain. Curio announces per piece instead of per deal to IPNI nodes. This removes unnecessary overhead of mapping deals to ads and instead takes a simpler approach of advertising only for the piece we have regardless of how many times they are onboarded. The chunking in Curio process leverages a database to ensure fast and efficient chunk creation and reconstruction. By storing the first CID of each chunk along with metadata like offsets and chunk numbers, the system can quickly locate and retrieve chunks. Sorted entries minimize redundancy by eliminating duplicates and allow for efficient querying and retrieval. Additionally, caching mechanisms and database indexing further enhance speed, making the system optimized for rapid chunking and reconstruction. This makes IPNI sync much faster in Curio compared to Boost. # Storage Market A comprehensive guide to configuring and managing storage deals in Curio ## Overview The Curio Storage Market provides a comprehensive framework for managing deals, data retrieval, and storage through decentralized protocols. This page details the different configurations, commands, and workflows available for managing storage deals, both online and offline. The storage market in Curio is built around several key concepts: * **Deal Protocols**: Protocols and workflows for deal-making, sealing, and data transfers. * **Tasks**: Various tasks managed by the storage market, such as commP, PSD, indexing deals, IPNI advertisement etc. * **Deal Flows**: The workflows for processing online and offline deals, each of which has specific tasks and checks to ensure the deal is successfully completed. ## Configuration The Curio storage market is configurable through the `StorageMarketConfig` structure. This section outlines the main configuration parameters and the implications of setting them. ```go type MarketConfig struct { StorageMarketConfig StorageMarketConfig } type StorageMarketConfig struct { MK12 MK12Config IPNI IPNIConfig Indexing IndexingConfig PieceLocator []PieceLocatorConfig } ``` ### **MK12 Configuration** The `MK12` configuration encompasses all deal-related settings for the **MK1.2.0** and **MK1.2.1** deal protocols (commonly referred to as **Boost deals**). This configuration controls key parameters like batching, sealing time, and the number of deals that can be published at once. ```go type MK12Config struct { PublishMsgPeriod Duration MaxDealsPerPublishMsg uint64 MaxPublishDealFee types.FIL ExpectedPoRepSealDuration Duration ExpectedSnapSealDuration Duration SkipCommP bool DisabledMiners []string MaxConcurrentDealSizeGiB int64 DenyUnknownClients bool DenyOnlineDeals bool DenyOfflineDeals bool CIDGravityToken string DefaultCIDGravityAccept bool } ``` **Key Parameters** 1. **PublishMsgPeriod**:\ Specifies the time to wait before publishing deals as a batch. Increasing this period allows more deals to be included in a single message but delays the publishing of deals. Lowering this period will result in faster publishing but fewer deals being batched together, increasing chain overhead. 2. **MaxDealsPerPublishMsg**:\ Controls the maximum number of deals to include in one batch. If set too high, the publish message may become too large and expensive. Setting it too low might reduce efficiency, as the node sends more messages. 3. **MaxPublishDealFee**:\ This defines the maximum fee you’re willing to pay per deal when sending the `PublishStorageDeals` message. The consequence of setting a low fee is that deal publishing may be delayed or fail if network congestion raises gas costs. 4. **ExpectedPoRepSealDuration**:\ This value controls how long you expect the Proof of Replication (PoRep) sealing process to take. Deals that cannot be sealed within this time will fail. 5. **ExpectedSnapSealDuration**:\ Similar to PoRep, this defines the expected time for snap sealing. The duration should account for hardware speed and network delays. 6. **SkipCommP**:\ If set to `true`, the CommP (Commitment Proof) check is skipped before the `PublishDealMessage` is sent on-chain. Skipping this step is risky because if there’s a mismatch, all deals in the sector may need to be resent. 7. **DisabledMiners**:\ A list of miner addresses excluded from participating in deal-making. Use this option to prevent specific miners from handling deals if needed. 8. **MaxConcurrentDealSizeGiB:**\ MaxConcurrentDealSizeGiB is a sum of all size of all deals which are waiting to be added to a sector when the cumulative size of all deals in process reaches this number, new deals will be rejected. (Default: 0 = unlimited) 9. **DenyUnknownClients:**\ DenyUnknownClients determines the default behaviour for the deal of clients which are not in allow/deny list. If True then all deals coming from unknown clients will be rejected. 10. **DenyOnlineDeals**: Determines whether the storage provider **accepts online deals**. 11. **DenyOfflineDeals**: Determines whether the storage provider **accepts offline deals**. 12. **CIDGravityToken**:\ The authorization token used for **CIDGravity filters**, a service that filters deal proposals based on custom policies. If empty (`""`), **CIDGravity filtering is disabled**. If set, the miner will **query CIDGravity** for each deal proposal before accepting it. 13. **DefaultCIDGravityAccept**:\ Defines what happens if the **CIDGravity service is unavailable**. If`true`: **Accepts deals** even if CIDGravity is unreachable. If`false`: **Rejects deals** when CIDGravity is unavailable (**default**). ### **PieceLocator Configuration** This configuration allows you to set up remote HTTP servers that provide piece data for offline deals. A `PieceLocator` config is a combination of a URL and headers for fetching pieces when requested by the miner. This is crucial for handling offline deals where data is not available immediately and must be retrieved during the commP and sealing phase. ```go type PieceLocatorConfig struct { URL string Headers http.Header } ``` * **URL**: The endpoint where the piece data can be located. * **Headers**: Any custom headers needed for the HTTP request, such as authorization tokens. {% hint style="warning" %} PieceLocator service will allow Curio to lookup details of a piece automatically for an offline deal. The `add-url` command should not be used for deal which are expected to fetch the data from PieceLocator services. {% endhint %} Consequences: If the piece data is not available at the specified URL, the offline deal will fail. Make sure that the remote server is properly configured and available. ## Enabling Storage Market To enable the Curio market on a Curio node, the following configuration changes are required: 1. **Enable the Deal Market**: * Set `EnableDealMarket` to `true` in the `CurioSubsystemsConfig` for at least one node. This enables deal-making capabilities on the node. 2. **Enable CommP**: * On one of the nodes where `EnableDealMarket` is set to `true`, ensure that `EnableCommP` is also set to `true`. This allows the node to compute piece commitments (CommP) before publishing storage deal messages. 3. **Enable HTTP**: * At least one node must have HTTP enabled to support: * Retrievals. * IPNI sync. * Handling storage deals. * To enable HTTP, set the `Enable` flag in the `HTTPConfig` to `true` and specify the `ListenAddress` for the HTTP server. 4. **Set a Domain Name**: * Ensure that a valid `DomainName` is specified in the `HTTPConfig`. This is mandatory for proper HTTP server functionality and essential for enabling TLS. The domain name cannot be an IP address. * In case `DelegateTLS` is `False` , the domain name must point to the public IP address your curio node is listening on. The purpose of setting this field is to allow lets encrypt ACME protocol to automatically issue a certificate to use TLS for encrypting access to the curio api. For let's encrypt policy reasons this will only work if curio listens on port 443. * Domain name should be specified in the base layer 5. **HTTP Configuration Details**: * If TLS is managed by a reverse proxy, enable `DelegateTLS` in the `HTTPConfig` to allow the HTTP server to run without handling TLS directly. * Configure additional parameters such as `ReadTimeout`, `IdleTimeout`, and `CompressionLevels` to ensure the server operates efficiently. 6. **Libp2p Activation**: * The `libp2p` service will automatically start on one of the servers running the HTTP server where `EnableDealMarket` is set to `true`. If more than 1 node satsifies the condition and the node running libp2p goes down then it will switch over to another node after 5 minutes. 7. **Other Considerations**: * Ensure the `MK12Config` settings under `StorageMarketConfig` are properly configured for deal publishing. Key parameters include: * `PublishMsgPeriod` for deal batching frequency. * `MaxDealsPerPublishMsg` for the maximum number of deals per message. * `MaxPublishDealFee` to set the fee limit for publishing deals. * If handling offline deals, configure `PieceLocator` to specify the endpoints for piece retrieval. 8. Verify that HTTP server is working: * Curl to your domain name and verify that server is reachable from outside\\ ```shell curl https:// Hello, World! -Curio ``` {% hint style="warning" %} If you do not get above output then something went wrong with configuration and you should not proceed with migration from Boost or Deal making. {% endhint %} By applying these changes, the Curio market subsystem will be activated on the specified node(s), enabling storage deals, IPNI synchronization, and retrieval functionality. ## MK12 Deals (Boost Deals) The **MK12** protocol governs the entire deal process, from proposing and publishing deals to sealing and validating them. It's designed to work efficiently for both online and offline deals. Key tasks within MK12 include: * **Piece Commitment**: Ensuring that the piece has been added to a sector. * **Publish Storage Deals (PSD)**: Sending the on-chain message to register the deal. * **Finding Deals**: Identifying the deal ID on-chain and adding it to the sector. ### MK12 Tasks Tasks refer to operations that the system performs on deals to ensure their success. Tasks include: * **CommP tasks**: Commitment proof tasks, ensuring the piece information is correct by calculating the commitment locally. * **PSD tasks**: Tasks for sending and validating the PublishStorageDeals message. * **Find Deal tasks**: These tasks poll the blockchain to identify the deal’s status and retrieve its ID after the deal has been published successfully with PSD task. ### Online Deal Flow 1. **Deal Proposal**:\ A client proposes a deal, specifying the amount of data and terms. 2. **Data Transfer**:\ Data is transferred immediately. The system checks that the entire piece has been received. 3. **CommP Task**:\ Once data is received, a commitment proof is generated. 4. **PSD Task**:\ The deal is published on-chain. 5. **Sector Assignment**:\ The deal is assigned to a sector and sealed. ### Offline Deal Flow 1. **Deal Proposal**:\ The client proposes a deal for data that is not yet available on the miner’s node. 2. **PieceLocator**:\ The miner is provided with a URL where the data can be fetched later. 3. **Data Fetch**:\ The miner fetches the piece from the provided URL using the `PieceLocator` configuration or [local database](#add-data-url-for-offline-deals). 4. **CommP Task**:\ Once the piece is retrieved, a commitment proof is generated. 5. **PSD Task**:\ The deal is published on-chain. 6. **Sector Assignment**:\ The deal is added to a sector and sealed. ### Add data URL for offline deals Why this exists (plain language): * Curio needs a way to *fetch the bytes* for the piece when it’s time to compute CommP / index / snap. * For offline deals, there isn’t necessarily a live HTTP endpoint by default—so you must provide one (or otherwise ensure the bytes are accessible via your chosen ingestion method). If you skip this step, you will typically see ParkPiece failures such as: * `no suitable data URL found for piece_id ...` The `add-url` command allows you to specify a URL from which the miner can fetch piece data for offline deals. This is essential for deals where the client does not transfer the data immediately upon deal acceptance. {% hint style="warning" %} The `add-url` command should not be used for deal which are expected to fetch the data from PieceLocator services. {% endhint %} ```bash curio market add-url [command options] ``` **Example Usage**: ```bash curio market add-url --url "https://data.server/pieces?id=pieceCID" --header "Authorization: Bearer token" OR curio market add-url --url "https://data.server/filename" --header "Authorization: Bearer token" OR curio market add-url --url "https://data.server/filename" ``` **Options**: * **--url**: The URL where the piece data can be fetched. * **--header**: Custom headers to include in the HTTP request. Consequences: If the URL is not accessible or the headers are incorrect, the deal will fail to retrieve the data and will not be able to complete. ### Move funds to escrow The `move-to-escrow` command moves funds from the deal collateral wallet to the escrow account with the storage market actor. This is necessary to lock in collateral for a deal. ```bash curio market move-to-escrow [command options] ``` **Example Usage**: ```bash curio market move-to-escrow --actor --max-fee 3 ``` **Options**: * **--actor**: Specifies the actor address that should start sealing sectors for the deal. * **--max-fee**: Maximum fee in FIL you’re willing to pay for this message. Consequences: If insufficient funds are moved to escrow, the deal may not be processed, and the collateral may not be secured. ### Start Sealing Early The `seal` command allows you to start sealing a deal's sector early, before all the deals have been batched. ```bash curio market seal [command options] ``` **Example Usage**: ```bash curio market seal --actor ``` **Options**: * **--actor**: Specifies the actor address. Consequences: Sealing early can speed up the process, but it may result in inefficiencies if all deals are not batched correctly. ## Offline Verified DDO deals Curio only supports offline verified DDO deals as of now. The allocation must be created by the client for the piece and handed over to the SP alongside the data. ### How to create allocation Clients can create allocation using the `sptool toolbox` or other methods. ```shell sptool --actor t01000 toolbox mk12-client allocate -p --piece-cid --piece-size ``` ### Start a DDO deal Storage providers can onboard the DDO deal using the below command. ```shell curio market ddo --actor ``` This command does **not** fetch any data. It only: 1. Validates the allocation against the chain and the actor. 2. Generates a fresh deal UUID. 3. Inserts a row into `market_direct_deals` and a corresponding offline-deal row into `market_mk12_deal_pipeline` (with `offline=true` and `started=false`). 4. Prints the deal UUID to stdout, for example: ``` Direct deals inserted successfully: 0f1b2c3d-... ``` Because the pipeline row is created with `started=false`, the commP task will not run until Curio has resolved a source URL for the piece. There are two supported ways to provide that URL: #### Option A: PieceLocator (preferred for bulk / automated ingestion) If `[Market.StorageMarketConfig.PieceLocator]` is configured, Curio will automatically discover the piece by issuing a `HEAD ?id=` against each configured locator. The first locator that returns `200 OK` with a valid `Content-Length` wins, and Curio writes that URL and `raw_size` into the pipeline row, flips `started=true`, and the commP task picks it up on its next tick. This is the only path you need if your locator already has the piece. You do not need to (and should not) also call `add-url` for the same deal; see the warning in [Add data URL for offline deals](#add-data-url-for-offline-deals). #### Option B: `curio market add-url` (manual / one-off) If you do not run a PieceLocator (or the locator does not have this particular piece), pass the UUID printed by `curio market ddo` to `add-url`: ```shell curio market add-url \ --url "https://data.server/pieces?id=" \ \ ``` This inserts a row into `market_offline_urls` keyed on the UUID. On its next pass, the storage-market poller calls `findURLForOfflineDeals`, joins `market_offline_urls` into `market_mk12_deal_pipeline`, flips `started=true`, and the commP task proceeds. #### Troubleshooting CommP mismatches on DDO deals If the commP task finishes much faster than expected (for example, a few seconds for a 32 GiB piece) and reports `commP mismatch calculated and supplied `, the most common causes are: * **PieceLocator returned a misleading response.** A locator that answers `200 OK` with a `Content-Length` on HEAD but serves wrong, partial, or unrelated bytes on GET will look healthy to Curio's URL resolver and only fail later during commP. This shows up as a subset of deals failing while most succeed, because the locator happens to have valid data for some PieceCIDs and not others. Verify each locator with: ```bash curl -I '?id=' curl -s -o /tmp/piece.bin '?id=' && wc -c /tmp/piece.bin ``` Confirm the body length matches the expected raw size and that `Content-Type` is what you expect. Bad or stale `PieceLocator` entries are a frequent culprit. * **Wrong `raw size` passed to `add-url`.** The commP task pads the fetched stream up to the declared raw size using `padreader.New(reader, rawSize)`. If `rawSize` is smaller than the real piece payload, the padder will truncate; if it is larger, the padder will append zeros. Either way the final CommP will not match the allocation's PieceCID. Always use the unpadded CAR / raw byte size from the same source that produced the PieceCID. * **Both `add-url` and a PieceLocator hit on the same deal.** Only one source is used per deal; if you call `add-url` for a deal that the PieceLocator can already serve, the two URL writers can race depending on which poller pass runs first. Pick one method per deal. You can inspect the resolved state directly: ```sql SELECT uuid, started, after_commp, url, raw_size, piece_cid, piece_size FROM market_mk12_deal_pipeline WHERE uuid = ''; ``` If `url` is `NULL` after several poller cycles, neither PieceLocator nor `add-url` has produced a source yet, and the deal will not advance. # Deal filters How to setup and use storage deal filters ## Deal filters ### Overview Curio provides a flexible filtering system to manage storage deals effectively. User have an option to choose from external filter like [CIDGravity](#cidgravity-filter) and built-in filters. The built-in filters allow you to: * Set pricing rules based on deal duration, size, and whether the data is verified. * Define client-specific rules, including rate limits and acceptable wallets or peers. * Explicitly allow or deny deals from certain wallets, overriding the default acceptance behavior. By configuring these filters, you can optimize your storage provisioning according to your business needs and network policies.

Deal filters

### Pricing Filters Pricing Filters determine the price at which you are willing to accept storage deals based on various criteria such as duration, size, and verification status. ### Adding a Pricing Filter 1. **Navigate to the Pricing Filters section** in the Curio UI. 2. **Click on the "Add Pricing Filter"** button. 3. **Fill in the form fields**: * **Min Duration (Days)**: The minimum deal duration in days. * **Max Duration (Days)**: The maximum deal duration in days. * **Min Size (Bytes)**: The minimum deal size in bytes. * **Max Size (Bytes)**: The maximum deal size in bytes. * **Price (FIL/TiB/Month)**: The price per TiB per month in FIL. * **Verified Deal**: Check this box if the filter applies to verified deals. 4. **Submit the form** by clicking the "Add" button. #### Editing a Pricing Filter 1. **Locate the pricing filter** you wish to edit in the table. 2. **Click the "Edit" button** next to it. 3. **Modify the desired fields** in the form. 4. **Submit the changes** by clicking the "Update" button. #### Removing a Pricing Filter 1. **Find the pricing filter** you want to remove in the table. 2. **Click the "Remove" button** next to it. 3. **Confirm the deletion** when prompted. ### Client Filters Client Filters allow you to define rules for specific clients, including rate limits and acceptable wallets or peers. #### Adding a Client Filter 1. **Go to the Client Filters section** in the Curio UI. 2. **Click on "Add Client Filter"**. 3. **Complete the form**: * **Name**: A unique identifier for the client filter. * **Active**: Check to activate the filter. * **Wallets**: Comma-separated list of client wallet addresses. * **Peers**: Comma-separated list of peer IDs. * **Pricing Filters**: Comma-separated list of pricing filter numbers to apply. * **Max Deals per Hour**: Maximum number of deals allowed per hour. * **Max Deal Size per Hour (GiB)**: Maximum total deal size per hour in GiB. * **Additional Info**: Any extra information or notes. 4. **Click "Add"** to save the filter. #### Editing a Client Filter 1. **Identify the client filter** you want to edit in the list. 2. **Click the "Edit" button** next to it. 3. **Update the fields** as needed. 4. **Click "Update"** to apply the changes. #### Removing a Client Filter 1. **Find the client filter** in the list. 2. **Click the "Remove" button**. 3. **Confirm the deletion** when asked. ### Allow/Deny List The Allow/Deny List lets you explicitly permit or reject deals from specific wallets, overriding the default client acceptance behavior. **Note:** Specify wallets to explicitly allow or deny deals from, overriding the default client acceptance behavior. #### Adding an Allow/Deny Entry 1. **Navigate to the Allow/Deny List section**. 2. **Click "Add Allow/Deny Entry"**. 3. **Fill in the details**: * **Wallet**: The wallet address to allow or deny. * **Allow**: Check to allow, uncheck to deny. 4. **Click "Add"** to save the entry. #### Editing an Allow/Deny Entry 1. **Locate the entry** in the list. 2. **Click the "Edit" button**. 3. **Modify the fields** as necessary. 4. **Click "Update"** to save changes. #### Removing an Allow/Deny Entry 1. **Find the entry** you wish to remove. 2. **Click the "Remove" button**. 3. **Confirm the action** when prompted. ### Default Allow Behaviour The default allow behaviour determines whether deals from unknown clients are accepted or denied by default. This can be modified in Curio configuration. ``` // DenyUnknownClients determines the default behaviour for the deal of clients which are not in allow/deny list // If True then all deals coming from unknown clients will be rejected. DenyUnknownClients bool ``` * **False**: Deals from unknown clients are accepted unless explicitly denied in the Allow/Deny List. * **True**: Deals from unknown clients are rejected unless explicitly allowed in the Allow/Deny List. **To view the current default behaviour**: * The Curio UI displays the default behaviour in the relevant section, indicating whether deals from unknown clients are allowed or denied. ### Usage Examples **Example 1: Accepting Large Verified Deals Only** * **Pricing Filter**: * Min Size: 1 TiB * Max Size: 10 TiB * Verified Deal: Yes * Price: 0 FIL/TiB/Month (or your desired price) * **Client Filter**: * Active: Yes * Max Deals per Hour: 5 * Max Deal Size per Hour: 50 GiB **Example 2: Denying Deals from a Specific Wallet** * **Allow/Deny Entry**: * Wallet: `t01234...` * Allow: Unchecked (deny) **Example 3: Rate Limiting a Client** * **Client Filter**: * Name: `HighVolumeClient` * Active: Yes * Wallets: `t05678...` * Max Deals per Hour: 10 * Max Deal Size per Hour: 100 GiB #### Best Practices * **Use Specific Filters**: Create filters that closely match your desired deal parameters to avoid unwanted deals. * **Regularly Review Filters**: Update your filters as your storage provisioning strategy evolves. * **Test New Filters**: Before deploying filters widely, test them with a small subset to ensure they behave as expected. * **Monitor Deal Flow**: Keep an eye on incoming deals to adjust filters proactively. #### Troubleshooting * **No Deals Being Accepted**: * Check if the default allow behaviour is set to deny. * Ensure your filters are not overly restrictive. * **Error Messages When Saving Filters**: * Review the error displayed in the UI. * Ensure all required fields are filled and values are within acceptable ranges. * Check for duplicates in names or wallet addresses. * **Unexpected Deals Being Accepted or Rejected**: * Verify the order and specificity of your filters. * Check the Allow/Deny List for conflicting entries. * Review client filters to ensure they are active and correctly configured. ## CIDGravity Filter ### What is CIDGravity? [CIDGravity](https://www.cidgravity.com/) is a powerful pricing and client management tool designed for Filecoin storage providers. It enables storage providers to efficiently filter storage and retrieval deals through a user-friendly interface. With CIDGravity, providers can set rules and policies for accepting or rejecting deals based on their business preferences, ensuring better control over their storage operations. For more details, refer to the [CIDGravity documentation](https://docs.cidgravity.com/). ### How to Enable CIDGravity in Curio CIDGravity integration in Curio is controlled through Curio configuration. To enable CIDGravity, you need to set the below parameters in the configuration. We highly recommend setting these values in "base" layer to control the market behaviour and correct UI rendering. ```toml # CIDGravityTokens is the list of authorization tokens to use for CIDGravity filters. # These should be in the format "minerID1:Token1", "minerID2:Token2". # If a token for a minerID within the cluster is not provided, CIDGravity filters will not be applied to deals associated with that miner ID. # # type: []string #CIDGravityTokens = [] # DefaultCIDGravityAccept when set to true will accept deals when CIDGravity service is not available. # Default behavior is to reject the deals (Default: false) # # type: bool #DefaultCIDGravityAccept = false ``` #### Configuration Options: **1. `CIDGravityTokens`** * **Description**: A list of authorization tokens used for CIDGravity filters. Each entry should be formatted as `"minerID:Token"`. * **Default Behavior**: If no token is provided for a `minerID`, CIDGravity filters will **not** be applied to deals associated with that miner ID. * **Type**: `[]string` **Example Configuration:** ```toml CIDGravityTokens = ["t01234:your-auth-token1", "t05678:your-auth-token2"] ``` {% hint style="info" %} To generate a CIDGravity token [claim your miner](https://docs.cidgravity.com/storage-providers/get-started/claim-a-miner/) in CIDGravity. If you already have an existing miner then you can use the same token. {% endhint %} **2. `DefaultCIDGravityAccept`** * **Description**: Defines the default behavior when the CIDGravity service is unavailable. * **Default Behavior**: If set to `false`, deals will be rejected when CIDGravity is not reachable. If set to `true`, deals will be accepted even if CIDGravity is not available. * **Type**: `bool` **Example Configuration:** ```toml DefaultCIDGravityAccept = false ``` #### Steps to Enable CIDGravity in Curio: 1. Obtain a **CIDGravityToken** from the [CIDGravity platform](https://app.cidgravity.com/). 2. Add the token(s) to the Curio configuration file under `CIDGravityTokens` in the format `"minerID:Token"`.

CID Gravity Disabled

3. Set `DefaultCIDGravityAccept` based on your preference: * `true` to accept deals when CIDGravity is unreachable. * `false` to reject deals when CIDGravity is unreachable.

Reject Deals when CID Gravity is unreachable

4. Restart Curio for the changes to take effect. 5. Verify that CIDGravity is enabled via UI Market Settings page.

CID Gravity Enabled

Once enabled, Curio will automatically interact with CIDGravity to apply deal filtering and pricing rules according to the policies set in your CIDGravity account. # Curio HTTP Server This page provides an overview of the Curio HTTP Server's key features, including HTTPS support, security, middleware, routing capabilities, and instructions for attaching custom service routes. The Curio HTTP Server is a secure, flexible, and high-performance HTTP server designed for use with Curio cluster. It comes with built-in support for HTTPS using Let's Encrypt certificates, advanced middleware features like logging and compression, and the ability to integrate various Curio services, including IPNI, retrieval providers, and LibP2P. ## Key Features ### 1. **HTTPS with Let's Encrypt** * Automatic certificate management using Let's Encrypt, ensuring that all traffic is encrypted. * Supports automatic renewal of certificates and domain validation through the `autocert.Manager`. ### 2. **Security-First Design** * **Strict Security Headers**: Adds essential security headers like Strict-Transport-Security, Content-Security-Policy, X-Frame-Options, and XSS protection to mitigate common vulnerabilities. * Protects from clickjacking, content-type sniffing attacks, and more by enforcing best practices via HTTP headers. ### 3. **Flexible Routing with Chi** * Uses the `chi` router for lightweight, flexible routing. Easily extend routes or handle custom paths by attaching new service modules. * Out-of-the-box support for path handling and easy extension for future service needs. ### 4. **Built-in Middleware** * **Compression**: Utilizes the `httpcompression` package to support GZIP, Brotli, and Deflate, optimizing bandwidth usage based on configurable compression levels. * **Logging**: Logs every incoming request with details such as request method, path, and duration, aiding in easier debugging and monitoring. * **CORS Support**: Conditional CORS support based on configuration for handling cross-origin requests securely. ### 5. **WebSocket and LibP2P Support** * Specialized handling for WebSocket upgrade requests, with support for forwarding them to the `/libp2p` endpoint. * Facilitates smooth communication between nodes and services in distributed Curio cluster running a single LibP2P. ### 6. **Health Check Endpoint** * A built-in `/health` endpoint provides an easy way to check the status of the HTTP server, ensuring it is running smoothly. ## Performance and Scalability * The server is optimized for handling large numbers of concurrent requests efficiently with appropriate timeouts (read, write, idle) to prevent overload. * Integrated compression ensures minimal bandwidth usage, even for large data exchanges. ## Database Integration for TLS Certificate Cache The server stores Let's Encrypt certificates and cache information in `Harmonydb`. This ensures persistence and fast access to TLS certificates in a Curio cluster with multiple HTTP servers. ### Database Operations * **Get**: Retrieves TLS certificates from the `autocert_cache` table. * **Put**: Inserts or updates certificates in the database. * **Delete**: Removes expired or invalid certificates from the cache. ## Attaching Routes to Extend Server Functionality Curio HTTP Server is designed to allow easy integration of various services by attaching custom routes. The `attachRouters` function enables the attachment of specific service routes to the Chi router. Examples include: 1. **Retrieval Provider**: Attaches routes to handle data retrieval using the `retrieval` module. * Creates a `RetrievalProvider` and registers the necessary HTTP endpoints. 2. **IPNI (Interplanetary Network Indexer)**: Integrates IPNI-specific routes. * The IPNI provider is instantiated, and routes are attached for IPNI services to handle data advertisement publishing. 3. **LibP2P Redirector**: Handles WebSocket connections for LibP2P communication. * Redirects WebSocket upgrade requests from `/` to `/libp2p` for seamless peer-to-peer communication. Here’s how the routes are attached: ```go attachRouters(ctx context.Context, r *chi.Mux, d *deps.Deps) (*chi.Mux, error) { // Attach retrievals rp := retrieval.NewRetrievalProvider(ctx, d.DB, d.IndexStore, d.CachedPieceReader) retrieval.Router(r, rp) // Attach IPNI ipp, err := ipni_provider.NewProvider(d) if err != nil { return nil, xerrors.Errorf("failed to create new IPNI provider: %w", err) } ipni_provider.Routes(r, ipp) go ipp.StartPublishing(ctx) // Attach LibP2P redirector rd := libp2p.NewRedirector(d.DB) libp2p.Router(r, rd) return r, nil } ``` This flexibility allows the server to be easily extended with new services without modifying the core server logic. ## Configuration The Curio HTTP Server can be customized using the `HTTPConfig` structure, which allows you to configure timeouts, compression levels, and security settings to suit your application. Below are the key configuration options along with their default values and explanations of their impact. ### **HTTPConfig** The list below is aligned with the actual `HTTPConfig` struct in `deps/config/types.go`. * **Enable**: Enables/disables the HTTP server on this node. * **DomainName**: DNS name used for requests and (when Curio terminates TLS) certificate issuance. Must be a real domain (not an IP). Default: `""`. * **ListenAddress**: IP:port to bind. Default: `"0.0.0.0:12310"`. * **DelegateTLS**: When `true`, Curio serves **plain HTTP** on `ListenAddress` and expects a reverse proxy to terminate TLS. * **ReadTimeout**: Max time to read request body. Default: `10s`. * **IdleTimeout**: Max keep-alive idle time. Default: `1h`. * **ReadHeaderTimeout**: Max time to read headers. Default: `5s`. * **CORSOrigins**: Allowed origins; empty disables CORS. Default: `[]`. * **CSP**: Content Security Policy mode for `/piece/` content. Values: `off`, `self`, `inline`. Default: `inline`. * **CompressionLevels**: Response compression tuning. Defaults: gzip=6, brotli=4, deflate=6. ### **Impact of Compression Levels** The compression levels directly affect server performance and bandwidth usage: * Higher compression levels (e.g., `GzipLevel 9`, `BrotliLevel 11`) reduce the size of responses, which can save bandwidth but require more CPU processing time, especially for large responses. * Lower levels (e.g., `GzipLevel 1`) are faster but provide less compression, meaning higher bandwidth usage but reduced server load. For most applications, the default values of `6` for GZIP and Deflate, and `4` for Brotli provide a good trade-off between compression efficiency and CPU load, especially for responses that contain text or JSON. *** ## HTTPS setup (Let’s Encrypt / autocert) — operational guide Curio can terminate TLS itself (autocert / Let’s Encrypt) or you can terminate TLS in a reverse proxy. ### Prerequisites (both modes) * You must use a real **domain name** (not an IP) in `HTTPConfig.DomainName`. * DNS must point your domain to the host that will receive traffic. * Inbound access must be allowed for: * **80/tcp** (ACME HTTP-01) and * **443/tcp** (HTTPS) If ports 80/443 are blocked (cloud firewall, NAT, or corporate network), Let’s Encrypt cannot validate your domain. ### Mode A: Curio terminates TLS (DelegateTLS = false) Use this when you want Curio to handle certificates directly. Checklist: * `DomainName` is set and resolvable publicly. * Curio must be reachable from the internet on 80/443. Operational notes: * If your OS restricts binding to privileged ports, you may need one of: * run the Curio HTTP service with permissions to bind 80/443, or * terminate TLS in a reverse proxy (Mode B), or * forward 80/443 to Curio’s listen port via firewall/NAT. ### Mode B: Reverse proxy terminates TLS (DelegateTLS = true) Use this when you already run Nginx/Caddy/Traefik or cannot (or do not want to) expose Curio directly. In this mode: * Curio serves **plain HTTP** internally. * Your reverse proxy handles TLS + Let’s Encrypt. * When the proxy connects over loopback, Curio treats the rightmost `X-Forwarded-For` entry as the client IP for rate limiting and abuse controls. * The proxy must overwrite or append `X-Forwarded-For` on every request. Forwarding headers from non-loopback peers are ignored. Minimal Nginx example: ```nginx server { listen 443 ssl; server_name example.com; # TLS config here (certbot/caddy/managed) location / { proxy_pass http://127.0.0.1:12310; proxy_set_header Host $host; proxy_set_header X-Forwarded-Proto https; proxy_set_header X-Forwarded-For $remote_addr; } } ``` ### Troubleshooting HTTPS If `curl https://` fails: * Confirm DNS A/AAAA is correct. * Confirm ports 80/443 are reachable externally. * Confirm `DomainName` matches the hostname you’re curling. * Confirm you didn’t enable `DelegateTLS=true` without actually configuring a reverse proxy. * Check Curio logs around certificate issuance / autocert. # Curio Market troubleshooting Troubleshooting Curio Market ingestion, indexing, retrieval, and IPNI. This page targets the common “deal is stuck” / “indexing is failing” cases seen in support. Before proceeding, collect the basics: * [Collect debug info](/troubleshooting/collect-debug-info) * deal UUID + piece CID * failing task IDs from the UI *** ## 1) Deal stuck at indexing / “CheckIndex” errors What CheckIndex is: * Curio runs task-based indexing and health checks; a recurring task checks for missing indexes / announcements and schedules follow-up work. * Implementation reference: `tasks/indexing/task_check_indexes.go` (task `CheckIndex`). If you see errors in CheckIndex: * Determine whether indexing is still progressing (tasks completing) vs stalled. * Many “indexing failures” are actually DB health/latency problems. Verify Yugabyte health first. ### Common error: `duplicate key value violates unique constraint ... (SQLSTATE 23505)` What it often means: * Concurrent retries inserted the same identity row. What to do: 1. Collect: deal UUID, piece CID, task ID(s), and the full error span. 2. Confirm whether the failing rows keep reappearing or are transient. 3. If stuck, restart only the market/indexing layer process (not the whole cluster). *** ## 2) IPNI failures Symptoms: * Indexing seems OK locally, but IPNI publishing is failing. Checklist: * Outbound HTTPS works from Curio host. * The “public URL” for your provider is correct (reachable from the internet). * DNS + firewall allow inbound connectivity to your HTTP server. Temporary mitigation: * If you need onboarding to proceed while IPNI is unstable, you may temporarily disable IPNI publishing (only if supported by your release/config). Document and keep track of the change so it isn’t forgotten. *** ## 3) ParkPiece: `no data URL found for piece_id` What it means: * Curio cannot find the configured URL+headers for fetching the piece data. Most common causes: * The URL was never registered, or it was registered on a different node than the one running ParkPiece. * Ingest layer is not enabled on the node you think. What to collect: * deal UUID, piece CID * whether offline/online deal * the exact command used to add the URL (if online) *** ## 4) Retrieval looks broken, but indexing is fine Checklist: * Ensure unsealed data is available on at least one node that can serve retrieval. * Confirm the Curio HTTP server is enabled and reachable. * If behind a reverse proxy, confirm TLS delegation mode matches your config. See also: * [Curio HTTP Server](/curio-market/curio-http-server) # Curio libp2p Server This page outlines the main features of the Libp2p server, including its configuration and components. ## Overview The Curio Libp2p server facilitates network communication between nodes and clients using the Libp2p protocol. It primarily handles deal proposals, deal status updates, and miner information in the Filecoin network. This server is tightly integrated with Curio's HTTP server and ensures the network operates smoothly for Filecoin deal processing. ### Key Components: * **Host Setup**: The libp2p host is initialized with necessary configuration details, including listening addresses, identity keys, and a peerstore for managing peer connections. * **Deal Handling**: The libp2p server manages deal proposals and responses using the MK12 market protocol. It processes incoming streams for deal proposals, status checks, and query requests. * **WebSocket Proxy**: WebSocket connections from clients are proxied through the Curio HTTP server at the `/libp2p` path, directing traffic to the libp2p node's listening address for secure P2P communication. * **Miner Information Updates**: The libp2p provider also ensures that miner information, such as peer IDs and multi-addresses, is kept up-to-date on-chain, allowing for smooth communication during deal negotiations. ## libp2p Host A **Libp2p host** is initialized with randomized ports for listening to incoming connections. Unlike other networks where the listen address might be broadcasted, in Curio's implementation, the Libp2p node listens on a random port and does not advertise its address to other peers. The setup involves creating a peer identity, configuring listen addresses, and ensuring that the node is ready to handle connections and requests. Once the node is operational, its local listen address is stored in the database, which the WebSocket proxy uses for connecting clients. ## Network Deal Management Curio libp2p server plays a critical role in handling incoming deal proposals over the network. Deals are initiated using the `DealProtocolv120ID` and `DealProtocolv121ID` for handling storage market deals, and the `DealStatusV12ProtocolID` for tracking deal statuses. The deal provider continuously listens for deal requests, processes them, and sends responses back to the clients. ### Deal Proposal Handling When a deal proposal is received, the libp2p handler send the request to MK12(INSERT LINK HERE) provider to validates the request, checks miner permissions, and processes the deal. If the deal is valid, MK12 provider moves forward with execution, handling the storage and sealing process on the storage provider's side. ### Deal Status Tracking The libp2p server supports querying the status of a deal using the `DealStatusV12ProtocolID`. This allows clients to monitor the current state of their proposals, such as whether the deal is sealed, indexed, or still in progress. ## WebSocket Proxy The Curio libp2p server leverages the HTTP server's `/libp2p` path to proxy WebSocket connections between the client and the libp2p node. This mechanism ensures secure and direct P2P communication through WebSocket channels. By connecting via the WebSocket proxy, peers can securely exchange deal information, status updates, and more. This proxy system enables seamless data flow between Curio nodes and external clients or peers, facilitating decentralized communication across the network without exposing local node details directly to clients. ### Failover and Node Coordination To ensure proper coordination between multiple nodes, the libp2p server periodically updates the database with the node's status, such as the `running_on` field, and performs keep-alive checks. If the node fails to update its status in time, another node may take over, ensuring that there is no interruption in the decentralized network. # libp2p Protocols Curio exposes libp2p protocols so that clients can initiate storage deals with the SP The client makes a deal proposal over `v1.2.0` or `v1.2.1` of the Propose Storage Deal Protocol: - `/fil/storage/mk/1.2.0` or\ \- `/fil/storage/mk/1.2.1` It is a request / response protocol, where the request and response are CBOR-marshalled. There are two new fields in the request of `v1.2.1` of the protocol, described in the table below. ### Request | Field | Type | Description | | --------------------------- | ------------------ | -------------------------------------------------------------------------------------------------- | | DealUUID | uuid | A uuid for the deal specified by the client | | IsOffline | boolean | Indicates whether the deal is online or offline | | ClientDealProposal | ClientDealProposal | Same as `.DealProposal` | | DealDataRoot | cid | The root cid of the CAR file. Same as `.Piece.Root` | | Transfer.Type | string | eg "http" | | Transfer.ClientID | string | Any id the client wants (useful for matching logs between client and server) | | Transfer.Params | byte array | Interpreted according to `Type`. eg for "http" `Transfer.Params` contains the http headers as JSON | | Transfer.Size | integer | The size of the data that is sent across the network | | SkipIPNIAnnounce (v1.2.1) | boolean | Whether the provider should announce the deal to IPNI or not (default: false) | | RemoveUnsealedCopy (v1.2.1) | boolean | Whether the provider should keep an unsealed copy of the deal (default: false) | ### Response | Field | Type | Description | | -------- | ------- | -------------------------------------------------- | | Accepted | boolean | Indicates whether the deal proposal was accepted | | Message | string | A message about why the deal proposal was rejected | ## Storage Deal Status Protocol The client requests the status of a deal over `v1.2.0` of the Storage Deal Status Protocol: `/fil/storage/status/1.2.0` It is a request / response protocol, where the request and response are CBOR-marshalled. ### Request | Field | Type | Description | | --------- | ----------------------------------------------------------------------------------------------------------------------------------------- | -------------------------------------------------- | | DealUUID | uuid | The uuid of the deal | | Signature | [Signature](https://github.com/filecoin-project/go-state-types/blob/057cdfb837f7a0309c1607c7c4640f315e51d7af/crypto/signature.go#L36-L39) | A signature over the uuid with the client's wallet | ### Response | Field | Type | Description | | ------------------- | ------------ | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | DealUUID | uuid | The uuid of the deal | | Error | string | Non-empty if there's an error getting the deal status | | IsOffline | boolean | Indicates whether the deal is online or offline | | TransferSize | integer | The total size of the transfer in bytes | | NBytesReceived | integer | The number of bytes that have been downloaded | | DealStatus.Error | string | Non-empty if the deal has failed | | DealStatus.Status | string | The [checkpoint](https://github.com/filecoin-project/boost/blob/4fb17ba117784479e09db4012a3abf9862b8afd9/storagemarket/types/dealcheckpoints/checkpoints.go#L7-L15) that the deal has reached | | DealStatus.Proposal | DealProposal | | | SignedProposalCid | cid | cid of the client deal proposal + signature | | PublishCid | cid | The cid of the publish message, if the deal has been published | | ChainDealID | integer | The ID of the deal on chain, if it's been published | # Indexing This page covers the local indexing process in Curio, detailing how data is stored, indexed, and managed for efficient retrieval and processing. Curio uses a local index store to manage the mapping of multihashes to content pieces. This system allows efficient retrieval of content across multiple nodes in the Curio cluster. The index store does not interact with IPNI directly, but Curio may use index data to create and serve advertisements for the IPNI network when enabled. ## Key Components * **Local Index Store**: A Cassandra-based local store that maps multihashes → piece CIDs + offsets/sizes. * **Concurrency & Batching**: Configurable concurrency and batch sizing for index inserts. * **Integration with Retrievals**: Retrieval performance depends on correct, timely indexing. * **Task-Based Indexing**: Indexing work is performed asynchronously via tasks. ## Index Store Curio’s indexing mechanism relies on a Cassandra-based storage system. The local copy of the indexes allows efficient lookups for retrieval operations. The IndexStore component handles all interactions with the underlying Cassandra database, including creating, adding, and removing index entries, and querying pieces by multihashes. ### Concurrency and Batching The IndexStore allows configurable concurrency and batching for operations. It uses multiple workers to process index entries, enabling parallel indexing to scale with the size of the dataset. ### Configuration The `IndexingConfig` struct defines parameters that control how indexing operations are performed: * **InsertBatchSize**: number of records per batch insert. * **InsertConcurrency**: number of concurrent insert workers. ## Task-Based Indexing Indexing tasks are created and executed asynchronously, allowing Curio to manage indexing without blocking other subsystems. Tasks are assigned to nodes within the Curio cluster, and each node executes indexing tasks based on the available system resources and the availability of the local copy of unsealed sector containing the deal. *** ## How “CheckIndex” works (and why it fails) Operators frequently see errors on a task called **CheckIndex** and aren’t sure what it does. What it does (high level): * Periodically scans for indexing and announcement work that should exist (or be retried). * Schedules follow-up tasks when it finds missing/failed items. Code reference: * `tasks/indexing/task_check_indexes.go` implements the `CheckIndex` task. ### Common failure mode: DB health/latency If Yugabyte is slow/unhealthy, downstream tasks (including market/indexing workflows) can appear broken. Before chasing indexing logic: * run the DB connectivity check from the Curio host * check tserver logs for FATALs See: [Yugabyte troubleshooting](/administration/yugabyte-troubleshooting) ### Common failure mode: duplicate key (SQLSTATE 23505) You may see: * `duplicate key value violates unique constraint ... (SQLSTATE 23505)` This often indicates concurrency/retries inserting the same identity row. What to do: * Determine whether the pipeline is still progressing. * If tasks are stuck, collect deal UUID + piece CID + failing task IDs and consult: * [Curio Market troubleshooting](/curio-market/troubleshooting) # IPNI (Interplanetary Network Indexer) Provider This page details the IPNI provider's sync process over HTTP, covering proactive announcements, indexer polling, and advertisement chain retrieval. The IPNI provider in Curio is designed to manage HTTP-based content announcements and indexing for decentralized discovery through external indexing nodes. It facilitates content chunking, advertisement generation, and HTTP-based announcements to indexers. The following sections explain the provider design, configuration, tasks, advertisements, and the synchronization process based on the [IPNI HTTP Provider specification](https://github.com/ipni/specs/blob/main/IPNI_HTTP_PROVIDER.md). ## IPNI Provider Identification **Curio's Approach to PeerID Management** In Curio, a **single PeerID is used across all MinerIDs** and is maintained on-chain. This ensures a consistent identity for all interactions and prevents unauthorized modifications. If the PeerID is changed manually on-chain, Curio will revert it to the expected value upon restart, ensuring stability and automation. This peerID is used only for mk1.2 deals which use Graphsync protocol. For **IPNI integration**, Curio assigns **unique PeerIDs per MinerID**, but these are **not stored on-chain**. These unique PeerIDs are essential for retrieval verification. Spark uses them to correctly identify miners and validate retrieval operations. This dual approach—using a **single on-chain PeerID for general operations** and **unique off-chain PeerIDs for IPNI**—ensures compatibility across different systems while maintaining efficiency. **Implementation Details** To support this architecture, the **MinerPeerIDMapping** smart contract has been deployed at **0x40721e8Ef366375492ee63c54e881068b15C8633**. This contract provides a decentralized and secure way to manage MinerID-to-PeerID mappings, ensuring that only authorized entities can modify these associations. * **Public State Mapping**: The contract maintains a **gas-free lookup table** mapping Filecoin MinerIDs (`uint64`) to PeerIDs. * **Secure Update Mechanism**: Mappings can be modified through three operations using one of the control addresses: * **Add**: Associates a new PeerID with a MinerID. * **Update**: Replaces an existing PeerID with a new one. * **Delete**: Removes the binding between a MinerID and a PeerID. * **Signed Message Verification**: Updates require a JSON-encoded signed message, ensuring authenticity. The message is signed using the on-chain PeerID. * **Automated Integration with Curio**: Curio ensures that the on-chain PeerID remains updated automatically, preventing unauthorized changes. This allows Spark to correctly associate Ads with MinerIDs, even when different PeerIDs are used in other systems. ## Provider Design The Curio IPNI provider operates over HTTP, managing content updates through advertisement creation and announcement. It does not use libp2p; instead, it interacts with indexing nodes by sending HTTP requests to announce content and expose the advertisement chain for indexing. ### Key Components * **Chunking**: Large datasets are divided into smaller chunks using the `Chunker` to create manageable entries for advertisement. Each chunk contains a subset of the data for efficient indexing by external nodes. * **Advertisement Creation**: After chunking the content, the provider generates IPLD-based advertisements. These advertisements contain essential metadata, including the content's addresses, the provider's identity, and a chain linking the advertisement to previous content updates. * **HTTP Announcements**: Once an advertisement is created, HTTP requests are sent to specific indexing nodes via the configured `DirectAnnounceURLs`, informing them about new advertisements. This proactive method ensures that indexers are notified of content changes quickly. * **Sync Mechanism**: Indexers receive announcements about new advertisements via HTTP. In cases where no new announcements are received for a while, the indexers poll the provider’s `/ipni/v1/ad/head` endpoint to retrieve the latest advertisement head CID. From there, they traverse the advertisement chain, fetching the missing updates. This ensures that indexers stay up-to-date with the provider's latest content, even if no explicit announcements are sent for a period of time. ## Configuration The behaviour of the Curio IPNI provider is controlled through the `IPNIConfig` structure, which defines how content announcements and synchronization with indexing nodes are handled. ### Default Configuration ```go [Market.StorageMarketConfig.IPNI] # Disable set whether to disable indexing announcement to the network and expose endpoints that # allow indexer nodes to process announcements. Default: False # # type: bool #Disable = false # The network indexer web UI URL for viewing published announcements # TODO: should we use this for checking published heads before publishing? Later commit # # type: []string #ServiceURL = ["https://cid.contact"] # The list of URLs of indexing nodes to announce to. This is a list of hosts we talk to tell them about new # heads. # # type: []string #DirectAnnounceURLs = ["https://cid.contact/ingest/announce"] ``` ### **IPNIConfig Fields** * **Disable**: Disables indexing announcements if set to `true`. Default: `false`. IPNI should be disabled on base layer. * **ServiceURL**: URLs for accessing the indexer web UI to view published advertisements. * **DirectAnnounceURLs**: URLs of indexing nodes where the provider sends HTTP announcements of new advertisements. ## IPNI Task The **IPNI task** is responsible for reading content, chunking it, creating advertisements, and sending HTTP announcements to indexing nodes. #### Task Lifecycle 1. **Content Reading**: The task reads content (sectors) from storage. 2. **Chunking**: The content is divided into smaller chunks using a`chunker`, organizing it for efficient indexing. 3. **Advertisement Creation**: A signed IPLD-based advertisement is generated for the chunked content, linking it to previous advertisements in the chain. 4. **Database Update**: The task marks itself as complete in the database once the content is advertised and announced to the indexer. The task ensures that content is indexed and made available to indexing nodes by creating manageable advertisement entries and sending updates via HTTP. ## Chunker The **chunker** generates entries for advertisements by leveraging the [**index store**](/curio-market/indexing#index-store) 1. **First CID for Speed Optimization**: * The chunking process begins with multihashes (`multihash.Multihash`) stored in the database. * For each chunk, the **first CID** (Content Identifier) is crucial as it is used to quickly locate and reconstruct the chunk when needed. This CID is often stored as the `first_cid` in the database along with other metadata like chunk number, piece CID, and offsets. 2. **Sorted Entries in Database**: * During chunk creation, the entries (multihashes) are first sorted in ascending order based on their binary value for efficient processing and retrieval. * Sorting ensures that duplicates are identified and removed, minimizing redundancy. This process involves: * Sorting the multihashes. * Removing duplicates. * After sorting, the entries are split into chunks of predefined size (`EntriesChunkSize`, e.g., 16,384 multihashes per chunk). 3. **Chunk Metadata Storage**: * Each chunk is associated with metadata stored in the database, including: * `cid`: Unique identifier for the chunk. * `piece_cid`: Identifier linking the chunk to its corresponding data piece. * `chunk_num`: Chunk sequence number. * `first_cid`: First multihash in the chunk for quick lookup. * `start_offset`: Byte offset (if applicable). * `num_blocks`: Number of entries in the chunk. 4. **Chunk Retrieval and Reconstruction**: * To reconstruct a chunk: * The database is queried using the `cid` to fetch metadata, including the `first_cid`. * The system then either reads data directly from the database (sorted entries). * For each chunk, a linked structure (`schema.EntryChunk`) is created, linking to the next chunk via an IPLD link. 5. **Linked Chunks**: * Chunks are linked to each other using the `Next` field in the IPLD node structure, forming a chain that can be navigated from the head. 6. **Efficient Querying**: * The system leverages the database for rapid querying of metadata and uses caching (e.g., LRU caches) to store recent chunk data for reuse and speculative pre-fetching. This design ensures fast chunking and reconstruction by leveraging the sorted entries and storing the first CID, allowing efficient access and reusability. ## Advertisements Advertisements describe the content available for indexing and discovery. Each advertisement is represented as an IPLD node containing the following fields: * **PreviousID**: The CID of the previous advertisement, forming a chain of advertisements. It’s empty for the genesis (first) advertisement. * **Provider**: The unique identifier of the content provider (peerID). This peerID is unique for IPNI per miner ID. * **Addresses**: Multiaddresses where clients can access the provider’s content i.e. retrievals. * **Entries**: A link to the multihashes of the advertised content. * **ContextID**: An identifier used to track updates or removals associated with the advertisement. * **Metadata**: Additional protocol-specific data used for retrieval. * **IsRm**: A flag indicating whether the advertisement removes previously published content. ### Entries Structure A linked list of multihashes in an advertisement, where `Next` links to the next chunk. Each chunk is kept under 4MB, allowing up to 16384 multihashes per chunk. ## Announcement The provider sends HTTP announcements to notify indexing nodes of new advertisements. This is a proactive method for updating indexers about content changes without relying solely on polling. ### Announcement Workflow 1. **Ad Creation**: The provider creates IPLD-based advertisements containing content metadata and multiaddresses. 2. **HTTP Announcements**: These advertisements are announced to the indexer nodes specified in the `DirectAnnounceURLs` via HTTP requests. 3. **Sync:** Indexer nodes query the IPNI provider for new advertisements based on the announcements and sync the available context indexes. 4. **Client Access**: Clients query the indexer nodes to discover content based on the newly announced advertisements. Announcements ensure that indexers are aware of updates quickly, reducing the time it takes to ingest new content. ## Serving IPNI Ads and Entries To serve IPNI advertisements and entries in Curio, the HTTP server exposes specific route. This route allow the retrieval of advertisement data and entry chunks through the following paths: 1. **Head Request Path:** * **Endpoint:** `/ipni-provider/{providerId}/ipni/v1/ad/head` * **Description:** This endpoint allows indexers to fetch the latest advertisement from a provider by requesting the head of the advertisement chain. 2. **Advertisement and Entry Request Path:** * **Endpoint:** `/ipni-provider/{providerId}/ipni/v1/ad/{cid}` * **Description:** This endpoint serves both advertisements and entry chunks based on the requested CID. The type of content returned (advertisement or entry) is determined by the CID and schema provided in the request headers. If the schema is not specified, the server defaults to checking for an advertisement and, if not found, falls back to serving the entry chunk associated with the given CID. These routes are registered in the Curio HTTP server as part of the IPNI integration, enabling smooth and efficient data sharing between providers and indexers. The server also periodically publishes the latest advertisement head for each provider. ## IPNI Sync The synchronization process ensures that indexing nodes stay updated with the latest content announcements from the provider. #### Sync Process: 1. **Proactive Announcements**: Indexing nodes are notified via HTTP announcements whenever the provider has new advertisements. This is the primary mechanism for keeping indexers in sync with the provider. 2. **Head Resource for Polling**: If no announcements are received for a period of time, indexers may poll the provider’s `/ipni/v1/ad/head` endpoint to check for new content. This head CID represents the latest advertisement in the chain. 3. **Chain Retrieval**: When indexers receive a new head CID, they traverse the advertisement chain backward, starting from the head, fetching each advertisement via HTTP (`/ipni/v1/ad/{CID}`). The indexer processes the advertisement chain in order from the oldest unseen advertisement to the newest. 4. **Continuous Updates**: Through a combination of announcements and periodic polling, indexers ensure they have the most up-to-date content from the provider, even if an explicit announcement was not sent. This process ensures that indexers remain synchronized with the provider’s content, maintaining up-to-date knowledge of available advertisements and content retrieval addresses. By using this mechanism, indexers can stay synchronized with the provider’s content in real-time or through periodic polling, ensuring continuous updates. ### Serving Advertisements: * **Advertisement Fetching**: Indexers fetch advertisements and entries directly from the provider via HTTP, making the advertised IPLD objects available for ingestion. * **Head Requests**: The provider exposes the latest advertisement through the `/ipni/v1/ad/head` endpoint, allowing indexers to know the most recent state of the advertisement chain. * **Serving Entries:** When indexers need to fetch advertised entries, they request specific entry chunks through their corresponding CID by making a GET request to the `/ipni/v1/ad/{CID}` endpoint. The provider serves these entries by reading either from the CAR file or from the stored index data, depending on how the entries were chunked during advertisement creation. This ensures efficient retrieval of multihashes for large datasets. This synchronization process ensures that indexers can efficiently track the latest updates from the provider, enabling quicker content discovery across the network. # Market UI Curio market related UI pages and their content description The "storage market" page is the main page for storage deal market. The page provides quick summary of market balances, piece status, storage ask and deal pipeline status.

Storage market UI page

You can set the storage ask for each miner ID in the Curio cluster individually from the UI.

Storage ask UI

By clocking on the UUID column of the deal pipeline in the storage market page, you can find more detailed information about the deal.

Deal detail page

The "Storage Deals" page list outs all the details in the Curio market. This can be used to get the summary of latest deals or to lookup a specific deal with unique identifier (UUID) using the search function.

Storage Deals page

The "Piece Info" contains all the deals about a piece onboarded by the Curio market. This is a one stop shop for all details about a piece and the deal containing the piece. It provides information about the piece itself, indexing, IPNI announcement status of the piece along with all the deals which contains this piece with their processing status. It also lists out the sectors which contain this piece. This is the single most important page for debugging issues with data onboarding. This page can be opened by clicking on the "Piece Cid" from multiple pages including the deal detail and deal list pages.

Piece Info page

Piece Info deal details

Piece Info storage pipeline status

All the IPNI provider and advertisement related details can be found on the "IPNI" page. The current status of IPNI provider (Curio) can be found on this page. The status is listed for each miner ID for each IPNI (ex: cid.contact) node.

IPNI page

The page also allows searching IPNI advertisements using the Cid. Users can scan the original piece to rebuild the entry list to debug issues with advertisement sync.

IPNI advertisement search page

# Retrievals This page explains Curio's HTTP-based content retrieval system. ## Overview The Curio HTTP server serves as the primary interface for handling content retrieval requests. In a Curio cluster, multiple nodes within the Curio cluster can host HTTP servers. Users can retrieve data using any of the available URLs, allowing for redundancy and ensuring high availability. ### Key Components: * **Piece-Based Retrievals**: Data can be fetched by a piece CID, enabling users to retrieve content directly from specific pieces stored within the Curio ecosystem. * **IPFS-Based Retrievals**: The Curio HTTP server also integrates with IPFS, allowing users to request content using an IPFS CID via the `/ipfs` route. * **Caching and Immutable Data**: Retrieved content is served with caching headers, ensuring immutability and efficiency for repeat requests. ## Piece-Based Retrievals The retrieval server allows users to fetch content by providing a piece CID through the `/piece/{cid}` route. Upon receiving such a request, the server looks up the requested piece and serves the content directly from the provider's unsealed sector. Any errors encountered during retrieval, such as the requested piece not being found, result in appropriate HTTP error responses (404, 500, etc.). ### IPFS Gateway Integration In addition to supporting piece-based retrievals, the Curio HTTP server also integrates IPFS gateway functionality. The `/ipfs/{cid}` route allows users to request content by CID. This is handled by the `frisbii` server backed by a `blockstore`, which fetches and returns the content corresponding to the requested IPFS CID. ### Caching and Headers Content served by the Curio retrieval server is immutable. For this reason, the server sets caching headers to allow clients and CDNs to cache content for extended periods. Specifically: * **ETag**: A unique identifier based on the piece CID ensures that cached content can be validated efficiently. * **Cache-Control**: Responses are marked as immutable and cacheable for up to one year, reducing the load on the retrieval server for repeat requests. By leveraging these caching mechanisms, the Curio HTTP server ensures that frequently accessed content is delivered quickly and with minimal resource consumption. # Migrating From Boost A step-by-step Guide to Migrating From Boost to Curio Migrating from Boost to Curio involves transitioning all deal data, including Boost deals, legacy deals, and Direct Data Onboarding (DDO) deals, to Curio’s storage and indexing system. Below is a detailed guide that walks through each step of the migration process. ## Build the `migrate-curio` CLI Tool * First, you need to build the `migrate-curio` tool, which is part of the Boost code base. This tool will handle the migration of deals. You must be running Boost version v2.4.7 or later. * Run the following command to build the tool. ```bash make migrate-curio ``` ## Preparation for Migration {% hint style="warning" %} Please read the [Curio market documentation](/curio-market/storage-market) carefully before proceeding. Curio market should be fully [configured](/curio-market/storage-market#enabling-storage-market) before proceeding with migration. {% endhint %} * Ensure that both Boost and Curio setups are ready for the migration. **Boost must be shutdown and no deal should be in process**. You will need the following details: * Path to the Boost repository (default is `~/.boost`) * Backup your Boost repository * `boostd-data` service must be up and running as migration needs to get some data from it if required. * Database credentials for Curio’s HarmonyDB (host, username, password, port) ## Run the Migration Tool Use the `migrate-curio` tool to begin the migration process. ```bash ./migrate-curio --boost-repo /path/to/boost-repo \ --db-host "127.0.0.1" \ --db-name "yugabyte" \ --db-user "yugabyte" \ --db-password "yugabyte" \ --db-port "5433" \ migrate ``` This command starts the migration, pulling data from the Boost repository and pushing it into Curio’s database. ## Migration Details The migration consists of 3 phases: ### Migrate Boost Deals 1. **Retrieve Active and Completed Deals:** * The tool will first query Boost’s database (`boost.db`) for active and completed deals. * It retrieves all Boost deals, including those that are still active or recently completed. 2. **Filtering Deals:** * Some deals may not be eligible for migration. The following deals are **skipped**: * Deals where the checkpoint is below the "add piece" stage. * Deals with a fatal retry error. * Deals with a sector ID of 0 or where the sector is no longer alive. * Deals that have already been migrated (tracked in `migrate-curio.db`). 3. **Processing Each Deal:** * For each eligible deal: * The deal is inserted into the `market_mk12_deals` table in Curio. * Additional information about the deal, such as proposal details, client peer ID, and the CID of the published deal, is migrated. * If the deal's sector is unsealed, it is added to the indexing and announcement pipeline (`market_mk12_deal_pipeline_migration`). ### Migrate Legacy Deals 1. **Load Legacy Deals:** * Legacy deals are stored in Boost’s LevelDB (`LID`). The tool queries the deals and processes each one. * It uses the `go-ds-versioning` library to access the old FSM (Finite State Machine) for managing legacy deals. 2. **Skipping Unnecessary Legacy Deals:** * Deals that have expired, have invalid sector numbers, or are already migrated are skipped. * Deals that do not have a chain deal ID or are past their expiration epoch are excluded from migration. 3. **Processing Legacy Deals:** * Legacy deals are inserted into Curio’s database: * Signed proposal CID, piece size, start and end epochs, and other deal-specific details are migrated. * Deals are not indexed. ### Migrate Direct Data Onboarding (DDO) Deals 1. **Retrieve DDO Deals:** * The tool queries the DDO database for all deals created using the Direct Data Onboarding method. 2. **Validating Claims:** * For each DDO deal, the tool verifies that the sector matches the claim in the deal. If the sector is no longer active, the deal is skipped. 3. **Processing DDO Deals:** * Eligible DDO deals are migrated to Curio’s `market_direct_deals` table. * If the sector is unsealed, the deal is also added to the `market_mk12_deal_pipeline_migration` table for indexing and announcement. ### **Deal Announcements** For all deals that have been migrated and are in unsealed sectors, Curio handles their indexing and IPNI (Indexing Provider Network Interface) announcements, ensuring they are publicly discoverable. ## Cleanup of Local Index Directory (LID) (optional) * After the migration, you can clean up the old LevelDB (`LID`) data that was used by Boost:\\ ```bash migrate-curio --boost-repo cleanup leveldb ``` * If the `LID` store was using YugabyteDB for storing indexes, use the following command\\ ```bash ./migrate-curio cleanup yugabyte \ --hosts "127.0.0.1" \ --username "yugabyte" \ --password "yugabyte" \ --connect-string "postgresql://postgres:postgres@localhost" \ --i-know-what-i-am-doing ``` This command will drop the `idx` keyspace and remove relevant tables in YugabyteDB. ## Monitoring and Finalizing Migration * Once the migration is complete, monitor the logs to ensure all deals have been correctly migrated and indexed. * Verify that the new Curio storage system has all the Boost, legacy, and DDO deals. * Verify the Curio is creating new indexing and IPNI jobs for migrated deals in batches. # Market 2.0 Market 2.0 is Curio’s unified, extensible deal interface between clients and storage providers. It exists to abstract away deal-type-specific integration.\ A client can request PoRep-style storage, PDP operations, or future specialized services through one consistent model and one consistent API surface. ## Deal Model Every deal is expressed as: * `id`: deal reference * `client`: requester identity * `products`: requested services * `data source` (optional): operation input data when required This model reflects real deal-making: who is asking, what they want, how it is referenced, and what inputs are needed. ## Clients Use these guides to submit and manage deals: 1. [HTTP API: create, update, and query deals](/market-2.0/http-api) 2. [Deal intake paths and upload ordering](/market-2.0/deal-processing#deal-intake-paths) 3. [Lifecycle and status tracking](/market-2.0/deal-processing) 4. [Data model and source/format requirements](/market-2.0/architecture#data-model) 5. [Error codes and troubleshooting](/market-2.0/http-api#endpoint-details) Contact your storage provider when: 1. You receive repeated `500` or `503` responses. 2. A deal stays non-terminal longer than expected. 3. A contract-related rejection needs provider policy confirmation. ## Storage Providers Use these guides to operate and support Market 2.0: 1. [Lifecycle and operational behavior](/market-2.0/deal-processing) 2. [Contract integration and provider allowlisting](/market-2.0/contracts) 3. [DDO contract review and allowlisting](/market-2.0/contracts/ddo-contract-review) 4. [Error codes for support triage](/market-2.0/http-api#endpoint-details) Operational note: 1. Product/source controls and contract allowlisting are provider policy decisions managed through the Curio GUI. 2. DDO contracts must be vetted before allowlisting. ## Developers ### SDK and Client Library Developers 1. [HTTP interaction model](/market-2.0/http-api) 2. [Lifecycle semantics](/market-2.0/deal-processing) 3. [Error handling contract](/market-2.0/http-api#endpoint-details) 4. [Data model and validation context](/market-2.0/architecture) 5. [Data source and format constraints](/market-2.0/architecture#data-model) ### Contract Integrators 1. [Contract integration guide](/market-2.0/contracts) 2. [Current DDO contract interface (CurioDealView v1)](/market-2.0/contracts/curiodealview) 3. [DDO contract review and builder guide](/market-2.0/contracts/ddo-contract-review) Current integration surface: 1. DDO uses `CurioDealViewV1` today. 2. Additional interfaces may be introduced in future product/version evolution without changing the top-level Market 2.0 deal model. ### Curio Product Developers 1. [Extending products](https://github.com/filecoin-project/curio/blob/main/documentation/en/market-2.0/extending-mk20.md) 2. [DDO product behavior](/market-2.0/products/ddo_v1) 3. [Retrieval product behavior](/market-2.0/products/retrieval_v1) 4. [PDP product behavior](/market-2.0/products/pdp_v1) Design rule: 1. Extend an existing product when behavior remains backward compatible and JSON evolution is safe. 2. Introduce a new product when behavior, lifecycle, or compatibility contracts diverge. # Architecture ## What This Page Covers This page explains what Market 2.0 is designed to do, why key model choices were made, how the system works end to end, and what problems it does not try to solve. ## Purpose Market 2.0 is the common interaction layer between clients and storage providers. It is designed so PoRep-style storage requests, PDP requests, and future specialized requests all use one stable deal envelope instead of separate APIs per request type. ## Core Deal Envelope ```go type Deal struct { Identifier ulid.ULID `json:"identifier"` Client string `json:"client"` Data *DataSource `json:"data,omitempty"` Products Products `json:"products"` } type Products struct { DDOV1 *DDOV1 `json:"ddo_v1,omitempty"` RetrievalV1 *RetrievalV1 `json:"retrieval_v1,omitempty"` PDPV1 *PDPV1 `json:"pdp_v1,omitempty"` } ``` Field meaning: 1. `identifier`: deal reference. 2. `client`: requester identity. 3. `products`: requested behavior. 4. `data`: operation input when required. Product-specific structs and fields are documented in: 1. [DDO v1](/market-2.0/products/ddo_v1) 2. [Retrieval v1](/market-2.0/products/retrieval_v1) 3. [PDP v1](/market-2.0/products/pdp_v1) ## Why `client` Is Text `client` is intentionally a text field, not a strict Filecoin address type. Why: 1. Market 2.0 must support identity schemes beyond native Filecoin address formats. 2. Current CurioAuth support is Filecoin-address based. 3. L2 or product-specific identity formats can be supported later without changing the deal envelope. Identity safety comes from authentication and identity matching, not from enforcing one address type in the schema. ## Deal Identifier (ULID) `identifier` uses ULID. 1. ULID specification: 2. Implementations (many languages): 3. Go implementation: 4. JavaScript/TypeScript implementation: ULID is used as a portable, sortable, globally unique deal reference. ## Authentication and Authorization This section documents current auth behavior. Header format: `Authorization: CurioAuth ::` Supported key types: 1. `secp256k1` 2. `bls` 3. `delegated` Signing input: 1. Build message bytes as `addressBytes || uppercaseRequestMethod || escapedRequestPath || RFC3339MinuteTimestamp`. 2. Hash with SHA-256. 3. Sign the digest. `addressBytes` means Filecoin address bytes (`address.Address.Bytes()`), not the human-readable address string. `escapedRequestPath` is the request URL path without the query string. The request body and query string are not part of the signed message. The auth header is therefore a short-lived path-scoped credential and must be protected in transit. Verification window: 1. Current minute. 2. Previous minute, to tolerate requests crossing a minute boundary in transit. Authorization checks: 1. Signature must verify for the request method, request path, and one of the accepted timestamps. 2. Client allow/deny policy is read from `market_mk20_clients`. 3. If client is not listed, fallback behavior uses `DenyUnknownClients` config. 4. For routes with `{id}`, authenticated client must own that deal. Route scope: 1. Market API routes are authenticated. 2. `/info/*` OpenAPI routes are public. ## Data Model ```go type DataSource struct { PieceCID cid.Cid `json:"piece_cid"` Format PieceDataFormat `json:"format"` SourceHTTP *DataSourceHTTP `json:"source_http,omitempty"` SourceAggregate *DataSourceAggregate `json:"source_aggregate,omitempty"` SourceOffline *DataSourceOffline `json:"source_offline,omitempty"` SourceHttpPut *DataSourceHttpPut `json:"source_http_put,omitempty"` } type PieceDataFormat struct { Car *FormatCar `json:"car,omitempty"` Aggregate *FormatAggregate `json:"aggregate,omitempty"` Raw *FormatBytes `json:"raw,omitempty"` } ``` Validation rules: 1. `piece_cid` must be valid PieceCID v2. 2. Exactly one source type is allowed. 3. Exactly one format is allowed. 4. Source type must be enabled by provider policy. ## Aggregation Current aggregation support: 1. `AggregateTypeV1` only. 2. This is datasegment/PODSI-style aggregation based on FRC-0058: Why aggregation matters in Market 2.0: 1. It allows subpiece-aware indexing and retrieval flows. 2. It enables retrieval of subpieces, including IPFS-style single-CID usage from subpiece data paths when retrieval/indexing settings are enabled. Current constraints: 1. Aggregate-of-aggregate is rejected. 2. `source_aggregate` mode and pre-aggregated `format.aggregate.sub` mode have different validation rules. 3. Subpiece order is meaningful and must match segment order. Future direction: 1. Additional aggregation types can be added as new FRCs are finalized. 2. The top-level Market 2.0 deal envelope remains unchanged. ## Product Composition Rules 1. At least one top-level product must be present. 2. `ddo_v1` and `pdp_v1` cannot be used together in one deal. 3. `ddo_v1` requires `retrieval_v1`. 4. With `ddo_v1`, `retrieval_v1.announce_piece` must be false. ## How a Deal Moves Through the System 1. Client submits deal intent. 2. Market 2.0 validates identity, payload, and product composition. 3. Market 2.0 selects product execution path. 4. Market 2.0 selects ingest path (immediate source or upload-first). 5. Processing continues through common lifecycle states. 6. Deal reaches terminal success (`complete`) or terminal failure (`failed`). Lifecycle states: 1. `accepted` 2. `uploading` 3. `processing` 4. `sealing` 5. `indexing` 6. `failed` 7. `complete` ## External Contract Boundary Market 2.0 supports product-specific contract verification without changing the outer deal model. Example: 1. DDO verifies through `CurioDealViewV1`, primarily with `version()` and `verifyDeal(...)`. 2. The interface also exposes `getDealState(...)` for deal-state queries. 3. Provider allowlist controls which contracts are accepted. 4. Interface versioning is explicit. ## Best Practices 1. Keep `identifier` immutable. 2. Keep `client` stable within the chosen identity namespace. 3. Query provider capabilities before creating deals. 4. Keep product payloads explicit. 5. Vet contracts before allowlisting. 6. Extend existing products only when backward compatibility is preserved. 7. Add a new product when behavior or lifecycle contracts diverge. ## What Market 2.0 Does Not Solve 1. Commercial negotiation and legal agreement terms. 2. Automatic trust in external contracts. 3. Universal payment/settlement semantics across all products. 4. Product-specific business policy decisions outside declared product contracts. 5. Operational monitoring and recovery by itself. # HTTP API Base path: `/market/mk20`\ `{id}` is the deal identifier (ULID). ## Authentication All Market 2.0 API routes require auth except `/info/*`. Auth details are documented in: [Architecture: Authentication and Authorization](/market-2.0/architecture#authentication-and-authorization) ## API Compatibility Market 2.0 APIs are expected to evolve conservatively, with backward compatibility as a primary consideration. ## Swagger and OpenAPI Specs Use the running storage provider spec first for integration and debugging. Running provider endpoints: 1. `GET /market/mk20/info/` 2. `GET /market/mk20/info/swagger.yaml` 3. `GET /market/mk20/info/swagger.json` Source tree specs: 1. [`market/mk20/http/swagger.yaml`](https://github.com/filecoin-project/curio/blob/main/market/mk20/http/swagger.yaml) 2. [`market/mk20/http/swagger.json`](https://github.com/filecoin-project/curio/blob/main/market/mk20/http/swagger.json) Version note: 1. A running provider may be on a different Curio version than this documentation. 2. If behavior differs, use that provider's `/info/swagger.*` as the operational reference. ## Endpoint Map 1. `POST /deal` 2. `GET /status/{id}` 3. `POST /update/{id}` 4. `GET /products` 5. `GET /sources` 6. `GET /contracts` 7. `POST /uploads/{id}` 8. `GET /uploads/{id}` 9. `PUT /uploads/{id}/{chunkNum}` 10. `POST /uploads/finalize/{id}` 11. `PUT /upload/{id}` 12. `POST /upload/{id}` ## Endpoint Details ## `POST /deal` Creates a deal. Request: 1. Body: `mk20.Deal` JSON. Success: 1. `200` `Ok` Error codes: 1. `400` `ErrBadProposal` 2. `401` `ErrUnAuthorized` 3. `404` `ErrDealNotFound` 4. `422` `ErrUnsupportedDataSource` 5. `423` `ErrUnsupportedProduct` 6. `424` `ErrProductNotEnabled` 7. `425` `ErrProductValidationFailed` 8. `426` `ErrDealRejectedByMarket` 9. `429` `ErrServiceOverloaded` 10. `430` `ErrMalformedDataSource` 11. `440` `ErrMarketNotEnabled` 12. `441` `ErrDurationTooShort` 13. `500` `ErrServerInternalError` 14. `503` `ErrServiceMaintenance` Response note: 1. Error body is plain text: `Reason: `. ## `GET /status/{id}` Returns deal status by product. Success: 1. `200` `mk20.DealProductStatusResponse` Error codes: 1. `400` invalid/missing id 2. `401` `ErrUnAuthorized` 3. `404` `ErrDealNotFound` 4. `500` `ErrServerInternalError` ## `POST /update/{id}` Updates an existing deal in supported update paths. The update path can fill missing `data` or add product fields that are not already present. Existing `data` and existing product fields are not replaced. Request: 1. Body: `mk20.Deal` JSON. Success: 1. `200` `Ok` Error codes: 1. `400` `ErrBadProposal` 2. `401` `ErrUnAuthorized` 3. `404` `ErrDealNotFound` 4. `422` `ErrUnsupportedDataSource` 5. `423` `ErrUnsupportedProduct` 6. `424` `ErrProductNotEnabled` 7. `425` `ErrProductValidationFailed` 8. `426` `ErrDealRejectedByMarket` 9. `429` `ErrServiceOverloaded` 10. `430` `ErrMalformedDataSource` 11. `440` `ErrMarketNotEnabled` 12. `441` `ErrDurationTooShort` 13. `500` `ErrServerInternalError` 14. `503` `ErrServiceMaintenance` ## `GET /products` Returns enabled products for this provider. Success: 1. `200` `{ "products": [...] }` Error codes: 1. `401` `ErrUnAuthorized` 2. `500` `ErrServerInternalError` ## `GET /sources` Returns enabled data source types for this provider. Success: 1. `200` `{ "sources": [...] }` Error codes: 1. `401` `ErrUnAuthorized` 2. `500` `ErrServerInternalError` ## `GET /contracts` Returns allowlisted DDO market contracts for this provider. Success: 1. `200` `{ "contracts": [...] }` Error codes: 1. `401` `ErrUnAuthorized` 2. `404` no supported contracts found 3. `500` `ErrServerInternalError` ## `POST /uploads/{id}` Starts chunked upload session. Request: 1. Body: `StartUpload` (`raw_size`, `chunk_size`). Success: 1. `200` `UploadStartCodeOk` Error codes: 1. `400` `UploadStartCodeBadRequest` 2. `401` `ErrUnAuthorized` 3. `404` `UploadStartCodeDealNotFound` 4. `409` `UploadStartCodeAlreadyStarted` 5. `500` `UploadStartCodeServerError` ## `GET /uploads/{id}` Returns chunked upload progress. Success: 1. `200` `UploadStatusCodeOk` Error codes: 1. `400` invalid/missing id 2. `401` `ErrUnAuthorized` 3. `404` `UploadStatusCodeDealNotFound` 4. `425` `UploadStatusCodeUploadNotStarted` 5. `500` `UploadStatusCodeServerError` ## `PUT /uploads/{id}/{chunkNum}` Uploads one chunk. Success: 1. `200` `UploadOk` Error codes: 1. `400` `UploadBadRequest` 2. `401` `ErrUnAuthorized` 3. `404` `UploadNotFound` 4. `409` `UploadChunkAlreadyUploaded` 5. `429` `UploadRateLimit` 6. `500` `UploadServerError` ## `POST /uploads/finalize/{id}` Finalizes chunked upload. Body may be empty or full `mk20.Deal`. Success: 1. `200` `Ok` Error codes: 1. `400` `ErrBadProposal` 2. `401` `ErrUnAuthorized` 3. `404` `ErrDealNotFound` 4. `422` `ErrUnsupportedDataSource` 5. `423` `ErrUnsupportedProduct` 6. `424` `ErrProductNotEnabled` 7. `425` `ErrProductValidationFailed` 8. `426` `ErrDealRejectedByMarket` 9. `429` `ErrServiceOverloaded` 10. `430` `ErrMalformedDataSource` 11. `440` `ErrMarketNotEnabled` 12. `441` `ErrDurationTooShort` 13. `500` `ErrServerInternalError` 14. `503` `ErrServiceMaintenance` ## `PUT /upload/{id}` Uploads full payload in serial flow. Success: 1. `200` `UploadOk` Error codes: 1. `400` `UploadBadRequest` 2. `401` `ErrUnAuthorized` 3. `404` `UploadStartCodeDealNotFound` 4. `500` `UploadServerError` ## `POST /upload/{id}` Finalizes serial upload. Body may be empty or full `mk20.Deal`. Success: 1. `200` `Ok` Error codes: 1. `400` `ErrBadProposal` 2. `401` `ErrUnAuthorized` 3. `404` `ErrDealNotFound` 4. `422` `ErrUnsupportedDataSource` 5. `423` `ErrUnsupportedProduct` 6. `424` `ErrProductNotEnabled` 7. `425` `ErrProductValidationFailed` 8. `426` `ErrDealRejectedByMarket` 9. `429` `ErrServiceOverloaded` 10. `430` `ErrMalformedDataSource` 11. `440` `ErrMarketNotEnabled` 12. `441` `ErrDurationTooShort` 13. `500` `ErrServerInternalError` 14. `503` `ErrServiceMaintenance` ## Operational Notes 1. Upload finalize endpoints accept an empty body when the deal already has data details. If the accepted deal omitted data details, finalize must include a full deal payload so MK20 can bind the uploaded piece to the deal. 2. Provider policies for products, sources, and contracts affect acceptance behavior. 3. When behavior differs from docs, trust the running provider `/info/swagger.*`. # Deal Processing This page defines the deal intake paths, lifecycle progression, status semantics, and pipeline behavior for Market 2.0. ## Deal Intake Paths All intake paths use the same deal model and the same deal identifier (`id`, ULID). They differ only in when and how piece bytes are provided. ### 1. Provider-Pull Deal Use this path when the storage provider should fetch data from client-supplied sources. API sequence: 1. `POST /deal` 2. `GET /status/{id}` until terminal state No upload endpoints are used. ### 2. Accept-Then-Upload Deal Use this path when a client wants acceptance first, then uploads bytes. API sequence: 1. `POST /deal` 2. Upload bytes using one mode: 3. Serial: `PUT /upload/{id}` then `POST /upload/{id}` 4. Chunked: `POST /uploads/{id}` then `PUT /uploads/{id}/{chunkNum}` (repeat) then `POST /uploads/finalize/{id}` 5. `GET /status/{id}` until terminal state ### 3. Upload-First-Then-Describe Deal Use this path when bytes are uploaded first and full deal details are supplied at finalize. This requires an initial product flow that can be accepted without data details. DDO deals require `data` during initial acceptance, so DDO upload flows should use `source_http_put` in the initial deal instead. API sequence: 1. `POST /deal` 2. Upload bytes using one mode: 3. Serial: `PUT /upload/{id}` 4. Chunked: `POST /uploads/{id}` then `PUT /uploads/{id}/{chunkNum}` (repeat) 5. Finalize with full deal payload: 6. Serial: `POST /upload/{id}` 7. Chunked: `POST /uploads/finalize/{id}` 8. `GET /status/{id}` until terminal state Current behavior: this path is supported in both serial and chunked finalize flows for products that can enter upload-waiting without initial data details. ### 4. Control-Only Operation (No Piece Ingestion) Use this path for PDP lifecycle operations that do not ingest piece data (for example dataset create, dataset delete, or piece-remove operations). API sequence: 1. `POST /deal` 2. `GET /status/{id}` until terminal state No upload endpoints are used. ## Deal Lifecycle Once accepted, deals follow a common lifecycle model: 1. `accepted`: proposal admitted. 2. `uploading` (optional): waiting for or receiving client-uploaded bytes. 3. `processing`: product execution has started. 4. `sealing` (when applicable): sealing work in progress. 5. `indexing` (when applicable): indexing and retrieval preparation in progress. 6. Terminal outcome: `complete` or `failed`. Not every deal uses every intermediate state. Upload-less operations typically skip `uploading`. ## Status Checks Use `GET /status/{id}` for lifecycle tracking. Status is returned per product present in the deal. Status values: 1. `accepted` 2. `uploading` 3. `processing` 4. `sealing` 5. `indexing` 6. `complete` (terminal success) 7. `failed` (terminal failure) Use upload endpoints for byte-transfer progress (`/upload*`, `/uploads*`). Use `/status/{id}` for lifecycle state. ## Pipelines Pipelines are the asynchronous execution layer behind lifecycle transitions. What pipelines do: 1. Run ingest work for pull and upload flows. 2. Run product-specific execution once data is ready. 3. Advance deals to terminal outcome. When pipeline execution starts: 1. Pull-based deals: after acceptance. 2. Upload-based deals: after upload finalize. 3. Control-only operations: after acceptance. Operational guidance: 1. If an upload-based deal appears stalled, verify finalize was called. 2. If status remains non-terminal unexpectedly, provider-side operational inspection is required. # Products Products are the service contract between client intent and provider execution in Market 2.0. This section documents the currently supported product families, how they fit into the common deal model, and how to evolve product behavior without breaking compatibility. In practice, products answer one question: what operation should Curio perform for this deal. ## Current Products 1. [DDO v1](/market-2.0/products/ddo_v1) 2. [Retrieval v1](/market-2.0/products/retrieval_v1) 3. [PDP v1](/market-2.0/products/pdp_v1) ## Product Model Basics 1. Product-specific payloads live under `deal.products`. 2. Product-specific validation must be explicit and deterministic. 3. Product behavior should map cleanly to Market 2.0 lifecycle/status behavior. 4. Product errors should map to existing MK20 error codes where possible. ## Product Definition Checklist A product definition should clearly specify: 1. Payload schema under `products.`. 2. Product name and compatibility scope. 3. Required vs optional fields. 4. Validation rules and rejection conditions. 5. Source/format assumptions (if data is required). 6. Lifecycle/status effects. 7. Error mapping expectations. 8. Operational controls required by storage providers. ## Extending Products ### When to Extend an Existing Product Extend an existing product when: 1. Behavior remains backward compatible. 2. Existing clients can continue to parse and use payloads safely. 3. JSON marshal/unmarshal compatibility is preserved. ### When to Add a New Product Add a new product when: 1. Behavior or lifecycle differs materially. 2. Compatibility contracts diverge from existing product semantics. 3. Overloading an existing product would create ambiguous behavior. ### Implementation Checklist 1. Add product type in MK20 deal/product types. 2. Register product name and validation rules. 3. Add intake/processing/finalize handling. 4. Add upload-path handling if product supports `source_http_put`. 5. Extend status derivation if lifecycle differs. 6. Add DB migrations for product state. 7. Add WebRPC/UI support if operator controls are required. 8. Add/update product docs in this directory. ### Compatibility Rules 1. Do not weaken auth and deal ownership guarantees. 2. Keep piece identity invariants intact. 3. Avoid breaking existing product behavior. ### Documentation Requirements For each product, document: 1. Use case 2. Payload fields 3. Validation rules 4. Lifecycle/status effects 5. Common rejection scenarios # DDO v1 ## What `ddo_v1` Is For `ddo_v1` is the storage onboarding product for MK20. Use it when a client wants a provider to ingest a piece and run normal sealing/indexing flow, optionally tied to an external market contract. ## Struct Definition ```go type DDOV1 struct { Provider address.Address `json:"provider"` StartEpoch *abi.ChainEpoch `json:"start_epoch"` Duration abi.ChainEpoch `json:"duration"` AllocationId *verifreg.AllocationId `json:"allocation_id,omitempty"` MarketAddress string `json:"market_address"` MarketDealID *uint64 `json:"market_deal_id"` NotificationAddress address.Address `json:"notification_address"` NotificationPayload []byte `json:"notification_payload,omitempty"` } ``` ## Required And Optional Fields * Required: * `provider` * `duration` * Optional: * `start_epoch` * `allocation_id` * `market_address` * `market_deal_id` (required if `market_address` is set) * `notification_address` and `notification_payload` (must be provided together) ## Field Behavior * `start_epoch` is optional. If set, Curio validates it against chain head and `ExpectedPoRepSealDuration`, and uses it as the deal schedule start during ingestion. * `duration` is always part of the DDO payload. For verified allocations, it must still satisfy allocation term bounds. * `market_address` and `market_deal_id` opt the deal into external contract verification. * `notification_address` and `notification_payload`, when set, are attached to the eventual piece activation manifest during ingestion. They are not evaluated by `verifyDeal(...)`. ## Core Validation Rules * `ddo_v1` must be enabled by the provider. * Provider must be a valid address and not in MK20 disabled-miner config. * If `start_epoch` is set, it must be greater than `0`. * If `allocation_id` is missing, minimum duration is enforced (`>= 518400`). * If `allocation_id` is set, it must not be `NoAllocationID`. * If `market_address` is set, it must be a valid `0x` hex address. * Notification address and payload must either both be set or both be unset. ## Market Contract Verification When `market_address` is provided, MK20 performs read-only contract verification before acceptance. Verification behavior: 1. Contract must exist in `ddo_contracts` and be allowed. 2. Contract must implement `CurioDealViewV1` and return `version() == 1`. 3. MK20 builds `CurioDealView` from local deal values and calls `verifyDeal(...)`. 4. `verifyDeal(...)` must return `true`. This call is a read-only intake gate. It checks whether the market contract accepts the proposed deal. It does not by itself guarantee payment, payout, or notification success. Curio passes these values into `verifyDeal(...)`: * `dealId` * `state` = `Open` * provider actor id * client identity bytes * piece CID v2 * `start_epoch` or `0` * duration * `allocation_id` or `0` * `finalizedEpoch` = `0` If `verifyDeal` reverts with `DealNotFound(uint256)`, MK20 rejects the deal as market-missing. ## Notification Callback Behavior If both `notification_address` and `notification_payload` are set, Curio carries them into the piece activation manifest during ingestion as a `DataActivationNotification`. Operationally: 1. This path runs later than `verifyDeal(...)`, during sector activation work. 2. Contracts that rely on notification callbacks for settlement or state transitions should be reviewed separately from intake-time contract verification. 3. Provider setups that depend on notification callbacks generally keep `Subsystems.RequireNotificationSuccess = true` (the default). See [Default Curio Configuration](/configuration/default-curio-configuration). ## Additional Sanitize Checks Before Pipeline Entry * `retrieval_v1` must be present. * `retrieval_v1.announce_piece` must be false. * Provider must be in Curio miner set. * Deal data must be present. * Piece size must fit provider sector size. * Raw format cannot be indexed. * Notification address must resolve on chain when set. * Client must pass provider allow/deny policy. * If `start_epoch` is set, it must be greater than or equal to current chain height plus `ExpectedPoRepSealDuration`. * If `allocation_id` is set, MK20 validates allocation ownership, provider, term, data, size, and expiration consistency. * For verified allocations, the allocation owner may be either the client or the market contract address. * If both `allocation_id` and `start_epoch` are set, allocation expiration must be greater than or equal to `start_epoch`. # Retrieval v1 ## What `retrieval_v1` Is For `retrieval_v1` controls indexing and announcement behavior for data submitted through MK20 products. ## Struct Definition ```go type RetrievalV1 struct { Indexing bool `json:"indexing"` AnnouncePayload bool `json:"announce_payload"` AnnouncePiece bool `json:"announce_piece"` } ``` ## Fields * `indexing`: enable indexing flow. * `announce_payload`: announce payload-level information. * `announce_piece`: announce piece-level information. ## Validation Rules * `retrieval_v1` must be enabled by provider. * `announce_payload` requires `indexing = true`. ## Product Compatibility Rules * `ddo_v1` requires `retrieval_v1`. * With `ddo_v1`, `announce_piece` must be false. * Additional context-dependent checks are applied during DDO/PDP sanitize stages. # PDP v1 ## What `pdp_v1` Is For `pdp_v1` drives PDP dataset and piece lifecycle operations through MK20. ## Struct Definition ```go type PDPV1 struct { CreateDataSet bool `json:"create_data_set"` DeleteDataSet bool `json:"delete_data_set"` AddPiece bool `json:"add_piece"` DeletePiece bool `json:"delete_piece"` DataSetID *uint64 `json:"data_set_id,omitempty"` RecordKeeper string `json:"record_keeper"` PieceIDs []uint64 `json:"piece_ids,omitempty"` ExtraData []byte `json:"extra_data,omitempty"` } ``` ## Action Model Exactly one of these flags must be set in each deal: * `create_data_set` * `delete_data_set` * `add_piece` * `delete_piece` ## Key Fields * `data_set_id`: required for all actions except `create_data_set`. * `record_keeper`: required when creating a dataset. * `piece_ids`: required for `delete_piece`. * `extra_data`: verifier/service payload. It is required for `create_data_set`, `add_piece`, and `delete_piece`, and must be absent for `delete_data_set`. ## Validation Rules * `pdp_v1` must be enabled by provider. * Provider must have PDP signing key configured (`eth_keys.role = 'pdp'`). * Only one action flag is allowed per deal. * `create_data_set`: * `data_set_id` must be absent * `record_keeper` must be a valid hex address * `extra_data` must be present and no larger than 4 KiB * `delete_data_set` / `add_piece`: * `data_set_id` must be present * dataset must exist and be active * `delete_data_set`: * `extra_data` must be absent * `add_piece`: * `extra_data` must be present and no larger than 8 KiB * `delete_piece`: * `data_set_id` must be present * `piece_ids` must be present * all listed pieces must exist and be active in that dataset * `extra_data` must be present and no larger than 256 bytes ## Runtime Sanitize Rules * `add_piece` requires `retrieval_v1`. * `offline` source is rejected for PDP flows. * Raw-format data cannot use `announce_payload`. * Dataset and piece ownership checks are revalidated. ## Processing Behavior * `add_piece` with HTTP/aggregate data goes directly into PDP processing pipelines. * `add_piece` with `put` source (or upload-first mode) enters upload-waiting first. * Create/delete operations enqueue PDP lifecycle actions without upload flow. # Contracts This section covers Market 2.0 contract integration for product developers and contract integrators. ## Contract Development Basics When designing a contract to integrate with Curio: 1. Prefer read-only verification interfaces for deal checks. 2. Use explicit interface versioning. 3. Keep verification input and deal-state semantics stable and deterministic. 4. Use a deterministic not-found contract error for missing deal references. 5. Treat provider allowlisting as a separate operational policy. ## DDO Current DDO integration uses: 1. [CurioDealView v1](/market-2.0/contracts/curiodealview) 2. [DDO contract review and allowlisting](/market-2.0/contracts/ddo-contract-review) Future products can add their own contract interfaces without changing the top-level Market 2.0 deal envelope. # DDO: CurioDealView v1 This page defines the DDO market contract interface expected by Market 2.0. ## Contract Control for Storage Providers Storage providers decide which DDO market contracts Curio will trust. From the Curio UI, each contract is either: 1. Allowed: new DDO deals can reference this contract. 2. Blocked: new DDO deals using this contract are rejected. 3. Removed: contract is no longer configured; new DDO deals using this contract are rejected. ### What This Means Operationally 1. Allowing a contract enables new intake for that contract. 2. Blocking or removing a contract stops new intake immediately. 3. This policy applies to new submissions and deal updates that introduce DDO market verification. ### Adding a New Contract Safely Before allowing a contract, providers should verify: 1. The contract implements `CurioDealViewV1` correctly. 2. `verifyDeal` checks the same deal semantics Curio will pass in (`provider`, `client`, `piece CID`, `start epoch`, `duration`, `allocation`, finalization fields). 3. `getDealState` uses the expected `Open` / `Active` / `Finalized` meanings. 4. The contract's state transitions and finalization behavior are clear for payout and replay safety. Only after this review should the contract be set to Allowed. Operational review guidance is documented in: 1. [DDO contract review and allowlisting](/market-2.0/contracts/ddo-contract-review) ## Interface Contracts integrated with Curio for DDO verification should implement `ICurioDealViewV1`. ```solidity interface ICurioDealViewV1 { error DealNotFound(uint256 dealId); enum DealState { Open, Active, Finalized } struct CurioDealView { uint256 dealId; DealState state; uint256 providerActorId; bytes clientId; bytes pieceCidV2; uint256 startEpoch; uint256 duration; uint256 allocationId; uint256 finalizedEpoch; } function version() external pure returns (uint256); function verifyDeal(CurioDealView calldata deal) external view returns (bool); function getDealState(uint256 dealId) external view returns (DealState); } ``` ## Versioning `version()` must return `1` for this interface. ## `CurioDealView` Field Semantics 1. `dealId`: the market deal identifier referenced by `market_deal_id`. 2. `state`: Curio sends `Open` during intake verification. 3. `startEpoch`: must be `0` when unused. 4. `allocationId`: must be `0` when unused. 5. `finalizedEpoch`: must be `0` for non-finalized intake verification. ## Curio Verification Checks When `market_address` is set for a DDO deal, Curio performs read-only verification: 1. Contract is allowlisted by the provider. 2. `version()` is supported and returns `1`. 3. Curio builds a `CurioDealView` from local deal details and calls `verifyDeal(...)`. 4. `verifyDeal(...)` must return `true`. If `verifyDeal` reverts with `DealNotFound(uint256)`, Curio treats the market deal as missing. The interface also includes `getDealState(...)` for deal-state queries. ## Read-Only Boundary 1. Curio does not call write methods for this verification path. 2. This verification path only decides whether the contract accepts the proposed deal during intake. 3. Payment, settlement, and notification-driven state transitions are outside this read-only boundary. # DDO Contract Review and Allowlisting This page is for storage providers reviewing DDO market contracts, and for builders deploying them for Market 2.0. ## What Curio Actually Checks When a DDO deal includes `market_address`, Curio performs read-only intake verification before accepting the deal. Current checks: 1. `market_address` must exist in `ddo_contracts` and be marked `allowed = true`. 2. `market_deal_id` must be present. 3. Curio calls `version()` and requires `1`. 4. Curio builds `ICurioDealViewV1.CurioDealView` from the local deal and calls `verifyDeal(...)` via `eth_call`. 5. `verifyDeal(...)` must return `true`. 6. If `verifyDeal(...)` reverts with `DealNotFound(uint256)`, Curio rejects the deal as market-missing. What this does not mean: 1. This call only answers whether the contract accepts the proposed deal at intake time. 2. It does not by itself prove payment, payout, callback success, or settlement behavior. 3. `AddMarketContract` only validates address syntax and that the address resolves to a Filecoin actor before inserting it into Curio policy state. 4. Curio does not inspect proxy admin, upgrade controls, ownership, or source verification for the contract. ## Provider Control States Provider policy is separate from contract code compatibility. Current states: 1. Added and allowed: new DDO deals can reference the contract. 2. Added but blocked (`allowed = false`): new DDO deals using that contract are rejected. 3. Removed: contract is no longer configured; new DDO deals using it are rejected. 4. Existing accepted deals continue through processing even if the contract is later blocked or removed. 5. `GET /contracts` returns only allowed contracts. These controls are managed through Curio operator tooling and UI. ## Provider Review Checklist Before allowing a DDO market contract, providers should review: 1. Verified contract source and ABI from a source they trust. 2. The deployed address, preferably confirmed through a trusted builder channel. 3. `verifyDeal(...)` behavior, including validation of `providerActorId`, `clientId`, `pieceCidV2`, `startEpoch`, `duration`, `allocationId`, and the expected `state` / `finalizedEpoch` semantics. 4. Notification-driven settlement or callback paths separately from the intake verification path. 5. Ownership, upgrade authority, and any governance or timelock around the contract if it is proxy-based or upgradeable. Allocation note: 1. For allocation-backed DDO flows, Curio can resolve the allocation against either the deal client or the market contract address. 2. If the market contract allocates on behalf of end users, providers should make sure that ownership model is intentional and documented. ## Upgradeable Contract Risk Curio does not distinguish immutable and upgradeable contracts during allowlisting. Providers should treat upgradeable contracts as an ongoing trust decision: 1. Review who controls upgrades. 2. Review whether upgrades are gated by a timelock or multisig. 3. Re-review the contract when implementation or ownership changes. 4. Block or remove the contract if governance or implementation changes become unclear. ## Builder Guidance Builders integrating a DDO market contract should: 1. Publish verified source and a reliable ABI reference. 2. Implement `CurioDealViewV1` exactly and return `version() == 1`. 3. Keep `verifyDeal(...)` deterministic and read-only. 4. Validate the fields Curio sends rather than blindly accepting every call. 5. Use `DealNotFound(uint256)` for missing market deal identifiers. 6. Keep notification-driven callbacks efficient and document the expected failure behavior for providers if the contract depends on them. 7. Communicate upgrade process and governance model clearly if the contract is upgradeable. ## Related Pages 1. [CurioDealView v1 interface](/market-2.0/contracts/curiodealview) 2. [DDO product behavior](/market-2.0/products/ddo_v1) # Snap Deals This guide explains how to enable snap-deals in Curio. ## Simplified explanation Snap-deals allow storage providers to accept deals from users and place that user’s data into a block of storage that had already been committed. That was a bit of a mouthful, so picture it like this. Imagine there is a town with a very long shelf. Anyone in this town can store anything they want on this shelf. When a townsperson wants to store something, they give that *thing* to a storage provider. The storage provider builds a wooden box, puts the townsperson’s stuff into the box, and then puts the box on the shelf.
A shelf representing the Filecoin network.

Sector as a shelf

Some of the boxes have useful stuff in them, like photographs, music, or videos. But sometimes, the storage providers don’t have any townspeople lining up to put useful stuff into the boxes. So instead, they put packing peanuts in the box and put that on the shelf. This means that there are a lot of boxes being made to just hold packing peanuts. Making boxes takes a long time and takes a lot of work from the storage provider.
Types of data in a Filecoin sector.

Data boxes

Instead of creating a new box every time someone wants to store something, it’d be better if we could just replace the packing peanuts with useful stuff! Since nobody cares about the packing peanuts, nobody is going to be unhappy with throwing them out. And the storage provider gets to put useful stuff on the shelf without having to create a new box! Things are better for the townsperson, too, since they don’t have to wait for the storage provider to create a new box!
Emptying sectors of dummy data to fill them with real data.

Replacing data

This is a simplified view of how Snap-deals work. Instead of a storage provider creating an entirely new sector to store a client’s data, they can put the client’s data into a committed capacity sector. The data becomes available faster, things are less expensive for the storage provider, and more of the network’s storage capacity gets utilised! ## How to enable snap-deals To enable the snap deals pipeline in a Curio cluster, user needs to enable the snap deal specific tasks on the machines that have GPU resources. Apart from this, the deal ingestion pipeline needs to be updated to pass the deals to the snap deal pipeline instead of the PoRep sealing pipeline. {% hint style="warning" %} Data can be ingested using either the Snap Deals pipeline or the PoRep pipeline at any given time, but not both simultaneously. {% endhint %} ## FastSnap (SnapDeals UpdateEncode acceleration) Curio’s SnapDeals `UpdateEncode` path has a **fast mode** (“fastsnap”) that uses the batch sealing CUDA toolchain (`extern/supraseal`) to accelerate TreeR generation and uses Curio-native snap encoding. * **Capability check**: run ```bash curio test supra system-info ``` Look for **“Can run fast TreeR: yes”**. * **Fallback mode**: if the host lacks AVX-512 (AMD64v4) or a usable CUDA GPU, Curio automatically falls back to a CPU path for TreeR generation. * **Troubleshooting / force fallback**: set: ```bash export DISABLE_SUPRA_TREE_R=1 ``` This forces the CPU fallback TreeR path (useful to isolate batch sealing/toolchain issues). ### Configuration {% hint style="warning" %} When switching between Snap and PoRep deal pipeline, you must ensure that no sectors are being sealed or snapped. All pipelines must be empty before making a switch. {% endhint %} #### Curio Market 1. Enable snap deals on base layer. Enabling it on base layer is very important so that no node in the cluster accidentally forwards deal to PoRep pipeline. 2. Save the layer and exit. [Enable snap tasks](#enable-snap-tasks) and restart all the nodes. ``` [Ingest] DoSnap = true ``` #### Boost Adapter (Deprecated) {% hint style="warning" %} Boost adapter is no longer supported with new Curio releases. {% endhint %} 1. Create or update the market layer ([if one is already created](https://github.com/filecoin-project/curio/blob/main/documentation/en/enabling-market.md#enable-market-adapter-in-curio)) for the minerID where you wish to use snap deals pipeline.
```shell curio config add --title mt01000 ``` \ Add an entry like:
``` [Subsystems] EnableParkPiece = true BoostAdapters = ["t10000:127.0.0.1:32100"] [Ingest] DoSnap = true ``` \ Press `ctrl + D` to save and exit.\ Or edit the existing layer.
```shell curio config edit mt01000 ``` \ Enable the snap deals for ingestion:
``` [Subsystems] EnableParkPiece = true BoostAdapters = ["t10000:127.0.0.1:32100"] [Ingest] DoSnap = true ``` \ Save the layer and exit. 2. Add the new market configuration layer to the appropriate nodes based on the [best practices](/best-practices). ### Enable snap tasks 1. Add the `upgrade` layer already shipped with Curio to the `/etc/curio.env` file on the Curio nodes where GPU resources are available.
``` CURIO_LAYERS=gui,seal,post,upgrade <----- Add the "upgrade" layer CURIO_ALL_REMAINING_FIELDS_ARE_OPTIONAL=true CURIO_DB_HOST=yugabyte1,yugabyte2,yugabyte3 CURIO_DB_USER=yugabyte CURIO_DB_PASSWORD=yugabyte CURIO_DB_PORT=5433 CURIO_DB_NAME=yugabyte CURIO_REPO_PATH=~/.curio CURIO_NODE_NAME=ChangeMe FIL_PROOFS_USE_MULTICORE_SDR=1 ```
2. Restart the Curio services on the node.
``` systemctl restart curio ``` *** ## Troubleshooting Snap Deals ingestion ### Error: `allocating sector numbers: no suitable sectors found` Plain-English meaning: * Curio tried to pick an existing **CC sector** to upgrade (snap), but none matched the constraints. Common causes: * There are no CC sectors available to upgrade for that miner. * Expiration constraints: the candidate sectors cannot satisfy the deal end-epoch requirements. * Snap pipeline not actually enabled where ingestion runs (layer mismatch). * Required snap tasks are not running on any GPU-capable node. What to check first: 1. Confirm ingestion is routed to Snap: ```toml [Ingest] DoSnap = true ``` 2. Confirm at least one node is running the `upgrade` layer (or equivalent snap task enablement) and has GPU resources. 3. Check in the Curio UI whether the miner has CC sectors expected to be eligible for upgrades. What to include when asking for help: * deal UUID + piece CID * the full error line + surrounding logs * your `CURIO_LAYERS` for the ingest node and the GPU node ### Error: `skipped scheduling ParkPiece ... out of available Storage` Plain-English meaning: * Curio refused to start ParkPiece because there isn’t enough eligible free space on any attached storage path. What to do: * Verify free space on the storage paths attached to the node running ParkPiece. * Check `AllowTypes` / `DenyTypes` in `/sectorstorage.json` (if `unsealed` is effectively disallowed everywhere, ParkPiece can’t place data). * Check the Curio config knob `ParkPieceMinFreeStoragePercent` (default is 5%) and whether your current free space is below that threshold. See: * [Storage Configuration](/storage-configuration) ### Error: `no suitable data URL found for piece_id ` Plain-English meaning: * Curio has a piece to ingest, but can’t find any usable location for the bytes (most commonly: no data URL was stored for the parked piece). What to do: * Ensure you have added a data URL (and any required headers) for the deal/piece. * Verify the market/ingest layer is enabled on the node that runs ParkPiece. See: * [Storage Market](/curio-market/storage-market) * [Curio Market troubleshooting](/curio-market/troubleshooting) # Batch Sealing This page explains how to set up Curio batch sealing (extern/supraseal) {% hint style="danger" %} **Disclaimer:** Batch sealing is currently in **BETA**. Use with caution and expect potential issues or changes in future versions. Some additional manual system configuration is required. Batch sealing only supports **CC sectors** at the moment. The recommended workflow for onboarding deal data is to batch-seal CC sectors and then upgrade them with SnapDeals. If you enable batch sealing on a node but do **not** enable SnapDeals in the cluster, deals may be routed into the CC/batch pipeline which will seal empty sectors (discarding deal data). Make sure SnapDeals are enabled if you intend to onboard real data deals. {% endhint %} ## CC Scheduler (batch sealing only) Curio includes a **CC Scheduler** UI and DB table (`sectors_cc_scheduler`) used to decide how many CC sectors to queue for batch sealing per SP. * The **CC Scheduler is only used for batch sealing**. * Do not enable or rely on it for deals. Curio’s web UI exposes this as the **CC Scheduler** page/tab. ## Key Features * Seals multiple sectors (up to 128) in a single batch * Up to 16x better core utilisation efficiency * Optimized to utilize CPU and GPU resources efficiently * Uses raw NVMe devices for layer storage instead of RAM ## Requirements * CPU with at least 4 cores per CCX (AMD) or equivalent * NVMe drives with high IOPS (10-20M total IOPS recommended) * GPU for PC2 phase (NVIDIA RTX 3090 or better recommended) * 1GB hugepages configured (minimum 36 pages) * Ubuntu or compatible Linux distribution (**GCC 12 or 13 required**, doesn't need to be system-wide) * At least 256GB RAM, ALL MEMORY CHANNELS POPULATED * Without **all** memory channels populated sealing **performance will suffer drastically** * NUMA-Per-Socket (NPS) set to 1 ## Storage Recommendations You need 2 sets of NVMe drives: 1. Drives for layers: * Total 10-20M IOPS * Capacity for 11 x 32G x batchSize x pipelines * Raw unformatted block devices (SPDK will take them over) * Each drive should be able to sustain \~2GiB/s of writes * This requirement isn't understood well yet, it's possible that lower write rates are fine. More testing is needed. 2. Drives for P2 output: * With a filesystem * Fast with sufficient capacity (\~70G x batchSize x pipelines) * Can be remote storage if fast enough (\~500MiB/s/GPU) The following table shows the number of NVMe drives required for different batch sizes. The drive count column indicates `N + M` where `N` is the number of drives for layer data (SPDK) and `M` is the number of drives for P2 output (filesystem). The iops/drive column shows the minimum iops **per drive** required for the batch size. Batch size indicated with `2x` means dual-pipeline drive setup. IOPS requirements are calculated simply by dividing total target 10M IOPS by the number of drives. In reality, depending on CPU core speed this may be too low or higher than necessary. When ordering a system with barely enough IOPS plan to have free drive slots in case you need to add more drives later. | Batch Size | 3.84TB | 7.68TB | 12.8TB | 15.36TB | 30.72TB | | ------------ | ------ | ------ | ------ | ------- | ------- | | 32 | 4 + 1 | 2 + 1 | 1 + 1 | 1 + 1 | 1 + 1 | | ^ iops/drive | 2500K | 5000K | 10000K | 10000K | 10000K | | 64 (2x 32) | 7 + 2 | 4 + 1 | 2 + 1 | 2 + 1 | 1 + 1 | | ^ iops/drive | 1429K | 2500K | 5000K | 5000K | 10000K | | 128 (2x 64) | 13 + 3 | 7 + 2 | 4 + 1 | 4 + 1 | 2 + 1 | | ^ iops/drive | 770K | 1429K | 2500K | 2500K | 5000K | | 2x 128 | 26 + 6 | 13 + 3 | 8 + 2 | 7 + 2 | 4 + 1 | | ^ iops/drive | 385K | 770K | 1250K | 1429K | 2500K | ## Hardware Recommendations Currently, the community is trying to determine the best hardware configurations for batch sealing. Some general observations are: * Single socket systems will be easier to use at full capacity * You want a lot of NVMe slots, on PCIe Gen4 platforms with large batch sizes you may use 20-24 3.84TB NVMe drives * In general you'll want to make sure all memory channels are populated * You need 4\~8 physical cores (not threads) for batch-wide compute, then on each CCX you'll lose 1 core for a "coordinator" * Each thread computes 2 sectors * On zen2 and earlier hashers compute only one sector per thread * Large (many-core) CCX-es are typically better {% hint style="info" %} Please consider contributing to the [batch sealing hardware examples](https://github.com/filecoin-project/curio/discussions/140). {% endhint %} ## Setup ### Check NUMA setup: ```bash numactl --hardware ``` You should expect to see `available: 1 nodes (0)`. If you see more than one node you need to go into your UEFI and set `NUMA Per Socket` (or a similar setting) to 1. ### Configure hugepages: This can be done by adding the following to `/etc/default/grub`. You need 36 1G hugepages for the batch sealer. ```bash GRUB_CMDLINE_LINUX_DEFAULT="hugepages=36 default_hugepagesz=1G hugepagesz=1G" ``` Then run `sudo update-grub` and reboot the machine. Or at runtime: ```bash sudo sysctl -w vm.nr_hugepages=36 ``` Then check /proc/meminfo to verify the hugepages are available: ```bash cat /proc/meminfo | grep Huge ``` Expect output like: ``` AnonHugePages: 0 kB ShmemHugePages: 0 kB FileHugePages: 0 kB HugePages_Total: 36 HugePages_Free: 36 HugePages_Rsvd: 0 HugePages_Surp: 0 Hugepagesize: 1048576 kB ``` Check that `HugePages_Free` is equal to 36, the kernel can sometimes use some of the hugepages for other purposes. ### Dependencies CUDA 12.x or later is required (11.x won't work). The build process requires **GCC 12 or 13** — system-wide or as `gcc-12`/`g++-12` (or `gcc-13`/`g++-13`) installed locally. {% hint style="warning" %} **CUDA / GCC compatibility:** Your CUDA toolkit version determines which GCC it supports as a host compiler: * **CUDA 12.0–12.5:** supports GCC up to 12.x only → use `gcc-12`/`g++-12` * **CUDA 12.6+:** supports GCC up to 13.2 → either GCC 12 or 13 works * **CUDA 13.0+:** supports GCC 13+ → use `gcc-13`/`g++-13` If you get nvcc errors about unsupported compiler versions, check your CUDA/GCC pairing. {% endhint %} * On Arch install GCC 12 or 13 via your distro/AUR as appropriate * On Ubuntu/Debian install `gcc-12` and `g++-12` (or `gcc-13`/`g++-13`) ```shell # For CUDA 12.0–12.5: sudo apt install gcc-12 g++-12 # For CUDA 12.6+ or CUDA 13+: sudo apt install gcc-13 g++-13 ``` * In addition to general build dependencies (listed on the [installation page](/installation)), you need `libgmp-dev` and `libconfig++-dev` ```shell sudo apt install libgmp-dev libconfig++-dev ``` {% hint style="info" %} For SnapDeals “fastsnap” troubleshooting (fast TreeR path), you can check CPU/CUDA prerequisites with: ```bash curio test supra system-info ``` {% endhint %} ### Building Build and install the batch-capable Curio binary: ```bash make curio make sptool ``` ```shell make install ``` For calibnet ```bash make calibnet make calibnet-sptool ``` ```shell make install ``` {% hint style="warning" %} The build should be run on the target machine. Binaries won't be portable between CPU generations due to different AVX512 support. {% endhint %} ### Setup NVMe devices for SPDK: {% hint style="success" %} SPDK setup can be done automatically using the Curio CLI command: {% endhint %} ```bash sudo curio batch setup ``` This command will: * Download SPDK if not already available * Configure 1GB hugepages (36 pages by default) * Bind NVMe devices for use with batch sealing You can customize the number of hugepages: ```bash sudo curio batch setup --hugepages 36 --min-pages 36 ``` Alternatively, if you need to manually check SPDK status or unbind devices, you can use: ```bash cd extern/supraseal/deps/spdk-v24.05/ # Check status sudo ./scripts/setup.sh status # Manually run setup (not normally needed) sudo env NRHUGE=36 ./scripts/setup.sh ``` ### Benchmark NVME IOPS Please make sure to benchmark the raw NVME IOPS before proceeding with further configuration to verify that IOPS requirements are fulfilled. ```bash cd extern/supraseal/deps/spdk-v24.05/ # repeat -b with all devices you plan to use with supraseal # NOTE: You want to test with ALL devices so that you can see if there are any bottlenecks in the system ./build/examples/perf -b 0000:85:00.0 -b 0000:86:00.0... -q 64 -o 4096 -w randread -t 10 ``` The output should look like below ``` ======================================================== Latency(us) Device Information : IOPS MiB/s Average min max PCIE (0000:04:00.0) NSID 1 from core 0: 889422.78 3474.31 71.93 10.05 1040.94 PCIE (0000:41:00.0) NSID 1 from core 0: 890028.08 3476.67 71.88 10.69 1063.32 PCIE (0000:42:00.0) NSID 1 from core 0: 890035.08 3476.70 71.88 10.66 1001.86 PCIE (0000:86:00.0) NSID 1 from core 0: 889259.28 3473.67 71.95 10.62 1003.83 PCIE (0000:87:00.0) NSID 1 from core 0: 889179.58 3473.36 71.95 10.55 993.32 PCIE (0000:88:00.0) NSID 1 from core 0: 889272.18 3473.72 71.94 10.38 974.63 PCIE (0000:c1:00.0) NSID 1 from core 0: 889815.08 3475.84 71.90 10.97 1044.70 PCIE (0000:c2:00.0) NSID 1 from core 0: 889691.08 3475.36 71.91 11.04 1036.57 PCIE (0000:c3:00.0) NSID 1 from core 0: 890082.78 3476.89 71.88 10.44 1023.32 ======================================================== Total : 8006785.90 31276.51 71.91 10.05 1063.32 ``` With ideally >10M IOPS total for all devices. ### PC2 output storage Attach scratch space storage for PC2 output (batch sealer needs \~70GB per sector in batch - 32GiB for the sealed sector, and 36GiB for the cache directory with TreeC/TreeR and aux files) ## Usage 1. Start the Curio node with the batch sealer layer ```bash curio run --layers batch-machine1 ``` 2. Add a batch of CC sectors: ```bash curio seal start --now --cc --count 32 --actor f01234 --duration-days 365 ``` 3. Monitor progress - you should see a "Batch..." task running in the [Curio GUI](/curio-gui) 4. PC1 will take 3.5-5 hours, followed by PC2 on GPU 5. After batch completion, the storage will be released for the next batch ## Configuration * Run `curio calc batch-cpu` on the target machine to determine supported batch sizes for your CPU
Example batch-cpu output ``` # EPYC 7313 16-Core Processor root@udon:~# ./curio calc batch-cpu Basic CPU Information Processor count: 1 Core count: 16 Thread count: 32 Threads per core: 2 Cores per L3 cache (CCX): 4 L3 cache count (CCX count): 4 Hasher Threads per CCX: 6 Sectors per CCX: 12 --------- Batch Size: 16 sectors Required Threads: 8 Required CCX: 2 Required Cores: 6 hasher (+4 minimum for non-hashers) Enough cores available for hashers ✔ Non-hasher cores: 10 Enough cores for coordination ✔ pc1 writer: 1 pc1 reader: 2 pc1 orchestrator: 3 pc2 reader: 4 pc2 hasher: 5 pc2 hasher_cpu: 6 pc2 writer: 7 pc2 writer_cores: 3 c1 reader: 7 Unoccupied Cores: 0 { sectors = 16; coordinators = ( { core = 10; hashers = 2; }, { core = 12; hashers = 6; } ) } --------- Batch Size: 32 sectors Required Threads: 16 Required CCX: 3 Required Cores: 11 hasher (+4 minimum for non-hashers) Enough cores available for hashers ✔ Non-hasher cores: 5 Enough cores for coordination ✔ ! P2 hasher will share a core with P1 writer, performance may be impacted ! P2 hasher_cpu will share a core with P2 reader, performance may be impacted pc1 writer: 1 pc1 reader: 2 pc1 orchestrator: 3 pc2 reader: 0 pc2 hasher: 1 pc2 hasher_cpu: 0 pc2 writer: 4 pc2 writer_cores: 1 c1 reader: 0 Unoccupied Cores: 0 { sectors = 32; coordinators = ( { core = 5; hashers = 4; }, { core = 8; hashers = 6; }, { core = 12; hashers = 6; } ) } --------- Batch Size: 64 sectors Required Threads: 32 Required CCX: 6 Required Cores: 22 hasher (+4 minimum for non-hashers) Not enough cores available for hashers ✘ Batch Size: 128 sectors Required Threads: 64 Required CCX: 11 Required Cores: 43 hasher (+4 minimum for non-hashers) Not enough cores available for hashers ✘ ```
* Create a new layer configuration for the batch sealer, e.g. batch-machine1: ```toml [Subsystems] EnableBatchSeal = true [Seal] # This field is optional. In most setups, NVMe devices can be inferred automatically if this configuration is omitted. LayerNVMEDevices = [ "0000:88:00.0", "0000:86:00.0", # Add PCIe addresses for all NVMe devices to use ] # Set to your desired batch size (what the batch-cpu command says your CPU supports AND what you have nvme space for) BatchSealBatchSize = 32 # pipelines can be either 1 or 2; 2 pipelines double storage requirements but in correctly balanced systems makes # layer hashing run 100% of the time, nearly doubling throughput BatchSealPipelines = 2 # Set to true for Zen2 or older CPUs for compatibility SingleHasherPerThread = false ``` ### Environment Variables | Variable | Description | | ---------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `DISABLE_SPDK_SETUP=1` | When set, disables automatic SPDK setup (hugepage configuration and NVMe device binding) during supraseal initialization. Useful for advanced users who want to manually manage SPDK configuration, map drives, or control hugepage-numa assignment. | ## Optimization * Balance batch size, CPU cores, and NVMe drives to keep PC1 running constantly * Ensure sufficient GPU capacity to complete PC2 before next PC1 batch finishes * Monitor CPU, GPU and NVMe utilization to identify bottlenecks * Monitor hasher core utilisation ## Troubleshooting ### Node doesn't start / isn't visible in the UI * Ensure hugepages are configured correctly * Check NVMe device IOPS and capacity * If spdk setup fails, try to `wipefs -a` the NVMe devices (this will wipe partitions from the devices, be careful!) ### Performance issues You can monitor performance by looking at "hasher" core utilisation in e.g. `htop`. To identify hasher cores, call `curio calc supraseal-config --batch-size 128` (with the correct batch size), and look for `coordinators` ```go topology: ... { pc1: { writer = 1; ... hashers_per_core = 2; sector_configs: ( { sectors = 128; coordinators = ( { core = 59; hashers = 8; }, { core = 64; hashers = 14; }, { core = 72; hashers = 14; }, { core = 80; hashers = 14; }, { core = 88; hashers = 14; } ) } ) }, pc2: { ... } ``` In this example, cores 59, 64, 72, 80, and 88 are "coordinators", with two hashers per core, meaning that * In first group core 59 is a coordinator, cores 60-63 are hashers (4 hasher cores / 8 hasher threads) * In second group core 64 is a coordinator, cores 65-71 are hashers (7 hasher cores / 14 hasher threads) * And so on Coordinator cores will usually sit at 100% utilisation, hasher threads **SHOULD** sit at 100% utilisation, anything less indicates a bottleneck in the system, like not enough NVMe IOPS, not enough Memory bandwidth, or incorrect NUMA setup. To troubleshoot: * Read the requirements at the top of this page very carefully * [Benchmark NVME IOPS](#benchmark-nvme-iops) * Validate GPU setup if PC2 is slow * Review logs for any errors during batch processing ### Slower than expected NVMe speed If the [NVME Benchmark](#benchmark-nvme-iops) shows lower than expected IOPS, you can try formatting the NVMe devices with SPDK: ```bash cd extern/supraseal/deps/spdk-v24.05/ ./build/examples/nvme_manage ``` Go through the menus like this ``` NVMe Management Options [1: list controllers] [2: create namespace] [3: delete namespace] [4: attach namespace to controller] [5: detach namespace from controller] [6: format namespace or controller] [7: firmware update] [8: opal] [9: quit] 6 0000:04:00.00 SAMSUNG MZQL23T8HCLS-00A07 S64HNG0W829861 6 0000:41:00.00 SAMSUNG MZQL23T8HCLS-00A07 S64HNG0W829862 6 0000:42:00.00 SAMSUNG MZQL23T8HCLS-00A07 S64HNG0W829860 6 0000:86:00.00 SAMSUNG MZQL23T8HCLS-00A07 S64HNG0W829794 6 0000:87:00.00 SAMSUNG MZQL23T8HCLS-00A07 S64HNG0W829798 6 0000:88:00.00 SAMSUNG MZQL23T8HCLS-00A07 S64HNG0W829837 6 0000:c1:00.00 SAMSUNG MZQL23T8HCLS-00A07 S64HNG0W829795 6 0000:c2:00.00 SAMSUNG MZQL23T8HCLS-00A07 S64HNG0W829836 6 0000:c3:00.00 SAMSUNG MZQL23T8HCLS-00A07 S64HNG0W829797 6 0000:c4:00.00 SAMSUNG MZQL23T8HCLS-00A07 S64HNG0W829850 6 Please Input PCI Address(domain:bus:dev.func): 0000:c4:00.00 Please Input Namespace ID (1 - 32): 1 ## Select 1 Please Input Secure Erase Setting: 0: No secure erase operation requested 1: User data erase 2: Cryptographic erase 0 Supported LBA formats: 0: 512 data bytes 1: 4096 data bytes Please input LBA format index (0 - 1): 1 ## Select 4096 data bytes Warning: use this utility at your own risk. This command will format your namespace and all data will be lost. This command may take several minutes to complete, so do not interrupt the utility until it completes. Press 'Y' to continue with the format operation. y ``` Then you might see a difference in performance like this: ``` Latency(us) Device Information : IOPS MiB/s Average min max PCIE (0000:c1:00.0) NSID 1 from core 0: 721383.71 2817.91 88.68 11.20 591.51 ## before PCIE (0000:86:00.0) NSID 1 from core 0: 1205271.62 4708.09 53.07 11.87 446.84 ## after ``` *** ## Troubleshooting batch commit / all-or-nothing failures Support frequently sees failures of the form: * `all-or-nothing: Batch successes 67/68, Batch failing: [code=.. at idx=..] ...` What it means: * Some batch operations are **atomic**: a single failing entry can cause the whole batch to fail. How to debug (operator workflow): 1. In the Curio UI, locate the failing **CommitBatch / PreCommitBatch** task and open its details. 2. Correlate the batch failure with the per-sector tasks immediately before it (often the underlying failure is earlier). 3. Check chain/message errors: many failures are actually wallet funding, fee, or chain state errors. 4. Collect: * `sp_id` * the task ID(s) * the full error message including the `idx=...` How to recover safely: * If you suspect one “bad” sector is poisoning the batch, temporarily reduce batching (batch size / concurrency) to isolate. * Avoid manual DB edits unless you know exactly what you are doing (take a backup first). If you’re stuck: * follow `documentation/en/troubleshooting/collect-debug-info.md` and include the task IDs. # Scaling Curio cluster This page describes how to add additional nodes or miner IDs to a Curio cluster ## Migrating additional lotus-miner to Curio To migrate your second or later `lotus-miner` to an existing Curio cluster, you need to follow the same steps as before. The only exception would be that there is no need to install a new YugabyteDB cluster. After installing the `curio` binary on the `lotus-miner` node, you can run `curio guided-setup` to start the migration. ## Initialising additional Miner IDs in existing Curio cluster The process to initiate a new minerID on the network is same as when you [initialise a new Curio cluster with a new minerID.](/setup#initiating-a-new-curio-cluster) The only exception would be that there is no need to install a new YugabyteDB cluster ## Migrating lotus-worker to Curio cluster Once you have migrated a minerID to the Curio cluster, you would need to repurpose all of your `lotus-worker` nodes attached to the migrated minerID as Curio nodes. 1. [Install](/installation) `curio` binary on the `lotus-worker` node. 2. [Configure the service ENV file](/curio-service#environment-variables-configuration) with correct details. 3. Start the new `curio` node and verify in GUI that the new node is now part of the cluster. 4. [Attach the existing storage](/storage-configuration#attach-existing-storage-to-curio) to the Curio node. 5. Repeat for the rest of the workers. ## Adding nodes to Curio cluster To add new nodes to an existing Curio cluster, please follow the below process. * [Install](/installation) `curio` binary. * [Configure the service ENV file](/curio-service#environment-variables-configuration) with correct details. * Start the new `curio` node and verify in GUI that the new node is now part of the cluster. * [Attach any new or existing storage](/storage-configuration) if required. * Repeat for any additional nodes to be attached. # Curio GUI This page describe how to access the Curio GUI and what information is available there. ## Accessing Curio GUI The default port is `4701` to access the Curio GUI. To enable GUI on a Curio node user must start a Curio node with `gui` layer. This is a pre-built layer shipped with Curio binaries. ### Changing default GUI port You can change the default GUI port by setting a different IP address and port in the "base" layer of the configuration. We highly recommend not specifying the GUI address in other layers to avoid confusion. ``` curio config edit base ``` This will open the "base" layer in your default text editor. ``` # The address that should listen for Web GUI requests. # # type: string #GuiAddress = "0.0.0.0:4701" should be changed to below # The address that should listen for Web GUI requests. # # type: string GuiAddress = "127.0.0.1:4702" ``` Save the configuration and restart the Curio service on the node running GUI layer to access the GUI on the new address and port. ## GUI menu and dashboards {% hint style="danger" %} Curio web UI is currently under development. Some UI pages might change over time and could be different from the screenshots and description below. {% endhint %} ### Home Page

Curio home page

Chain Connectivity: Chain sync status of all available Lotus daemon nodes Cluster Machines: A quick list of all Curio nodes in the cluster PoRep Pipeline: A quick summary of cluster wide sealing sectors Actor Summary: Summary of minerIDs served by this Curio cluster ### Configuration page All configuration layers can be found and edited from the `configuration` page of the GUI. It also allows adding new configuration layers via GUI.

Configuration page

Editing a configuration layer

Editing a config layer

### Sectors Curio GUI can be used to browse the list of all sectors from all the minerID served by the Curio cluster. This is the GUI replacement of `lotus-miner sectors list` from `lotus-miner`.

Curio sector list

### PoRep Pipeline This page can be used to glance as the sectors current being sealed and historically sealed by Curio cluster. It details sector moving through the various sealing stages and status of each stage.

PoRep pipeline

Users can click on "DETAILS" and get more detailed information about a sector. This page will tell you about the pieces, storage etc.

Sector details

### Node details On the home page, in the "Cluster Machines" list, a user can click on the machine name to get a more detailed view of each node. It will list attached storage and status of tasks handled by that particular machine.

Node details

# Garbage Collection Garbage collection and cleanup process in Curio ## Sealing Pipeline cleanup The **SDRPipelineGC** is a periodic task in the system that ensures the efficiency and effectiveness of the storage sealing process. It is responsible for cleaning up completed entries in the sealing pipeline. ### Process The GC operates by removing sealing pipeline entries that have finished the sealing process. These entries have their metadata already stored in the long-term sector metadata table (also known as the `sectors_meta` table). This operation aids in maintaining the pipeline fluid and uncluttered, enhancing overall system performance. ### Handling failed sector In case a sector fails the sealing process, its corresponding entry can be manually removed via the web user interface (WebUI). This function allows for active management of pipeline entries and ensures that failed entries do not stall the pipeline. ## Storage Cleanup The `StorageGCMark` component is responsible for sweeping through all sector files in the system. A sector file will be marked in the `storage_removal_marks` table, under the following conditions: * The sector is not "pinned" in the `storage_gc_pins` table. (A sector pin indicates that the sector should not be removed, even after it has expired.) * The sector is not present in the sealing table called `sectors_sdr_pipeline`. * The sector has been labeled as a "failed" sector. (Note that "failed" sectors must first be removed from the pipeline table before they can have their data garbage collected.) * The sector is not present in the miner actor precommit sector set. * The sector is not present in the `Live` or `Unproven` sector sets. ### Approval and Removal Removal marks from the `StorageGCMark` process need to be approved separately. This approval is currently available only through the WebUI. In the future, there may be extensions to allow auto-approval, backed by custom policies for selection.

Storage GC approval

Once a removal mark has been granted approval, the periodic `StorageGCSweep` task will review all approved removal marks. This task will then proceed to delete the files which have been approved for removal. This final stage ensures that only necessary data remains in the system, optimizing storage and improving the overall system's functionality. ### Removing a failed sector For the removal of a sector that has failed the sealing process, users should go to the "PoRep" page on the WebUI and select the "DETAILS" link corresponding to the sector in question. This action will redirect them to a page where a "Remove" button is available. On clicking this button, the failed sector will be extricated from the SDR pipeline table, making it available for the StorageGCMark process to mark it for garbage collection. However, the removal of the sector will only ensue after it gains the requisite approval. This approval can be provided on the "Storage GC Info" page. On receipt of the approval, the StorageGCSweep task will review the mark and proceed to delete the sector files, effectively removing the failed sector from the system

How GC a failed sector

# Best Practices Curio best practices 1. YugybteDB backing the Curio cluster should be multi node to avoid single point of failure. 2. All miner IDs should be part of the base layer. We highly recommend not creating separate layers for different MinerIDs but using different layers for control addresses if required. 3. No worker should be dedicated to specific minerIDs. All Curio nodes should be setup to allow jobs for any minerIDs. 4. Multiple workers should be started with `--post` layer to allow fast wdPost and winPost turn around time. 5. We recommend running 1 GUI layer enabled node. A cluster wide GUI can be access via this node without putting any additional strain of read operations on the DB. 6. The unsealed and sealed copies should not be stored in the same storage location. Curio will allow automatic regeneration of sealed and unsealed in future if one is lost. 7. The **`DoSnap`** configuration should be set at the **base layer** to ensure that each node has the correct deal handling. This ensures consistency across all nodes and prevents misconfiguration. 8. Market requires a domain name in Curio. A domain name should be prepared before configuring the market. 9. Users utilizing a **reverse proxy** for delegated TLS configuration should enable the **market** service on multiple nodes to ensure **high availability** for the HTTP server. Additionally, **libp2p** should be enabled on multiple nodes to maintain **high availability** for peer-to-peer communications. 10. It is recommended to create a distinct layer for each market adapter (Deprecated), corresponding to each minerID. This configuration enables precise control, allowing for the assignment of specific minerIDs to either the Snap Deals pipeline or the PoRep pipeline. 11. It is advised to run the market adapter (Deprecated) on the same node that will execute the TreeD task for the PoRep pipeline or the Encode task for the Snap Deals pipeline. # Administration Administration guides for managing your Curio cluster This section covers administrative tasks for maintaining and managing your Curio cluster. ## Guides * [Node Maintenance & Cordoning](/administration/node-maintenance) - How to safely cordon nodes for maintenance without disrupting sealing pipelines * [YugabyteDB Backup](/administration/yugabyte-backup) - How to backup and restore your database * [YugabyteDB Troubleshooting](/administration/yugabyte-troubleshooting) - Common YB migration/runtime issues and how to resolve them * [Indexing / CheckIndex Troubleshooting](/administration/indexing-checkindex-troubleshooting) - Diagnose deal indexing and CheckIndex pipeline errors # Node Maintenance & Cordoning How to safely cordon Curio nodes for maintenance without disrupting in-progress sealing pipelines. ## What cordoning does Running `curio cordon` (or setting `unschedulable = true` in the WebUI) tells the Harmony scheduler to **stop scheduling new tasks** on a node. Tasks that are already running will finish, but no new work will be picked up. This is intentionally simple: cordon → wait for running tasks to finish → do maintenance → restart → uncordon. ## How cordoning affects the sealing pipeline Cordoning blocks **all** new task scheduling on the node, including pipeline continuation tasks like TreeD, TreeRC, SyntheticProofs, and Finalize. If a sector's data is **location-bound** to the cordoned node (the sector cache lives on local storage), those follow-up tasks cannot run on another node either. **What this means in practice:** * If you cordon a node that has sectors mid-pipeline (e.g., SDR completed but TreeRC not yet started), those sectors will be **paused** until the node is uncordoned. * The sectors are not lost — they will resume once the node is uncordoned and the scheduler picks them up again. * However, if the node stays cordoned for too long, sectors may **expire** (miss their precommit deadline), wasting the SDR work. {% hint style="warning" %} **Non-batched sealing operators:** SDR takes many hours. If you cordon a node and leave it cordoned past the sector's precommit deadline, that SDR work is lost. Plan maintenance windows accordingly. {% endhint %} ## Recommended workflows ### Quick maintenance (restart, upgrade, config change) For short interruptions where downtime is minutes, not hours: 1. **Check the pipeline** — in the WebUI, verify what stages are in progress on the node. 2. `curio cordon ` — stop new work from being scheduled. 3. **Wait** for currently-running tasks to complete (watch the WebUI or logs). 4. Do your maintenance (restart, upgrade, etc.). 5. `curio uncordon ` — resume scheduling. Pipeline sectors that were paused (waiting for their next stage) will resume automatically after uncordon. ### Planned extended maintenance (hours) If the node will be down for an extended period: 1. **Stop starting new sectors** first — pause deal intake or CC sector creation so no new SDR work begins on the node. 2. **Wait for in-progress pipelines to clear** — let sectors finish through Finalize before cordoning. Monitor via the pipeline view in the WebUI. 3. `curio cordon ` once pipelines are drained. 4. Do your maintenance. 5. `curio uncordon ` when ready. ### Decommissioning a node or long outage If you know a node will be offline for a long time (longer than precommit deadlines): 1. **Attach the node's storage to another node** — use `curio attach` on a different machine to make the sector data accessible elsewhere. 2. Other nodes with access to the storage paths can then pick up the remaining pipeline tasks. 3. Cordon and shut down the original node. See [Storage Configuration](/storage-configuration) for details on attaching storage. ## Key points to remember | Aspect | Behavior | | ----------------------------- | ---------------------------------------------------------------------------- | | Running tasks | Finish normally on the cordoned node | | New task scheduling | Blocked — no new work starts | | Location-bound pipeline tasks | Paused until uncordon (data is on local storage, other nodes can't run them) | | Sector data | Safe — nothing is deleted or moved by cordoning | | Precommit deadlines | Still apply — sectors can expire if cordoned too long | ## Common mistakes * **Cordoning during active SDR without a plan to uncordon quickly.** SDR is the longest pipeline stage. If you cordon right after SDR completes, the follow-up stages (TreeD, TreeRC, etc.) are blocked. Either wait for the full pipeline to clear first, or ensure you'll uncordon in time. * **Forgetting to uncordon after maintenance.** The node will sit idle, and any paused pipelines will remain stuck. Set a reminder or check the WebUI after maintenance. * **Cordoning when you should be migrating storage.** If the node is going away permanently, cordon alone won't help — you need to move or re-attach the storage paths so other nodes can access the sector data. # YugabyteDB Backup How to backup and restore your YugabyteDB database for Curio Maintaining regular backups of your YugabyteDB database is critical for disaster recovery and before performing operations like software downgrades. This guide covers essential backup and restore procedures for your Curio cluster's database. {% hint style="danger" %} **Always create a backup before running `curio toolbox downgrade`** or performing any major cluster operations. Database schema changes during upgrades may not be reversible without a backup. {% endhint %} ## Important: Curio uses a DB *and* a schema Curio connects to YugabyteDB over **YSQL** (Postgres protocol). Two configuration values matter: * **Database name** (`CURIO_DB_NAME` / `--db-name`) – default is `yugabyte` * **Schema** inside that database – Curio uses schema **`curio`** by default So, on a default setup, you typically want to back up the **`yugabyte`** database (and it will include the `curio` schema). Before running backups, confirm what Curio is configured to use: ```bash # If you run Curio with env vars echo "$CURIO_DB_NAME" # If Curio is installed on PATH, you can inspect flags/defaults curio --help | grep -E "db-(name|host|port)" ``` If you’re running Curio via systemd or containers, check the env/flags there: ```bash # systemd (example) systemctl cat curio | sed -n '1,200p' # docker compose (example) docker compose config | sed -n '1,200p' ``` ## Prerequisites * Access to your YugabyteDB cluster * The `ysql_dump` and `ysqlsh` utilities (included with YugabyteDB installation) * Sufficient disk space for backup files ## Backup Methods ### Method 1: Using ysql\_dump (Recommended) The `ysql_dump` utility creates a logical backup of your database that can be restored to any YugabyteDB cluster. #### Full database backup ```bash ysql_dump -h -p 5433 -U -d -F c -f curio_backup_$(date +%Y%m%d_%H%M%S).dump ``` **Parameters:** * `-h`: YugabyteDB host address * `-p`: YSQL port (default: 5433) * `-U`: Database username * `-d`: Database name (often `yugabyte` for Curio defaults) * `-F c`: Custom format (compressed, supports parallel restore) * `-f`: Output filename #### Example with typical Curio defaults ```bash ysql_dump -h 127.0.0.1 -p 5433 -U yugabyte -d yugabyte -F c -f curio_backup_$(date +%Y%m%d_%H%M%S).dump ``` #### Curio schema backup If you want to back up only Curio’s schema (and its data), not other schemas in the DB: ```bash ysql_dump -h -p 5433 -U -d --schema=curio -F c -f curio_schema_$(date +%Y%m%d_%H%M%S).dump ``` If you want **schema-only (DDL only)** for Curio (no data): ```bash ysql_dump -h -p 5433 -U -d --schema=curio --schema-only -f curio_schema_$(date +%Y%m%d_%H%M%S).sql ``` ### Method 2: Using ysqlsh with COPY For smaller exports or specific tables: ```bash ysqlsh -h -p 5433 -U -d -c "\COPY TO 'table_backup.csv' WITH CSV HEADER" ``` ## Restore Procedures ### Restore from ysql\_dump backup {% hint style="warning" %} Ensure all Curio nodes are **stopped** before restoring a backup. {% endhint %} #### Full restore (entire database) ```bash # Drop and recreate the database (ONLY if you intend to restore the full DB) ysqlsh -h -p 5433 -U -c "DROP DATABASE IF EXISTS ;" ysqlsh -h -p 5433 -U -c "CREATE DATABASE ;" # Restore from backup pg_restore -h -p 5433 -U -d -F c curio_backup_YYYYMMDD_HHMMSS.dump ``` #### Restore only the Curio schema If you created a Curio-schema dump (`--schema=curio`), restore into the existing DB: ```bash pg_restore -h -p 5433 -U -d -F c curio_schema_YYYYMMDD_HHMMSS.dump ``` If you created a Curio schema-only SQL file (`--schema-only`), apply it with ysqlsh: ```bash ysqlsh -h -p 5433 -U -d -f curio_schema_YYYYMMDD_HHMMSS.sql ``` ## Automated Backup Script (example) ```bash #!/bin/bash # curio-db-backup.sh BACKUP_DIR="/path/to/backups" YB_HOST="127.0.0.1" YB_PORT="5433" YB_USER="yugabyte" YB_DB="yugabyte" # Curio default DB name RETENTION_DAYS=7 mkdir -p "$BACKUP_DIR" BACKUP_FILE="$BACKUP_DIR/curio_backup_$(date +%Y%m%d_%H%M%S).dump" ysql_dump -h "$YB_HOST" -p "$YB_PORT" -U "$YB_USER" -d "$YB_DB" -F c -f "$BACKUP_FILE" if [ $? -eq 0 ]; then echo "Backup created successfully: $BACKUP_FILE" find "$BACKUP_DIR" -name "curio_backup_*.dump" -mtime +$RETENTION_DAYS -delete else echo "Backup failed!" exit 1 fi ``` ## Best Practices 1. **Regular backups**: schedule automated daily backups for production clusters 2. **Test restores**: periodically verify backups by performing test restores 3. **Off-site storage**: store backup copies in a different location or cloud storage 4. **Pre-upgrade backups**: always create a fresh backup before upgrading or downgrading Curio 5. **Monitor backup size**: ensure adequate storage capacity ## Troubleshooting ### Connection issues ```bash # Verify YugabyteDB is running yugabyted status # Check connectivity ysqlsh -h -p 5433 -U yugabyte -c "SELECT version();" ``` ### Permission errors Ensure your database user has sufficient privileges: ```sql GRANT ALL PRIVILEGES ON DATABASE TO ; ``` You may also need privileges on the `curio` schema and its objects depending on how your cluster is secured. ## Additional Resources * [YugabyteDB Backup and Restore Documentation](https://docs.yugabyte.com/preview/manage/backup-restore/) * [ysql\_dump Reference](https://docs.yugabyte.com/preview/admin/ysql-dump/) * [YugabyteDB Best Practices](https://docs.yugabyte.com/preview/develop/best-practices-ysql/) *** ## Multi-node cluster caveats (practical ops notes) If you run a multi-node Yugabyte cluster: * Validate that your backup method matches your recovery goal. * Logical dumps (`ysql_dump`) are typically easiest for portability. * If you rely on Yugabyte-native distributed backups/snapshots, ensure you test restores. Before a restore/downgrade: * stop all Curio writers * confirm no nodes are still writing tasks/metrics See also: * `documentation/en/administration/yugabyte-troubleshooting.md` # YugabyteDB troubleshooting Operational troubleshooting for YugabyteDB when used as Curio’s control plane. Many “Curio issues” are actually **control-plane issues** (YugabyteDB health, schema, connectivity, or performance). Before doing anything risky: * take a backup (see [YugabyteDB Backup](/administration/yugabyte-backup)) * stop Curio writers if you are restoring/downgrading *** ## 1) Quick health checklist On the Yugabyte nodes: * `yugabyted status` * check `yb-master` and `yb-tserver` logs for FATALs From a Curio node: ```bash ysqlsh -h "$CURIO_DB_HOST" -p "${CURIO_DB_PORT:-5433}" -U "$CURIO_DB_USER" -d "${CURIO_DB_NAME:-yugabyte}" -c "select 1;" ``` If the above is slow/intermittent, fix DB health first. *** ## 2) Connection string / multi-host setups Curio commonly uses a comma-separated host list: ```bash CURIO_DB_HOST=yugabyte1,yugabyte2,yugabyte3 CURIO_DB_PORT=5433 CURIO_DB_NAME=yugabyte ``` Tips: * Ensure all hosts are reachable from all Curio nodes. * If you disable load balancing in Curio, it may pin to the first host; keep that host stable. *** ## 3) ulimit / file descriptor exhaustion Symptoms: * DB is unstable under load, or tserver logs mention file descriptor issues. What to do: * Ensure you applied recommended `nofile` limits (see PDP guide and Yugabyte docs). * Restart services after changing persistent limits. *** ## 4) Upgrade / downgrade safety Rules of thumb: * Back up before changing Curio or Yugabyte versions. * Stop Curio before restoring a DB dump. * If a schema migration fails, capture: * the migration filename * full error output * Yugabyte version Common migration blocker: * `Rewriting of YB table is not yet implemented (SQLSTATE 0A000)` *** ## 5) Common SQLSTATE errors in support ### `SQLSTATE 23505` (duplicate key) Often indicates: * concurrency/retries inserting the same identity row. Action: * verify whether the app is progressing or stuck. * if stuck, correlate with task errors and consider a targeted restart of the relevant Curio subsystem. ### `relation "curio." does not exist` Usually indicates: * schema migrations didn’t run, or you are connected to the wrong DB name/schema. Action: * confirm `CURIO_DB_NAME` and that schema `curio` exists. *** ## 6) Backups Follow the backup guidance here: * [YugabyteDB Backup](/administration/yugabyte-backup) # Indexing & CheckIndex troubleshooting Troubleshooting indexing/IPNI and the CheckIndex task in Curio. This page is built from recurring support incidents in `#fil-curio-help`. ## What indexing is (plain English) Curio maintains a local index so retrieval can answer: “Which piece contains this multihash, and at what offset?”. If you use IPNI, Curio may also publish advertisements so the wider retrieval ecosystem can discover your content. ## What `CheckIndex` does `CheckIndex` is a background task that periodically checks whether indexing and announcements are complete and schedules follow-up work when something is missing or needs retrying. Code reference: * `tasks/indexing/task_check_indexes.go` ## First response playbook (safe steps) When indexing looks broken, do these in order: 1. Check DB health first * Many “indexing errors” are actually Yugabyte slowness/unavailability. * See: `documentation/en/administration/yugabyte-troubleshooting.md` 2. Check whether you *have unsealed data* available * Re-indexing typically requires access to an **unsealed copy**. * If your cluster intentionally does not keep unsealed data, expect errors like “no unsealed copy found” when attempting reindex. 3. Reduce overload (temporary) * If tasks are piling up, temporarily reduce indexing concurrency on the node(s) that run market/indexing. * Knobs live in Curio configuration (see `configuration/default-curio-configuration.md`). ## Common symptoms and what they usually mean ### Many `CheckIndex` tasks at once / “storm” Typical causes: * DB health issues causing retries. * A node repeatedly restarting and re-enqueueing work. * A configuration mismatch causing work to never complete. What to collect: * counts of tasks by type * the oldest CheckIndex task ID + its error logs ### `SQLSTATE 23505 duplicate key value violates unique constraint ...` Plain-English meaning: * Two workers attempted to create the same “identity” row concurrently. What to do: * Determine if it’s transient noise (pipeline still progresses) or if the same deal/piece is stuck. * Collect: deal UUID, piece CID, task IDs, and the full error span. ### `piece missing in indexstore` / `no unsealed copy of sector found for reindexing` Plain-English meaning: * Curio is being asked to build/serve an index for content, but cannot find the bytes required to construct it. What to do: * Confirm at least one storage path allows `unsealed` and has the data. * Confirm the node doing indexing can access the path. ## Quick DB inspection (YSQL) > These queries assume default schema `curio`. Count tasks by name: ```sql select name, count(*) from curio.harmony_task group by name order by count(*) desc; ``` List recent CheckIndex tasks: ```sql select id, posted_time, update_time, owner_id, retries from curio.harmony_task where name = 'CheckIndex' order by posted_time desc limit 50; ``` ## Last resort: deleting stuck CheckIndex tasks (use with caution) Only do this if you understand the impact and have a DB backup. 1. Stop Curio on the node running indexing/market. 2. Delete tasks: ```sql delete from curio.harmony_task where name = 'CheckIndex'; ``` 3. Start Curio again and monitor task creation rate. If the storm returns immediately, the root cause is still present (DB health, configuration, or missing data). # Logging This guide describes how to update logging preferences in Curio. ## Log file configuration Each Curio node generates Go logs which are directed to `/var/log/curio/curio.log` file by default if you are running Curio as a systemd service. ### Redirect Go logs to a file By default, Curio redirect all logs to the standard output if not running as a systemd service. To change this behaviour, add the following variable to the `.bashrc` file and restart the `curio` process to start redirecting all logs to the file. ```shell export GOLOG_OUTPUT=FILE >> ~/.bashrc export GOLOG_FILE="$HOME/curio.log" >> ~/.bashrc && source ~/.bashrc ``` ### Redirect Rust logs to a standard output By default the `fil_logger` library used by `rust-fil-proof` doesn’t log anything. You can change this by setting the RUST\_LOG environment variable to another level. This will show log output on stderr which can be redirected to a file either by systemd or in the shell while launching the `curio` process manually. Using systemd service file: ```bash export RUST_LOG=info >> /etc/curio.env systemctl restart curio.service ``` Running Curio manually: ```shell export RUST_LOG=info >> ~/.bashrc && source ~/.bashrc ``` The log-level can be chosen between 5 options: * trace * debug * info * warn * error ### Change logging verbosity The verbosity of the `curio` logs can be changed without restarting the service or process. The following command can be used to list different subsystems within the `curio` process and change the verbosity of individual subsystem to get more/less detailed logs. ```shell curio cli --machine log list ``` To change the verbosity, please run: ```shell curio cli --machine log set-level --system chain debug ``` The log-level can be chosen between 4 options: * debug * info * warn * error You can specify multiple subsystems to change the log level of multiple subsystems at once. ```shell curio cli --machine log set-level --system chain --system chainxchg debug ``` # Curio CLI Curio command-line interface Curio ships with 2 binaries called `curio` and `sptool` by default. ## Curio Binary The command line interface (CLI) for Curio operates slightly differently from typical software. Some commands, such as those related to storage, make API calls in the backend, while others interact directly with the database to perform the required actions. Commands that make API calls require authentication since the Curio API is permissioned. This authentication involves passing a token and address with the API call. The Curio CLI handles this authentication seamlessly, eliminating the need to set environment variables for the token and address. It retrieves the authentication and actor details from the database and generates the API call accordingly. This design allows you to make changes to a remote node without having direct access to it. For example, you can detach storage in Node 2 from Node 1. All such commands are nested under the `cli` subcommand. To perform actions on a remote machine, you must provide the correct IP address and port using the `--machine` flag in the format `--machine=10.0.0.1:12300`. This documentation explains the unique aspects of the Curio CLI, including its authentication process and how to interact with remote nodes. The `curio` CLI references can be found [here](/curio-cli/curio). ## Sptool Binary Certain administrative and monitoring operations require updating or fetching information about the minerID from the chain. These operations do not need access to the database and therefore are not included in the Curio binary. Instead, these commands are hosted under the `sptool` binary. The `sptool` binary provides an interface with the Filecoin blockchain for operations required by a storage provider. The `sptool` CLI references can be found [here](/curio-cli/sptool).
# Curio ``` NAME: curio - Filecoin decentralized storage network provider USAGE: curio [global options] command [command options] VERSION: 1.28.3 COMMANDS: cli Execute cli commands run Start a Curio process config Manage node config by layers. The layer 'base' will always be applied at Curio start-up. test Utility functions for testing web, gui Start Curio web interface guided-setup Run the guided setup for migrating from lotus-miner to Curio or Creating a new Curio miner seal Manage the sealing pipeline unseal Manage unsealed data market fetch-params Fetch proving parameters calc Math Utils toolbox Tool Box for Curio batch Manage batch sealing operations help, h Shows a list of commands or help for one command GLOBAL OPTIONS: --color use color in display output (default: depends on output being a TTY) --db-host value Comma-separated list of hostnames for yugabyte cluster (default: "127.0.0.1") [$CURIO_DB_HOST, $CURIO_HARMONYDB_HOSTS] --db-host-cql value Comma-separated list of hostnames for yugabyte cluster (default: <--db-host>) [$CURIO_DB_HOST_CQL] --db-name value Name of the Postgres database in Yugabyte cluster (default: "yugabyte") [$CURIO_DB_NAME, $CURIO_HARMONYDB_NAME] --db-user value Username for connecting to the Postgres database in Yugabyte cluster (default: "yugabyte") [$CURIO_DB_USER, $CURIO_HARMONYDB_USERNAME] --db-password value Password for connecting to the Postgres database in Yugabyte cluster (default: "yugabyte") [$CURIO_DB_PASSWORD, $CURIO_HARMONYDB_PASSWORD] --db-port value Port for connecting to the Postgres database in Yugabyte cluster (default: "5433") [$CURIO_DB_PORT, $CURIO_HARMONYDB_PORT] --db-cassandra-port value Port for connecting to the Cassandra database in Yugabyte cluster (default: 9042) [$CURIO_DB_CASSANDRA_PORT, $CURIO_INDEXDB_PORT] --db-load-balance Enable load balancing for connecting to the Postgres database in Yugabyte cluster (default: true) [$CURIO_DB_LOAD_BALANCE, $CURIO_HARMONYDB_LOAD_BALANCE] --db-readonly Open the database in read-only mode (skip schema upgrades and harmony_machines writes) (default: false) [$CURIO_DB_READONLY] --repo-path value (default: "~/.curio") [$CURIO_REPO_PATH] --vv enables very verbose mode, useful for debugging the CLI (default: false) --help, -h show help --version, -v print the version ``` ## curio cli ``` NAME: curio cli - Execute cli commands USAGE: curio cli [command options] COMMANDS: info Get Curio node info storage manage sector storage log Manage logging wait-api Wait for Curio api to come online stop Stop a running Curio process cordon Cordon a machine, set it to maintenance mode uncordon Uncordon a machine, resume scheduling index-sample Provides a sample of CIDs from an indexed piece help, h Shows a list of commands or help for one command OPTIONS: --machine value machine host:port (curio run --listen address) --help, -h show help ``` ### curio cli info ``` NAME: curio cli info - Get Curio node info USAGE: curio cli info [command options] OPTIONS: --help, -h show help ``` ### curio cli storage ``` NAME: curio cli storage - manage sector storage USAGE: curio cli storage [command options] DESCRIPTION: Sectors can be stored across many filesystem paths. These commands provide ways to manage the storage a Curio node will use to store sectors long term for proving (references as 'store') as well as how sectors will be stored while moving through the sealing pipeline (references as 'seal'). COMMANDS: attach attach local storage path detach detach local storage path list list local storage paths find find sector in the storage system generate-vanilla-proof generate vanilla proof for a sector redeclare redeclare sectors in a local storage path help, h Shows a list of commands or help for one command OPTIONS: --help, -h show help ``` #### curio cli storage attach ``` NAME: curio cli storage attach - attach local storage path USAGE: curio cli storage attach [command options] [path] DESCRIPTION: Storage can be attached to a Curio node using this command. The storage volume list is stored local to the Curio node in storage.json set in curio run. We do not recommend manually modifying this value without further understanding of the storage system. Each storage volume contains a configuration file which describes the capabilities of the volume. When the '--init' flag is provided, this file will be created using the additional flags. Weight A high weight value means data will be more likely to be stored in this path Seal Data for the sealing process will be stored here Store Finalized sectors that will be moved here for long term storage and be proven over time OPTIONS: --init initialize the path first (default: false) --weight value (for init) path weight (default: 10) --seal (for init) use path for sealing (default: false) --store (for init) use path for long-term storage (default: false) --max-storage value (for init) limit storage space for sectors (expensive for very large paths!) --groups value [ --groups value ] path group names --allow-to value [ --allow-to value ] path groups allowed to pull data from this path (allow all if not specified) --allow-types value [ --allow-types value ] file types to allow storing in this path --deny-types value [ --deny-types value ] file types to deny storing in this path --allow-miners value [ --allow-miners value ] miners to allow storing in this path --deny-miners value [ --deny-miners value ] miners to deny storing in this path --help, -h show help ``` #### curio cli storage detach ``` NAME: curio cli storage detach - detach local storage path USAGE: curio cli storage detach [command options] [path] OPTIONS: --really-do-it (default: false) --help, -h show help ``` #### curio cli storage list ``` NAME: curio cli storage list - list local storage paths USAGE: curio cli storage list [command options] OPTIONS: --local only list local storage paths (default: false) --help, -h show help ``` #### curio cli storage find ``` NAME: curio cli storage find - find sector in the storage system USAGE: curio cli storage find [command options] [miner address] [sector number] OPTIONS: --help, -h show help ``` #### curio cli storage generate-vanilla-proof ``` NAME: curio cli storage generate-vanilla-proof - generate vanilla proof for a sector USAGE: curio cli storage generate-vanilla-proof [command options] [miner address] [sector number] OPTIONS: --help, -h show help ``` #### curio cli storage redeclare ``` NAME: curio cli storage redeclare - redeclare sectors in a local storage path USAGE: curio cli storage redeclare [command options] [id] DESCRIPTION: --machine flag in cli command should point to the node where storage to redeclare is attached OPTIONS: --all redeclare all storage paths (default: false) --drop-missing Drop index entries with missing files (default: true) --help, -h show help ``` ### curio cli log ``` NAME: curio cli log - Manage logging USAGE: curio cli log [command options] COMMANDS: list List log systems set-level Set log level help, h Shows a list of commands or help for one command OPTIONS: --help, -h show help ``` #### curio cli log list ``` NAME: curio cli log list - List log systems USAGE: curio cli log list [command options] OPTIONS: --help, -h show help ``` #### curio cli log set-level ``` NAME: curio cli log set-level - Set log level USAGE: curio cli log set-level [command options] [level] DESCRIPTION: Set the log level for logging systems: The system flag can be specified multiple times. eg) log set-level --system chain --system chainxchg debug Available Levels: debug info warn error Environment Variables: GOLOG_LOG_LEVEL - Default log level for all log systems GOLOG_LOG_FMT - Change output log format (json, nocolor) GOLOG_FILE - Write logs to file GOLOG_OUTPUT - Specify whether to output to file, stderr, stdout or a combination, i.e. file+stderr OPTIONS: --system value [ --system value ] limit to log system --help, -h show help ``` ### curio cli wait-api ``` NAME: curio cli wait-api - Wait for Curio api to come online USAGE: curio cli wait-api [command options] OPTIONS: --timeout value duration to wait till fail (default: 30s) --help, -h show help ``` ### curio cli stop ``` NAME: curio cli stop - Stop a running Curio process USAGE: curio cli stop [command options] OPTIONS: --help, -h show help ``` ### curio cli cordon ``` NAME: curio cli cordon - Cordon a machine, set it to maintenance mode USAGE: curio cli cordon [command options] OPTIONS: --help, -h show help ``` ### curio cli uncordon ``` NAME: curio cli uncordon - Uncordon a machine, resume scheduling USAGE: curio cli uncordon [command options] OPTIONS: --help, -h show help ``` ### curio cli index-sample ``` NAME: curio cli index-sample - Provides a sample of CIDs from an indexed piece USAGE: curio cli index-sample [command options] piece-cid OPTIONS: --json output in json format (default: false) --help, -h show help ``` ## curio run ``` NAME: curio run - Start a Curio process USAGE: curio run [command options] OPTIONS: --listen value host address and port the worker api will listen on (default: "0.0.0.0:12300") [$CURIO_LISTEN] --nosync don't check full-node sync status (default: false) --manage-fdlimit manage open file limit (default: true) --layers value, -l value, --layer value [ --layers value, -l value, --layer value ] list of layers to be interpreted (atop defaults). Default: base [$CURIO_LAYERS] --name value custom node name [$CURIO_NODE_NAME] --help, -h show help ``` ## curio config ``` NAME: curio config - Manage node config by layers. The layer 'base' will always be applied at Curio start-up. USAGE: curio config [command options] COMMANDS: default, defaults Print default node config set, add, update, create Set a config layer or the base by providing a filename or stdin. get, cat, show Get a config layer by name. You may want to pipe the output to a file, or use 'less' list, ls List config layers present in the DB. interpret, view, stacked, stack Interpret stacked config layers by this version of curio, with system-generated comments. remove, rm, del, delete Remove a named config layer. edit edit a config layer new-cluster Create new configuration for a new cluster help, h Shows a list of commands or help for one command OPTIONS: --help, -h show help ``` ### curio config default ``` NAME: curio config default - Print default node config USAGE: curio config default [command options] OPTIONS: --no-comment don't comment default values (default: false) --help, -h show help ``` ### curio config set ``` NAME: curio config set - Set a config layer or the base by providing a filename or stdin. USAGE: curio config set [command options] a layer's file name OPTIONS: --title value title of the config layer (req'd for stdin) --help, -h show help ``` ### curio config get ``` NAME: curio config get - Get a config layer by name. You may want to pipe the output to a file, or use 'less' USAGE: curio config get [command options] layer name OPTIONS: --help, -h show help ``` ### curio config list ``` NAME: curio config list - List config layers present in the DB. USAGE: curio config list [command options] OPTIONS: --help, -h show help ``` ### curio config interpret ``` NAME: curio config interpret - Interpret stacked config layers by this version of curio, with system-generated comments. USAGE: curio config interpret [command options] a list of layers to be interpreted as the final config OPTIONS: --layers value [ --layers value ] comma or space separated list of layers to be interpreted (base is always applied) --help, -h show help ``` ### curio config remove ``` NAME: curio config remove - Remove a named config layer. USAGE: curio config remove [command options] OPTIONS: --help, -h show help ``` ### curio config edit ``` NAME: curio config edit - edit a config layer USAGE: curio config edit [command options] [layer name] OPTIONS: --editor value editor to use (default: "vim") [$EDITOR] --source value source config layer (default: ) --allow-overwrite allow overwrite of existing layer if source is a different layer (default: false) --no-source-diff save the whole config into the layer, not just the diff (default: false) --no-interpret-source do not interpret source layer (default: true if --source is set) --help, -h show help ``` ### curio config new-cluster ``` NAME: curio config new-cluster - Create new configuration for a new cluster USAGE: curio config new-cluster [command options] [SP actor address...] OPTIONS: --help, -h show help ``` ## curio test ``` NAME: curio test - Utility functions for testing USAGE: curio test [command options] COMMANDS: window-post, wd, windowpost, wdpost Compute a proof-of-spacetime for a sector (requires the sector to be pre-sealed). These will not send to the chain. debug Collection of debugging utilities supra Supra consensus testing utilities help, h Shows a list of commands or help for one command OPTIONS: --help, -h show help ``` ### curio test window-post ``` NAME: curio test window-post - Compute a proof-of-spacetime for a sector (requires the sector to be pre-sealed). These will not send to the chain. USAGE: curio test window-post [command options] COMMANDS: here, cli Compute WindowPoSt for performance and configuration testing. task, scheduled, schedule, async, asynchronous Test the windowpost scheduler by running it on the next available curio. If tasks fail all retries, you will need to ctrl+c to exit. vanilla Compute WindowPoSt vanilla proofs and verify them. help, h Shows a list of commands or help for one command OPTIONS: --help, -h show help ``` #### curio test window-post here ``` NAME: curio test window-post here - Compute WindowPoSt for performance and configuration testing. USAGE: curio test window-post here [command options] [deadline index] DESCRIPTION: Note: This command is intended to be used to verify PoSt compute performance. It will not send any messages to the chain. Since it can compute any deadline, output may be incorrectly timed for the chain. OPTIONS: --deadline value deadline to compute WindowPoSt for (default: 0) --layers value [ --layers value ] list of layers to be interpreted (atop defaults). Default: base --partition value partition to compute WindowPoSt for (default: 0) --addr value SP ID to compute WindowPoSt for --help, -h show help ``` #### curio test window-post task ``` NAME: curio test window-post task - Test the windowpost scheduler by running it on the next available curio. If tasks fail all retries, you will need to ctrl+c to exit. USAGE: curio test window-post task [command options] OPTIONS: --deadline value deadline to compute WindowPoSt for (default: 0) --layers value [ --layers value ] list of layers to be interpreted (atop defaults). Default: base --addr value SP ID to compute WindowPoSt for --help, -h show help ``` #### curio test window-post vanilla ``` NAME: curio test window-post vanilla - Compute WindowPoSt vanilla proofs and verify them. USAGE: curio test window-post vanilla [command options] OPTIONS: --deadline value deadline to compute WindowPoSt for (default: 0) --layers value [ --layers value ] list of layers to be interpreted (atop defaults). Default: base --partition value partition to compute WindowPoSt for (default: 0) --addr value SP ID to compute WindowPoSt for --help, -h show help ``` ### curio test debug ``` NAME: curio test debug - Collection of debugging utilities USAGE: curio test debug [command options] COMMANDS: ipni-piece-chunks generate ipni chunks from a file debug-snsvc proofsvc-client Interact with the remote proof service help, h Shows a list of commands or help for one command OPTIONS: --help, -h show help ``` #### curio test debug ipni-piece-chunks ``` NAME: curio test debug ipni-piece-chunks - generate ipni chunks from a file USAGE: curio test debug ipni-piece-chunks [command options] OPTIONS: --help, -h show help ``` #### curio test debug debug-snsvc ``` NAME: curio test debug debug-snsvc USAGE: curio test debug debug-snsvc [command options] COMMANDS: deposit Deposit FIL into the Router contract (client) client-initiate-withdrawal Initiate a withdrawal request from the client's deposit client-complete-withdrawal Complete a pending client withdrawal after the withdrawal window elapses client-cancel-withdrawal Cancel a pending client withdrawal request redeem-client Redeem a client voucher (service role) redeem-provider Redeem a provider voucher (provider role) service-initiate-withdrawal Initiate a withdrawal request from the service pool service-complete-withdrawal Complete a pending service withdrawal after the withdrawal window elapses service-cancel-withdrawal Cancel a pending service withdrawal request service-deposit Deposit funds into the service pool (service role) get-client-state Query the state of a client get-provider-state Query the state of a provider get-service-state Query the service state create-client-voucher Create a client voucher create-provider-voucher Create a provider voucher propose-service-actor Propose a new service actor accept-service-actor Accept a proposed service actor validate-client-voucher Validate a client voucher signature validate-provider-voucher Validate a provider voucher signature help, h Shows a list of commands or help for one command OPTIONS: --help, -h show help ``` **curio test debug debug-snsvc deposit** ``` NAME: curio test debug debug-snsvc deposit - Deposit FIL into the Router contract (client) USAGE: curio test debug debug-snsvc deposit [command options] OPTIONS: --from value Sender address --amount value Amount in FIL --help, -h show help ``` **curio test debug debug-snsvc client-initiate-withdrawal** ``` NAME: curio test debug debug-snsvc client-initiate-withdrawal - Initiate a withdrawal request from the client's deposit USAGE: curio test debug debug-snsvc client-initiate-withdrawal [command options] OPTIONS: --from value Client sender address --amount value Withdrawal amount (in FIL) --help, -h show help ``` **curio test debug debug-snsvc client-complete-withdrawal** ``` NAME: curio test debug debug-snsvc client-complete-withdrawal - Complete a pending client withdrawal after the withdrawal window elapses USAGE: curio test debug debug-snsvc client-complete-withdrawal [command options] OPTIONS: --from value Client sender address --help, -h show help ``` **curio test debug debug-snsvc client-cancel-withdrawal** ``` NAME: curio test debug debug-snsvc client-cancel-withdrawal - Cancel a pending client withdrawal request USAGE: curio test debug debug-snsvc client-cancel-withdrawal [command options] OPTIONS: --from value Client sender address --help, -h show help ``` **curio test debug debug-snsvc redeem-client** ``` NAME: curio test debug debug-snsvc redeem-client - Redeem a client voucher (service role) USAGE: curio test debug debug-snsvc redeem-client [command options] OPTIONS: --from value Service sender address --client value Client actor --amount value Cumulative amount (FIL) --nonce value Voucher nonce (default: 0) --sig value Voucher signature (hex) --help, -h show help ``` **curio test debug debug-snsvc redeem-provider** ``` NAME: curio test debug debug-snsvc redeem-provider - Redeem a provider voucher (provider role) USAGE: curio test debug debug-snsvc redeem-provider [command options] OPTIONS: --from value Provider sender address --provider value Provider actor --amount value Cumulative amount (FIL) --nonce value Voucher nonce (default: 0) --sig value Voucher signature (hex) --help, -h show help ``` **curio test debug debug-snsvc service-initiate-withdrawal** ``` NAME: curio test debug debug-snsvc service-initiate-withdrawal - Initiate a withdrawal request from the service pool USAGE: curio test debug debug-snsvc service-initiate-withdrawal [command options] OPTIONS: --amount value Withdrawal amount (in FIL) --from value Service sender address --help, -h show help ``` **curio test debug debug-snsvc service-complete-withdrawal** ``` NAME: curio test debug debug-snsvc service-complete-withdrawal - Complete a pending service withdrawal after the withdrawal window elapses USAGE: curio test debug debug-snsvc service-complete-withdrawal [command options] OPTIONS: --from value Service sender address --help, -h show help ``` **curio test debug debug-snsvc service-cancel-withdrawal** ``` NAME: curio test debug debug-snsvc service-cancel-withdrawal - Cancel a pending service withdrawal request USAGE: curio test debug debug-snsvc service-cancel-withdrawal [command options] OPTIONS: --from value Service sender address --help, -h show help ``` **curio test debug debug-snsvc service-deposit** ``` NAME: curio test debug debug-snsvc service-deposit - Deposit funds into the service pool (service role) USAGE: curio test debug debug-snsvc service-deposit [command options] OPTIONS: --from value Service sender address --amount value Amount to deposit (FIL) --help, -h show help ``` **curio test debug debug-snsvc get-client-state** ``` NAME: curio test debug debug-snsvc get-client-state - Query the state of a client USAGE: curio test debug debug-snsvc get-client-state [command options] OPTIONS: --client value Client actor address --help, -h show help ``` **curio test debug debug-snsvc get-provider-state** ``` NAME: curio test debug debug-snsvc get-provider-state - Query the state of a provider USAGE: curio test debug debug-snsvc get-provider-state [command options] OPTIONS: --provider value Provider actor address --help, -h show help ``` **curio test debug debug-snsvc get-service-state** ``` NAME: curio test debug debug-snsvc get-service-state - Query the service state USAGE: curio test debug debug-snsvc get-service-state [command options] OPTIONS: --help, -h show help ``` **curio test debug debug-snsvc create-client-voucher** ``` NAME: curio test debug debug-snsvc create-client-voucher - Create a client voucher USAGE: curio test debug debug-snsvc create-client-voucher [command options] OPTIONS: --client value Client actor address --amount value Amount to redeem (FIL) --help, -h show help ``` **curio test debug debug-snsvc create-provider-voucher** ``` NAME: curio test debug debug-snsvc create-provider-voucher - Create a provider voucher USAGE: curio test debug debug-snsvc create-provider-voucher [command options] OPTIONS: --provider value Provider actor address --amount value Amount to redeem (FIL) --nonce value Voucher nonce (default: 0) --service value Service sender address --help, -h show help ``` **curio test debug debug-snsvc propose-service-actor** ``` NAME: curio test debug debug-snsvc propose-service-actor - Propose a new service actor USAGE: curio test debug debug-snsvc propose-service-actor [command options] OPTIONS: --from value Service sender address --new-service-actor value New service actor address --help, -h show help ``` **curio test debug debug-snsvc accept-service-actor** ``` NAME: curio test debug debug-snsvc accept-service-actor - Accept a proposed service actor USAGE: curio test debug debug-snsvc accept-service-actor [command options] OPTIONS: --from value Service sender address --help, -h show help ``` **curio test debug debug-snsvc validate-client-voucher** ``` NAME: curio test debug debug-snsvc validate-client-voucher - Validate a client voucher signature USAGE: curio test debug debug-snsvc validate-client-voucher [command options] OPTIONS: --client value Client actor address --amount value Cumulative amount (FIL) --nonce value Voucher nonce (default: 0) --sig value Voucher signature (hex) --help, -h show help ``` **curio test debug debug-snsvc validate-provider-voucher** ``` NAME: curio test debug debug-snsvc validate-provider-voucher - Validate a provider voucher signature USAGE: curio test debug debug-snsvc validate-provider-voucher [command options] OPTIONS: --provider value Provider actor address --amount value Cumulative amount (FIL) --nonce value Voucher nonce (default: 0) --sig value Voucher signature (hex) --help, -h show help ``` #### curio test debug proofsvc-client ``` NAME: curio test debug proofsvc-client - Interact with the remote proof service USAGE: curio test debug proofsvc-client [command options] COMMANDS: create-voucher Create a client voucher submit Submit a proof request status Check proof status help, h Shows a list of commands or help for one command OPTIONS: --help, -h show help ``` **curio test debug proofsvc-client create-voucher** ``` NAME: curio test debug proofsvc-client create-voucher - Create a client voucher USAGE: curio test debug proofsvc-client create-voucher [command options] OPTIONS: --client value --amount value --help, -h show help ``` **curio test debug proofsvc-client submit** ``` NAME: curio test debug proofsvc-client submit - Submit a proof request USAGE: curio test debug proofsvc-client submit [command options] OPTIONS: --c1 value path to lotus-bench c1 json --miner value miner address --client-id value (default: 0) --nonce value (default: 0) --amount value --sig value --help, -h show help ``` **curio test debug proofsvc-client status** ``` NAME: curio test debug proofsvc-client status - Check proof status USAGE: curio test debug proofsvc-client status [command options] OPTIONS: --id value --help, -h show help ``` ### curio test supra ``` NAME: curio test supra - Supra consensus testing utilities USAGE: curio test supra [command options] COMMANDS: system-info Display CPU and CUDA information relevant for supraseal tree-r-file Test tree-r-file snap-encode Test snap-encode help, h Shows a list of commands or help for one command OPTIONS: --help, -h show help ``` #### curio test supra system-info ``` NAME: curio test supra system-info - Display CPU and CUDA information relevant for supraseal USAGE: curio test supra system-info [command options] OPTIONS: --help, -h show help ``` #### curio test supra tree-r-file ``` NAME: curio test supra tree-r-file - Test tree-r-file USAGE: curio test supra tree-r-file [command options] OPTIONS: --last-layer-filename value Last layer filename --data-filename value Data filename --output-dir value Output directory --sector-size value Sector size (default: 0) --help, -h show help ``` #### curio test supra snap-encode ``` NAME: curio test supra snap-encode - Test snap-encode USAGE: curio test supra snap-encode [command options] OPTIONS: --sealed-filename value Sealed filename --unsealed-filename value Unsealed filename --update-filename value Update filename --sector-size value Sector size (bytes). Supported: 2048, 8388608, 549755813888, 34359738368, 68719476736 (default: 0) --commd value Unsealed CommD CID (v1) --commk value SectorKey CommR (commK) CID (v1) --membuffer Use memory buffer instead of disk (load and store) (default: false) --help, -h show help ``` ## curio web ``` NAME: curio web - Start Curio web interface USAGE: curio web [command options] DESCRIPTION: Start an instance of Curio web interface. This creates the 'web' layer if it does not exist, then calls run with that layer. In --db-readonly / CURIO_DB_READONLY mode, no config layer is written; the GUI is enabled in-memory. OPTIONS: --gui-listen value Address to listen for the GUI on (default: "0.0.0.0:4701") --nosync don't check full-node sync status (default: false) --layers value [ --layers value ] list of layers to be interpreted (atop defaults). Default: base --help, -h show help ``` ## curio guided-setup ``` NAME: curio guided-setup - Run the guided setup for migrating from lotus-miner to Curio or Creating a new Curio miner USAGE: curio guided-setup [command options] OPTIONS: --help, -h show help ``` ## curio seal ``` NAME: curio seal - Manage the sealing pipeline USAGE: curio seal [command options] COMMANDS: start Start new sealing operations manually events List pipeline events help, h Shows a list of commands or help for one command OPTIONS: --help, -h show help ``` ### curio seal start ``` NAME: curio seal start - Start new sealing operations manually USAGE: curio seal start [command options] OPTIONS: --actor value Specify actor address to start sealing sectors for --now Start sealing sectors for all actors now (not on schedule) (default: false) --cc Start sealing new CC sectors (default: false) --count value Number of sectors to start (default: 1) --synthetic Use synthetic PoRep (default: false) --layers value [ --layers value ] list of layers to be interpreted (atop defaults). Default: base --duration-days value, -d value How long to commit sectors for (default: 1278 (3.5 years)) --help, -h show help ``` ### curio seal events ``` NAME: curio seal events - List pipeline events USAGE: curio seal events [command options] OPTIONS: --actor value Filter events by actor address; lists all if not specified --sector value Filter events by sector number; requires --actor to be specified (default: 0) --last value Limit output to the last N events (default: 100) --help, -h show help ``` ## curio unseal ``` NAME: curio unseal - Manage unsealed data USAGE: curio unseal [command options] COMMANDS: info Get information about unsealed data list-sectors List data from the sectors_unseal_pipeline and sectors_meta tables set-target-state Set the target unseal state for a sector set-target-by-pieces Set the target unseal state for sectors containing the given piece CIDs check Check data integrity in unsealed sector files help, h Shows a list of commands or help for one command OPTIONS: --help, -h show help ``` ### curio unseal info ``` NAME: curio unseal info - Get information about unsealed data USAGE: curio unseal info [command options] [minerAddress] [sectorNumber] OPTIONS: --help, -h show help ``` ### curio unseal list-sectors ``` NAME: curio unseal list-sectors - List data from the sectors_unseal_pipeline and sectors_meta tables USAGE: curio unseal list-sectors [command options] OPTIONS: --sp-id value, -s value Filter by storage provider ID (default: 0) --output value, -o value Output file path (default: stdout) --help, -h show help ``` ### curio unseal set-target-state ``` NAME: curio unseal set-target-state - Set the target unseal state for a sector USAGE: curio unseal set-target-state [command options] DESCRIPTION: Set the target unseal state for a specific sector. : The storage provider ID : The sector number : The target state (true, false) The unseal target state indicates to curio how an unsealed copy of the sector should be maintained. If the target state is true, curio will ensure that the sector is unsealed. If the target state is false, curio will ensure that there is no unsealed copy of the sector. If the target state is none, curio will not change the current state of the sector. Currently when the curio will only start new unseal processes when the target state changes from another state to true. When the target state is false, and an unsealed sector file exists, the GC mark step will create a removal mark for the unsealed sector file. The file will only be removed after the removal mark is accepted. OPTIONS: --help, -h show help ``` ### curio unseal set-target-by-pieces ``` NAME: curio unseal set-target-by-pieces - Set the target unseal state for sectors containing the given piece CIDs USAGE: curio unseal set-target-by-pieces [command options] [piece-cid ...] DESCRIPTION: Resolve each piece CID to sector(s) via market_piece_deal, then set target_unseal_state for those sectors. Accepts piece CID v1 or v2. Use --target-state to specify the desired state (true or false). Request all at once to minimize the sectors needed to be unsealed (if you have pieces stored in multiple sectors). Use --stdin to read piece CIDs one per line from stdin (for large lists). OPTIONS: --target-state Target state: true (ensure unsealed), false (ensure no unsealed copy) (default: true) --stdin Read piece CIDs one per line from stdin instead of from arguments (default: false) --help, -h show help ``` ### curio unseal check ``` NAME: curio unseal check - Check data integrity in unsealed sector files USAGE: curio unseal check [command options] DESCRIPTION: Create a check task for a specific sector, wait for its completion, and output the result. : The storage provider ID : The sector number OPTIONS: --help, -h show help ``` ## curio market ``` NAME: curio market USAGE: curio market [command options] COMMANDS: seal start sealing a deal sector early add-url Add URL to fetch data for offline deals move-to-escrow Moves funds from the deal collateral wallet into escrow with the storage market actor ddo Create a new offline verified DDO deal for Curio help, h Shows a list of commands or help for one command OPTIONS: --help, -h show help ``` ### curio market seal ``` NAME: curio market seal - start sealing a deal sector early USAGE: curio market seal [command options] OPTIONS: --actor value Specify actor address to start sealing sectors for --synthetic Use synthetic PoRep (default: false) --help, -h show help ``` ### curio market add-url ``` NAME: curio market add-url - Add URL to fetch data for offline deals USAGE: curio market add-url [command options] OPTIONS: --file value CSV file location to use for multiple deal input. Each line in the file should be in the format 'uuid,raw size,url,header1,header2...' --header HEADER, -H HEADER [ --header HEADER, -H HEADER ] Custom HEADER to include in the HTTP request --url URL, -u URL URL to send the request to --help, -h show help ``` ### curio market move-to-escrow ``` NAME: curio market move-to-escrow - Moves funds from the deal collateral wallet into escrow with the storage market actor USAGE: curio market move-to-escrow [command options] OPTIONS: --actor value Specify actor address to start sealing sectors for --max-fee value maximum fee in FIL user is willing to pay for this message (default: "0.5") --wallet value Specify wallet address to send the funds from --help, -h show help ``` ### curio market ddo ``` NAME: curio market ddo - Create a new offline verified DDO deal for Curio USAGE: curio market ddo [command options] OPTIONS: --actor value Specify actor address for the deal --remove-unsealed-copy Remove unsealed copies of sector containing this deal (default: false) --skip-ipni-announce indicates that deal index should not be announced to the IPNI (default: false) --start-epoch value start epoch by when the deal should be proved by provider on-chain (default: 2 days from now) (default: 0) --help, -h show help ``` ## curio fetch-params ``` NAME: curio fetch-params - Fetch proving parameters USAGE: curio fetch-params [command options] [sectorSize] OPTIONS: --help, -h show help ``` ## curio calc ``` NAME: curio calc - Math Utils USAGE: curio calc [command options] COMMANDS: batch-cpu Analyze and display the layout of batch sealer threads supraseal-config Generate a supra_seal configuration help, h Shows a list of commands or help for one command OPTIONS: --actor value --help, -h show help ``` ### curio calc batch-cpu ``` NAME: curio calc batch-cpu - Analyze and display the layout of batch sealer threads USAGE: curio calc batch-cpu [command options] DESCRIPTION: Analyze and display the layout of batch sealer threads on your CPU. It provides detailed information about CPU utilization for batch sealing operations, including core allocation, thread distribution for different batch sizes. OPTIONS: --dual-hashers (default: true) --help, -h show help ``` ### curio calc supraseal-config ``` NAME: curio calc supraseal-config - Generate a supra_seal configuration USAGE: curio calc supraseal-config [command options] DESCRIPTION: Generate a supra_seal configuration for a given batch size. This command outputs a configuration expected by SupraSeal. Main purpose of this command is for debugging and testing. The config can be used directly with SupraSeal binaries to test it without involving Curio. OPTIONS: --dual-hashers Zen3 and later supports two sectors per thread, set to false for older CPUs (default: true) --batch-size value, -b value (default: 0) --help, -h show help ``` ## curio toolbox ``` NAME: curio toolbox - Tool Box for Curio USAGE: curio toolbox [command options] COMMANDS: fix-msg Updated DB with message data missing from chain node downgrade Downgrade a cluster's database to a previous software version. fix-boost-migration Fix Boost migration help, h Shows a list of commands or help for one command OPTIONS: --help, -h show help ``` ### curio toolbox fix-msg ``` NAME: curio toolbox fix-msg - Updated DB with message data missing from chain node USAGE: curio toolbox fix-msg [command options] OPTIONS: --all Update data for messages in wait queue (default: false) --help, -h show help ``` ### curio toolbox downgrade ``` NAME: curio toolbox downgrade - Downgrade a cluster's database to a previous software version. USAGE: curio toolbox downgrade [command options] DESCRIPTION: If, however, the upgrade has a serious bug and you need to downgrade, first shutdown all nodes in your cluster and then run this command. Finally, only start downgraded nodes. OPTIONS: --last_good_date value YYYYMMDD when your cluster had the preferred schema. Ex: 20251128 (default: 0) --help, -h show help ``` ### curio toolbox fix-boost-migration ``` NAME: curio toolbox fix-boost-migration - Fix Boost migration USAGE: curio toolbox fix-boost-migration [command options] OPTIONS: --check check how many entries need to be fixed (default: false) --db-file value location of boost.db file --boostd-data-hosts value [ --boostd-data-hosts value ] yugabyte hosts to connect to over cassandra interface eg '127.0.0.1' --boostd-data-username value yugabyte username to connect to over cassandra interface eg 'cassandra' --boostd-data-password value yugabyte password to connect to over cassandra interface eg 'cassandra' --help, -h show help ``` ## curio batch ``` NAME: curio batch - Manage batch sealing operations USAGE: curio batch [command options] COMMANDS: setup Setup SPDK for batch sealing (configures hugepages and binds NVMe devices) help, h Shows a list of commands or help for one command OPTIONS: --help, -h show help ``` ### curio batch setup ``` NAME: curio batch setup - Setup SPDK for batch sealing (configures hugepages and binds NVMe devices) USAGE: curio batch setup [command options] DESCRIPTION: Setup SPDK for batch sealing operations. This command automatically: - Downloads SPDK if not already available - Configures 1GB hugepages (36 pages minimum) - Binds NVMe devices for use with SupraSeal Requires root/sudo access for SPDK setup operations. OPTIONS: --hugepages value Number of 1GB hugepages to configure (default: 36) --min-pages value Minimum number of hugepages required (default: 36) --help, -h show help ``` # Sptool ``` NAME: sptool - Manage Filecoin Miner Actor USAGE: sptool [global options] command [command options] VERSION: 1.28.3 COMMANDS: actor Manage Filecoin Miner Actor Metadata info Print miner actor info sectors interact with sector store proving View proving information toolbox some tools to fix some problems help, h Shows a list of commands or help for one command GLOBAL OPTIONS: --log-level value (default: "info") --actor value miner actor to manage [$SP_ADDRESS] --verbose, --vv enable verbose logging (default: false) --help, -h show help --version, -v print the version ``` ## sptool actor ``` NAME: sptool actor - Manage Filecoin Miner Actor Metadata USAGE: sptool actor [command options] COMMANDS: set-addresses, set-addrs set addresses that your miner can be publicly dialed on withdraw withdraw available balance to beneficiary repay-debt pay down a miner's debt set-peer-id set the peer id of your miner set-owner Set owner address (this command should be invoked twice, first with the old owner as the senderAddress, and then with the new owner) control Manage control addresses propose-change-worker Propose a worker address change confirm-change-worker Confirm a worker address change compact-allocated compact allocated sectors bitfield propose-change-beneficiary Propose a beneficiary address change confirm-change-beneficiary Confirm a beneficiary address change new-miner Initializes a new miner actor help, h Shows a list of commands or help for one command OPTIONS: --help, -h show help ``` ### sptool actor set-addresses ``` NAME: sptool actor set-addresses - set addresses that your miner can be publicly dialed on USAGE: sptool actor set-addresses [command options] OPTIONS: --from value optionally specify the account to send the message from --gas-limit value set gas limit (default: 0) --unset unset address (default: false) --help, -h show help ``` ### sptool actor withdraw ``` NAME: sptool actor withdraw - withdraw available balance to beneficiary USAGE: sptool actor withdraw [command options] [amount (FIL)] OPTIONS: --confidence value number of block confirmations to wait for (default: 5) --beneficiary send withdraw message from the beneficiary address (default: false) --help, -h show help ``` ### sptool actor repay-debt ``` NAME: sptool actor repay-debt - pay down a miner's debt USAGE: sptool actor repay-debt [command options] [amount (FIL)] OPTIONS: --from value optionally specify the account to send funds from --help, -h show help ``` ### sptool actor set-peer-id ``` NAME: sptool actor set-peer-id - set the peer id of your miner USAGE: sptool actor set-peer-id [command options] OPTIONS: --gas-limit value set gas limit (default: 0) --help, -h show help ``` ### sptool actor set-owner ``` NAME: sptool actor set-owner - Set owner address (this command should be invoked twice, first with the old owner as the senderAddress, and then with the new owner) USAGE: sptool actor set-owner [command options] [newOwnerAddress senderAddress] OPTIONS: --really-do-it Actually send transaction performing the action (default: false) --help, -h show help ``` ### sptool actor control ``` NAME: sptool actor control - Manage control addresses USAGE: sptool actor control [command options] COMMANDS: list Get currently set control addresses. Note: This excludes most roles as they are not known to the immediate chain state. set Set control address(-es) help, h Shows a list of commands or help for one command OPTIONS: --help, -h show help ``` #### sptool actor control list ``` NAME: sptool actor control list - Get currently set control addresses. Note: This excludes most roles as they are not known to the immediate chain state. USAGE: sptool actor control list [command options] OPTIONS: --verbose (default: false) --help, -h show help ``` #### sptool actor control set ``` NAME: sptool actor control set - Set control address(-es) USAGE: sptool actor control set [command options] [...address] OPTIONS: --really-do-it Actually send transaction performing the action (default: false) --help, -h show help ``` ### sptool actor propose-change-worker ``` NAME: sptool actor propose-change-worker - Propose a worker address change USAGE: sptool actor propose-change-worker [command options] [address] OPTIONS: --really-do-it Actually send transaction performing the action (default: false) --help, -h show help ``` ### sptool actor confirm-change-worker ``` NAME: sptool actor confirm-change-worker - Confirm a worker address change USAGE: sptool actor confirm-change-worker [command options] [address] OPTIONS: --really-do-it Actually send transaction performing the action (default: false) --help, -h show help ``` ### sptool actor compact-allocated ``` NAME: sptool actor compact-allocated - compact allocated sectors bitfield USAGE: sptool actor compact-allocated [command options] OPTIONS: --mask-last-offset value Mask sector IDs from 0 to 'highest_allocated - offset' (default: 0) --mask-upto-n value Mask sector IDs from 0 to 'n' (default: 0) --really-do-it Actually send transaction performing the action (default: false) --help, -h show help ``` ### sptool actor propose-change-beneficiary ``` NAME: sptool actor propose-change-beneficiary - Propose a beneficiary address change USAGE: sptool actor propose-change-beneficiary [command options] [beneficiaryAddress quota expiration] OPTIONS: --really-do-it Actually send transaction performing the action (default: false) --overwrite-pending-change Overwrite the current beneficiary change proposal (default: false) --actor value specify the address of miner actor --help, -h show help ``` ### sptool actor confirm-change-beneficiary ``` NAME: sptool actor confirm-change-beneficiary - Confirm a beneficiary address change USAGE: sptool actor confirm-change-beneficiary [command options] [minerID] OPTIONS: --really-do-it Actually send transaction performing the action (default: false) --existing-beneficiary send confirmation from the existing beneficiary address (default: false) --new-beneficiary send confirmation from the new beneficiary address (default: false) --help, -h show help ``` ### sptool actor new-miner ``` NAME: sptool actor new-miner - Initializes a new miner actor USAGE: sptool actor new-miner [command options] OPTIONS: --worker value, -w value worker key to use for new miner initialisation --owner value, -o value owner key to use for new miner initialisation --from value, -f value address to send actor(miner) creation message from --sector-size value specify sector size to use for new miner initialisation --confidence value number of block confirmations to wait for (default: 5) --help, -h show help ``` ## sptool info ``` NAME: sptool info - Print miner actor info USAGE: sptool info [command options] OPTIONS: --help, -h show help ``` ## sptool sectors ``` NAME: sptool sectors - interact with sector store USAGE: sptool sectors [command options] COMMANDS: status Get the on-chain status of a sector by its number list List sectors precommits Print on-chain precommit info check-expire Inspect expiring sectors expired Get or cleanup expired sectors extend Extend expiring sectors while not exceeding each sector's max life terminate Forcefully terminate a sector (WARNING: This means losing power and pay a one-time termination penalty(including collateral) for the terminated sector) compact-partitions removes dead sectors from partitions and reduces the number of partitions used if possible help, h Shows a list of commands or help for one command OPTIONS: --help, -h show help ``` ### sptool sectors status ``` NAME: sptool sectors status - Get the on-chain status of a sector by its number USAGE: sptool sectors status [command options] OPTIONS: --help, -h show help ``` ### sptool sectors list ``` NAME: sptool sectors list - List sectors USAGE: sptool sectors list [command options] OPTIONS: --help, -h show help ``` ### sptool sectors precommits ``` NAME: sptool sectors precommits - Print on-chain precommit info USAGE: sptool sectors precommits [command options] OPTIONS: --help, -h show help ``` ### sptool sectors check-expire ``` NAME: sptool sectors check-expire - Inspect expiring sectors USAGE: sptool sectors check-expire [command options] OPTIONS: --cutoff value skip sectors whose current expiration is more than epochs from now, defaults to 60 days (default: 172800) --help, -h show help ``` ### sptool sectors expired ``` NAME: sptool sectors expired - Get or cleanup expired sectors USAGE: sptool sectors expired [command options] OPTIONS: --expired-epoch value epoch at which to check sector expirations (default: WinningPoSt lookback epoch) --help, -h show help ``` ### sptool sectors extend ``` NAME: sptool sectors extend - Extend expiring sectors while not exceeding each sector's max life USAGE: sptool sectors extend [command options] DESCRIPTION: NOTE: --new-expiration, --from and --to flags have multiple formats: 1. Absolute epoch number: 2. Relative epoch number: +, e.g. +1000, means 1000 epochs from now 3. Relative day number: +d, e.g. +10d, means 10 days from now The --extension flag has two formats: 1. Number of epochs to extend by: 2. Number of days to extend by: d Extensions will be clamped at either the maximum sector extension of 3.5 years/1278 days or the sector's maximum lifetime which currently is 5 years. OPTIONS: --from value only consider sectors whose current expiration epoch is in the range of [from, to], defaults to: now + 120 (1 hour) (default: "+120") --to value only consider sectors whose current expiration epoch is in the range of [from, to], defaults to: now + 92160 (32 days) (default: "+92160") --sector-file value provide a file containing one sector number in each line, ignoring above selecting criteria --exclude value optionally provide a file containing excluding sectors --extension value try to extend selected sectors by this number of epochs, defaults to 540 days (default: "540d") --new-expiration value try to extend selected sectors to this epoch, ignoring extension --only-cc only extend CC sectors (useful for making sector ready for snap upgrade) (default: false) --no-cc don't extend CC sectors (exclusive with --only-cc) (default: false) --drop-claims drop claims for sectors that can be extended, but only by dropping some of their verified power claims (default: false) --tolerance value don't try to extend sectors by fewer than this number of epochs, defaults to 7 days (default: 20160) --max-fee value use up to this amount of FIL for one message. pass this flag to avoid message congestion. (default: "0") --max-sectors value the maximum number of sectors contained in each message (default: 500) --really-do-it pass this flag to really extend sectors, otherwise will only print out json representation of parameters (default: false) --help, -h show help ``` ### sptool sectors terminate ``` NAME: sptool sectors terminate - Forcefully terminate a sector (WARNING: This means losing power and pay a one-time termination penalty(including collateral) for the terminated sector) USAGE: sptool sectors terminate [command options] [sectorNum1 sectorNum2 ...] OPTIONS: --actor value specify the address of miner actor --really-do-it pass this flag if you know what you are doing (default: false) --from value specify the address to send the terminate message from --help, -h show help ``` ### sptool sectors compact-partitions ``` NAME: sptool sectors compact-partitions - removes dead sectors from partitions and reduces the number of partitions used if possible USAGE: sptool sectors compact-partitions [command options] OPTIONS: --deadline value the deadline to compact the partitions in (default: 0) --partitions value [ --partitions value ] list of partitions to compact sectors in --really-do-it Actually send transaction performing the action (default: false) --help, -h show help ``` ## sptool proving ``` NAME: sptool proving - View proving information USAGE: sptool proving [command options] COMMANDS: info View current state information deadlines View the current proving period deadlines information deadline View the current proving period deadline information by its index faults View the currently known proving faulty sectors information help, h Shows a list of commands or help for one command OPTIONS: --help, -h show help ``` ### sptool proving info ``` NAME: sptool proving info - View current state information USAGE: sptool proving info [command options] OPTIONS: --help, -h show help ``` ### sptool proving deadlines ``` NAME: sptool proving deadlines - View the current proving period deadlines information USAGE: sptool proving deadlines [command options] OPTIONS: --all, -a Count all sectors (only live sectors are counted by default) (default: false) --help, -h show help ``` ### sptool proving deadline ``` NAME: sptool proving deadline - View the current proving period deadline information by its index USAGE: sptool proving deadline [command options] OPTIONS: --sector-nums, -n Print sector/fault numbers belonging to this deadline (default: false) --bitfield, -b Print partition bitfield stats (default: false) --help, -h show help ``` ### sptool proving faults ``` NAME: sptool proving faults - View the currently known proving faulty sectors information USAGE: sptool proving faults [command options] OPTIONS: --help, -h show help ``` ## sptool toolbox ``` NAME: sptool toolbox - some tools to fix some problems USAGE: sptool toolbox [command options] COMMANDS: spark Manage Smart Contract PeerID used by Spark mk12-client mk12 client for Curio mk20-client mk20 client for Curio stats Curio Node Stats help, h Shows a list of commands or help for one command OPTIONS: --help, -h show help ``` ### sptool toolbox spark ``` NAME: sptool toolbox spark - Manage Smart Contract PeerID used by Spark USAGE: sptool toolbox spark [command options] COMMANDS: delete-peer Delete PeerID from Spark Smart Contract help, h Shows a list of commands or help for one command OPTIONS: --help, -h show help ``` #### sptool toolbox spark delete-peer ``` NAME: sptool toolbox spark delete-peer - Delete PeerID from Spark Smart Contract USAGE: sptool toolbox spark delete-peer [command options] OPTIONS: --really-do-it Send the message to the smart contract (default: false) --help, -h show help ``` ### sptool toolbox mk12-client ``` NAME: sptool toolbox mk12-client - mk12 client for Curio USAGE: sptool toolbox mk12-client [command options] COMMANDS: init Initialise curio mk12 client repo deal Make an online deal with Curio deal-status offline-deal Make an offline deal with Curio allocate Create new allocation[s] for verified deals list-allocations Lists all allocations for a client address(wallet) market-add Add funds to the Storage Market actor market-withdraw Withdraw funds from the Storage Market actor commp generate-rand-car creates a randomly generated dense car wallet Manage mk12 client wallets help, h Shows a list of commands or help for one command OPTIONS: --mk12-client-repo value repo directory for mk12 client (default: "~/.curio-client") [$CURIO_MK12_CLIENT_REPO] --help, -h show help ``` #### sptool toolbox mk12-client init ``` NAME: sptool toolbox mk12-client init - Initialise curio mk12 client repo USAGE: sptool toolbox mk12-client init [command options] OPTIONS: --help, -h show help ``` #### sptool toolbox mk12-client deal ``` NAME: sptool toolbox mk12-client deal - Make an online deal with Curio USAGE: sptool toolbox mk12-client deal [command options] OPTIONS: --http-url value http url to CAR file --http-headers value [ --http-headers value ] http headers to be passed with the request (e.g key=value) --car-size value size of the CAR file: required for online deals (default: 0) --provider value storage provider on-chain address --commp value commp of the CAR file --piece-size value size of the CAR file as a padded piece (default: 0) --payload-cid value root CID of the CAR file --start-epoch-head-offset value start epoch by when the deal should be proved by provider on-chain after current chain head (default: 0) --start-epoch value start epoch by when the deal should be proved by provider on-chain (default: 0) --duration value duration of the deal in epochs (default: 518400) --provider-collateral value deal collateral that storage miner must put in escrow; if empty, the min collateral for the given piece size will be used (default: 0) --storage-price value storage price in attoFIL per epoch per GiB (default: 1) --verified whether the deal funds should come from verified client data-cap (default: false) --remove-unsealed-copy indicates that an unsealed copy of the sector in not required for fast retrieval (default: false) --wallet value wallet address to be used to initiate the deal --skip-ipni-announce indicates that deal index should not be announced to the IPNI(Network Indexer) (default: false) --http make the deal over HTTP instead of libp2p (default: false) --help, -h show help ``` #### sptool toolbox mk12-client deal-status ``` NAME: sptool toolbox mk12-client deal-status USAGE: sptool toolbox mk12-client deal-status [command options] OPTIONS: --provider value storage provider on-chain address --deal-uuid value --wallet value the wallet address that was used to sign the deal proposal --http make the deal over HTTP instead of libp2p (default: false) --help, -h show help ``` #### sptool toolbox mk12-client offline-deal ``` NAME: sptool toolbox mk12-client offline-deal - Make an offline deal with Curio USAGE: sptool toolbox mk12-client offline-deal [command options] OPTIONS: --provider value storage provider on-chain address --commp value commp of the CAR file --piece-size value size of the CAR file as a padded piece (default: 0) --payload-cid value root CID of the CAR file --start-epoch-head-offset value start epoch by when the deal should be proved by provider on-chain after current chain head (default: 0) --start-epoch value start epoch by when the deal should be proved by provider on-chain (default: 0) --duration value duration of the deal in epochs (default: 518400) --provider-collateral value deal collateral that storage miner must put in escrow; if empty, the min collateral for the given piece size will be used (default: 0) --storage-price value storage price in attoFIL per epoch per GiB (default: 1) --verified whether the deal funds should come from verified client data-cap (default: false) --remove-unsealed-copy indicates that an unsealed copy of the sector in not required for fast retrieval (default: false) --wallet value wallet address to be used to initiate the deal --skip-ipni-announce indicates that deal index should not be announced to the IPNI(Network Indexer) (default: false) --http make the deal over HTTP instead of libp2p (default: false) --help, -h show help ``` #### sptool toolbox mk12-client allocate ``` NAME: sptool toolbox mk12-client allocate - Create new allocation[s] for verified deals USAGE: sptool toolbox mk12-client allocate [command options] DESCRIPTION: The command can accept a CSV formatted file in the format 'pieceCid,pieceSize,miner,tmin,tmax,expiration' OPTIONS: --miner value, -m value, --provider value, -p value [ --miner value, -m value, --provider value, -p value ] storage provider address[es] --piece-cid value, --piece value data piece-cid to create the allocation --piece-size value, --size value piece size to create the allocation (default: 0) --wallet value the wallet address that will used create the allocation --quiet do not print the allocation list (default: false) --term-min value, --tmin value The minimum duration which the provider must commit to storing the piece to avoid early-termination penalties (epochs). Default is 180 days. (default: 518400) --term-max value, --tmax value The maximum period for which a provider can earn quality-adjusted power for the piece (epochs). Default is 5 years. (default: 5256000) --expiration value The latest epoch by which a provider must commit data before the allocation expires (epochs). Default is 60 days. (default: 172800) --piece-file value, --pf value file containing piece information to create the allocation. Each line in the file should be in the format 'pieceCid,pieceSize,miner,tmin,tmax,expiration' --batch-size value number of extend requests per batch. If set incorrectly, this will lead to out of gas error (default: 500) --confidence value number of block confirmations to wait for (default: 5) --assume-yes, -y, --yes automatic yes to prompts; assume 'yes' as answer to all prompts and run non-interactively (default: false) --evm-client-contract value f4 address of EVM contract to spend DataCap from --json, -j print output in JSON format (default: false) --help, -h show help ``` #### sptool toolbox mk12-client list-allocations ``` NAME: sptool toolbox mk12-client list-allocations - Lists all allocations for a client address(wallet) USAGE: sptool toolbox mk12-client list-allocations [command options] OPTIONS: --miner value, -m value, --provider value, -p value Storage provider address. If provided, only allocations against this minerID will be printed --wallet value the wallet address that will used create the allocation --json, -j print output in JSON format (default: false) --help, -h show help ``` #### sptool toolbox mk12-client market-add ``` NAME: sptool toolbox mk12-client market-add - Add funds to the Storage Market actor USAGE: sptool toolbox mk12-client market-add [command options] DESCRIPTION: Send signed message to add funds for the default wallet to the Storage Market actor. Uses 2x current BaseFee and a maximum fee of 1 nFIL. This is an experimental utility, do not use in production. OPTIONS: --assume-yes, -y, --yes automatic yes to prompts; assume 'yes' as answer to all prompts and run non-interactively (default: false) --wallet value move balance from this wallet address to its market actor --help, -h show help ``` #### sptool toolbox mk12-client market-withdraw ``` NAME: sptool toolbox mk12-client market-withdraw - Withdraw funds from the Storage Market actor USAGE: sptool toolbox mk12-client market-withdraw [command options] OPTIONS: --assume-yes, -y, --yes automatic yes to prompts; assume 'yes' as answer to all prompts and run non-interactively (default: false) --wallet value move balance to this wallet address from its market actor --help, -h show help ``` #### sptool toolbox mk12-client commp ``` NAME: sptool toolbox mk12-client commp USAGE: sptool toolbox mk12-client commp [command options] OPTIONS: --help, -h show help ``` #### sptool toolbox mk12-client generate-rand-car ``` NAME: sptool toolbox mk12-client generate-rand-car - creates a randomly generated dense car USAGE: sptool toolbox mk12-client generate-rand-car [command options] OPTIONS: --size value, -s value The size of the data to turn into a car (default: 8000000) --chunksize value, -c value Size of chunking that should occur (default: 512) --maxlinks value, -l value Max number of leaves per level (default: 8) --help, -h show help ``` #### sptool toolbox mk12-client wallet ``` NAME: sptool toolbox mk12-client wallet - Manage mk12 client wallets USAGE: sptool toolbox mk12-client wallet [command options] COMMANDS: new Generate a new key of the given type list List wallet address balance Get account balance export export keys import import keys default, get-default Get default wallet address set-default Set default wallet address delete Delete an account from the wallet sign Sign a message help, h Shows a list of commands or help for one command OPTIONS: --help, -h show help ``` **sptool toolbox mk12-client wallet new** ``` NAME: sptool toolbox mk12-client wallet new - Generate a new key of the given type USAGE: sptool toolbox mk12-client wallet new [command options] [bls|secp256k1|delegated (default secp256k1)] OPTIONS: --help, -h show help ``` **sptool toolbox mk12-client wallet list** ``` NAME: sptool toolbox mk12-client wallet list - List wallet address USAGE: sptool toolbox mk12-client wallet list [command options] OPTIONS: --addr-only, -a Only print addresses (default: false) --id, -i Output ID addresses (default: false) --help, -h show help ``` **sptool toolbox mk12-client wallet balance** ``` NAME: sptool toolbox mk12-client wallet balance - Get account balance USAGE: sptool toolbox mk12-client wallet balance [command options] [address] OPTIONS: --help, -h show help ``` **sptool toolbox mk12-client wallet export** ``` NAME: sptool toolbox mk12-client wallet export - export keys USAGE: sptool toolbox mk12-client wallet export [command options] [address] OPTIONS: --help, -h show help ``` **sptool toolbox mk12-client wallet import** ``` NAME: sptool toolbox mk12-client wallet import - import keys USAGE: sptool toolbox mk12-client wallet import [command options] [ (optional, will read from stdin if omitted)] OPTIONS: --format value specify input format for key (default: "hex-lotus") --as-default import the given key as your new default key (default: false) --help, -h show help ``` **sptool toolbox mk12-client wallet default** ``` NAME: sptool toolbox mk12-client wallet default - Get default wallet address USAGE: sptool toolbox mk12-client wallet default [command options] OPTIONS: --help, -h show help ``` **sptool toolbox mk12-client wallet set-default** ``` NAME: sptool toolbox mk12-client wallet set-default - Set default wallet address USAGE: sptool toolbox mk12-client wallet set-default [command options] [address] OPTIONS: --help, -h show help ``` **sptool toolbox mk12-client wallet delete** ``` NAME: sptool toolbox mk12-client wallet delete - Delete an account from the wallet USAGE: sptool toolbox mk12-client wallet delete [command options]
OPTIONS: --help, -h show help ``` **sptool toolbox mk12-client wallet sign** ``` NAME: sptool toolbox mk12-client wallet sign - Sign a message USAGE: sptool toolbox mk12-client wallet sign [command options] OPTIONS: --help, -h show help ``` ### sptool toolbox mk20-client ``` NAME: sptool toolbox mk20-client - mk20 client for Curio USAGE: sptool toolbox mk20-client [command options] COMMANDS: init Initialise curio mk12 client repo commp deal Make a mk20 deal with Curio pdp-deal Make a mk20 PDP deal with Curio aggregate Create a new aggregate from a list of CAR files upload Upload a file to the storage provider chunk-upload Upload a file in chunks to the storage provider deal-status Get status of a Mk20 deal help, h Shows a list of commands or help for one command OPTIONS: --mk12-client-repo value repo directory for mk12 client (default: "~/.curio-client") [$CURIO_MK12_CLIENT_REPO] --help, -h show help ``` #### sptool toolbox mk20-client init ``` NAME: sptool toolbox mk20-client init - Initialise curio mk12 client repo USAGE: sptool toolbox mk20-client init [command options] OPTIONS: --help, -h show help ``` #### sptool toolbox mk20-client commp ``` NAME: sptool toolbox mk20-client commp USAGE: sptool toolbox mk20-client commp [command options] OPTIONS: --help, -h show help ``` #### sptool toolbox mk20-client deal ``` NAME: sptool toolbox mk20-client deal - Make a mk20 deal with Curio USAGE: sptool toolbox mk20-client deal [command options] OPTIONS: --http-url value http url to CAR file --http-headers value [ --http-headers value ] http headers to be passed with the request (e.g key=value) --provider value storage provider on-chain address --pcidv2 value pcidv2 of the CAR file --duration value duration of the deal in epochs (default: 518400) --market-address value market contract address of the deal --market-deal-id value market deal ID (default: 0) --allocation value allocation id of the deal (default: 0) --indexing indicates that an deal should be indexed (default: true) --wallet value wallet address to be used to initiate the deal --announce indicates that deal should be announced to the IPNI(Network Indexer) (default: true) --aggregate value aggregate file path for the deal --put used HTTP put as data source (default: false) --help, -h show help ``` #### sptool toolbox mk20-client pdp-deal ``` NAME: sptool toolbox mk20-client pdp-deal - Make a mk20 PDP deal with Curio USAGE: sptool toolbox mk20-client pdp-deal [command options] OPTIONS: --http-url value http url to CAR file --http-headers value [ --http-headers value ] http headers to be passed with the request (e.g key=value) --provider value PDP providers's URL --pcidv2 value pcidv2 of the CAR file --wallet value wallet address to be used to initiate the deal --aggregate value aggregate file path for the deal --put used HTTP put as data source (default: false) --add-piece add piece (default: false) --add-dataset add dataset (default: false) --remove-piece remove piece (default: false) --remove-dataset remove dataset (default: false) --record-keeper value record keeper address --piece-id value [ --piece-id value ] root IDs --dataset-id value dataset IDs (default: 0) --help, -h show help ``` #### sptool toolbox mk20-client aggregate ``` NAME: sptool toolbox mk20-client aggregate - Create a new aggregate from a list of CAR files USAGE: sptool toolbox mk20-client aggregate [command options] OPTIONS: --files value [ --files value ] list of CAR files to aggregate --piece-size value piece size of the aggregate (default: 0) --out output the aggregate file (default: true) --help, -h show help ``` #### sptool toolbox mk20-client upload ``` NAME: sptool toolbox mk20-client upload - Upload a file to the storage provider USAGE: sptool toolbox mk20-client upload [command options] OPTIONS: --provider value PDP providers's URL --deal value deal id to upload to --help, -h show help ``` #### sptool toolbox mk20-client chunk-upload ``` NAME: sptool toolbox mk20-client chunk-upload - Upload a file in chunks to the storage provider USAGE: sptool toolbox mk20-client chunk-upload [command options] OPTIONS: --provider value storage provider on-chain address --deal value deal id to upload to --chunk-size value chunk size to be used for the upload (default: "4 MiB") --wallet value wallet address to be used to initiate the deal --help, -h show help ``` #### sptool toolbox mk20-client deal-status ``` NAME: sptool toolbox mk20-client deal-status - Get status of a Mk20 deal USAGE: sptool toolbox mk20-client deal-status [command options] OPTIONS: --provider value PDP providers's URL --id value deal id --wallet value wallet address to be used to initiate the deal --help, -h show help ``` ### sptool toolbox stats ``` NAME: sptool toolbox stats - Curio Node Stats USAGE: sptool toolbox stats [command options] OPTIONS: --help, -h show help ``` # Developer Developer-oriented references for Curio build, API, and tooling behavior. This section contains technical references intended for Curio developers and advanced operators. * [API](/developer/api) * [Build and Make Variables](/developer/make-variables) # API Curio API references ## Groups * [Allocate](#Allocate) * [AllocatePieceToSector](#AllocatePieceToSector) * [DefaultGroup](#DefaultGroup) * [Cordon](#Cordon) * [Info](#Info) * [Shutdown](#Shutdown) * [Uncordon](#Uncordon) * [Version](#Version) * [Index](#Index) * [IndexSamples](#IndexSamples) * [Log](#Log) * [LogList](#LogList) * [LogSetLevel](#LogSetLevel) * [Storage](#Storage) * [StorageAddLocal](#StorageAddLocal) * [StorageDetachLocal](#StorageDetachLocal) * [StorageFindSector](#StorageFindSector) * [StorageGenerateVanillaProof](#StorageGenerateVanillaProof) * [StorageInfo](#StorageInfo) * [StorageInit](#StorageInit) * [StorageList](#StorageList) * [StorageLocal](#StorageLocal) * [StorageRedeclare](#StorageRedeclare) * [StorageStat](#StorageStat) ### Allocate #### AllocatePieceToSector There are not yet any comments for this method. Perms: write Inputs: ```json [ "f01234", { "PublishCid": { "/": "bafy2bzacea3wsdh6y3a36tb3skempjoxqpuyompjbmfeyf34fi3uy6uue42v4" }, "DealID": 5432, "DealProposal": { "PieceCID": { "/": "bafy2bzacea3wsdh6y3a36tb3skempjoxqpuyompjbmfeyf34fi3uy6uue42v4" }, "PieceSize": 1032, "VerifiedDeal": true, "Client": "f01234", "Provider": "f01234", "Label": "", "StartEpoch": 10101, "EndEpoch": 10101, "StoragePricePerEpoch": "0", "ProviderCollateral": "0", "ClientCollateral": "0" }, "DealSchedule": { "StartEpoch": 10101, "EndEpoch": 10101 }, "PieceActivationManifest": { "CID": { "/": "bafy2bzacea3wsdh6y3a36tb3skempjoxqpuyompjbmfeyf34fi3uy6uue42v4" }, "Size": 1032, "VerifiedAllocationKey": { "Client": 1000, "ID": 0 }, "Notify": [ { "Address": "f01234", "Payload": "Ynl0ZSBhcnJheQ==" } ] }, "KeepUnsealed": true }, 9, { "Scheme": "string value", "Opaque": "string value", "User": {}, "Host": "string value", "Path": "string value", "Fragment": "string value", "RawQuery": "string value", "RawPath": "string value", "RawFragment": "string value", "ForceQuery": true, "OmitHost": true }, { "Authorization": [ "Bearer ey.." ] } ] ``` Response: ```json { "Sector": 9, "Offset": 1032 } ``` ### DefaultGroup #### Cordon Perms: admin Inputs: `null` Response: `{}` #### Info Perms: read Inputs: `null` Response: ```json { "ID": 123, "CPU": 123, "RAM": 9, "GPU": 1, "HostPort": "string value", "LastContact": "0001-01-01T00:00:00Z", "Unschedulable": true, "Name": { "String": "string value", "Valid": true }, "StartupTime": { "Time": "0001-01-01T00:00:00Z", "Valid": true }, "Tasks": { "String": "string value", "Valid": true }, "Layers": { "String": "string value", "Valid": true }, "Miners": { "String": "string value", "Valid": true } } ``` #### Shutdown Perms: admin Inputs: `null` Response: `{}` #### Uncordon Perms: admin Inputs: `null` Response: `{}` #### Version There are not yet any comments for this method. Perms: admin Inputs: `null` Response: ```json [ 123 ] ``` ### Index #### IndexSamples Perms: admin Inputs: ```json [ { "/": "bafy2bzacea3wsdh6y3a36tb3skempjoxqpuyompjbmfeyf34fi3uy6uue42v4" } ] ``` Response: ```json [ "Bw==" ] ``` ### Log The log method group has logging methods #### LogList There are not yet any comments for this method. Perms: read Inputs: `null` Response: ```json [ "string value" ] ``` #### LogSetLevel Perms: admin Inputs: ```json [ "string value", "string value" ] ``` Response: `{}` ### Storage The storage method group contains are storage administration method #### StorageAddLocal Perms: admin Inputs: ```json [ "string value" ] ``` Response: `{}` #### StorageDetachLocal Perms: admin Inputs: ```json [ "string value" ] ``` Response: `{}` #### StorageFindSector Perms: admin Inputs: ```json [ { "Miner": 1000, "Number": 9 }, 1, 34359738368, true ] ``` Response: ```json [ { "ID": "76f1988b-ef30-4d7e-b3ec-9a627f4ba5a8", "URLs": [ "string value" ], "BaseURLs": [ "string value" ], "Weight": 42, "CanSeal": true, "CanStore": true, "Primary": true, "AllowTypes": [ "string value" ], "DenyTypes": [ "string value" ], "AllowMiners": [ "string value" ], "DenyMiners": [ "string value" ] } ] ``` #### StorageGenerateVanillaProof Perms: admin Inputs: ```json [ "f01234", 9 ] ``` Response: `"Ynl0ZSBhcnJheQ=="` #### StorageInfo Perms: admin Inputs: ```json [ "76f1988b-ef30-4d7e-b3ec-9a627f4ba5a8" ] ``` Response: ```json { "ID": "76f1988b-ef30-4d7e-b3ec-9a627f4ba5a8", "URLs": [ "string value" ], "Weight": 42, "MaxStorage": 42, "CanSeal": true, "CanStore": true, "Groups": [ "string value" ], "AllowTo": [ "string value" ], "AllowTypes": [ "string value" ], "DenyTypes": [ "string value" ], "AllowMiners": [ "string value" ], "DenyMiners": [ "string value" ] } ``` #### StorageInit There are not yet any comments for this method. Perms: admin Inputs: ```json [ "string value", { "ID": "76f1988b-ef30-4d7e-b3ec-9a627f4ba5a8", "Weight": 42, "CanSeal": true, "CanStore": true, "MaxStorage": 42, "Groups": [ "string value" ], "AllowTo": [ "string value" ], "AllowTypes": [ "string value" ], "DenyTypes": [ "string value" ], "AllowMiners": [ "string value" ], "DenyMiners": [ "string value" ] } ] ``` Response: `{}` #### StorageList Perms: admin Inputs: `null` Response: ```json { "76f1988b-ef30-4d7e-b3ec-9a627f4ba5a8": [ { "Miner": 1000, "Number": 100, "SectorFileType": 2 } ] } ``` #### StorageLocal Perms: admin Inputs: `null` Response: ```json { "76f1988b-ef30-4d7e-b3ec-9a627f4ba5a8": "/data/path" } ``` #### StorageRedeclare Perms: admin Inputs: ```json [ "string value", true ] ``` Response: `{}` #### StorageStat Perms: admin Inputs: ```json [ "76f1988b-ef30-4d7e-b3ec-9a627f4ba5a8" ] ``` Response: ```json { "Capacity": 9, "Available": 9, "FSAvailable": 9, "Reserved": 9, "Max": 9, "Used": 9 } ``` # Build and Make Variables This project uses a modular Makefile layout: * `Makefile`: entrypoint; includes all `scripts/makefiles/*.mk` fragments. * `scripts/makefiles/00-vars.mk`: shared defaults and derived variables. * `scripts/makefiles/05-help.mk`: `make help` and quick usage guidance. * `scripts/makefiles/10-deps.mk`: dependency bootstrap (`ffi`, `blst`, `supraseal`, submodules). * `scripts/makefiles/20-test.mk`: test and coverage targets. * `scripts/makefiles/30-build.mk`: binary build/install/cleanup targets. * `scripts/makefiles/40-gen.mk`: code/doc generation targets. * `scripts/makefiles/50-docker.mk`: docker/devnet targets. * `scripts/makefiles/60-abi.mk`: ABI to Go generation pattern rule. The refactor preserves behavior but makes execution paths easier to reason about. ## How overrides work You can override variables at invocation time: ```bash make build FFI_USE_OPENCL=1 make gen GOCC=gotip GOCACHE_CLEAN=1 make docker/devnet build_lotus=1 lotus_version=v1.35.0 ``` You can also export them from the environment: ```bash export GOFLAGS='-mod=mod' export FFI_USE_OPENCL=1 make build ``` Command-line values take precedence over file defaults. ## Build target matrix These targets share the same pipeline and differ only in tags: * `make build` * Builds: `curio`, `sptool` * Tag behavior: uses `CURIO_TAGS` as-is. * Default tags: `CURIO_TAGS_BASE` (`cunative`) plus conditional extras. * `make calibnet` * Builds: `curio`, `sptool` (mainnet-calibration network) * Tag behavior: appends `calibnet` to `CURIO_TAGS` then runs `build`. * `make debug` * Builds: `curio`, `sptool` * Tag behavior: appends `debug` to `CURIO_TAGS` then runs `build`. * `make 2k` * Builds: `curio`, `sptool` (dev-net / 2k) * Tag behavior: appends `2k` to `CURIO_TAGS` then runs `build`. * `make curio-pdp` (synonym: `make skiff`) * Builds: `curio` (PDP-only daemon from `./cmd/skiff`; no worker RPC or deal market) * Uses `SKIFF_TAGS` (`CURIO_TAGS` + `nosupraseal` + `skiff`) and skips filecoin-ffi `BUILD_DEPS`. * Overwrites the `curio` binary produced by `make curio`. Equivalent one-off forms are also available: * `make calibnet-curio` * `make calibnet-sptool` * `make calibnet-curio-pdp` (synonym: `make calibnet-skiff`) * `make cu2k` * `make 2k-curio-pdp` (synonym: `make 2k-skiff`) ## Important default context * On Linux, the default `make build` path does **not** add the `nosupraseal` tag. That means `lib/supraffi` Linux implementations (`//go:build linux && !nosupraseal`) are compiled in. * On CUDA-capable Linux hosts, Curio can use the SupraSeal-backed fast TreeR path for snap encode. If prerequisites are not met, Curio falls back to filecoin-ffi TreeR logic at runtime. * See `lib/ffiselect/ffidirect/ffi-direct.go` (`TreeRFile`) for the fast-path + fallback decision. * Even if you do not plan to run SupraSeal batch sealing, keeping default (non-`nosupraseal`) builds is still useful for snap encode performance paths. * On Linux CUDA builds, filecoin-ffi now defaults `FFI_USE_CUDA_SUPRASEAL=1` (FFI-internal feature selection only). ## `FFI_USE_OPENCL` and `FFI_USE_CUDA_SUPRASEAL` behavior by OS and CUDA availability `FFI_USE_OPENCL` controls Curio-side build routing: * filecoin-ffi backend selection inputs (`FFI_USE_CUDA`/OpenCL direction), * whether linux supraseal dependency build is included, * whether `nosupraseal` is auto-added to `CURIO_TAGS`. `FFI_USE_CUDA_SUPRASEAL` controls only filecoin-ffi's internal GPU feature choice: * it does not affect Curio `nosupraseal` tagging, * it does not affect whether linux `build/.supraseal-install` is included. Defaults and guardrails: * `FFI_USE_CUDA` is derived as `1` unless `FFI_USE_OPENCL=1`. * `FFI_USE_CUDA_SUPRASEAL` defaults to `1` only when `UNAME_S=Linux` and `FFI_USE_CUDA=1`; otherwise default `0`. * effective value is forced to `0` whenever `FFI_USE_CUDA=0`. ### Behavior matrix | OS | `nvcc` in `PATH` | `FFI_USE_OPENCL` value | default `FFI_USE_CUDA_SUPRASEAL` | Result | | ----- | ---------------- | ---------------------- | -------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | Linux | Yes | unset/empty | `1` | Build succeeds. `FFI_USE_CUDA=1`, FFI defaults to `cuda-supraseal`, linux supraseal dep included unless `DISABLE_SUPRASEAL=1`, tags default to `cunative` (or `cunative nosupraseal` when `DISABLE_SUPRASEAL=1`). | | Linux | No | unset/empty | `1` | Build fails early in `curio-libfilecoin` with CUDA-required error. | | Linux | Yes/No | `1` | `0` | Build succeeds. `FFI_USE_CUDA=0`, effective `FFI_USE_CUDA_SUPRASEAL=0`, direct filecoin-ffi OpenCL install path (prebuilt eligible), supraseal dep skipped, `nosupraseal` auto-added to tags, `build.IsOpencl=1`. | | Linux | Yes | `0` | `1` | Build succeeds. `FFI_USE_CUDA=1` (unambiguous CUDA), FFI defaults to `cuda-supraseal`, supraseal dep included unless `DISABLE_SUPRASEAL=1`. | | Linux | No | `0` | `1` | Build fails (same CUDA-required guard as default). | | macOS | Yes/No | unset/empty | `0` | Build succeeds. Darwin filecoin-ffi install path is used, linux supraseal dep block not applicable, tags default to `cunative` (or `cunative nosupraseal` when `DISABLE_SUPRASEAL=1`). | | macOS | Yes/No | `1` | `0` | Build succeeds. Mainly affects Curio tags/ldflags (`nosupraseal`, `build.IsOpencl=1`). Darwin path does not have Linux CUDA guard. | Notes: * On macOS, presence/absence of CUDA libraries does not drive Curio's Makefile decision flow. * In filecoin-ffi source builds, Darwin already chooses OpenCL feature path by default. * If either `FFI_USE_OPENCL=1` or `DISABLE_SUPRASEAL=1`, linux `build/.supraseal-install` is not added. * `FFI_USE_CUDA_SUPRASEAL` can be overridden (for example `FFI_USE_CUDA_SUPRASEAL=0`) without changing Curio tags/dependency gating. ## Detailed `make build` scenarios This section answers "what exactly is in the binary and what path was taken?" for common cases. ### Quick outcome table | Scenario | Example command | `CURIO_TAGS` used by `curio`/`sptool` | FFI dependency path | FFI GPU feature | `build/.supraseal-install` step | `build.IsOpencl` ldflag in `curio` | | ------------------------------------------------ | ------------------------------------------------- | ------------------------------------- | -------------------------------------------------------------- | -------------------------------------------- | ------------------------------- | ---------------------------------- | | Linux default (CUDA path) | `make build` | `cunative` | Linux `curio-libfilecoin` path, `FFI_USE_CUDA=1` | `cuda-supraseal` (default) | Yes | empty string | | macOS default | `make build` | `cunative` | Darwin direct `make -C extern/filecoin-ffi .install-filcrypto` | Darwin OpenCL-style default in filecoin-ffi | No (linux-only dep block) | empty string | | Linux OpenCL path | `make build FFI_USE_OPENCL=1` | `cunative nosupraseal` | Linux direct `make -C extern/filecoin-ffi .install-filcrypto` | `opencl` | No | `1` | | Linux CUDA but disable supraseal in Curio binary | `make build FFI_USE_OPENCL=0 DISABLE_SUPRASEAL=1` | `cunative nosupraseal` | Linux `curio-libfilecoin` path, `FFI_USE_CUDA=1` | `cuda-supraseal` (default unless overridden) | No | `0` | ### 1. Linux + `make build` (default CUDA-style path) Command: ```bash make build ``` Resulting behavior: * `build` invokes `curio` and `sptool`. * `BUILD_DEPS` includes: * `setup-cgo-env` * `build/.filecoin-install` * `ffi-version-check` * `build/.blst-install` * `build/.supraseal-install` (because linux and `FFI_USE_OPENCL != 1`) * `build/.filecoin-install` uses `curio-libfilecoin` (linux path), which: * requires `nvcc` on this CUDA path * passes `FFI_USE_CUDA=1` when `FFI_USE_OPENCL` is unset/empty * defaults `FFI_USE_CUDA_SUPRASEAL=1` on Linux CUDA path * tags default to `CURIO_TAGS="cunative"` (no `nosupraseal`) * `curio` compile specifics: * `GOAMD64=v3` is set in the build command * `CGO_LDFLAGS_ALLOW='.*'` is set on Linux for `curio` * `-X github.com/filecoin-project/curio/build.IsOpencl=` (empty) * `sptool` builds with the same tag set (`cunative`) * `sptool` uses the default `CGO_LDFLAGS_ALLOW` pattern (not Linux `curio` override) ### 2. macOS + `make build` Command: ```bash make build ``` Resulting behavior: * `BUILD_DEPS` still runs setup/version/blst checks. * `build/.filecoin-install` uses Darwin-specific path: * `make -C extern/filecoin-ffi .install-filcrypto` * it does **not** use `curio-libfilecoin` runtime `nvcc` gate. * linux-only supraseal dependency block is not added. * `build/.blst-install` still runs (`bash scripts/build-blst.sh`) because BLST is always in `BUILD_DEPS`. * tags are still `CURIO_TAGS="cunative"` by default. * `curio` ldflag `build.IsOpencl` remains empty unless you set `FFI_USE_OPENCL`. * even without `nosupraseal` tag, non-linux builds compile non-linux supraffi stubs by file-level build constraints (`!linux`). ### 3. Linux + OpenCL GPU path Command: ```bash make build FFI_USE_OPENCL=1 ``` Resulting behavior: * `FFI_USE_OPENCL=1` changes both deps and tags: * linux supraseal dependency block is skipped * `CURIO_TAGS_EXTRA` adds `nosupraseal` * final tags become `CURIO_TAGS="cunative nosupraseal"` * `build/.filecoin-install` uses direct filecoin-ffi install path on Linux OpenCL: * `make -C extern/filecoin-ffi .install-filcrypto` * prebuilt libfilcrypto is eligible (unless `FFI_BUILD_FROM_SOURCE=1` is set externally) * `curio` ldflag embeds `build.IsOpencl=1`. ### 4. Linux + CUDA, but you do not want supraseal in the Curio binary If your goal is "CUDA filecoin-ffi path, but compile binary without supraseal code", use: ```bash make build FFI_USE_OPENCL=0 DISABLE_SUPRASEAL=1 ``` What this does: * `FFI_USE_OPENCL=0` keeps CUDA mode explicit (`FFI_USE_CUDA=1`). * `DISABLE_SUPRASEAL=1` adds `nosupraseal`, so supraffi linux implementations are excluded at compile time. * linux supraseal dependency build step is skipped. * FFI still defaults to `FFI_USE_CUDA_SUPRASEAL=1` unless you override it. ## What `nosupraseal` means `nosupraseal` is a Go build tag that selects non-supraseal code paths in `lib/supraffi`. Relevant files: * supraseal-enabled linux files: * `lib/supraffi/seal.go` * `lib/supraffi/cuda_linux.go` * `lib/supraffi/seal_nvme.go` * `lib/supraffi/spdk_setup.go` * fallback/stub files selected by `!linux || nosupraseal`: * `lib/supraffi/seal_nonlinux.go` * `lib/supraffi/cuda_nonlinux.go` Practical effect: * with `nosupraseal`, linux supraseal integration code is not linked into the binary. * calls into supraffi functionality will use stub behavior (errors/panics for unavailable operations). * on non-linux, those stub files are selected even without explicitly adding `nosupraseal`. ## Common `make build` use cases * Linux host without CUDA toolkit: ```bash make build FFI_USE_OPENCL=1 ``` * Build calibnet binary set with OpenCL path: ```bash make calibnet FFI_USE_OPENCL=1 ``` * Build debug variant with explicit Go flags: ```bash make debug GOFLAGS='-mod=mod -trimpath' ``` * Linux CUDA path, but disable filecoin-ffi `cuda-supraseal` feature only: ```bash make build FFI_USE_OPENCL=0 FFI_USE_CUDA_SUPRASEAL=0 ``` * Build with fully explicit tags (overrides computed tag composition): ```bash make build CURIO_TAGS='cunative debug calibnet' ``` * Build host-optimized native curio at a fixed ISA level: ```bash make curio-native GOAMD64_NATIVE=v2 ``` ## Variables you are expected to override ### Build and toolchain * `GOCC` (default: `go`) * Purpose: Go command used for `build/run/generate`. * Override when: testing with a non-default Go binary (`gotip`, custom wrapper). * Example: `make build GOCC=/usr/local/go/bin/go` * `GOFLAGS` (default: empty) * Purpose: standard Go flags propagated into `go build`/`go run`/`go generate`. * Override when: changing module behavior, enabling race, changing trimpath, etc. * Example: `make build GOFLAGS='-mod=mod -trimpath'` * `FFI_USE_OPENCL` (default: unset) * Purpose: controls GPU backend assumptions for FFI-related dependency builds. * Expected values: * unset/empty or any value other than `1`: CUDA path (`FFI_USE_CUDA=1`). * `1`: OpenCL path (`FFI_USE_CUDA=0`). * Override when: * Linux host has no CUDA toolkit (`nvcc`) and you want OpenCL. * Running `make gen` is already forced to `FFI_USE_OPENCL=1` by target export. * Examples: * `make deps FFI_USE_OPENCL=1` * `make build FFI_USE_OPENCL=0` * `FFI_USE_CUDA_SUPRASEAL` (default: computed) * Purpose: controls whether filecoin-ffi uses `cuda-supraseal` in CUDA builds. * Default behavior: * Linux with `FFI_USE_CUDA=1`: defaults to `1`. * all other cases: defaults to `0`. * if `FFI_USE_CUDA=0`, effective value is forced to `0`. * Override when: you want CUDA FFI build path but want the non-`cuda-supraseal` FFI feature set. * Example: `make build FFI_USE_OPENCL=0 FFI_USE_CUDA_SUPRASEAL=0` * Important: this variable does not affect Curio `nosupraseal` tags or linux supraseal dependency gating. * `DISABLE_SUPRASEAL` (default: `0`) * Purpose: force `nosupraseal` tag and skip Linux supraseal dependency build. * Expected values: * `0`: default behavior. * `1`: disable supraseal in Curio build/tag flow. * Override when: Linux host has CUDA and you want CUDA-backed FFI build but a Curio binary compiled without supraseal paths. * Example: `make build FFI_USE_OPENCL=0 DISABLE_SUPRASEAL=1` * `CGO_LDFLAGS_ALLOW` (default: quoted value of `CGO_LDFLAGS_ALLOW_PATTERN`) * Purpose: linker flag allowlist for cgo. * Override when: integrating custom toolchains requiring different allow patterns. * Usually do not override directly; prefer default. * `CURIO_TAGS_BASE` (default: `cunative`) * Purpose: baseline build tags for binaries. * Override when: customizing feature profile globally. * Example: `make build CURIO_TAGS_BASE='cunative debug'` * `CURIO_TAGS` (default: computed from `CURIO_TAGS_BASE` plus conditional extras) * Purpose: final tag string passed to most builds. * Override when: you need exact control and do not want computed defaults. * Example: `make curio CURIO_TAGS='cunative calibnet debug'` * `GOAMD64_NATIVE` (default: auto-detected on linux/amd64 via `/proc/cpuinfo`) * Purpose: ISA level for `curio-native` target. * Override when: reproducibility across hosts or intentionally targeting older CPUs. * Example: `make curio-native GOAMD64_NATIVE=v2` ### Generation and caching * `GOCACHE_CLEAN` (default: unset) * Purpose: when set to `1`, `make gensimple` runs `go clean -cache` first. * Override when: generation repeatedly fails due Go build cache contention/corruption. * Example: `make gen GOCACHE_CLEAN=1` ### Testing and coverage * `COVERAGE_DIR` (default: `coverage`) * Purpose: output directory for coverage artifacts. * Override when: CI artifact path conventions differ. * Example: `make cov COVERAGE_DIR=build/coverage` ### Docker/devnet * `build_lotus` (default: `0`) * Purpose: * `0`: use prebuilt lotus image. * `1`: clone/build lotus image locally. * Override when: testing against a custom lotus branch/tag implementation. * `lotus_version` (default: `v1.35.1`) * Purpose: lotus version for prebuilt image tag or local clone branch/tag. * Override when: validating compatibility with another lotus version. * `curio_docker_user` (default: `curio`) * Purpose: image name prefix for locally built images. * Override when: publishing/testing in your own registry namespace. * `curio_base_image` (default: `$(curio_docker_user)/curio-all-in-one:latest-debug`) * Purpose: base image reference passed into docker builds. * Override when: pinning to an alternate base image/tag. * `ffi_from_source` (default: `0`) * Purpose: docker build arg controlling FFI source build behavior in image builds. * Override when: image tests require source-built FFI. * `lotus_base_image` (default when `build_lotus=0`: prebuilt GHCR image) * Purpose: lotus image consumed by curio docker build flow. * Override when: testing against a custom already-built lotus image. * `docker_args` (default: empty) * Purpose: appended raw args to docker build commands. * Override when: injecting platform/buildkit/cache args. * Example: `make docker/curio docker_args='--platform linux/amd64 --no-cache'` ## Variables that are internal/derived (usually do not override) * `FFI_PATH`, `BLST_PATH`, `SUPRA_FFI_PATH` * `BUILD_DEPS`, `MODULES`, `BINS`, `CLEAN` * `UNAME_S`, `NVCC_PATH`, `CUDA_PATH`, `CUDA_LIB_PATH` * `FFI_USE_CUDA`, `FFI_USE_CUDA_SUPRASEAL_EFFECTIVE` * `CURIO_TAGS_EXTRA`, `CURIO_TAGS_CSV` * `TEST_ENV_VARS` * `CGO_LDFLAGS_ALLOW_PATTERN` If you override internal variables, do so only when actively developing the Makefile itself. If you need these overridden as a regular pattern, consider opening a Github issue or talking with support. ## About `make test` Current state in this repository: * `make test` Runs `go test -v -tags="cgo,fvm" -timeout 30m ./itests/...`. It can be used to run integration tests locally. ## Safe override recipes * Linux without CUDA: * `make deps FFI_USE_OPENCL=1` * `make build FFI_USE_OPENCL=1` * Build calibnet binaries with explicit tags: * `make calibnet CURIO_TAGS='cunative calibnet'` * Linux CUDA path but compile Curio without supraseal: * `make build FFI_USE_OPENCL=0 DISABLE_SUPRASEAL=1` * Linux CUDA path but disable only filecoin-ffi `cuda-supraseal`: * `make build FFI_USE_OPENCL=0 FFI_USE_CUDA_SUPRASEAL=0` * Recover from unstable generation cache: * `make gen GOCACHE_CLEAN=1` * Build docker devnet with locally built lotus: * `make docker/devnet build_lotus=1 lotus_version=v1.35.1` # Docker Devnet How to run a local network with Curio using docker ## Prerequisites To ensure a stable and functional network, the Curio devnet requires running multiple binaries in parallel. To simplify this process, we have packaged the devnet using Docker. Please make sure to install the latest version of Docker on your system before proceeding. * Install Docker - ## Building Docker images Build images from the root of the Curio repository ``` make clean docker/devnet ``` * If you need to build containers using a specific version of lotus then provide the version as a parameter. The version must be a tag of [Lotus git repo](https://github.com/filecoin-project/lotus). We are shipping images for all releases from Lotus in our [Github image repo](https://github.com/filecoin-shipyard/lotus-containers/pkgs/container/lotus-containers).\\ ```bash make clean docker/devnet lotus_version=v1.29.2 ``` \\ * If the branch or tag you requested does not exist in our [Github image repository](https://github.com/filecoin-shipyard/lotus-containers/pkgs/container/lotus-containers) then you can build the lotus image manually.\\ ```bash make clean docker/devnet lotus_version=test/branch1 build_lotus=1 ``` ## Start devnet Docker stack * Run ``` make devnet/up ``` * It will spin up `lotus`, `lotus-miner`, `yugabyte`, `curio` and `piece-server` containers. All temporary data will be saved in `./docker/data` folder. * The initial setup could take up to 5 min or more as it needs to download Filecoin proof parameters. During the initial setup, it is normal to see error messages in the log. Containers are waiting for the lotus to be ready. It may timeout several times. Restart is expected to be managed by `docker`. * Try opening the Curio GUI . Devnet is ready to operate when the URL opens and indicates no errors on the startup page. * You can inspect the status using `cd docker/devnet && docker compose logs -f`. ## Make a deal in devnet 1. Login to `piece-server` container either via docker desktop UI or with below command ```shell docker exec -it piece-server /bin/bash ``` 2. Run the below command to make a deal and follow the on-screen instructions. ```shell ./sample/make-a-deal.sh ``` # Experimental Features This section covers the current experimental features available in Curio Curio is developing new features on a regular basis as part of the overall development. This section covers the experimental features released by Curio along with details on how to use them. It is **not** recommended to run experimental features in production environments. The features should be tested as per your requirements, and any issues or requests should be reported to the team via Github or Slack. Once the new features have been tested and vetted, they may be released as part of a stable Curio release and all documentation concerning those features will be moved to an appropriate section of this site. Current experimental features are listed below. {% content-ref url="/pages/cG0uFxJMC1wm4sY0PEr7" %} [GPU Over Provisioning](/experimental-features/gpu-over-provisioning) {% endcontent-ref %} {% content-ref url="/pages/prgTowu8zIKmj20tUd9t" %} [Snark Market](/experimental-features/snark-market) {% endcontent-ref %} {% content-ref url="/pages/TRjFWDJ4OmcU4UXacVNm" %} [Snark Market (Consumer)](/experimental-features/snark-market-consumer) {% endcontent-ref %} {% content-ref url="/pages/K4tcL73v6kMmfmNNijuu" %} [cuzk Proving Daemon](/experimental-features/cuzk-proving-daemon) {% endcontent-ref %} # GPU Over Provisioning This page explains how to allow Curio to run more than multiple GPU tasks on a single GPU at the same time ## Overview The `HARMONY_GPU_OVERPROVISION_FACTOR` environment variable enables GPU over-provisioning by allowing each physical GPU to present itself as multiple logical GPUs. When set to a value greater than 1, this feature allows a single GPU to handle multiple independent processes concurrently. ## Usage ### Enabling Over provisioning Set the `HARMONY_GPU_OVERPROVISION_FACTOR` environment variable to the desired over-provisioning factor. #### **Example** ```bash export HARMONY_GPU_OVERPROVISION_FACTOR=2 ``` * **Effect**: Each physical GPU is treated as two logical GPUs. * **Application**: In a snap encode worker, this setting allows each GPU to handle two independent encode processes simultaneously. #### Example with Service File **/etc/curio.env File** ```sh CURIO_LAYERS=gui,post CURIO_ALL_REMAINING_FIELDS_ARE_OPTIONAL=true CURIO_DB_HOST=yugabyte1,yugabyte2,yugabyte3 CURIO_DB_USER=yugabyte CURIO_DB_PASSWORD=yugabyte CURIO_DB_PORT=5433 CURIO_DB_NAME=yugabyte CURIO_REPO_PATH=~/.curio CURIO_NODE_NAME=ChangeMe FIL_PROOFS_USE_MULTICORE_SDR=1 HARMONY_GPU_OVERPROVISION_FACTOR=2 ``` ## Considerations * **Workload Compatibility**: Ideal for workloads that are not heavily memory-bound. * **Snap Encode Workloads**: Generally suitable for over-provisioning. * **SNARK Workloads**: May encounter memory limitations, especially on GPUs with lower memory capacity. * **GPU Specifications**: Enterprise GPUs with higher memory are better suited for over-provisioning. * **Performance Testing**: It's important to test and validate the optimal over-provisioning factor for your specific hardware and workloads. ### Benefits * **Increased Throughput**: Potentially improves processing capacity per GPU. * **Enhanced Utilization**: Makes better use of GPU resources that might otherwise be underutilized. ### Limitations * **Memory Constraints**: Over-provisioning can lead to memory bottlenecks on GPUs with limited memory. * **Potential Instability**: Running multiple processes on a single GPU may affect system stability and performance. ### Recommendations * **Start with Lower Values**: Begin with an over-provisioning factor of 2 and monitor system performance. * **Monitor Resource Usage**: Keep an eye on GPU memory usage, temperatures, and overall system load. * **Increment Gradually**: Adjust the over-provisioning factor incrementally to find the optimal balance. ### Feedback and Support As this is an experimental feature, we encourage users to provide feedback on their experience. Your insights are valuable for improving GPU over-provisioning support in future releases. # Enable PDP This guide walks you through setting up a PDP-enabled Filecoin Storage Provider using Lotus, YugabyteDB, and Curio {% hint style="info" %} For a PDP-only node without PoRep/sealing (skiff), see the [Curio-PDP runbook](/experimental-features/curio-pdp) and [Skiff binary](https://github.com/filecoin-project/curio/blob/main/documentation/en/skiff-binary.md). {% endhint %} {% hint style="danger" %} **ALPHA FEATURE - UNDER DEVELOPMENT** This documentation covers the PDP (Proof of Data Possession) feature, which is currently in alpha and under active development. This tool is intended for testing and experimental use only. For production use and submitting real deals with live PDP Storage Providers, please use the [Synapse SDK](https://github.com/FilOzone/synapse-sdk). {% endhint %} ## 🚀 Prerequisites {% hint style="warning" %} **Note:** This guide is written specifically for **Ubuntu 22.04**. If you are using a different Linux distribution, refer to the relevant documentation for package installation and compatibility. {% endhint %} Before starting, make sure you have a user with **sudo privileges**. This section prepares your system for the PDP stack. *** ### ⚙️ Hardware requirements * **RAM**: 32 GiB+ * **CPU**: 8 Core+ * **Storage**: * 1 TiB Fast storage (NVMe/SSD) * 10 TiB Long-term storage (HDD) * **GPU**: Not required * **Connectivity**: Public HTTPS endpoint (domain) *** ### 🧰 System Package Installation ```sh sudo apt update && sudo apt upgrade -y && sudo apt install -y \ mesa-opencl-icd ocl-icd-opencl-dev gcc git jq pkg-config curl clang \ build-essential hwloc libhwloc-dev libarchive-dev wget ntp python-is-python3 aria2 ``` *** ### :hammer: Install Go (match `go.mod`) Curio’s minimum Go version is set in the Curio repo at `go.mod`. Example (current repo min is **1.26.2**): ```sh sudo rm -rf /usr/local/go wget https://go.dev/dl/go1.26.2.linux-amd64.tar.gz sudo tar -C /usr/local -xzf go1.26.2.linux-amd64.tar.gz echo 'export PATH=$PATH:/usr/local/go/bin' >> ~/.bashrc source ~/.bashrc go version ``` {% hint style="success" %} You should see something like: `go version go1.26.2 linux/amd64` {% endhint %} *** ### :wrench: Install Rust ```sh curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh ``` {% hint style="info" %} When prompted, choose the option 1) Proceed with standard installation (default — just press Enter). {% endhint %} ```sh source $HOME/.cargo/env rustc --version ``` {% hint style="success" %} You should see something like: `rustc 1.86.0 (05f9846f8 2025-03-31)` {% endhint %} *** ### 🔐 Add Go and Rust to Secure Sudo Path ```sh sudo tee /etc/sudoers.d/dev-paths <
Lotus Documentationhttps://lotus.filecoin.io/lotus/get-started/what-is-lotus//files/hYl19U06Px0ZUgI4sqRGhttps://lotus.filecoin.io/lotus/get-started/what-is-lotus/
Lotus Support ChannelsFilecoin Slack - #fil-lotus-help/files/9Nk64EGiSiQSyBWdnfwmhttps://filecoinproject.slack.com/archives/CPFTWMY7N
### 🔧 Build Lotus Daemon Clone and check out Lotus: ```sh git clone https://github.com/filecoin-project/lotus.git cd lotus git checkout $(curl -s https://api.github.com/repos/filecoin-project/lotus/releases/latest | jq -r .tag_name) ``` **Build and Install for Mainnet** ```sh make clean && make lotus sudo make install-daemon lotus --version ``` **Build and Install for Calibration** ```sh make clean && make GOFLAGS="-tags=calibnet" lotus sudo make install-daemon lotus --version ``` {% hint style="success" %} You should see something like: `lotus version 1.32.2+calibnet+git.ff88d8269` {% endhint %} *** ### 📦 Import a Snapshot and Start the Daemon Download the Snapshot **Mainnet:** ```sh aria2c -x5 -o snapshot.car.zst https://forest-archive.chainsafe.dev/latest/mainnet/ ``` **Calibration:** ```sh aria2c -x5 -o snapshot.car.zst https://forest-archive.chainsafe.dev/latest/calibnet/ ``` **Import and Start the Daemon** ```sh lotus daemon --import-snapshot snapshot.car.zst --remove-existing-chain --halt-after-import nohup lotus daemon > ~/lotus.log 2>&1 & ``` {% hint style="info" %} If you encounter errors related to `EnableEthRPC` or `EnableIndexer`, run the following command and restart Lotus {% endhint %} ```sh sed -i 's/^\( *\)#*EnableEthRPC = .*/\1EnableEthRPC = true/; s/^\( *\)#*EnableIndexer = .*/\1EnableIndexer = true/' ~/.lotus/config.toml ``` **Monitor Sync Progress** ```sh lotus sync wait ``` To monitor continuously: ```sh lotus sync wait --watch ``` **Monitor Logs** ```sh tail -f ~/lotus.log ``` *** ## 🐘 Running YugabyteDB 🧠 Curio uses YugabyteDB to store metadata about deals, sealing operations, and PDP submissions.
Cover image
Yugabyte Documentationhttps://docs.yugabyte.com/preview/tutorials/quick-start/linux//files/T8ayxg9WluJEOg1CwsVThttps://docs.yugabyte.com/preview/tutorials/quick-start/linux/
Yugabyte Support ChannelsFilecoin Slack - #fil-curio-help - Yugabyte Slack/files/JSzWtmVgzubPKREGWNrBhttps://filecoinproject.slack.com/archives/C06LF5YP8S3
### 🛠 Set ulimit configuration {% hint style="warning" %} Before starting Yugabyte, you must increase the default `ulimit` values to ensure system limits do not interfere with the database. {% endhint %} To do this: #### 🔁 **Persist new limits across reboots** Add these lines to `/etc/security/limits.conf`: ```sh echo "$(whoami) soft nofile 1048576" | sudo tee -a /etc/security/limits.conf echo "$(whoami) hard nofile 1048576" | sudo tee -a /etc/security/limits.conf ``` This ensures the increased limits are automatically applied to future sessions. #### ⚡ **Apply limit immediately (for current shell only)** ```sh ulimit -n 1048576 ``` Verify: ```sh ulimit -n ``` {% hint style="success" %} This should output `1048576`. {% endhint %} ### ⚙️ Install Yugabyte ```sh wget https://software.yugabyte.com/releases/2.25.1.0/yugabyte-2.25.1.0-b381-linux-x86_64.tar.gz tar xvfz yugabyte-2.25.1.0-b381-linux-x86_64.tar.gz cd yugabyte-2.25.1.0 ./bin/post_install.sh ``` ### 🚀 Start the DB ```sh ./bin/yugabyted start \ --advertise_address 127.0.0.1 \ --master_flags rpc_bind_addresses=127.0.0.1 \ --tserver_flags rpc_bind_addresses=127.0.0.1 ``` {% hint style="warning" %} If you encounter locale-related errors when starting Yugabyte for the first time, run: {% endhint %} ```sh sudo locale-gen en_US.UTF-8 ``` {% hint style="success" %} Visit `http://127.0.0.1:15433` to confirm successful installation. This is the YugabyteDB web UI — it should display the dashboard if the service is running correctly and all nodes are healthy. {% endhint %} {% hint style="info" %} You can also check your Yugabyte cluster details directly in the CLI with: {% endhint %} ```sh ./bin/yugabyted status ``` *** ## 🧱 Installing and Configuring Curio 🧠 Curio is the core PDP client that coordinates sealing, interacts with Lotus and submits PDP proofs.
Curio Documentationhttps://docs.curiostorage.org//files/JSzWtmVgzubPKREGWNrBhttps://docs.curiostorage.org/
Curio Support ChannelsFilecoin Slack - #fil-curio-help/files/9Nk64EGiSiQSyBWdnfwmhttps://filecoinproject.slack.com/archives/C06LF5YP8S3
### ⚙️ System Configuration Before you proceed with the installation, you should increase the UDP buffer size: ```sh sudo sysctl -w net.core.rmem_max=2097152 sudo sysctl -w net.core.rmem_default=2097152 ``` To make this change persistent across reboots: ```sh echo 'net.core.rmem_max=2097152' | sudo tee -a /etc/sysctl.conf echo 'net.core.rmem_default=2097152' | sudo tee -a /etc/sysctl.conf ``` ### 🔬 Build Curio Clone the repository and switch to the PDP branch: ```sh git clone https://github.com/filecoin-project/curio.git cd curio git checkout pdpM3d ``` {% hint style="info" %} Curio is compiled for a specific Filecoin network at build time. Choose the appropriate build command below. {% endhint %} Mainnet ```sh make clean build ``` Calibration ```sh make clean calibnet ``` {% hint style="info" %} This step will take a few minutes to complete. {% endhint %} ### ✅ Install and Verify Curio Run the following to install the compiled binary: ```sh sudo make install ``` This will place curio in `/usr/local/bin` Verify the installation: ```sh curio --version ``` Expected example output: ```sh curio version 1.24.4+calibnet+git_f954c0a_2025-04-06T15:46:32-04:00 ``` *** ### 🔧 Guided Setup Curio provides a utility to help you set up a new miner interactively. Run the following command: ```sh curio guided-setup ``` #### 1️⃣ Select Curio Installation Type Use the arrow keys to navigate the guided setup menu and select "**Setup non-Storage Provider cluster**". #### 2️⃣ Enter Your YugabyteDB Connection Details If you used the default installation steps from this guide, the following values should work: * Host: `127.0.0.1` * Port: `5433` * Username: `yugabyte` * Password: `yugabyte` * Database: `yugabyte` You can verify these settings by running the following command from the Yugabyte directory: ```sh ./bin/yugabyted status ``` After selecting "**Continue to connect and update schema**", Curio will automatically create the required tables and schema in the database. #### 3️⃣ Telemetry (Optional) You'll be asked whether to share anonymised or signed telemetry with the Curio team to help improve the software. Select your preference and continue. #### 4️⃣ Save Database Configuration At the final step of the guided setup, you'll be prompted to choose where to save your database configuration file. Use the arrow keys to select a location. A common default is: ```sh /home/your-username/curio.env ``` Once selected, setup will complete, and the miner configuration will be stored. #### 5️⃣ Launch the Curio Web GUI To explore the Curio interface visually, start the GUI layer: ```sh curio run --layers=gui ``` Then, open your browser and go to: ```sh http://127.0.0.1:4701 ``` This will launch the Curio web GUI locally. *** ## 🧪 Enabling FWSS PDP 🧠 This section enables **FWSS Proof of Data Possession (PDP)** on your SP node using Curio. These steps guide you through running a standalone PDP service using Curio and pdptool.
Cover image
PDP Support Channels/files/9Nk64EGiSiQSyBWdnfwmhttps://filecoinproject.slack.com/archives/C0717TGU7V2
### 📦 Attach Storage Locations With Curio running with the GUI layer: ```sh curio run --layers=gui ``` Run the following commands in your Curio CLI to attach storage paths: ```sh curio cli storage attach --init --seal /fast-storage/path curio cli storage attach --init --store /long-term-storage/path ``` {% hint style="info" %} Your fast-storage path should point to high-performance storage media such as NVMe or SSD {% endhint %} *** ### 🔧 Add a PDP Configuration Layer Browse to the **Configurations** page of the Curio GUI. Create a new layer named **pdp** and enable the following under Subsystems: {% hint style="info" %} You may find it helpful to search for the setting names in your browser. {% endhint %} * ✅ `EnableParkPiece` * ✅ `EnablePDP` * ✅ `EnableCommP` * ✅ `EnableMoveStorage` In the **HTTP** section: * ✅ Enable: `true` * 🌐 DomainName: `your domain (e.g., pdp.mydomain.com)` * 📡 ListenAddress: `0.0.0.0:443` {% hint style="info" %} **Tip:** You must point your domain's A record to your server's public IP address for Let's Encrypt to issue a certificate. {% endhint %} *** ### 💰 Import your Filecoin Wallet Private Key: {% hint style="warning" %} There are several ways to obtain private keys for Ethereum addresses. In this guide, we will use a new delegated FIL wallet address. {% endhint %} Create a new delegated wallet: ```sh lotus wallet new delegated ``` ```sh # Example output: t410fuo4dghaeiqzokiqnxruzdr6e3cjktnxprrc56bi ``` {% hint style="info" %} You can display your Lotus wallets at any time by running: {% endhint %} ```sh lotus wallet list ``` Export & convert your new delegated wallet address private key: ```sh lotus wallet export | xxd -r -p | jq -r '.PrivateKey' | base64 -d | xxd -p -c 32 ``` ```sh # Example output: d4c2e3f9a716bb0e47fa91b2cf4a29870be3c5982fd6eafed71e8ac3f9c0b127 ``` Browse to the **PDP** page of the Curio GUI and in the **Owner Address** section: * Select **Import Key** * Copy the previously generated private wallet key into the **Private Key (Hex)** field. * Select **Import Key** {% hint style="success" %} Your 0x wallet address - the delegated Ethereum address derived from your Filecoin delegated wallet private key - will be added to the **Owner Address** section of the Curio PDP page. {% endhint %} Make sure to send a small amount of FIL or tFIL (testnet FIL) to your 0x wallet - we recommend 8 FIL for Mainnet & 5 tFIL for Calibration to ensure uninterrupted PDP operation during initial setup and testing. [Calibration test FIL faucet information](https://docs.filecoin.io/smart-contracts/developing-contracts/get-test-tokens). {% hint style="warning" %} **Important:** Secure your private key material. Don't expose or store it in plain text without protection. {% endhint %} *** ### 🚀 Restart and Verify Restart Curio with both layers: ```sh curio run --layers=gui,pdp ``` {% hint style="info" %} If you encounter errors related to `EnableEthRPC` or `EnableIndexer`, run the following command and restart Lotus {% endhint %} ```sh sed -i 's/^\( *\)#*EnableEthRPC = .*/\1EnableEthRPC = true/; s/^\( *\)#*EnableIndexer = .*/\1EnableIndexer = true/' ~/.lotus/config.toml ``` {% hint style="info" %} If you encounter errors binding to port 443 when starting Curio with the pdp configuration layer, run: {% endhint %} ```sh sudo setcap 'cap_net_bind_service=+ep' /usr/local/bin/curio ``` Test the PDP service: {% hint style="info" %} If `pdptool` is not installed, clone and build Curio: {% endhint %} ```sh git clone https://github.com/filecoin-project/curio.git cd curio/cmd/pdptool go build . ``` ```sh ./pdptool ping --service-url https://your-domain.com --service-name public ``` {% hint style="info" %} Always use `public` for the `--service-name` flag {% endhint %} {% hint style="success" %} Expected output: {% endhint %} ```sh Ping successful: Service is reachable and JWT token is valid. ``` *** ## 🎉 You're Done! You've successfully launched a **PDP-enabled Filecoin Storage Provider** stack. Your system is now: * ✅ Syncing with the Filecoin network via Lotus * ✅ Recording deal and sector metadata in YugabyteDB * ✅ Operating Curio to manage sealing and coordination * ✅ Enabled Proof of Data Possession (PDP) * ✅ Connected to your PDP-enabled storage provider *** ## 🔜 Next Steps * :heavy\_check\_mark: Register your FWSS node * :link: Explore FWSS & PDP tools & resources at [https://www.filecoin.services](https://www.filecoin.services/) * 💬 Join the community - Filecoin Slack - [#fil-pdp](https://filecoinproject.slack.com/archives/C0717TGU7V2) # Curio-PDP runbook Curio-PDP is the lightweight PDP storage provider build (`make curio-pdp`, Go tag `skiff`). It runs PDP proving and the FWSS registration flow without PoRep/sealing, MK20 market code, or `filecoin-ffi`. For the skiff binary overview and build flags, see [Skiff binary](https://github.com/filecoin-project/curio/blob/main/documentation/en/skiff-binary.md). For full-stack Curio with Yugabyte and optional PDP alongside sealing, see [Enable PDP](/experimental-features/enable-pdp). ## Architecture and data stores | Deployment | HarmonyDB (tasks, config, PDP state) | Piece index (multihash → offset) | | --------------------- | ------------------------------------ | ------------------------------------ | | **Full Curio** | Yugabyte (YSQL) | Yugabyte YCQL / Cassandra-compatible | | **Curio-PDP (skiff)** | **Yugabyte (YSQL)** | **Yugabyte YCQL** | | **Tests / CI** | Postgres | Scylla (CQL) | Curio-PDP is intentionally lighter on compute and dependencies: no PoRep/sealing, MK20 market code, or `filecoin-ffi`. Operators still run **Dockerized Yugabyte** for HarmonyDB and piece indexing — the same YSQL + YCQL stack as full Curio, bundled via `docker/skiff`. Piece payload files live on disk under writable paths discovered under `/data` (see [Storage](#storage)). Index data lives in Yugabyte YCQL and must be backed up with the database (see [Yugabyte backup](/administration/yugabyte-backup)). ## Prerequisites * **Docker** and **Docker Compose** (recommended deployment path) * External **Lotus-compatible chain node** (Lotus, Forest, etc.) — set `FULLNODE_API_INFO` or `[APIs].ChainApiInfo` * Writable storage under `/data` (see [Storage](#storage)) * Optional public **HTTPS domain** when exposing the PDP HTTP API (`HTTP.DomainName` in config) * FIL/tFIL to fund the PDP signing wallet before FWSS registration ## First-time setup ### 1. Configure `docker/skiff/.env` Copy or edit `docker/skiff/.env` before starting the stack. **Chain node (required).** Skiff does not embed a chain node. Set `FULLNODE_API_INFO` to a Lotus-compatible RPC endpoint that matches your skiff build network (`skiff` = mainnet, `calibnet-skiff` = calibration, etc.). If Lotus (or another chain node) runs on the **same machine** as Docker, use an address the skiff **container** can reach — not `127.0.0.1` inside the container: ```bash # Lotus on the Docker host (macOS / Windows / Linux with host-gateway) FULLNODE_API_INFO=/ip4/host.docker.internal/tcp/1234/http # Lotus on another host on your LAN FULLNODE_API_INFO=/ip4/192.168.1.50/tcp/1234/http ``` See [Skiff binary — Chain API](https://github.com/filecoin-project/curio/blob/main/documentation/en/skiff-binary.md#chain-api) for config-layer alternatives (`[APIs].ChainApiInfo`). **Storage paths** default to `./data/` under `docker/skiff/`. Adjust `YUGABYTE_DATA`, `SKIFF_REPO_DATA`, and `SKIFF_STORAGE` if needed. ### 2. Start Yugabyte + Skiff (Docker) From the repo root: ```bash cd docker/skiff docker compose up -d ``` This starts: * **Yugabyte** — YSQL on port `5433`, YCQL on port `9042`, web UI on `15433` * **Skiff** — local admin GUI on `127.0.0.1:4701`; public PDP API on `80`/`443` only Persistent data defaults to `docker/skiff/data/` (Yugabyte, repo state, and piece storage). {% hint style="warning" %} **Public firewall: open only TCP 80 and 443.** The admin GUI on port `4701` is unauthenticated and for local operator access only — do not publish it to the internet. The Compose file maps host `127.0.0.1:4701` for the same reason. Use SSH port forwarding if you need remote GUI access (see [PDP signing wallet](#3-pdp-signing-wallet-admin-gui)). {% endhint %} HarmonyDB migrations run on connect and create the same `curio` schema as full Curio. The piece `IndexStore` connects to Yugabyte YCQL on the same host (port `9042` by default via `--db-cassandra-port` / `CURIO_DB_CASSANDRA_PORT`). For a native skiff binary against the same Yugabyte stack (without the skiff container), export: ```bash export CURIO_DB_HOST=127.0.0.1 export CURIO_DB_PORT=5433 export CURIO_DB_USER=yugabyte export CURIO_DB_PASSWORD=yugabyte export CURIO_DB_NAME=yugabyte export CURIO_REPO_PATH=~/.curio export SKIFF_MACHINE_HOST=127.0.0.1:skiff export FULLNODE_API_INFO=/ip4/127.0.0.1/tcp/1234/http ``` ### 3. PDP signing wallet (admin GUI) Skiff needs a **PDP signing key** stored in HarmonyDB (`eth_keys` with `role=pdp`) before FWSS registration. Configure it through the admin GUI — the key is **not** set in `.env`. **Open the GUI** * On the Docker host: **** * From a remote machine (SSH tunnel): ```bash ssh -L 4701:127.0.0.1:4701 user@your-server ``` Then browse to **** on your laptop. Go to **PDP** → wallet section. | Action | When to use | | ---------- | ---------------------------------------------------------------------------------------------------------------- | | **Create** | Generate a new secp256k1 key on this node; the private key is shown **once** — save it before closing the dialog | | **Import** | Paste a hex private key for an existing 0x address you already control | Only one PDP key is allowed per cluster. After create or import, fund the displayed **0x address** with enough FIL/tFIL for registration and ongoing on-chain messages (see [Enable PDP — Import your Filecoin Wallet Private Key](/experimental-features/enable-pdp#import-your-filecoin-wallet-private-key) for recommended amounts and a Lotus delegated-wallet import workflow). {% hint style="danger" %} The GUI has no login. Anyone who can reach port `4701` can manage keys and config. Keep it on localhost or behind an SSH tunnel only. {% endhint %} The wallet private key is stored in Yugabyte and survives container restarts as long as `YUGABYTE_DATA` is preserved. Back up Yugabyte before redeploying (see [Yugabyte backup](/administration/yugabyte-backup)). ### 4. Register with FWSS In the GUI **Register** tab, complete provider registration, then verify with: ```bash pdptool ping --service-url https://your-domain.com --service-name public ``` ### Native binary (optional) If running skiff outside Docker, start Yugabyte first (see `docker/skiff/docker-compose.yaml` for the reference single-node command), then: ```bash ./curio # curio-pdp build ``` On first start, skiff **auto-seeds the `base` config layer** with PDP defaults (`EnablePDP`, `EnableWebGui`, `GuiAddress`, `StorageRPCSecret`). If a separate `pdp` layer already exists from a prior full-Curio setup, it is merged into `base` once at startup. Configure the PDP wallet via the same GUI steps above. ## Configuration model Skiff reads **only the `base` layer** at runtime. Do not rely on separate `pdp` or `gui` layers — put operational settings in `base` (or let auto-seed populate defaults and edit via the GUI). Typical `base` values: * `Subsystems.EnablePDP = true` (forced on) * `Subsystems.EnableWebGui = true` * `Subsystems.GuiAddress = "127.0.0.1:4701"` (never bind the GUI to `0.0.0.0` on a host reachable from the internet) * `HTTP.Enable = false` until a domain is configured for the public API Only **TCP 80 and 443** should be exposed on your public firewall for FWSS registration and client traffic. See [Curio HTTP server](/curio-market/curio-http-server) for TLS and reverse-proxy options. Skiff requires an external Lotus-compatible chain node. Set `FULLNODE_API_INFO` or `[APIs].ChainApiInfo` in the `base` layer (see [Skiff binary — Chain API](https://github.com/filecoin-project/curio/blob/main/documentation/en/skiff-binary.md#chain-api)). ## Storage Curio-PDP stores piece payloads on local disk. Mount drives at **`/data`** (or bind-mount volumes beneath it). On startup the node scans `/data` and **every subdirectory**, probes each for write access, and uses every writable location as storage. Unwritable paths are skipped. Missing `sectorstore.json` files are created automatically in each writable location. Additionally, you can use a path other than `/data`: ```bash DATA_STORAGE=/var/lib/curio-data ./curio ``` You can also set `[Subsystems].DataPath` in the `base` config layer, or pass `--data=/var/lib/curio-data`. ## Moving between deployment profiles Full Curio and Curio-PDP both use **Yugabyte (YSQL + YCQL)**. CI uses Postgres + Scylla and is not an operator deployment profile. To move **relational PDP state and piece indexes**: 1. Back up Yugabyte YSQL and YCQL (see [Yugabyte backup](/administration/yugabyte-backup)). 2. Restore into the target Yugabyte instance. 3. Copy **piece files** separately; payloads are not in the database dump. 4. If the imported DB has a separate `pdp` config layer, skiff merges it into `base` on next startup. There is no dedicated migration tool — Yugabyte backup/restore plus file copy is sufficient. ## Troubleshooting | Symptom | Check | | ------------------------------------------------ | ------------------------------------------------------------------------------------------------------------------------------------------- | | `docker compose up` fails on `FULLNODE_API_INFO` | Set chain RPC in `docker/skiff/.env`; see [Configure .env](#1-configure-dockerskiffenv) | | Skiff cannot reach chain node | From inside the container, `127.0.0.1` is not the Docker host — use `host.docker.internal` or the node's LAN IP | | Alert: PDP wallet not configured | [PDP signing wallet](#3-pdp-signing-wallet-admin-gui) → Create or Import; verify `eth_keys` has `role=pdp` | | Yugabyte connection errors | `docker compose ps`, `CURIO_DB_*`, YSQL on `5433`, YCQL on `9042`; see [Yugabyte troubleshooting](/administration/yugabyte-troubleshooting) | | No storage paths | Drives mounted under `/data` (or `DATA_STORAGE` / `--data`); write permissions on discovered paths | | Registration fails | Wallet funded; `HTTP.DomainName` / TLS; chain node synced and reachable | | Startup warning about missing key | Expected until wallet is configured; clears after key insert | # Snark Market This page explains how to set up a provider for the Curio Snark Market (experimental). > ⚠️ **Experimental Feature in Testing**\ > This feature is currently experimental and under active testing. Interfaces, behaviors, and requirements **may change without notice**. *** ## 🔧 What is the Snark Market? The Snark Market allows any Curio node — **including Storage Providers with spare sealing/GPU capacity** — to sell or buy proof computation in exchange for FIL. It is designed for GPU nodes that want to participate in Filecoin proof offloading, enabling a decentralized proof marketplace. Storage Providers can also **consume** proof compute — buying proofs from the market to save GPU capacity. See [Snark Market (Consumer)](/experimental-features/snark-market-consumer) for that setup. This guide will walk you through how to: * Enable proof selling on GPU nodes * Set up the pricing and wallet * See activity and settlement stats in the UI *** ## ⚙️ Prerequisites Before enabling the Snark Market on your node: * You must be running a Curio node with GPU capabilities. * You must have a working web UI (Your browser can access ports exposed by Curio). * You need a Lotus node installed and the Filecoin mainnet chain synced.\ Refer to lotus documentation here:\ * You need **YugabyteDB** installed.\ 👉 Follow the official setup instructions here:\ *** ## ⚙️ System Requirements * Modern **NVIDIA GPU** (recommended 12GB+ VRAM) * 70GB base system RAM + \~220 GB Per GPU * Lower amount is acceptable, but you won't be able to use the much faster CUDA C2 feature (batch sealing toolchain)\ Meaning \~10mins/proof instead of 2. * Curio **v1.27.0 or later** (Snark Market is included in official releases) * FIL balance on Mainnet *** ## 🚀 Setup Instructions ### 0. (Optional/Recommended) Benchmark your system You can skip most of the complex setup and learn how your hardware performs with lotus-bench * Follow build instructions from * For faster CUDA C2 (batch sealing toolchain) lotus-bench do:\ `RUSTFLAGS="-C target-cpu=native -g" FFI_BUILD_FROM_SOURCE=1 FFI_USE_CUDA_SUPRASEAL=1 make clean deps lotus-bench` * Download example snark inputs from (\~51MB) * Run `./lotus-bench simple commit2 c1.json`, wait a few minutes for results\ On the first run lotus-bench may need to download the SNARK proving parameters ### 1. Install Curio First, install dependencies (as per the [main install guide](https://docs.curiostorage.org/setup)): ```bash sudo apt update sudo apt install -y build-essential pkg-config libssl-dev curl git clang cmake golang ``` Then, build Curio (for CUDA C2 support, build from source): ```bash git checkout v1.27.2 # or latest release tag git submodule update make clean RUSTFLAGS="-C target-cpu=native -g" FFI_BUILD_FROM_SOURCE=1 FFI_USE_CUDA_SUPRASEAL=1 make clean build all sudo make install ``` Run basic setup. * If needed set database variables (defaults work with local YugabyteDB install)\ `CURIO_DB_NAME`, `CURIO_DB_USER`, `CURIO_DB_HOST` (can be a comma separated list of tserver IPs, one is also ok), `CURIO_DB_PASSWORD` * Ensure your lotus-node is running * Run `curio guided-setup` -> select `Setup non-Storage Provider cluster` * Create empty snark-provider configuration layer -> `echo | curio config create --title snark-provider` After all is done, [setup, enable and start the curio service](https://docs.curiostorage.org/curio-service#service-file-for-curio) * For `CURIO_LAYERS` use `gui,snark-provider` *** ## 🛠️ Step-by-step Setup ### 2. Enable the Market in Layer Configuration Ensure you're **not running on a WindowPoSt node**. This is only supported on GPU-based PoRep or Snap nodes. In the Web UI: 1. Go to `Overview` → `Configuration`, select the `snark-provider` if created previously 2. Find the **Subsystems** section 3. Set `EnableProofShare` to true 4. Save and restart the node
Enable PROOFSHARE toggle in configuration

Enable PROOFSHARE from the configuration layer

*** ### 3. Configure Provider Settings Navigate to `Snark Market` in the sidebar. Under **Provider Settings**: * Enable the settings checkbox * **Create a new `f1` wallet** (do *not* reuse existing wallet)\ Use `lotus wallet new secp256k1` in the CLI\ ⚠️ *Please note: This wallet can be changed later, but it is tricky* * Set **Price (FIL/p)** to `0.005` (recommended for testing) * Single proof (`p`) should take roughly two minutes to compute, your price should be calculated based on how many proofs per hour per GPU you expect to compute and your cost to run the GPU. The snark marketplace automatically adjusts the market price to match supply to demand. * Click **Update Settings**
Snark Wallet Setup

Snark Market provider settings with wallet and price configured

*** ### 4. Verify Your Node Once you've configured the provider settings: * Your node will automatically begin queueing proof work * The dashboard will update with: * Active Asks * SNARK Queue * Payment Summaries * Recent Settlements * You can also view the [global public dashboard](https://mainnet.snass.fsp.sh/ui/)
Snark Market Dashboard

Overview showing queue, asks, settlements, and active proofs

*** ## 🪙 Wallet Setup Notes * Create a **new `f1` address** and fund it (e.g. `0.1 FIL`) * This wallet receives SNARK proof rewards * Ensure the wallet remains **unlocked** *** ## 📈 Pricing & Payments * Price is set per \~130M proof constraints (default granularity) * Suggested starting point price: `0.005 FIL` * Settlements occur automatically when network gas fee to settle is less than 0.2% of the balance to settle *** ## 🧪 Notes * Your provider must complete **50 challenge proofs** per "work-slot" * Each "work-slot" is allows for either one "work ask" in the market - listing of readiness to take on work on one proof for some minimum price, or one "assigned proof" which is being actively worked on. * Number of work slots is calculated by dividing "completed challenge proofs" by **50** * Failure to complete assigned work within the deadline reduces "completed challenge proofs" balance by **50** proofs * Withdrawing an ask from the market (i.e. in order to adjust ask price) reduces "completed challenge proofs" balance by **1** proof * "challenge proofs" are gained only by completing unpaid challenge proofs, which are assigned when attempting to create a work ask while the challenge balance is too low. * The service *may* assign real proofs as challenges, but only if they were failed by other providers and no other provider can be found who can take the payment * Proofs must complete within **45 minutes**, or your node will lose its active slot and need to **re-earn trust** * The system is **fault-tolerant** and retries failed work automatically * You may be assigned retry proofs which come with non-current, potentially lower than current, still higher than your minimum price per proof. * You can **scale horizontally** by running more GPU workers with the same setup *** Let us know on Slack if you’re testing `#fil-curio-help` — we’ll be actively monitoring for feedback and performance 🚀 # Snark Market (Consumer) This page explains how to set up a consumer (GPU saver) for the Curio Snark Market (experimental). > ⚠️ **Experimental Feature in Testing**\ > This feature is currently experimental and under active testing. Interfaces, behaviors, and requirements **may change without notice**. *** ## 🔧 What is the Snark Market (Consumer)? The Snark Market allows Storage Providers to **buy** proof computation from the market instead of running GPUs locally. You offload PoRep and Snap proof work to providers in exchange for FIL — a "GPU saver" approach that lets you complete the C2 (proof) phase of sealing without local GPU capacity. To sell proof compute (be a provider), see [Snark Market](/experimental-features/snark-market). *** ## ⚙️ Prerequisites Before enabling the Snark Market consumer on your node: * You must be running a Curio node with a **sealing pipeline** (PoRep or Snap tasks). * You must have a working web UI (your browser can access ports exposed by Curio). * You need a Lotus node installed and the Filecoin mainnet chain synced.\ Refer to lotus documentation here:\ * You need **YugabyteDB** installed.\ 👉 Follow the official setup instructions here:\ *** ## ⚙️ System Requirements * **No GPU required** for consumers — you buy proofs from the market. * Standard sealing node RAM and storage. * Curio **v1.27.0 or later** (Snark Market is included in official releases). * FIL balance on Mainnet (to pay for proof compute). *** ## 🛠️ Step-by-step Setup ### 1. Enable Remote Proofs in Configuration 1. Go to `Overview` → `Configuration`, and select your miner or sealing layer. 2. Find the **Subsystems** section. 3. Set `EnableRemoteProofs` to true. 4. Save and restart the node. *** ### 2. Add and Fund Client Wallets Navigate to **Snark Market** in the sidebar. Under **Client Wallets**: * **Add Wallet**: Click **Add Wallet** and enter an f1 address you control. You can use an existing wallet (e.g. worker, collateral) or create a dedicated one — SPs typically already have funded wallets. * **Deposit**: Click **Deposit** to move FIL from that wallet's chain balance into the payment router. You need available balance in the router to pay for proofs. *** ### 3. Configure Client Settings Under **Client Settings** (right side of the page): 1. Accept the **Client Terms of Service** when prompted. 2. Click **Add SP**. Enter your SP address and the client wallet address (use a wallet you added in Client Wallets). 3. For each SP row: * Check **Enabled**. * Set **Wallet** to the f1 address you will use for payments. * Set **buy\_delay\_secs** — delay before outsourcing work (allows local GPU to take it first if you have one). * Enable **do\_porep** and/or **do\_snap** for the proof types you want to buy. * Set **FIL/P** (max price per proof; market price must be ≤ this). 4. Click **Save**. *** ## 📈 Pricing **FIL/P** is the maximum you are willing to pay per **P**. One **P** (one "proof unit") equals the cost of a single **32 GiB C2 (PoRep) proof** — the SNARK proof generated after C1 during sector sealing. | Proof type | Multiplier | Cost formula | Example at 0.005 FIL/P | | ------------------------------ | ---------- | ------------- | ---------------------- | | **32 GiB C2** (PoRep) | 1× | 1 × (FIL/P) | **0.005 FIL** | | **32 GiB Snap** (UpdateEncode) | 1.6× | 1.6 × (FIL/P) | **0.008 FIL** | Example: If you set **FIL/P = 0.005** and the market price is at or below that, a 32 GiB C2 will cost \~0.005 FIL and a 32 GiB Snap proof will cost \~0.008 FIL. The market price fluctuates; your configured FIL/P is the maximum you will pay — proofs are only purchased when the current market price is at or below your limit. Check the [public dashboard](https://mainnet.snass.fsp.sh/ui/) for the current **Min price**; use it to set your FIL/P accordingly. *** ## 🔄 Balance Manager (Optional) To automatically top up client wallet balances: 1. Go to **Wallet** → **Balance Manager**. 2. Click **Add SnarkMarket Client Rule**. 3. Set **Subject** to your client wallet address. 4. Set **Low** and **High** watermarks (in FIL). 5. Save. The Balance Manager will deposit FIL from the wallet's chain balance into the router when the available balance falls below the low watermark. *** ## ✅ Verify * **View Requests**: For each SP, click **View Requests** to see in-flight and completed proof requests. * **Client Messages**: Shows deposit and withdrawal status for client wallets. *** ## 🧪 Notes * **buy\_delay\_secs** gives your local GPU (if any) time to take work before outsourcing. Set to 0 to outsource immediately when local capacity is idle. * Work is only sent to the market when the current price is ≤ your configured max FIL/P. * Ensure client wallets stay funded; otherwise proofs cannot be purchased. *** Let us know on Slack if you're testing `#fil-curio-help` — we'll be actively monitoring for feedback and performance 🚀 # Market 2.0 API OpenAPI spec of Market 2.0 API. {% hint style="warning" %} These APIs can change without any notice as they are still being developed. This page is meant to be a reference for client libraries and SDK to allow parallel development. {% endhint %} ## List of supported DDO contracts > List of supported DDO contracts ```json {"openapi":"3.1.1","info":{"title":"Curio Market 2.0 API","version":"0.0.1"},"paths":{"/contracts":{"get":{"description":"List of supported DDO contracts","summary":"List of supported DDO contracts","responses":{"200":{"description":"Array of contract addresses supported by a system or application.","content":{"application/json":{"schema":{"$ref":"#/components/schemas/mk20.SupportedContracts"}}}},"500":{"description":"Internal Server Error","content":{"application/json":{"schema":{"type":"string"}}}}}}}},"components":{"schemas":{"mk20.SupportedContracts":{"type":"object","properties":{"contracts":{"description":"Contracts represents a list of supported contract addresses in string format.","type":"array","items":{"type":"string"}}}}}}} ``` ## List of supported products > List of supported products ```json {"openapi":"3.1.1","info":{"title":"Curio Market 2.0 API","version":"0.0.1"},"paths":{"/products":{"get":{"description":"List of supported products","summary":"List of supported products","responses":{"200":{"description":"Array of products supported by the SP","content":{"application/json":{"schema":{"$ref":"#/components/schemas/mk20.SupportedProducts"}}}},"500":{"description":"Internal Server Error","content":{"application/json":{"schema":{"type":"string"}}}}}}}},"components":{"schemas":{"mk20.SupportedProducts":{"type":"object","properties":{"products":{"description":"Contracts represents a list of supported contract addresses in string format.","type":"array","items":{"type":"string"}}}}}}} ``` ## List of supported dats sources > List of supported data sources ```json {"openapi":"3.1.1","info":{"title":"Curio Market 2.0 API","version":"0.0.1"},"paths":{"/sources":{"get":{"description":"List of supported data sources","summary":"List of supported dats sources","responses":{"200":{"description":"Array of dats sources supported by the SP","content":{"application/json":{"schema":{"$ref":"#/components/schemas/mk20.SupportedDataSources"}}}},"500":{"description":"Internal Server Error","content":{"application/json":{"schema":{"type":"string"}}}}}}}},"components":{"schemas":{"mk20.SupportedDataSources":{"type":"object","properties":{"sources":{"description":"Contracts represents a list of supported contract addresses in string format.","type":"array","items":{"type":"string"}}}}}}} ``` ## List of supported DDO contracts > List of supported DDO contracts ```json {"openapi":"3.1.1","info":{"title":"Curio Market 2.0 API","version":"0.0.1"},"paths":{"/status/{id}":{"get":{"description":"List of supported DDO contracts","summary":"List of supported DDO contracts","parameters":[{"schema":{"type":"string"},"description":"id","name":"id","in":"path","required":true}],"responses":{"200":{"description":"the status response for deal products with their respective deal statuses","content":{"application/json":{"schema":{"$ref":"#/components/schemas/mk20.DealProductStatusResponse"}}}},"400":{"description":"Bad Request - Invalid input or validation error","content":{"application/json":{"schema":{"type":"string"}}}},"500":{"description":"Internal Server Error","content":{"application/json":{"schema":{"type":"string"}}}}}}}},"components":{"schemas":{"mk20.DealProductStatusResponse":{"type":"object","properties":{"ddo_v1":{"description":"DDOV1 holds the DealStatusResponse for product \"ddo_v1\".","allOf":[{"$ref":"#/components/schemas/mk20.DealStatusResponse"}]},"pdp_v1":{"description":"PDPV1 represents the DealStatusResponse for the product pdp_v1.","allOf":[{"$ref":"#/components/schemas/mk20.DealStatusResponse"}]}}},"mk20.DealStatusResponse":{"type":"object","properties":{"error_msg":{"description":"ErrorMsg is an optional field containing error details associated with the deal's current state if an error occurred.","type":"string"},"status":{"description":"State indicates the current processing state of the deal as a DealState value.","allOf":[{"$ref":"#/components/schemas/mk20.DealState"}]}}},"mk20.DealState":{"type":"string","enum":["accepted","uploading","processing","sealing","indexing","failed","complete"]}}}} ``` ## Update the deal details of existing deals. > Useful for adding adding additional products and updating PoRep duration ```json {"openapi":"3.1.1","info":{"title":"Curio Market 2.0 API","version":"0.0.1"},"paths":{"/update/{id}":{"get":{"description":"Useful for adding adding additional products and updating PoRep duration","summary":"Update the deal details of existing deals.","parameters":[{"schema":{"type":"string"},"description":"id","name":"id","in":"path","required":true}],"responses":{"200":{"description":"Ok represents a successful operation with an HTTP status code of 200","content":{"application/json":{"schema":{"$ref":"#/components/schemas/mk20.DealCode"}}}},"400":{"description":"Bad Request - Invalid input or validation error","content":{"application/json":{"schema":{"type":"string"}}}},"404":{"description":"ErrDealNotFound indicates that the specified deal could not be found, corresponding to the HTTP status code 404","content":{"application/json":{"schema":{"$ref":"#/components/schemas/mk20.DealCode"}}}},"422":{"description":"ErrUnsupportedDataSource indicates the specified data source is not supported or disabled for use in the current context","content":{"application/json":{"schema":{"$ref":"#/components/schemas/mk20.DealCode"}}}},"423":{"description":"ErrUnsupportedProduct indicates that the requested product is not supported by the provider","content":{"application/json":{"schema":{"$ref":"#/components/schemas/mk20.DealCode"}}}},"424":{"description":"ErrProductNotEnabled indicates that the requested product is not enabled on the provider","content":{"application/json":{"schema":{"$ref":"#/components/schemas/mk20.DealCode"}}}},"425":{"description":"ErrProductValidationFailed indicates a failure during product-specific validation due to invalid or missing data","content":{"application/json":{"schema":{"$ref":"#/components/schemas/mk20.DealCode"}}}},"426":{"description":"ErrDealRejectedByMarket indicates that a proposed deal was rejected by the market for not meeting its acceptance criteria or rules","content":{"application/json":{"schema":{"$ref":"#/components/schemas/mk20.DealCode"}}}},"429":{"description":"ErrServiceOverloaded indicates that the service is overloaded and cannot process the request at the moment","content":{"application/json":{"schema":{"$ref":"#/components/schemas/mk20.DealCode"}}}},"430":{"description":"ErrMalformedDataSource indicates that the provided data source is incorrectly formatted or contains invalid data","content":{"application/json":{"schema":{"$ref":"#/components/schemas/mk20.DealCode"}}}},"440":{"description":"ErrMarketNotEnabled indicates that the market is not enabled for the requested operation","content":{"application/json":{"schema":{"$ref":"#/components/schemas/mk20.DealCode"}}}},"441":{"description":"ErrDurationTooShort indicates that the provided duration value does not meet the minimum required threshold","content":{"application/json":{"schema":{"$ref":"#/components/schemas/mk20.DealCode"}}}},"500":{"description":"ErrServerInternalError indicates an internal server error with a corresponding error code of 500","content":{"application/json":{"schema":{"$ref":"#/components/schemas/mk20.DealCode"}}}},"503":{"description":"ErrServiceMaintenance indicates that the service is temporarily unavailable due to maintenance, corresponding to HTTP status code 503","content":{"application/json":{"schema":{"$ref":"#/components/schemas/mk20.DealCode"}}}}},"requestBody":{"content":{"application/json":{"schema":{"$ref":"#/components/schemas/mk20.Deal"}}},"description":"mk20.Deal in json format","required":true}}}},"components":{"schemas":{"mk20.DealCode":{"type":"integer","enum":[200,401,400,404,430,422,423,424,425,426,500,503,429,440,441]},"mk20.Deal":{"type":"object","properties":{"client":{"description":"Client wallet for the deal","allOf":[{"$ref":"#/components/schemas/address.Address"}]},"data":{"description":"Data represents the source of piece data and associated metadata.","allOf":[{"$ref":"#/components/schemas/mk20.DataSource"}]},"identifier":{"description":"Identifier represents a unique identifier for the deal in UUID format.","type":"array","items":{"type":"integer"}},"products":{"description":"Products represents a collection of product-specific information associated with a deal","allOf":[{"$ref":"#/components/schemas/mk20.Products"}]}}},"address.Address":{"type":"object"},"mk20.DataSource":{"type":"object","properties":{"format":{"description":"Format defines the format of the piece data, which can include CAR, Aggregate, or Raw formats.","allOf":[{"$ref":"#/components/schemas/mk20.PieceDataFormat"}]},"piece_cid":{"description":"PieceCID represents the unique identifier (pieceCID V2) for a piece of data, stored as a CID object.","allOf":[{"$ref":"#/components/schemas/cid.Cid"}]},"source_aggregate":{"description":"SourceAggregate represents an aggregated source, comprising multiple data sources as pieces.","allOf":[{"$ref":"#/components/schemas/mk20.DataSourceAggregate"}]},"source_http":{"description":"SourceHTTP represents the HTTP-based source of piece data within a deal, including raw size and URLs for retrieval.","allOf":[{"$ref":"#/components/schemas/mk20.DataSourceHTTP"}]},"source_httpput":{"description":"SourceHTTPPut // allow clients to push piece data after deal accepted, sort of like offline import","allOf":[{"$ref":"#/components/schemas/mk20.DataSourceHttpPut"}]},"source_offline":{"description":"SourceOffline defines the data source for offline pieces, including raw size information.","allOf":[{"$ref":"#/components/schemas/mk20.DataSourceOffline"}]}}},"mk20.PieceDataFormat":{"type":"object","properties":{"aggregate":{"description":"Aggregate holds a reference to the aggregated format of piece data.","allOf":[{"$ref":"#/components/schemas/mk20.FormatAggregate"}]},"car":{"description":"Car represents the optional CAR file format, including its metadata and versioning details.","allOf":[{"$ref":"#/components/schemas/mk20.FormatCar"}]},"raw":{"description":"Raw represents the raw format of the piece data, encapsulated as bytes.","allOf":[{"$ref":"#/components/schemas/mk20.FormatBytes"}]}}},"mk20.FormatAggregate":{"type":"object","properties":{"sub":{"description":"Sub holds a slice of DataSource, representing details of sub pieces aggregated under this format.\nThe order must be same as segment index to avoid incorrect indexing of sub pieces in an aggregate","type":"array","items":{"$ref":"#/components/schemas/mk20.DataSource"}},"type":{"description":"Type specifies the type of aggregation for data pieces, represented by an AggregateType value.","allOf":[{"$ref":"#/components/schemas/mk20.AggregateType"}]}}},"mk20.AggregateType":{"type":"integer","enum":[0,1]},"mk20.FormatCar":{"type":"object"},"mk20.FormatBytes":{"type":"object"},"cid.Cid":{"type":"object"},"mk20.DataSourceAggregate":{"type":"object","properties":{"pieces":{"type":"array","items":{"$ref":"#/components/schemas/mk20.DataSource"}}}},"mk20.DataSourceHTTP":{"type":"object","properties":{"urls":{"description":"URLs lists the HTTP endpoints where the piece data can be fetched.","type":"array","items":{"$ref":"#/components/schemas/mk20.HttpUrl"}}}},"mk20.HttpUrl":{"type":"object","properties":{"fallback":{"description":"Fallback indicates whether this URL serves as a fallback option when other URLs fail.","type":"boolean"},"headers":{"description":"HTTPHeaders represents the HTTP headers associated with the URL.","allOf":[{"$ref":"#/components/schemas/http.Header"}]},"priority":{"description":"Priority indicates the order preference for using the URL in requests, with lower values having higher priority.","type":"integer"},"url":{"description":"URL specifies the HTTP endpoint where the piece data can be fetched.","type":"string"}}},"http.Header":{"type":"object","additionalProperties":{"type":"array","items":{"type":"string"}}},"mk20.DataSourceHttpPut":{"type":"object"},"mk20.DataSourceOffline":{"type":"object"},"mk20.Products":{"type":"object","properties":{"ddo_v1":{"description":"DDOV1 represents a product v1 configuration for Direct Data Onboarding (DDO)","allOf":[{"$ref":"#/components/schemas/mk20.DDOV1"}]},"pdp_v1":{"description":"PDPV1 represents product-specific configuration for PDP version 1 deals.","allOf":[{"$ref":"#/components/schemas/mk20.PDPV1"}]},"retrieval_v1":{"description":"RetrievalV1 represents configuration for retrieval settings in the system, including indexing and announcement flags.","allOf":[{"$ref":"#/components/schemas/mk20.RetrievalV1"}]}}},"mk20.DDOV1":{"type":"object","properties":{"allocation_id":{"description":"AllocationId represents an aggregated allocation identifier for the deal.","allOf":[{"$ref":"#/components/schemas/github_com_filecoin-project_go-state-types_builtin_v16_verifreg.AllocationId"}]},"contract_address":{"description":"ContractAddress specifies the address of the contract governing the deal","type":"string"},"contract_verify_method":{"description":"ContractDealIDMethod specifies the method name to verify the deal and retrieve the deal ID for a contract","type":"string"},"contract_verify_method_params":{"description":"ContractDealIDMethodParams represents encoded parameters for the contract verify method if required by the contract","type":"array","items":{"type":"integer"}},"duration":{"description":"Duration represents the deal duration in epochs. This value is ignored for the deal with allocationID.\nIt must be at least 518400","type":"integer"},"notification_address":{"description":"NotificationAddress specifies the address to which notifications will be relayed to when sector is activated","type":"string"},"notification_payload":{"description":"NotificationPayload holds the notification data typically in a serialized byte array format.","type":"array","items":{"type":"integer"}},"piece_manager":{"description":"Actor providing AuthorizeMessage (like f1/f3 wallet) able to authorize actions such as managing ACLs","allOf":[{"$ref":"#/components/schemas/address.Address"}]},"provider":{"description":"Provider specifies the address of the provider","allOf":[{"$ref":"#/components/schemas/address.Address"}]}}},"github_com_filecoin-project_go-state-types_builtin_v16_verifreg.AllocationId":{"type":"integer","enum":[0]},"mk20.PDPV1":{"type":"object","properties":{"add_root":{"description":"AddRoot indicated that this deal is meant to add root to a given ProofSet. ProofSetID must be defined.","type":"boolean"},"create_proof_set":{"description":"CreateProofSet indicated that this deal is meant to create a new ProofSet for the client by storage provider.","type":"boolean"},"delete_proof_set":{"description":"DeleteProofSet indicated that this deal is meant to delete an existing ProofSet created by SP for the client.\nProofSetID must be defined.","type":"boolean"},"delete_root":{"description":"DeleteRoot indicates whether the root of the data should be deleted. ProofSetID must be defined.","type":"boolean"},"extra_data":{"description":"ExtraData can be used to send additional information to service contract when Verifier action like AddRoot, DeleteRoot etc. are performed.","type":"array","items":{"type":"integer"}},"proof_set_id":{"description":"ProofSetID is PDP verified contract proofset ID. It must be defined for all deals except when CreateProofSet is true.","type":"integer"},"record_keeper":{"description":"RecordKeeper specifies the record keeper contract address for the new PDP proofset.","type":"string"},"root_ids":{"description":"RootIDs is a list of root ids in a proof set.","type":"array","items":{"type":"integer"}}}},"mk20.RetrievalV1":{"type":"object","properties":{"announce_payload":{"description":"AnnouncePayload indicates whether the payload should be announced to IPNI.","type":"boolean"},"announce_piece":{"description":"AnnouncePiece indicates whether the piece information should be announced to IPNI.","type":"boolean"},"indexing":{"description":"Indexing indicates if the deal is to be indexed in the provider's system to support CIDs based retrieval","type":"boolean"}}}}}} ``` ## Finalizes the serial upload process > Finalizes the serial upload process once data has been uploaded ```json {"openapi":"3.1.1","info":{"title":"Curio Market 2.0 API","version":"0.0.1"},"paths":{"/upload/{id}":{"post":{"description":"Finalizes the serial upload process once data has been uploaded","summary":"Finalizes the serial upload process","parameters":[{"schema":{"type":"string"},"description":"id","name":"id","in":"path","required":true}],"responses":{"200":{"description":"Ok represents a successful operation with an HTTP status code of 200","content":{"application/json":{"schema":{"$ref":"#/components/schemas/mk20.DealCode"}}}},"400":{"description":"Bad Request - Invalid input or validation error","content":{"application/json":{"schema":{"type":"string"}}}},"404":{"description":"ErrDealNotFound indicates that the specified deal could not be found, corresponding to the HTTP status code 404","content":{"application/json":{"schema":{"$ref":"#/components/schemas/mk20.DealCode"}}}},"422":{"description":"ErrUnsupportedDataSource indicates the specified data source is not supported or disabled for use in the current context","content":{"application/json":{"schema":{"$ref":"#/components/schemas/mk20.DealCode"}}}},"423":{"description":"ErrUnsupportedProduct indicates that the requested product is not supported by the provider","content":{"application/json":{"schema":{"$ref":"#/components/schemas/mk20.DealCode"}}}},"424":{"description":"ErrProductNotEnabled indicates that the requested product is not enabled on the provider","content":{"application/json":{"schema":{"$ref":"#/components/schemas/mk20.DealCode"}}}},"425":{"description":"ErrProductValidationFailed indicates a failure during product-specific validation due to invalid or missing data","content":{"application/json":{"schema":{"$ref":"#/components/schemas/mk20.DealCode"}}}},"426":{"description":"ErrDealRejectedByMarket indicates that a proposed deal was rejected by the market for not meeting its acceptance criteria or rules","content":{"application/json":{"schema":{"$ref":"#/components/schemas/mk20.DealCode"}}}},"429":{"description":"ErrServiceOverloaded indicates that the service is overloaded and cannot process the request at the moment","content":{"application/json":{"schema":{"$ref":"#/components/schemas/mk20.DealCode"}}}},"430":{"description":"ErrMalformedDataSource indicates that the provided data source is incorrectly formatted or contains invalid data","content":{"application/json":{"schema":{"$ref":"#/components/schemas/mk20.DealCode"}}}},"440":{"description":"ErrMarketNotEnabled indicates that the market is not enabled for the requested operation","content":{"application/json":{"schema":{"$ref":"#/components/schemas/mk20.DealCode"}}}},"441":{"description":"ErrDurationTooShort indicates that the provided duration value does not meet the minimum required threshold","content":{"application/json":{"schema":{"$ref":"#/components/schemas/mk20.DealCode"}}}},"500":{"description":"ErrServerInternalError indicates an internal server error with a corresponding error code of 500","content":{"application/json":{"schema":{"$ref":"#/components/schemas/mk20.DealCode"}}}},"503":{"description":"ErrServiceMaintenance indicates that the service is temporarily unavailable due to maintenance, corresponding to HTTP status code 503","content":{"application/json":{"schema":{"$ref":"#/components/schemas/mk20.DealCode"}}}}},"requestBody":{"content":{"application/json":{"schema":{"$ref":"#/components/schemas/mk20.Deal"}}},"description":"mk20.deal in json format"}}}},"components":{"schemas":{"mk20.DealCode":{"type":"integer","enum":[200,401,400,404,430,422,423,424,425,426,500,503,429,440,441]},"mk20.Deal":{"type":"object","properties":{"client":{"description":"Client wallet for the deal","allOf":[{"$ref":"#/components/schemas/address.Address"}]},"data":{"description":"Data represents the source of piece data and associated metadata.","allOf":[{"$ref":"#/components/schemas/mk20.DataSource"}]},"identifier":{"description":"Identifier represents a unique identifier for the deal in UUID format.","type":"array","items":{"type":"integer"}},"products":{"description":"Products represents a collection of product-specific information associated with a deal","allOf":[{"$ref":"#/components/schemas/mk20.Products"}]}}},"address.Address":{"type":"object"},"mk20.DataSource":{"type":"object","properties":{"format":{"description":"Format defines the format of the piece data, which can include CAR, Aggregate, or Raw formats.","allOf":[{"$ref":"#/components/schemas/mk20.PieceDataFormat"}]},"piece_cid":{"description":"PieceCID represents the unique identifier (pieceCID V2) for a piece of data, stored as a CID object.","allOf":[{"$ref":"#/components/schemas/cid.Cid"}]},"source_aggregate":{"description":"SourceAggregate represents an aggregated source, comprising multiple data sources as pieces.","allOf":[{"$ref":"#/components/schemas/mk20.DataSourceAggregate"}]},"source_http":{"description":"SourceHTTP represents the HTTP-based source of piece data within a deal, including raw size and URLs for retrieval.","allOf":[{"$ref":"#/components/schemas/mk20.DataSourceHTTP"}]},"source_httpput":{"description":"SourceHTTPPut // allow clients to push piece data after deal accepted, sort of like offline import","allOf":[{"$ref":"#/components/schemas/mk20.DataSourceHttpPut"}]},"source_offline":{"description":"SourceOffline defines the data source for offline pieces, including raw size information.","allOf":[{"$ref":"#/components/schemas/mk20.DataSourceOffline"}]}}},"mk20.PieceDataFormat":{"type":"object","properties":{"aggregate":{"description":"Aggregate holds a reference to the aggregated format of piece data.","allOf":[{"$ref":"#/components/schemas/mk20.FormatAggregate"}]},"car":{"description":"Car represents the optional CAR file format, including its metadata and versioning details.","allOf":[{"$ref":"#/components/schemas/mk20.FormatCar"}]},"raw":{"description":"Raw represents the raw format of the piece data, encapsulated as bytes.","allOf":[{"$ref":"#/components/schemas/mk20.FormatBytes"}]}}},"mk20.FormatAggregate":{"type":"object","properties":{"sub":{"description":"Sub holds a slice of DataSource, representing details of sub pieces aggregated under this format.\nThe order must be same as segment index to avoid incorrect indexing of sub pieces in an aggregate","type":"array","items":{"$ref":"#/components/schemas/mk20.DataSource"}},"type":{"description":"Type specifies the type of aggregation for data pieces, represented by an AggregateType value.","allOf":[{"$ref":"#/components/schemas/mk20.AggregateType"}]}}},"mk20.AggregateType":{"type":"integer","enum":[0,1]},"mk20.FormatCar":{"type":"object"},"mk20.FormatBytes":{"type":"object"},"cid.Cid":{"type":"object"},"mk20.DataSourceAggregate":{"type":"object","properties":{"pieces":{"type":"array","items":{"$ref":"#/components/schemas/mk20.DataSource"}}}},"mk20.DataSourceHTTP":{"type":"object","properties":{"urls":{"description":"URLs lists the HTTP endpoints where the piece data can be fetched.","type":"array","items":{"$ref":"#/components/schemas/mk20.HttpUrl"}}}},"mk20.HttpUrl":{"type":"object","properties":{"fallback":{"description":"Fallback indicates whether this URL serves as a fallback option when other URLs fail.","type":"boolean"},"headers":{"description":"HTTPHeaders represents the HTTP headers associated with the URL.","allOf":[{"$ref":"#/components/schemas/http.Header"}]},"priority":{"description":"Priority indicates the order preference for using the URL in requests, with lower values having higher priority.","type":"integer"},"url":{"description":"URL specifies the HTTP endpoint where the piece data can be fetched.","type":"string"}}},"http.Header":{"type":"object","additionalProperties":{"type":"array","items":{"type":"string"}}},"mk20.DataSourceHttpPut":{"type":"object"},"mk20.DataSourceOffline":{"type":"object"},"mk20.Products":{"type":"object","properties":{"ddo_v1":{"description":"DDOV1 represents a product v1 configuration for Direct Data Onboarding (DDO)","allOf":[{"$ref":"#/components/schemas/mk20.DDOV1"}]},"pdp_v1":{"description":"PDPV1 represents product-specific configuration for PDP version 1 deals.","allOf":[{"$ref":"#/components/schemas/mk20.PDPV1"}]},"retrieval_v1":{"description":"RetrievalV1 represents configuration for retrieval settings in the system, including indexing and announcement flags.","allOf":[{"$ref":"#/components/schemas/mk20.RetrievalV1"}]}}},"mk20.DDOV1":{"type":"object","properties":{"allocation_id":{"description":"AllocationId represents an aggregated allocation identifier for the deal.","allOf":[{"$ref":"#/components/schemas/github_com_filecoin-project_go-state-types_builtin_v16_verifreg.AllocationId"}]},"contract_address":{"description":"ContractAddress specifies the address of the contract governing the deal","type":"string"},"contract_verify_method":{"description":"ContractDealIDMethod specifies the method name to verify the deal and retrieve the deal ID for a contract","type":"string"},"contract_verify_method_params":{"description":"ContractDealIDMethodParams represents encoded parameters for the contract verify method if required by the contract","type":"array","items":{"type":"integer"}},"duration":{"description":"Duration represents the deal duration in epochs. This value is ignored for the deal with allocationID.\nIt must be at least 518400","type":"integer"},"notification_address":{"description":"NotificationAddress specifies the address to which notifications will be relayed to when sector is activated","type":"string"},"notification_payload":{"description":"NotificationPayload holds the notification data typically in a serialized byte array format.","type":"array","items":{"type":"integer"}},"piece_manager":{"description":"Actor providing AuthorizeMessage (like f1/f3 wallet) able to authorize actions such as managing ACLs","allOf":[{"$ref":"#/components/schemas/address.Address"}]},"provider":{"description":"Provider specifies the address of the provider","allOf":[{"$ref":"#/components/schemas/address.Address"}]}}},"github_com_filecoin-project_go-state-types_builtin_v16_verifreg.AllocationId":{"type":"integer","enum":[0]},"mk20.PDPV1":{"type":"object","properties":{"add_root":{"description":"AddRoot indicated that this deal is meant to add root to a given ProofSet. ProofSetID must be defined.","type":"boolean"},"create_proof_set":{"description":"CreateProofSet indicated that this deal is meant to create a new ProofSet for the client by storage provider.","type":"boolean"},"delete_proof_set":{"description":"DeleteProofSet indicated that this deal is meant to delete an existing ProofSet created by SP for the client.\nProofSetID must be defined.","type":"boolean"},"delete_root":{"description":"DeleteRoot indicates whether the root of the data should be deleted. ProofSetID must be defined.","type":"boolean"},"extra_data":{"description":"ExtraData can be used to send additional information to service contract when Verifier action like AddRoot, DeleteRoot etc. are performed.","type":"array","items":{"type":"integer"}},"proof_set_id":{"description":"ProofSetID is PDP verified contract proofset ID. It must be defined for all deals except when CreateProofSet is true.","type":"integer"},"record_keeper":{"description":"RecordKeeper specifies the record keeper contract address for the new PDP proofset.","type":"string"},"root_ids":{"description":"RootIDs is a list of root ids in a proof set.","type":"array","items":{"type":"integer"}}}},"mk20.RetrievalV1":{"type":"object","properties":{"announce_payload":{"description":"AnnouncePayload indicates whether the payload should be announced to IPNI.","type":"boolean"},"announce_piece":{"description":"AnnouncePiece indicates whether the piece information should be announced to IPNI.","type":"boolean"},"indexing":{"description":"Indexing indicates if the deal is to be indexed in the provider's system to support CIDs based retrieval","type":"boolean"}}}}}} ``` ## Upload the deal data > Allows uploading data for deals in a single stream. Suitable for small deals. ```json {"openapi":"3.1.1","info":{"title":"Curio Market 2.0 API","version":"0.0.1"},"paths":{"/upload/{id}":{"put":{"description":"Allows uploading data for deals in a single stream. Suitable for small deals.","summary":"Upload the deal data","parameters":[{"schema":{"type":"string"},"description":"id","name":"id","in":"path","required":true}],"responses":{"200":{"description":"UploadOk indicates a successful upload operation, represented by the HTTP status code 200","content":{"application/json":{"schema":{"$ref":"#/components/schemas/mk20.UploadCode"}}}},"400":{"description":"Bad Request - Invalid input or validation error","content":{"application/json":{"schema":{"type":"string"}}}},"404":{"description":"UploadStartCodeDealNotFound represents a 404 status indicating the deal was not found during the upload start process","content":{"application/json":{"schema":{"$ref":"#/components/schemas/mk20.UploadStartCode"}}}},"500":{"description":"UploadServerError indicates a server-side error occurred during the upload process, represented by the HTTP status code 500","content":{"application/json":{"schema":{"$ref":"#/components/schemas/mk20.UploadCode"}}}}},"requestBody":{"content":{"application/json":{"schema":{"type":"array","items":{"type":"integer"}}}},"description":"raw binary","required":true}}}},"components":{"schemas":{"mk20.UploadCode":{"type":"integer","enum":[200,400,404,409,500]},"mk20.UploadStartCode":{"type":"integer","enum":[200,400,404,409,500]}}}} ``` ## Finalizes the upload process > Finalizes the upload process once all the chunks are uploaded. ```json {"openapi":"3.1.1","info":{"title":"Curio Market 2.0 API","version":"0.0.1"},"paths":{"/uploads/finalize/{id}":{"post":{"description":"Finalizes the upload process once all the chunks are uploaded.","summary":"Finalizes the upload process","parameters":[{"schema":{"type":"string"},"description":"id","name":"id","in":"path","required":true}],"responses":{"200":{"description":"Ok represents a successful operation with an HTTP status code of 200","content":{"application/json":{"schema":{"$ref":"#/components/schemas/mk20.DealCode"}}}},"400":{"description":"Bad Request - Invalid input or validation error","content":{"application/json":{"schema":{"type":"string"}}}},"404":{"description":"ErrDealNotFound indicates that the specified deal could not be found, corresponding to the HTTP status code 404","content":{"application/json":{"schema":{"$ref":"#/components/schemas/mk20.DealCode"}}}},"422":{"description":"ErrUnsupportedDataSource indicates the specified data source is not supported or disabled for use in the current context","content":{"application/json":{"schema":{"$ref":"#/components/schemas/mk20.DealCode"}}}},"423":{"description":"ErrUnsupportedProduct indicates that the requested product is not supported by the provider","content":{"application/json":{"schema":{"$ref":"#/components/schemas/mk20.DealCode"}}}},"424":{"description":"ErrProductNotEnabled indicates that the requested product is not enabled on the provider","content":{"application/json":{"schema":{"$ref":"#/components/schemas/mk20.DealCode"}}}},"425":{"description":"ErrProductValidationFailed indicates a failure during product-specific validation due to invalid or missing data","content":{"application/json":{"schema":{"$ref":"#/components/schemas/mk20.DealCode"}}}},"426":{"description":"ErrDealRejectedByMarket indicates that a proposed deal was rejected by the market for not meeting its acceptance criteria or rules","content":{"application/json":{"schema":{"$ref":"#/components/schemas/mk20.DealCode"}}}},"429":{"description":"ErrServiceOverloaded indicates that the service is overloaded and cannot process the request at the moment","content":{"application/json":{"schema":{"$ref":"#/components/schemas/mk20.DealCode"}}}},"430":{"description":"ErrMalformedDataSource indicates that the provided data source is incorrectly formatted or contains invalid data","content":{"application/json":{"schema":{"$ref":"#/components/schemas/mk20.DealCode"}}}},"440":{"description":"ErrMarketNotEnabled indicates that the market is not enabled for the requested operation","content":{"application/json":{"schema":{"$ref":"#/components/schemas/mk20.DealCode"}}}},"441":{"description":"ErrDurationTooShort indicates that the provided duration value does not meet the minimum required threshold","content":{"application/json":{"schema":{"$ref":"#/components/schemas/mk20.DealCode"}}}},"500":{"description":"ErrServerInternalError indicates an internal server error with a corresponding error code of 500","content":{"application/json":{"schema":{"$ref":"#/components/schemas/mk20.DealCode"}}}},"503":{"description":"ErrServiceMaintenance indicates that the service is temporarily unavailable due to maintenance, corresponding to HTTP status code 503","content":{"application/json":{"schema":{"$ref":"#/components/schemas/mk20.DealCode"}}}}},"requestBody":{"content":{"application/json":{"schema":{"$ref":"#/components/schemas/mk20.Deal"}}},"description":"mk20.deal in json format"}}}},"components":{"schemas":{"mk20.DealCode":{"type":"integer","enum":[200,401,400,404,430,422,423,424,425,426,500,503,429,440,441]},"mk20.Deal":{"type":"object","properties":{"client":{"description":"Client wallet for the deal","allOf":[{"$ref":"#/components/schemas/address.Address"}]},"data":{"description":"Data represents the source of piece data and associated metadata.","allOf":[{"$ref":"#/components/schemas/mk20.DataSource"}]},"identifier":{"description":"Identifier represents a unique identifier for the deal in UUID format.","type":"array","items":{"type":"integer"}},"products":{"description":"Products represents a collection of product-specific information associated with a deal","allOf":[{"$ref":"#/components/schemas/mk20.Products"}]}}},"address.Address":{"type":"object"},"mk20.DataSource":{"type":"object","properties":{"format":{"description":"Format defines the format of the piece data, which can include CAR, Aggregate, or Raw formats.","allOf":[{"$ref":"#/components/schemas/mk20.PieceDataFormat"}]},"piece_cid":{"description":"PieceCID represents the unique identifier (pieceCID V2) for a piece of data, stored as a CID object.","allOf":[{"$ref":"#/components/schemas/cid.Cid"}]},"source_aggregate":{"description":"SourceAggregate represents an aggregated source, comprising multiple data sources as pieces.","allOf":[{"$ref":"#/components/schemas/mk20.DataSourceAggregate"}]},"source_http":{"description":"SourceHTTP represents the HTTP-based source of piece data within a deal, including raw size and URLs for retrieval.","allOf":[{"$ref":"#/components/schemas/mk20.DataSourceHTTP"}]},"source_httpput":{"description":"SourceHTTPPut // allow clients to push piece data after deal accepted, sort of like offline import","allOf":[{"$ref":"#/components/schemas/mk20.DataSourceHttpPut"}]},"source_offline":{"description":"SourceOffline defines the data source for offline pieces, including raw size information.","allOf":[{"$ref":"#/components/schemas/mk20.DataSourceOffline"}]}}},"mk20.PieceDataFormat":{"type":"object","properties":{"aggregate":{"description":"Aggregate holds a reference to the aggregated format of piece data.","allOf":[{"$ref":"#/components/schemas/mk20.FormatAggregate"}]},"car":{"description":"Car represents the optional CAR file format, including its metadata and versioning details.","allOf":[{"$ref":"#/components/schemas/mk20.FormatCar"}]},"raw":{"description":"Raw represents the raw format of the piece data, encapsulated as bytes.","allOf":[{"$ref":"#/components/schemas/mk20.FormatBytes"}]}}},"mk20.FormatAggregate":{"type":"object","properties":{"sub":{"description":"Sub holds a slice of DataSource, representing details of sub pieces aggregated under this format.\nThe order must be same as segment index to avoid incorrect indexing of sub pieces in an aggregate","type":"array","items":{"$ref":"#/components/schemas/mk20.DataSource"}},"type":{"description":"Type specifies the type of aggregation for data pieces, represented by an AggregateType value.","allOf":[{"$ref":"#/components/schemas/mk20.AggregateType"}]}}},"mk20.AggregateType":{"type":"integer","enum":[0,1]},"mk20.FormatCar":{"type":"object"},"mk20.FormatBytes":{"type":"object"},"cid.Cid":{"type":"object"},"mk20.DataSourceAggregate":{"type":"object","properties":{"pieces":{"type":"array","items":{"$ref":"#/components/schemas/mk20.DataSource"}}}},"mk20.DataSourceHTTP":{"type":"object","properties":{"urls":{"description":"URLs lists the HTTP endpoints where the piece data can be fetched.","type":"array","items":{"$ref":"#/components/schemas/mk20.HttpUrl"}}}},"mk20.HttpUrl":{"type":"object","properties":{"fallback":{"description":"Fallback indicates whether this URL serves as a fallback option when other URLs fail.","type":"boolean"},"headers":{"description":"HTTPHeaders represents the HTTP headers associated with the URL.","allOf":[{"$ref":"#/components/schemas/http.Header"}]},"priority":{"description":"Priority indicates the order preference for using the URL in requests, with lower values having higher priority.","type":"integer"},"url":{"description":"URL specifies the HTTP endpoint where the piece data can be fetched.","type":"string"}}},"http.Header":{"type":"object","additionalProperties":{"type":"array","items":{"type":"string"}}},"mk20.DataSourceHttpPut":{"type":"object"},"mk20.DataSourceOffline":{"type":"object"},"mk20.Products":{"type":"object","properties":{"ddo_v1":{"description":"DDOV1 represents a product v1 configuration for Direct Data Onboarding (DDO)","allOf":[{"$ref":"#/components/schemas/mk20.DDOV1"}]},"pdp_v1":{"description":"PDPV1 represents product-specific configuration for PDP version 1 deals.","allOf":[{"$ref":"#/components/schemas/mk20.PDPV1"}]},"retrieval_v1":{"description":"RetrievalV1 represents configuration for retrieval settings in the system, including indexing and announcement flags.","allOf":[{"$ref":"#/components/schemas/mk20.RetrievalV1"}]}}},"mk20.DDOV1":{"type":"object","properties":{"allocation_id":{"description":"AllocationId represents an aggregated allocation identifier for the deal.","allOf":[{"$ref":"#/components/schemas/github_com_filecoin-project_go-state-types_builtin_v16_verifreg.AllocationId"}]},"contract_address":{"description":"ContractAddress specifies the address of the contract governing the deal","type":"string"},"contract_verify_method":{"description":"ContractDealIDMethod specifies the method name to verify the deal and retrieve the deal ID for a contract","type":"string"},"contract_verify_method_params":{"description":"ContractDealIDMethodParams represents encoded parameters for the contract verify method if required by the contract","type":"array","items":{"type":"integer"}},"duration":{"description":"Duration represents the deal duration in epochs. This value is ignored for the deal with allocationID.\nIt must be at least 518400","type":"integer"},"notification_address":{"description":"NotificationAddress specifies the address to which notifications will be relayed to when sector is activated","type":"string"},"notification_payload":{"description":"NotificationPayload holds the notification data typically in a serialized byte array format.","type":"array","items":{"type":"integer"}},"piece_manager":{"description":"Actor providing AuthorizeMessage (like f1/f3 wallet) able to authorize actions such as managing ACLs","allOf":[{"$ref":"#/components/schemas/address.Address"}]},"provider":{"description":"Provider specifies the address of the provider","allOf":[{"$ref":"#/components/schemas/address.Address"}]}}},"github_com_filecoin-project_go-state-types_builtin_v16_verifreg.AllocationId":{"type":"integer","enum":[0]},"mk20.PDPV1":{"type":"object","properties":{"add_root":{"description":"AddRoot indicated that this deal is meant to add root to a given ProofSet. ProofSetID must be defined.","type":"boolean"},"create_proof_set":{"description":"CreateProofSet indicated that this deal is meant to create a new ProofSet for the client by storage provider.","type":"boolean"},"delete_proof_set":{"description":"DeleteProofSet indicated that this deal is meant to delete an existing ProofSet created by SP for the client.\nProofSetID must be defined.","type":"boolean"},"delete_root":{"description":"DeleteRoot indicates whether the root of the data should be deleted. ProofSetID must be defined.","type":"boolean"},"extra_data":{"description":"ExtraData can be used to send additional information to service contract when Verifier action like AddRoot, DeleteRoot etc. are performed.","type":"array","items":{"type":"integer"}},"proof_set_id":{"description":"ProofSetID is PDP verified contract proofset ID. It must be defined for all deals except when CreateProofSet is true.","type":"integer"},"record_keeper":{"description":"RecordKeeper specifies the record keeper contract address for the new PDP proofset.","type":"string"},"root_ids":{"description":"RootIDs is a list of root ids in a proof set.","type":"array","items":{"type":"integer"}}}},"mk20.RetrievalV1":{"type":"object","properties":{"announce_payload":{"description":"AnnouncePayload indicates whether the payload should be announced to IPNI.","type":"boolean"},"announce_piece":{"description":"AnnouncePiece indicates whether the piece information should be announced to IPNI.","type":"boolean"},"indexing":{"description":"Indexing indicates if the deal is to be indexed in the provider's system to support CIDs based retrieval","type":"boolean"}}}}}} ``` ## Status of deal upload > Return a json struct detailing the current status of a deal upload. ```json {"openapi":"3.1.1","info":{"title":"Curio Market 2.0 API","version":"0.0.1"},"paths":{"/uploads/{id}":{"get":{"description":"Return a json struct detailing the current status of a deal upload.","summary":"Status of deal upload","parameters":[{"schema":{"type":"string"},"description":"id","name":"id","in":"path","required":true}],"responses":{"200":{"description":"The status of a file upload process, including progress and missing chunks","content":{"application/json":{"schema":{"$ref":"#/components/schemas/mk20.UploadStatus"}}}},"400":{"description":"Bad Request - Invalid input or validation error","content":{"application/json":{"schema":{"type":"string"}}}},"404":{"description":"UploadStatusCodeDealNotFound indicates that the requested deal was not found, corresponding to status code 404","content":{"application/json":{"schema":{"$ref":"#/components/schemas/mk20.UploadStatusCode"}}}},"425":{"description":"UploadStatusCodeUploadNotStarted indicates that the upload process has not started yet","content":{"application/json":{"schema":{"$ref":"#/components/schemas/mk20.UploadStatusCode"}}}},"500":{"description":"UploadStatusCodeServerError indicates an internal server error occurred during the upload process, corresponding to status code 500","content":{"application/json":{"schema":{"$ref":"#/components/schemas/mk20.UploadStatusCode"}}}}}}}},"components":{"schemas":{"mk20.UploadStatus":{"type":"object","properties":{"missing":{"description":"Missing represents the number of chunks that are not yet uploaded.","type":"integer"},"missing_chunks":{"description":"MissingChunks is a slice containing the indices of missing chunks.","type":"array","items":{"type":"integer"}},"total_chunks":{"description":"TotalChunks represents the total number of chunks required for the upload.","type":"integer"},"uploaded":{"description":"Uploaded represents the number of chunks successfully uploaded.","type":"integer"},"uploaded_chunks":{"description":"UploadedChunks is a slice containing the indices of successfully uploaded chunks.","type":"array","items":{"type":"integer"}}}},"mk20.UploadStatusCode":{"type":"integer","enum":[200,404,425,500]}}}} ``` ## Starts the upload process > Initializes the upload for a deal. Each upload must be initialized before chunks can be uploaded for a deal. ```json {"openapi":"3.1.1","info":{"title":"Curio Market 2.0 API","version":"0.0.1"},"paths":{"/uploads/{id}":{"post":{"description":"Initializes the upload for a deal. Each upload must be initialized before chunks can be uploaded for a deal.","summary":"Starts the upload process","parameters":[{"schema":{"type":"string"},"description":"id","name":"id","in":"path","required":true}],"responses":{"200":{"description":"UploadStartCodeOk indicates a successful upload start request with status code 200","content":{"application/json":{"schema":{"$ref":"#/components/schemas/mk20.UploadStartCode"}}}},"400":{"description":"Bad Request - Invalid input or validation error","content":{"application/json":{"schema":{"type":"string"}}}},"404":{"description":"UploadStartCodeDealNotFound represents a 404 status indicating the deal was not found during the upload start process","content":{"application/json":{"schema":{"$ref":"#/components/schemas/mk20.UploadStartCode"}}}},"409":{"description":"UploadStartCodeAlreadyStarted indicates that the upload process has already been initiated and cannot be started again","content":{"application/json":{"schema":{"$ref":"#/components/schemas/mk20.UploadStartCode"}}}},"500":{"description":"UploadStartCodeServerError indicates an error occurred on the server while processing an upload start request","content":{"application/json":{"schema":{"$ref":"#/components/schemas/mk20.UploadStartCode"}}}}},"requestBody":{"content":{"application/json":{"schema":{"$ref":"#/components/schemas/mk20.StartUpload"}}},"description":"Metadata for initiating an upload operation","required":true}}}},"components":{"schemas":{"mk20.UploadStartCode":{"type":"integer","enum":[200,400,404,409,500]},"mk20.StartUpload":{"type":"object","properties":{"chunk_size":{"description":"ChunkSize defines the size of each data chunk to be used during the upload process.","type":"integer"},"raw_size":{"description":"RawSize indicates the total size of the data to be uploaded in bytes.","type":"integer"}}}}}} ``` ## Upload a file chunk > Allows uploading chunks for a deal file. Method can be called in parallel to speed up uploads. ```json {"openapi":"3.1.1","info":{"title":"Curio Market 2.0 API","version":"0.0.1"},"paths":{"/uploads/{id}/{chunkNum}":{"put":{"description":"Allows uploading chunks for a deal file. Method can be called in parallel to speed up uploads.","summary":"Upload a file chunk","parameters":[{"schema":{"type":"string"},"description":"id","name":"id","in":"path","required":true},{"schema":{"type":"string"},"description":"chunkNum","name":"chunkNum","in":"path","required":true}],"responses":{"200":{"description":"UploadOk indicates a successful upload operation, represented by the HTTP status code 200","content":{"application/json":{"schema":{"$ref":"#/components/schemas/mk20.UploadCode"}}}},"400":{"description":"Bad Request - Invalid input or validation error","content":{"application/json":{"schema":{"type":"string"}}}},"404":{"description":"UploadNotFound represents an error where the requested upload chunk could not be found, typically corresponding to HTTP status 404","content":{"application/json":{"schema":{"$ref":"#/components/schemas/mk20.UploadCode"}}}},"409":{"description":"UploadChunkAlreadyUploaded indicates that the chunk has already been uploaded and cannot be re-uploaded","content":{"application/json":{"schema":{"$ref":"#/components/schemas/mk20.UploadCode"}}}},"500":{"description":"UploadServerError indicates a server-side error occurred during the upload process, represented by the HTTP status code 500","content":{"application/json":{"schema":{"$ref":"#/components/schemas/mk20.UploadCode"}}}}},"requestBody":{"content":{"application/json":{"schema":{"type":"array","items":{"type":"integer"}}}},"description":"raw binary","required":true}}}},"components":{"schemas":{"mk20.UploadCode":{"type":"integer","enum":[200,400,404,409,500]}}}} ``` ## The address.Address object ```json {"openapi":"3.1.1","info":{"title":"Curio Market 2.0 API","version":"0.0.1"},"components":{"schemas":{"address.Address":{"type":"object"}}}} ``` ## The cid.Cid object ```json {"openapi":"3.1.1","info":{"title":"Curio Market 2.0 API","version":"0.0.1"},"components":{"schemas":{"cid.Cid":{"type":"object"}}}} ``` ## The github\_com\_filecoin-project\_go-state-types\_builtin\_v16\_verifreg.AllocationId object ```json {"openapi":"3.1.1","info":{"title":"Curio Market 2.0 API","version":"0.0.1"},"components":{"schemas":{"github_com_filecoin-project_go-state-types_builtin_v16_verifreg.AllocationId":{"type":"integer","enum":[0]}}}} ``` ## The http.Header object ```json {"openapi":"3.1.1","info":{"title":"Curio Market 2.0 API","version":"0.0.1"},"components":{"schemas":{"http.Header":{"type":"object","additionalProperties":{"type":"array","items":{"type":"string"}}}}}} ``` ## The mk20.AggregateType object ```json {"openapi":"3.1.1","info":{"title":"Curio Market 2.0 API","version":"0.0.1"},"components":{"schemas":{"mk20.AggregateType":{"type":"integer","enum":[0,1]}}}} ``` ## The mk20.DDOV1 object ```json {"openapi":"3.1.1","info":{"title":"Curio Market 2.0 API","version":"0.0.1"},"components":{"schemas":{"mk20.DDOV1":{"type":"object","properties":{"allocation_id":{"description":"AllocationId represents an aggregated allocation identifier for the deal.","allOf":[{"$ref":"#/components/schemas/github_com_filecoin-project_go-state-types_builtin_v16_verifreg.AllocationId"}]},"contract_address":{"description":"ContractAddress specifies the address of the contract governing the deal","type":"string"},"contract_verify_method":{"description":"ContractDealIDMethod specifies the method name to verify the deal and retrieve the deal ID for a contract","type":"string"},"contract_verify_method_params":{"description":"ContractDealIDMethodParams represents encoded parameters for the contract verify method if required by the contract","type":"array","items":{"type":"integer"}},"duration":{"description":"Duration represents the deal duration in epochs. This value is ignored for the deal with allocationID.\nIt must be at least 518400","type":"integer"},"notification_address":{"description":"NotificationAddress specifies the address to which notifications will be relayed to when sector is activated","type":"string"},"notification_payload":{"description":"NotificationPayload holds the notification data typically in a serialized byte array format.","type":"array","items":{"type":"integer"}},"piece_manager":{"description":"Actor providing AuthorizeMessage (like f1/f3 wallet) able to authorize actions such as managing ACLs","allOf":[{"$ref":"#/components/schemas/address.Address"}]},"provider":{"description":"Provider specifies the address of the provider","allOf":[{"$ref":"#/components/schemas/address.Address"}]}}},"github_com_filecoin-project_go-state-types_builtin_v16_verifreg.AllocationId":{"type":"integer","enum":[0]},"address.Address":{"type":"object"}}}} ``` ## The mk20.DataSource object ```json {"openapi":"3.1.1","info":{"title":"Curio Market 2.0 API","version":"0.0.1"},"components":{"schemas":{"mk20.DataSource":{"type":"object","properties":{"format":{"description":"Format defines the format of the piece data, which can include CAR, Aggregate, or Raw formats.","allOf":[{"$ref":"#/components/schemas/mk20.PieceDataFormat"}]},"piece_cid":{"description":"PieceCID represents the unique identifier (pieceCID V2) for a piece of data, stored as a CID object.","allOf":[{"$ref":"#/components/schemas/cid.Cid"}]},"source_aggregate":{"description":"SourceAggregate represents an aggregated source, comprising multiple data sources as pieces.","allOf":[{"$ref":"#/components/schemas/mk20.DataSourceAggregate"}]},"source_http":{"description":"SourceHTTP represents the HTTP-based source of piece data within a deal, including raw size and URLs for retrieval.","allOf":[{"$ref":"#/components/schemas/mk20.DataSourceHTTP"}]},"source_httpput":{"description":"SourceHTTPPut // allow clients to push piece data after deal accepted, sort of like offline import","allOf":[{"$ref":"#/components/schemas/mk20.DataSourceHttpPut"}]},"source_offline":{"description":"SourceOffline defines the data source for offline pieces, including raw size information.","allOf":[{"$ref":"#/components/schemas/mk20.DataSourceOffline"}]}}},"mk20.PieceDataFormat":{"type":"object","properties":{"aggregate":{"description":"Aggregate holds a reference to the aggregated format of piece data.","allOf":[{"$ref":"#/components/schemas/mk20.FormatAggregate"}]},"car":{"description":"Car represents the optional CAR file format, including its metadata and versioning details.","allOf":[{"$ref":"#/components/schemas/mk20.FormatCar"}]},"raw":{"description":"Raw represents the raw format of the piece data, encapsulated as bytes.","allOf":[{"$ref":"#/components/schemas/mk20.FormatBytes"}]}}},"mk20.FormatAggregate":{"type":"object","properties":{"sub":{"description":"Sub holds a slice of DataSource, representing details of sub pieces aggregated under this format.\nThe order must be same as segment index to avoid incorrect indexing of sub pieces in an aggregate","type":"array","items":{"$ref":"#/components/schemas/mk20.DataSource"}},"type":{"description":"Type specifies the type of aggregation for data pieces, represented by an AggregateType value.","allOf":[{"$ref":"#/components/schemas/mk20.AggregateType"}]}}},"cid.Cid":{"type":"object"},"mk20.DataSourceAggregate":{"type":"object","properties":{"pieces":{"type":"array","items":{"$ref":"#/components/schemas/mk20.DataSource"}}}},"mk20.DataSourceHTTP":{"type":"object","properties":{"urls":{"description":"URLs lists the HTTP endpoints where the piece data can be fetched.","type":"array","items":{"$ref":"#/components/schemas/mk20.HttpUrl"}}}},"mk20.HttpUrl":{"type":"object","properties":{"fallback":{"description":"Fallback indicates whether this URL serves as a fallback option when other URLs fail.","type":"boolean"},"headers":{"description":"HTTPHeaders represents the HTTP headers associated with the URL.","allOf":[{"$ref":"#/components/schemas/http.Header"}]},"priority":{"description":"Priority indicates the order preference for using the URL in requests, with lower values having higher priority.","type":"integer"},"url":{"description":"URL specifies the HTTP endpoint where the piece data can be fetched.","type":"string"}}},"http.Header":{"type":"object","additionalProperties":{"type":"array","items":{"type":"string"}}},"mk20.DataSourceHttpPut":{"type":"object"},"mk20.DataSourceOffline":{"type":"object"},"mk20.AggregateType":{"type":"integer","enum":[0,1]},"mk20.FormatCar":{"type":"object"},"mk20.FormatBytes":{"type":"object"}}}} ``` ## The mk20.DataSourceAggregate object ```json {"openapi":"3.1.1","info":{"title":"Curio Market 2.0 API","version":"0.0.1"},"components":{"schemas":{"mk20.DataSourceAggregate":{"type":"object","properties":{"pieces":{"type":"array","items":{"$ref":"#/components/schemas/mk20.DataSource"}}}},"mk20.DataSource":{"type":"object","properties":{"format":{"description":"Format defines the format of the piece data, which can include CAR, Aggregate, or Raw formats.","allOf":[{"$ref":"#/components/schemas/mk20.PieceDataFormat"}]},"piece_cid":{"description":"PieceCID represents the unique identifier (pieceCID V2) for a piece of data, stored as a CID object.","allOf":[{"$ref":"#/components/schemas/cid.Cid"}]},"source_aggregate":{"description":"SourceAggregate represents an aggregated source, comprising multiple data sources as pieces.","allOf":[{"$ref":"#/components/schemas/mk20.DataSourceAggregate"}]},"source_http":{"description":"SourceHTTP represents the HTTP-based source of piece data within a deal, including raw size and URLs for retrieval.","allOf":[{"$ref":"#/components/schemas/mk20.DataSourceHTTP"}]},"source_httpput":{"description":"SourceHTTPPut // allow clients to push piece data after deal accepted, sort of like offline import","allOf":[{"$ref":"#/components/schemas/mk20.DataSourceHttpPut"}]},"source_offline":{"description":"SourceOffline defines the data source for offline pieces, including raw size information.","allOf":[{"$ref":"#/components/schemas/mk20.DataSourceOffline"}]}}},"mk20.PieceDataFormat":{"type":"object","properties":{"aggregate":{"description":"Aggregate holds a reference to the aggregated format of piece data.","allOf":[{"$ref":"#/components/schemas/mk20.FormatAggregate"}]},"car":{"description":"Car represents the optional CAR file format, including its metadata and versioning details.","allOf":[{"$ref":"#/components/schemas/mk20.FormatCar"}]},"raw":{"description":"Raw represents the raw format of the piece data, encapsulated as bytes.","allOf":[{"$ref":"#/components/schemas/mk20.FormatBytes"}]}}},"mk20.FormatAggregate":{"type":"object","properties":{"sub":{"description":"Sub holds a slice of DataSource, representing details of sub pieces aggregated under this format.\nThe order must be same as segment index to avoid incorrect indexing of sub pieces in an aggregate","type":"array","items":{"$ref":"#/components/schemas/mk20.DataSource"}},"type":{"description":"Type specifies the type of aggregation for data pieces, represented by an AggregateType value.","allOf":[{"$ref":"#/components/schemas/mk20.AggregateType"}]}}},"mk20.AggregateType":{"type":"integer","enum":[0,1]},"mk20.FormatCar":{"type":"object"},"mk20.FormatBytes":{"type":"object"},"cid.Cid":{"type":"object"},"mk20.DataSourceHTTP":{"type":"object","properties":{"urls":{"description":"URLs lists the HTTP endpoints where the piece data can be fetched.","type":"array","items":{"$ref":"#/components/schemas/mk20.HttpUrl"}}}},"mk20.HttpUrl":{"type":"object","properties":{"fallback":{"description":"Fallback indicates whether this URL serves as a fallback option when other URLs fail.","type":"boolean"},"headers":{"description":"HTTPHeaders represents the HTTP headers associated with the URL.","allOf":[{"$ref":"#/components/schemas/http.Header"}]},"priority":{"description":"Priority indicates the order preference for using the URL in requests, with lower values having higher priority.","type":"integer"},"url":{"description":"URL specifies the HTTP endpoint where the piece data can be fetched.","type":"string"}}},"http.Header":{"type":"object","additionalProperties":{"type":"array","items":{"type":"string"}}},"mk20.DataSourceHttpPut":{"type":"object"},"mk20.DataSourceOffline":{"type":"object"}}}} ``` ## The mk20.DataSourceHTTP object ```json {"openapi":"3.1.1","info":{"title":"Curio Market 2.0 API","version":"0.0.1"},"components":{"schemas":{"mk20.DataSourceHTTP":{"type":"object","properties":{"urls":{"description":"URLs lists the HTTP endpoints where the piece data can be fetched.","type":"array","items":{"$ref":"#/components/schemas/mk20.HttpUrl"}}}},"mk20.HttpUrl":{"type":"object","properties":{"fallback":{"description":"Fallback indicates whether this URL serves as a fallback option when other URLs fail.","type":"boolean"},"headers":{"description":"HTTPHeaders represents the HTTP headers associated with the URL.","allOf":[{"$ref":"#/components/schemas/http.Header"}]},"priority":{"description":"Priority indicates the order preference for using the URL in requests, with lower values having higher priority.","type":"integer"},"url":{"description":"URL specifies the HTTP endpoint where the piece data can be fetched.","type":"string"}}},"http.Header":{"type":"object","additionalProperties":{"type":"array","items":{"type":"string"}}}}}} ``` ## The mk20.DataSourceHttpPut object ```json {"openapi":"3.1.1","info":{"title":"Curio Market 2.0 API","version":"0.0.1"},"components":{"schemas":{"mk20.DataSourceHttpPut":{"type":"object"}}}} ``` ## The mk20.DataSourceOffline object ```json {"openapi":"3.1.1","info":{"title":"Curio Market 2.0 API","version":"0.0.1"},"components":{"schemas":{"mk20.DataSourceOffline":{"type":"object"}}}} ``` ## The mk20.Deal object ```json {"openapi":"3.1.1","info":{"title":"Curio Market 2.0 API","version":"0.0.1"},"components":{"schemas":{"mk20.Deal":{"type":"object","properties":{"client":{"description":"Client wallet for the deal","allOf":[{"$ref":"#/components/schemas/address.Address"}]},"data":{"description":"Data represents the source of piece data and associated metadata.","allOf":[{"$ref":"#/components/schemas/mk20.DataSource"}]},"identifier":{"description":"Identifier represents a unique identifier for the deal in UUID format.","type":"array","items":{"type":"integer"}},"products":{"description":"Products represents a collection of product-specific information associated with a deal","allOf":[{"$ref":"#/components/schemas/mk20.Products"}]}}},"address.Address":{"type":"object"},"mk20.DataSource":{"type":"object","properties":{"format":{"description":"Format defines the format of the piece data, which can include CAR, Aggregate, or Raw formats.","allOf":[{"$ref":"#/components/schemas/mk20.PieceDataFormat"}]},"piece_cid":{"description":"PieceCID represents the unique identifier (pieceCID V2) for a piece of data, stored as a CID object.","allOf":[{"$ref":"#/components/schemas/cid.Cid"}]},"source_aggregate":{"description":"SourceAggregate represents an aggregated source, comprising multiple data sources as pieces.","allOf":[{"$ref":"#/components/schemas/mk20.DataSourceAggregate"}]},"source_http":{"description":"SourceHTTP represents the HTTP-based source of piece data within a deal, including raw size and URLs for retrieval.","allOf":[{"$ref":"#/components/schemas/mk20.DataSourceHTTP"}]},"source_httpput":{"description":"SourceHTTPPut // allow clients to push piece data after deal accepted, sort of like offline import","allOf":[{"$ref":"#/components/schemas/mk20.DataSourceHttpPut"}]},"source_offline":{"description":"SourceOffline defines the data source for offline pieces, including raw size information.","allOf":[{"$ref":"#/components/schemas/mk20.DataSourceOffline"}]}}},"mk20.PieceDataFormat":{"type":"object","properties":{"aggregate":{"description":"Aggregate holds a reference to the aggregated format of piece data.","allOf":[{"$ref":"#/components/schemas/mk20.FormatAggregate"}]},"car":{"description":"Car represents the optional CAR file format, including its metadata and versioning details.","allOf":[{"$ref":"#/components/schemas/mk20.FormatCar"}]},"raw":{"description":"Raw represents the raw format of the piece data, encapsulated as bytes.","allOf":[{"$ref":"#/components/schemas/mk20.FormatBytes"}]}}},"mk20.FormatAggregate":{"type":"object","properties":{"sub":{"description":"Sub holds a slice of DataSource, representing details of sub pieces aggregated under this format.\nThe order must be same as segment index to avoid incorrect indexing of sub pieces in an aggregate","type":"array","items":{"$ref":"#/components/schemas/mk20.DataSource"}},"type":{"description":"Type specifies the type of aggregation for data pieces, represented by an AggregateType value.","allOf":[{"$ref":"#/components/schemas/mk20.AggregateType"}]}}},"mk20.AggregateType":{"type":"integer","enum":[0,1]},"mk20.FormatCar":{"type":"object"},"mk20.FormatBytes":{"type":"object"},"cid.Cid":{"type":"object"},"mk20.DataSourceAggregate":{"type":"object","properties":{"pieces":{"type":"array","items":{"$ref":"#/components/schemas/mk20.DataSource"}}}},"mk20.DataSourceHTTP":{"type":"object","properties":{"urls":{"description":"URLs lists the HTTP endpoints where the piece data can be fetched.","type":"array","items":{"$ref":"#/components/schemas/mk20.HttpUrl"}}}},"mk20.HttpUrl":{"type":"object","properties":{"fallback":{"description":"Fallback indicates whether this URL serves as a fallback option when other URLs fail.","type":"boolean"},"headers":{"description":"HTTPHeaders represents the HTTP headers associated with the URL.","allOf":[{"$ref":"#/components/schemas/http.Header"}]},"priority":{"description":"Priority indicates the order preference for using the URL in requests, with lower values having higher priority.","type":"integer"},"url":{"description":"URL specifies the HTTP endpoint where the piece data can be fetched.","type":"string"}}},"http.Header":{"type":"object","additionalProperties":{"type":"array","items":{"type":"string"}}},"mk20.DataSourceHttpPut":{"type":"object"},"mk20.DataSourceOffline":{"type":"object"},"mk20.Products":{"type":"object","properties":{"ddo_v1":{"description":"DDOV1 represents a product v1 configuration for Direct Data Onboarding (DDO)","allOf":[{"$ref":"#/components/schemas/mk20.DDOV1"}]},"pdp_v1":{"description":"PDPV1 represents product-specific configuration for PDP version 1 deals.","allOf":[{"$ref":"#/components/schemas/mk20.PDPV1"}]},"retrieval_v1":{"description":"RetrievalV1 represents configuration for retrieval settings in the system, including indexing and announcement flags.","allOf":[{"$ref":"#/components/schemas/mk20.RetrievalV1"}]}}},"mk20.DDOV1":{"type":"object","properties":{"allocation_id":{"description":"AllocationId represents an aggregated allocation identifier for the deal.","allOf":[{"$ref":"#/components/schemas/github_com_filecoin-project_go-state-types_builtin_v16_verifreg.AllocationId"}]},"contract_address":{"description":"ContractAddress specifies the address of the contract governing the deal","type":"string"},"contract_verify_method":{"description":"ContractDealIDMethod specifies the method name to verify the deal and retrieve the deal ID for a contract","type":"string"},"contract_verify_method_params":{"description":"ContractDealIDMethodParams represents encoded parameters for the contract verify method if required by the contract","type":"array","items":{"type":"integer"}},"duration":{"description":"Duration represents the deal duration in epochs. This value is ignored for the deal with allocationID.\nIt must be at least 518400","type":"integer"},"notification_address":{"description":"NotificationAddress specifies the address to which notifications will be relayed to when sector is activated","type":"string"},"notification_payload":{"description":"NotificationPayload holds the notification data typically in a serialized byte array format.","type":"array","items":{"type":"integer"}},"piece_manager":{"description":"Actor providing AuthorizeMessage (like f1/f3 wallet) able to authorize actions such as managing ACLs","allOf":[{"$ref":"#/components/schemas/address.Address"}]},"provider":{"description":"Provider specifies the address of the provider","allOf":[{"$ref":"#/components/schemas/address.Address"}]}}},"github_com_filecoin-project_go-state-types_builtin_v16_verifreg.AllocationId":{"type":"integer","enum":[0]},"mk20.PDPV1":{"type":"object","properties":{"add_root":{"description":"AddRoot indicated that this deal is meant to add root to a given ProofSet. ProofSetID must be defined.","type":"boolean"},"create_proof_set":{"description":"CreateProofSet indicated that this deal is meant to create a new ProofSet for the client by storage provider.","type":"boolean"},"delete_proof_set":{"description":"DeleteProofSet indicated that this deal is meant to delete an existing ProofSet created by SP for the client.\nProofSetID must be defined.","type":"boolean"},"delete_root":{"description":"DeleteRoot indicates whether the root of the data should be deleted. ProofSetID must be defined.","type":"boolean"},"extra_data":{"description":"ExtraData can be used to send additional information to service contract when Verifier action like AddRoot, DeleteRoot etc. are performed.","type":"array","items":{"type":"integer"}},"proof_set_id":{"description":"ProofSetID is PDP verified contract proofset ID. It must be defined for all deals except when CreateProofSet is true.","type":"integer"},"record_keeper":{"description":"RecordKeeper specifies the record keeper contract address for the new PDP proofset.","type":"string"},"root_ids":{"description":"RootIDs is a list of root ids in a proof set.","type":"array","items":{"type":"integer"}}}},"mk20.RetrievalV1":{"type":"object","properties":{"announce_payload":{"description":"AnnouncePayload indicates whether the payload should be announced to IPNI.","type":"boolean"},"announce_piece":{"description":"AnnouncePiece indicates whether the piece information should be announced to IPNI.","type":"boolean"},"indexing":{"description":"Indexing indicates if the deal is to be indexed in the provider's system to support CIDs based retrieval","type":"boolean"}}}}}} ``` ## The mk20.DealCode object ```json {"openapi":"3.1.1","info":{"title":"Curio Market 2.0 API","version":"0.0.1"},"components":{"schemas":{"mk20.DealCode":{"type":"integer","enum":[200,401,400,404,430,422,423,424,425,426,500,503,429,440,441]}}}} ``` ## The mk20.DealProductStatusResponse object ```json {"openapi":"3.1.1","info":{"title":"Curio Market 2.0 API","version":"0.0.1"},"components":{"schemas":{"mk20.DealProductStatusResponse":{"type":"object","properties":{"ddo_v1":{"description":"DDOV1 holds the DealStatusResponse for product \"ddo_v1\".","allOf":[{"$ref":"#/components/schemas/mk20.DealStatusResponse"}]},"pdp_v1":{"description":"PDPV1 represents the DealStatusResponse for the product pdp_v1.","allOf":[{"$ref":"#/components/schemas/mk20.DealStatusResponse"}]}}},"mk20.DealStatusResponse":{"type":"object","properties":{"error_msg":{"description":"ErrorMsg is an optional field containing error details associated with the deal's current state if an error occurred.","type":"string"},"status":{"description":"State indicates the current processing state of the deal as a DealState value.","allOf":[{"$ref":"#/components/schemas/mk20.DealState"}]}}},"mk20.DealState":{"type":"string","enum":["accepted","uploading","processing","sealing","indexing","failed","complete"]}}}} ``` ## The mk20.DealState object ```json {"openapi":"3.1.1","info":{"title":"Curio Market 2.0 API","version":"0.0.1"},"components":{"schemas":{"mk20.DealState":{"type":"string","enum":["accepted","uploading","processing","sealing","indexing","failed","complete"]}}}} ``` ## The mk20.DealStatusResponse object ```json {"openapi":"3.1.1","info":{"title":"Curio Market 2.0 API","version":"0.0.1"},"components":{"schemas":{"mk20.DealStatusResponse":{"type":"object","properties":{"error_msg":{"description":"ErrorMsg is an optional field containing error details associated with the deal's current state if an error occurred.","type":"string"},"status":{"description":"State indicates the current processing state of the deal as a DealState value.","allOf":[{"$ref":"#/components/schemas/mk20.DealState"}]}}},"mk20.DealState":{"type":"string","enum":["accepted","uploading","processing","sealing","indexing","failed","complete"]}}}} ``` ## The mk20.FormatAggregate object ```json {"openapi":"3.1.1","info":{"title":"Curio Market 2.0 API","version":"0.0.1"},"components":{"schemas":{"mk20.FormatAggregate":{"type":"object","properties":{"sub":{"description":"Sub holds a slice of DataSource, representing details of sub pieces aggregated under this format.\nThe order must be same as segment index to avoid incorrect indexing of sub pieces in an aggregate","type":"array","items":{"$ref":"#/components/schemas/mk20.DataSource"}},"type":{"description":"Type specifies the type of aggregation for data pieces, represented by an AggregateType value.","allOf":[{"$ref":"#/components/schemas/mk20.AggregateType"}]}}},"mk20.DataSource":{"type":"object","properties":{"format":{"description":"Format defines the format of the piece data, which can include CAR, Aggregate, or Raw formats.","allOf":[{"$ref":"#/components/schemas/mk20.PieceDataFormat"}]},"piece_cid":{"description":"PieceCID represents the unique identifier (pieceCID V2) for a piece of data, stored as a CID object.","allOf":[{"$ref":"#/components/schemas/cid.Cid"}]},"source_aggregate":{"description":"SourceAggregate represents an aggregated source, comprising multiple data sources as pieces.","allOf":[{"$ref":"#/components/schemas/mk20.DataSourceAggregate"}]},"source_http":{"description":"SourceHTTP represents the HTTP-based source of piece data within a deal, including raw size and URLs for retrieval.","allOf":[{"$ref":"#/components/schemas/mk20.DataSourceHTTP"}]},"source_httpput":{"description":"SourceHTTPPut // allow clients to push piece data after deal accepted, sort of like offline import","allOf":[{"$ref":"#/components/schemas/mk20.DataSourceHttpPut"}]},"source_offline":{"description":"SourceOffline defines the data source for offline pieces, including raw size information.","allOf":[{"$ref":"#/components/schemas/mk20.DataSourceOffline"}]}}},"mk20.PieceDataFormat":{"type":"object","properties":{"aggregate":{"description":"Aggregate holds a reference to the aggregated format of piece data.","allOf":[{"$ref":"#/components/schemas/mk20.FormatAggregate"}]},"car":{"description":"Car represents the optional CAR file format, including its metadata and versioning details.","allOf":[{"$ref":"#/components/schemas/mk20.FormatCar"}]},"raw":{"description":"Raw represents the raw format of the piece data, encapsulated as bytes.","allOf":[{"$ref":"#/components/schemas/mk20.FormatBytes"}]}}},"mk20.AggregateType":{"type":"integer","enum":[0,1]},"mk20.FormatCar":{"type":"object"},"mk20.FormatBytes":{"type":"object"},"cid.Cid":{"type":"object"},"mk20.DataSourceAggregate":{"type":"object","properties":{"pieces":{"type":"array","items":{"$ref":"#/components/schemas/mk20.DataSource"}}}},"mk20.DataSourceHTTP":{"type":"object","properties":{"urls":{"description":"URLs lists the HTTP endpoints where the piece data can be fetched.","type":"array","items":{"$ref":"#/components/schemas/mk20.HttpUrl"}}}},"mk20.HttpUrl":{"type":"object","properties":{"fallback":{"description":"Fallback indicates whether this URL serves as a fallback option when other URLs fail.","type":"boolean"},"headers":{"description":"HTTPHeaders represents the HTTP headers associated with the URL.","allOf":[{"$ref":"#/components/schemas/http.Header"}]},"priority":{"description":"Priority indicates the order preference for using the URL in requests, with lower values having higher priority.","type":"integer"},"url":{"description":"URL specifies the HTTP endpoint where the piece data can be fetched.","type":"string"}}},"http.Header":{"type":"object","additionalProperties":{"type":"array","items":{"type":"string"}}},"mk20.DataSourceHttpPut":{"type":"object"},"mk20.DataSourceOffline":{"type":"object"}}}} ``` ## The mk20.FormatBytes object ```json {"openapi":"3.1.1","info":{"title":"Curio Market 2.0 API","version":"0.0.1"},"components":{"schemas":{"mk20.FormatBytes":{"type":"object"}}}} ``` ## The mk20.FormatCar object ```json {"openapi":"3.1.1","info":{"title":"Curio Market 2.0 API","version":"0.0.1"},"components":{"schemas":{"mk20.FormatCar":{"type":"object"}}}} ``` ## The mk20.HttpUrl object ```json {"openapi":"3.1.1","info":{"title":"Curio Market 2.0 API","version":"0.0.1"},"components":{"schemas":{"mk20.HttpUrl":{"type":"object","properties":{"fallback":{"description":"Fallback indicates whether this URL serves as a fallback option when other URLs fail.","type":"boolean"},"headers":{"description":"HTTPHeaders represents the HTTP headers associated with the URL.","allOf":[{"$ref":"#/components/schemas/http.Header"}]},"priority":{"description":"Priority indicates the order preference for using the URL in requests, with lower values having higher priority.","type":"integer"},"url":{"description":"URL specifies the HTTP endpoint where the piece data can be fetched.","type":"string"}}},"http.Header":{"type":"object","additionalProperties":{"type":"array","items":{"type":"string"}}}}}} ``` ## The mk20.PDPV1 object ```json {"openapi":"3.1.1","info":{"title":"Curio Market 2.0 API","version":"0.0.1"},"components":{"schemas":{"mk20.PDPV1":{"type":"object","properties":{"add_root":{"description":"AddRoot indicated that this deal is meant to add root to a given ProofSet. ProofSetID must be defined.","type":"boolean"},"create_proof_set":{"description":"CreateProofSet indicated that this deal is meant to create a new ProofSet for the client by storage provider.","type":"boolean"},"delete_proof_set":{"description":"DeleteProofSet indicated that this deal is meant to delete an existing ProofSet created by SP for the client.\nProofSetID must be defined.","type":"boolean"},"delete_root":{"description":"DeleteRoot indicates whether the root of the data should be deleted. ProofSetID must be defined.","type":"boolean"},"extra_data":{"description":"ExtraData can be used to send additional information to service contract when Verifier action like AddRoot, DeleteRoot etc. are performed.","type":"array","items":{"type":"integer"}},"proof_set_id":{"description":"ProofSetID is PDP verified contract proofset ID. It must be defined for all deals except when CreateProofSet is true.","type":"integer"},"record_keeper":{"description":"RecordKeeper specifies the record keeper contract address for the new PDP proofset.","type":"string"},"root_ids":{"description":"RootIDs is a list of root ids in a proof set.","type":"array","items":{"type":"integer"}}}}}}} ``` ## The mk20.PieceDataFormat object ```json {"openapi":"3.1.1","info":{"title":"Curio Market 2.0 API","version":"0.0.1"},"components":{"schemas":{"mk20.PieceDataFormat":{"type":"object","properties":{"aggregate":{"description":"Aggregate holds a reference to the aggregated format of piece data.","allOf":[{"$ref":"#/components/schemas/mk20.FormatAggregate"}]},"car":{"description":"Car represents the optional CAR file format, including its metadata and versioning details.","allOf":[{"$ref":"#/components/schemas/mk20.FormatCar"}]},"raw":{"description":"Raw represents the raw format of the piece data, encapsulated as bytes.","allOf":[{"$ref":"#/components/schemas/mk20.FormatBytes"}]}}},"mk20.FormatAggregate":{"type":"object","properties":{"sub":{"description":"Sub holds a slice of DataSource, representing details of sub pieces aggregated under this format.\nThe order must be same as segment index to avoid incorrect indexing of sub pieces in an aggregate","type":"array","items":{"$ref":"#/components/schemas/mk20.DataSource"}},"type":{"description":"Type specifies the type of aggregation for data pieces, represented by an AggregateType value.","allOf":[{"$ref":"#/components/schemas/mk20.AggregateType"}]}}},"mk20.DataSource":{"type":"object","properties":{"format":{"description":"Format defines the format of the piece data, which can include CAR, Aggregate, or Raw formats.","allOf":[{"$ref":"#/components/schemas/mk20.PieceDataFormat"}]},"piece_cid":{"description":"PieceCID represents the unique identifier (pieceCID V2) for a piece of data, stored as a CID object.","allOf":[{"$ref":"#/components/schemas/cid.Cid"}]},"source_aggregate":{"description":"SourceAggregate represents an aggregated source, comprising multiple data sources as pieces.","allOf":[{"$ref":"#/components/schemas/mk20.DataSourceAggregate"}]},"source_http":{"description":"SourceHTTP represents the HTTP-based source of piece data within a deal, including raw size and URLs for retrieval.","allOf":[{"$ref":"#/components/schemas/mk20.DataSourceHTTP"}]},"source_httpput":{"description":"SourceHTTPPut // allow clients to push piece data after deal accepted, sort of like offline import","allOf":[{"$ref":"#/components/schemas/mk20.DataSourceHttpPut"}]},"source_offline":{"description":"SourceOffline defines the data source for offline pieces, including raw size information.","allOf":[{"$ref":"#/components/schemas/mk20.DataSourceOffline"}]}}},"mk20.FormatCar":{"type":"object"},"mk20.FormatBytes":{"type":"object"},"cid.Cid":{"type":"object"},"mk20.DataSourceAggregate":{"type":"object","properties":{"pieces":{"type":"array","items":{"$ref":"#/components/schemas/mk20.DataSource"}}}},"mk20.DataSourceHTTP":{"type":"object","properties":{"urls":{"description":"URLs lists the HTTP endpoints where the piece data can be fetched.","type":"array","items":{"$ref":"#/components/schemas/mk20.HttpUrl"}}}},"mk20.HttpUrl":{"type":"object","properties":{"fallback":{"description":"Fallback indicates whether this URL serves as a fallback option when other URLs fail.","type":"boolean"},"headers":{"description":"HTTPHeaders represents the HTTP headers associated with the URL.","allOf":[{"$ref":"#/components/schemas/http.Header"}]},"priority":{"description":"Priority indicates the order preference for using the URL in requests, with lower values having higher priority.","type":"integer"},"url":{"description":"URL specifies the HTTP endpoint where the piece data can be fetched.","type":"string"}}},"http.Header":{"type":"object","additionalProperties":{"type":"array","items":{"type":"string"}}},"mk20.DataSourceHttpPut":{"type":"object"},"mk20.DataSourceOffline":{"type":"object"},"mk20.AggregateType":{"type":"integer","enum":[0,1]}}}} ``` ## The mk20.Products object ```json {"openapi":"3.1.1","info":{"title":"Curio Market 2.0 API","version":"0.0.1"},"components":{"schemas":{"mk20.Products":{"type":"object","properties":{"ddo_v1":{"description":"DDOV1 represents a product v1 configuration for Direct Data Onboarding (DDO)","allOf":[{"$ref":"#/components/schemas/mk20.DDOV1"}]},"pdp_v1":{"description":"PDPV1 represents product-specific configuration for PDP version 1 deals.","allOf":[{"$ref":"#/components/schemas/mk20.PDPV1"}]},"retrieval_v1":{"description":"RetrievalV1 represents configuration for retrieval settings in the system, including indexing and announcement flags.","allOf":[{"$ref":"#/components/schemas/mk20.RetrievalV1"}]}}},"mk20.DDOV1":{"type":"object","properties":{"allocation_id":{"description":"AllocationId represents an aggregated allocation identifier for the deal.","allOf":[{"$ref":"#/components/schemas/github_com_filecoin-project_go-state-types_builtin_v16_verifreg.AllocationId"}]},"contract_address":{"description":"ContractAddress specifies the address of the contract governing the deal","type":"string"},"contract_verify_method":{"description":"ContractDealIDMethod specifies the method name to verify the deal and retrieve the deal ID for a contract","type":"string"},"contract_verify_method_params":{"description":"ContractDealIDMethodParams represents encoded parameters for the contract verify method if required by the contract","type":"array","items":{"type":"integer"}},"duration":{"description":"Duration represents the deal duration in epochs. This value is ignored for the deal with allocationID.\nIt must be at least 518400","type":"integer"},"notification_address":{"description":"NotificationAddress specifies the address to which notifications will be relayed to when sector is activated","type":"string"},"notification_payload":{"description":"NotificationPayload holds the notification data typically in a serialized byte array format.","type":"array","items":{"type":"integer"}},"piece_manager":{"description":"Actor providing AuthorizeMessage (like f1/f3 wallet) able to authorize actions such as managing ACLs","allOf":[{"$ref":"#/components/schemas/address.Address"}]},"provider":{"description":"Provider specifies the address of the provider","allOf":[{"$ref":"#/components/schemas/address.Address"}]}}},"github_com_filecoin-project_go-state-types_builtin_v16_verifreg.AllocationId":{"type":"integer","enum":[0]},"address.Address":{"type":"object"},"mk20.PDPV1":{"type":"object","properties":{"add_root":{"description":"AddRoot indicated that this deal is meant to add root to a given ProofSet. ProofSetID must be defined.","type":"boolean"},"create_proof_set":{"description":"CreateProofSet indicated that this deal is meant to create a new ProofSet for the client by storage provider.","type":"boolean"},"delete_proof_set":{"description":"DeleteProofSet indicated that this deal is meant to delete an existing ProofSet created by SP for the client.\nProofSetID must be defined.","type":"boolean"},"delete_root":{"description":"DeleteRoot indicates whether the root of the data should be deleted. ProofSetID must be defined.","type":"boolean"},"extra_data":{"description":"ExtraData can be used to send additional information to service contract when Verifier action like AddRoot, DeleteRoot etc. are performed.","type":"array","items":{"type":"integer"}},"proof_set_id":{"description":"ProofSetID is PDP verified contract proofset ID. It must be defined for all deals except when CreateProofSet is true.","type":"integer"},"record_keeper":{"description":"RecordKeeper specifies the record keeper contract address for the new PDP proofset.","type":"string"},"root_ids":{"description":"RootIDs is a list of root ids in a proof set.","type":"array","items":{"type":"integer"}}}},"mk20.RetrievalV1":{"type":"object","properties":{"announce_payload":{"description":"AnnouncePayload indicates whether the payload should be announced to IPNI.","type":"boolean"},"announce_piece":{"description":"AnnouncePiece indicates whether the piece information should be announced to IPNI.","type":"boolean"},"indexing":{"description":"Indexing indicates if the deal is to be indexed in the provider's system to support CIDs based retrieval","type":"boolean"}}}}}} ``` ## The mk20.RetrievalV1 object ```json {"openapi":"3.1.1","info":{"title":"Curio Market 2.0 API","version":"0.0.1"},"components":{"schemas":{"mk20.RetrievalV1":{"type":"object","properties":{"announce_payload":{"description":"AnnouncePayload indicates whether the payload should be announced to IPNI.","type":"boolean"},"announce_piece":{"description":"AnnouncePiece indicates whether the piece information should be announced to IPNI.","type":"boolean"},"indexing":{"description":"Indexing indicates if the deal is to be indexed in the provider's system to support CIDs based retrieval","type":"boolean"}}}}}} ``` ## The mk20.StartUpload object ```json {"openapi":"3.1.1","info":{"title":"Curio Market 2.0 API","version":"0.0.1"},"components":{"schemas":{"mk20.StartUpload":{"type":"object","properties":{"chunk_size":{"description":"ChunkSize defines the size of each data chunk to be used during the upload process.","type":"integer"},"raw_size":{"description":"RawSize indicates the total size of the data to be uploaded in bytes.","type":"integer"}}}}}} ``` ## The mk20.SupportedContracts object ```json {"openapi":"3.1.1","info":{"title":"Curio Market 2.0 API","version":"0.0.1"},"components":{"schemas":{"mk20.SupportedContracts":{"type":"object","properties":{"contracts":{"description":"Contracts represents a list of supported contract addresses in string format.","type":"array","items":{"type":"string"}}}}}}} ``` ## The mk20.SupportedDataSources object ```json {"openapi":"3.1.1","info":{"title":"Curio Market 2.0 API","version":"0.0.1"},"components":{"schemas":{"mk20.SupportedDataSources":{"type":"object","properties":{"sources":{"description":"Contracts represents a list of supported contract addresses in string format.","type":"array","items":{"type":"string"}}}}}}} ``` ## The mk20.SupportedProducts object ```json {"openapi":"3.1.1","info":{"title":"Curio Market 2.0 API","version":"0.0.1"},"components":{"schemas":{"mk20.SupportedProducts":{"type":"object","properties":{"products":{"description":"Contracts represents a list of supported contract addresses in string format.","type":"array","items":{"type":"string"}}}}}}} ``` ## The mk20.UploadCode object ```json {"openapi":"3.1.1","info":{"title":"Curio Market 2.0 API","version":"0.0.1"},"components":{"schemas":{"mk20.UploadCode":{"type":"integer","enum":[200,400,404,409,500]}}}} ``` ## The mk20.UploadStartCode object ```json {"openapi":"3.1.1","info":{"title":"Curio Market 2.0 API","version":"0.0.1"},"components":{"schemas":{"mk20.UploadStartCode":{"type":"integer","enum":[200,400,404,409,500]}}}} ``` ## The mk20.UploadStatus object ```json {"openapi":"3.1.1","info":{"title":"Curio Market 2.0 API","version":"0.0.1"},"components":{"schemas":{"mk20.UploadStatus":{"type":"object","properties":{"missing":{"description":"Missing represents the number of chunks that are not yet uploaded.","type":"integer"},"missing_chunks":{"description":"MissingChunks is a slice containing the indices of missing chunks.","type":"array","items":{"type":"integer"}},"total_chunks":{"description":"TotalChunks represents the total number of chunks required for the upload.","type":"integer"},"uploaded":{"description":"Uploaded represents the number of chunks successfully uploaded.","type":"integer"},"uploaded_chunks":{"description":"UploadedChunks is a slice containing the indices of successfully uploaded chunks.","type":"array","items":{"type":"integer"}}}}}}} ``` ## The mk20.UploadStatusCode object ```json {"openapi":"3.1.1","info":{"title":"Curio Market 2.0 API","version":"0.0.1"},"components":{"schemas":{"mk20.UploadStatusCode":{"type":"integer","enum":[200,404,425,500]}}}} ``` # Wallet Exporter The **Wallet Exporter** is an optional telemetry component that periodically exposes comprehensive statistics about Curio wallets, miners and the messages they send. The metrics are exported via the built-in Prometheus endpoint and can be scraped by any Prometheus compatible monitoring stack. > ⚠️ IMPORTANT: **Enable the exporter on exactly *****one***** Curio node in the cluster.** Enabling it on multiple nodes would cause duplicated metric series which leads to incorrect dashboards and aggregated values. *** ## Why would I enable it? * Track wallet balances for every on-chain address that appears in the `wallet_names` table. Those are wallets with a name on the Wallets page. * Observe storage-provider balances and power in real-time. * Monitor message throughput, gas usage and fees broken down by sender, receiver and reason. * Build Grafana dashboards that correlate wallet activity with other Curio subsystems. ## How does it work? 1. Every 30 seconds (`WalletExporterInterval`) Curio executes one exporter cycle. 2. During a cycle Curio gathers the following information: * Wallet balances * Miner available balance and power (for the SP IDs configured on this node) * Newly sent messages that are waiting to land * Messages that have already landed (executed) and their execution result 3. For every data-point the exporter records an *OpenCensus measure* which is immediately made available as a Prometheus metric. ## Enabling the exporter Add the following to one layer which is active on *exactly one* node: ```toml [CurioSubsystems] EnableWalletExporter = true ``` Restart the node. The new metrics will appear at the standard metrics endpoint within \~30 seconds. ## Exported metrics | Metric name | Type | Description | | -------------------------------------- | --------- | -------------------------------------------------------------------------- | | `wallet_balance_nfil` | gauge | Wallet or miner balance in **NanoFIL** | | `wallet_power` | gauge | Storage-provider power in **bytes** (label `type` = `raw` or `qap`) | | `wallet_message_sent` | counter | Number of messages that have been **submitted** | | `wallet_message_landed` | counter | Number of messages that have **executed** on-chain | | `wallet_gas_units_requested` | counter | Gas units requested by submitted messages | | `wallet_gas_units_used` | counter | Gas units used by executed messages | | `wallet_sent_nfil` | counter | NanoFIL **value** transferred by executed messages | | `wallet_gas_paid_nfil` | counter | NanoFIL **gas fee** paid by executed messages | | `wallet_message_land_duration_seconds` | histogram | Distribution of time (in seconds) between message submission and execution | # cuzk Proving Daemon This page explains how to set up the cuzk persistent GPU SNARK proving daemon to accelerate proof computation in Curio. > **Experimental Feature**\ > This feature is currently experimental and under active development. Configuration, behavior, and interfaces may change without notice. *** ## What is cuzk? cuzk is a persistent GPU-resident SNARK proving daemon. It acts as a "proving server" that Curio delegates proof computations to over gRPC. The key difference from the default proving path (`ffiselect`) is that cuzk loads Groth16 SRS parameters **once at startup** and keeps them resident in CUDA-pinned host memory across all proofs. The default Curio code path spawns a fresh child process per proof, each of which loads the SRS from disk (30-90 seconds for 32 GiB PoRep), runs one proof, and exits. cuzk eliminates this repeated loading overhead entirely. ### Supported proof types | Proof type | Curio task | Description | | --------------- | ------------- | -------------------------------- | | PoRep C2 | `PoRep` | Seal commit phase 2 SNARK | | SnapDeals Prove | `UpdateProve` | CC sector update proof | | PSProve | `PSProve` | Snark Market proof share compute | ### How integration works When cuzk is enabled in Curio's configuration: 1. **Resource accounting bypassed**: `TypeDetails()` reports zero GPU and minimal RAM for proving tasks. Curio's harmony scheduler no longer gates these tasks on local GPU availability. 2. **Backpressure via polling**: `CanAccept()` queries the cuzk daemon's queue via `GetStatus` and rejects tasks when the queue is full (controlled by `MaxPending`). 3. **Vanilla proofs stay local**: The `Do()` method generates vanilla proofs locally (requires sector data on disk), sends them to cuzk for SNARK computation, then verifies the returned proof locally. When cuzk is **not** configured (default), all three tasks behave exactly as before. There is no behavioral change for existing deployments. *** ## Requirements * **NVIDIA GPU** with CUDA support (the cuzk daemon itself runs on the GPU machine) * **CUDA toolkit** (`nvcc` must be in PATH) * **Rust toolchain** (1.86.0 or later; managed automatically via `rust-toolchain.toml`) * **Filecoin proof parameters** downloaded (same parameters as standard Curio proving) * Sufficient system RAM for SRS residency (minimum 128 GiB, recommended 256+ GiB) *** ## Building From the Curio repository root: ```bash # Build both Curio and the cuzk daemon make curio cuzk ``` The `make cuzk` target: * Checks for `cargo` (Rust) and `nvcc` (CUDA) in PATH * Runs `cargo build --release` in `extern/cuzk/` * Copies the resulting binary to `./cuzk` To install: ```bash sudo make install # installs curio and sptool sudo make install-cuzk # installs cuzk to /usr/local/bin/cuzk ``` Note: `make cuzk` is intentionally **not** part of `make build` or `make buildall` since it requires CUDA and Rust, which are not available in all build environments (e.g., CI). *** ## Daemon Configuration The cuzk daemon reads its configuration from a TOML file (default: `/data/zk/cuzk.toml`). An example configuration is provided at `extern/cuzk/cuzk.example.toml`. ### Minimal configuration ```toml [daemon] listen = "0.0.0.0:9820" [srs] param_cache = "/var/tmp/filecoin-proof-parameters" preload = ["porep-32g"] [synthesis] partition_workers = 7 # Adjust based on RAM (see table below) ``` ### RAM-based tuning The primary tuning knob is `partition_workers`, which controls how many PoRep partitions are synthesized concurrently on the CPU. More workers keep the GPU fed but use more RAM. | System RAM | partition\_workers | gpu\_workers\_per\_device | Peak RSS | Throughput | | ---------- | ------------------ | ------------------------- | --------- | ------------- | | 128 GiB | 2 | 1 | \~110 GiB | \~152 s/proof | | 256 GiB | 7 | 1 | \~208 GiB | \~53 s/proof | | 384 GiB | 10 | 2 | \~271 GiB | \~43 s/proof | | 512+ GiB | 12 | 2 | \~400 GiB | \~38 s/proof | Memory formula: `Peak RSS = 69 + (partition_workers x 20) GiB` ### Running the daemon ```bash # With default config path cuzk # With custom config cuzk --config /path/to/cuzk.toml # Override listen address cuzk --listen unix:///run/curio/cuzk.sock # Override log level cuzk --log-level debug ``` For production, run cuzk as a systemd service: ```ini [Unit] Description=cuzk SNARK proving daemon After=network.target [Service] Type=simple ExecStart=/usr/local/bin/cuzk --config /data/zk/cuzk.toml Restart=on-failure RestartSec=10 LimitNOFILE=1048576 # Ensure CUDA libraries are available Environment=LD_LIBRARY_PATH=/usr/local/cuda/lib64 [Install] WantedBy=multi-user.target ``` *** ## Curio Configuration Add the following to your Curio configuration layer to connect to the cuzk daemon: ```toml [Cuzk] # gRPC endpoint of the cuzk daemon. # TCP: "127.0.0.1:9820" # Unix: "unix:///run/curio/cuzk.sock" Address = "127.0.0.1:9820" # Maximum total pending proofs in the cuzk queue before Curio stops # sending new tasks (backpressure). When the daemon's queue reaches # this level, CanAccept rejects new tasks until capacity frees up. MaxPending = 10 # Maximum time to wait for a proof result from the daemon. # If exceeded, the task is retried. ProveTimeout = "30m" ``` When `Address` is empty (the default), cuzk integration is disabled and all proving tasks use the standard local GPU path. ### Which Curio subsystems are affected The cuzk client is used by tasks on nodes that have these subsystems enabled: * `EnablePoRepProof = true` -- PoRep C2 proving * `EnableUpdateProve = true` -- SnapDeals update proving * `EnableProofShare = true` -- Snark Market proof computation These subsystems must still be enabled as usual. The `[Cuzk]` configuration only changes *how* the SNARK computation is performed (local GPU vs. remote daemon). *** ## Deployment Patterns ### Co-located (single machine) Run both Curio and cuzk on the same GPU machine. Use TCP localhost or a Unix socket: ``` Curio (Go) --gRPC--> cuzk (Rust) --CUDA--> GPU ``` ```toml # Curio config [Cuzk] Address = "127.0.0.1:9820" ``` This is the simplest deployment and avoids any network overhead. ### Dedicated prover (separate machines) Run Curio on CPU-only machines for sealing tasks (SDR, TreeD, etc.) and cuzk on a dedicated GPU machine. Curio connects over the network: ``` Curio (CPU node) --gRPC/TCP--> cuzk (GPU node) ``` ```toml # Curio config (on CPU node) [Cuzk] Address = "gpu-prover.internal:9820" ``` Note: vanilla proof data (up to \~200 MB for PoRep C2) is sent over gRPC, so ensure sufficient network bandwidth between the nodes. *** ## Monitoring The cuzk daemon exposes its status via the gRPC `GetStatus` RPC. Curio queries this automatically for backpressure. You can also query it manually: ```bash grpcurl -plaintext 127.0.0.1:9820 cuzk.v1.ProvingEngine/GetStatus ``` This returns the current queue state for each proof type (pending count, in-progress count). *** ## Troubleshooting ### cuzk build fails * Verify `nvcc` is in PATH: `nvcc --version` * Verify Rust toolchain: `rustup show` (should show 1.86.0 or later) * The Cargo workspace in `extern/cuzk/` depends on vendored forks of `bellperson`, `bellpepper-core`, and `supraseal-c2` under `extern/`. If you see missing crate errors, ensure `git submodule update --init --recursive` was run. ### Curio cannot connect to cuzk * Check that the daemon is running: `systemctl status cuzk` * Verify the address matches between Curio's `[Cuzk].Address` and the daemon's `[daemon].listen` * Check firewall rules if using TCP across machines * Look at Curio logs for `cuzk` entries: `journalctl -u curio | grep cuzk` ### Proofs are slow or timing out * Increase `ProveTimeout` in Curio's config if proofs legitimately take longer * Check daemon logs for queue depth. If proofs pile up, reduce `MaxPending` or add more GPU capacity * Tune `partition_workers` based on the RAM table above. Too many workers can cause memory pressure; too few starve the GPU ### Curio tasks rejected (backpressure) If Curio logs show "cuzk pipeline full, backpressuring", the daemon's queue is at capacity. Either: * Increase `MaxPending` (allows more queued proofs, uses more memory) * Add GPU capacity (second GPU, second daemon instance) * Reduce the rate of incoming sealing/snap work # 什么是Curio？| What is Curio? 什么是Curio，它与Lotus-Miner有何不同？ ## What is Curio? ## Curio是什么？ ### Overview ### 概述 Curio是Filecoin存储协议的新实现。它旨在简化存储提供商的设置和操作。 {% hint style="danger" %} 请注意，Curio集群不能在不同的网络之间共享。 示例：单个Curio集群不能同时托管来自主网和校准网的矿工ID。 {% endhint %} ### Key Features ### 主要特性 **High Availability** **高可用性** Curio设计用于高可用性。您可以运行多个Curio节点实例来处理类似类型的任务。分布式调度器和贪婪工作器设计将确保即使在大多数部分故障的情况下，任务也能按时完成。您可以安全地更新其中一台Curio机器，而不会中断其他机器的运行。 **Node Heartbeat** **节点心跳** 集群中的每个Curio节点必须每10分钟在HarmonyDB中发布一次心跳消息，更新其状态。如果错过心跳，该节点将被视为丢失，所有任务现在可以在剩余节点上调度。 **Task Retry** **任务重试** Curio中的每个任务都有一个限制，即在被声明为丢失之前应该尝试多少次。这确保了Curio不会无限期地重试坏任务。这可以防止计算时间和存储的浪费。 **Polling** **轮询** Curio通过轮询系统避免节点过载。节点检查它们可以处理的任务，优先考虑空闲节点以实现均衡的工作负载分配。 **Simple Configuration Management** **简单的配置管理** 配置以层的形式存储在数据库中。这些层可以堆叠在一起创建最终配置。用户可以重用这些层来控制多台机器的行为，而无需维护每个节点的配置。使用适当的标志启动二进制文件以连接YugabyteDB并指定使用哪些配置层以获得所需的行为。 **Running Curio with Multiple GPUs** **使用多个GPU运行Curio** Curio可以同时处理多个GPU，而无需运行多个Curio进程实例。因此，Curio可以作为单个systemd服务进行管理，而无需担心GPU分配问题。 ### Curio vs Lotus Miner ### Curio与Lotus Miner对比 | 特性 | Curio | Lotus-Miner | | ------- | --------------------- | --------------------- | | 调度 | 协作式（优先贪婪） | 单点故障 | | 高可用性 | 可用 | 单一控制进程 | | 冗余Post | 可用 | 不可用 | | 任务重试控制 | 任务重试有截止限制（每个任务） | 无限重试导致资源耗尽 | | 多个矿工ID | Curio集群可以支持多个矿工ID | 每个Lotus-Miner只有单个矿工ID | | 共享任务节点 | Curio节点可以处理多个矿工ID的任务 | 附加的工作器只处理单个矿工ID的任务 | | 分布式配置管理 | 配置存储在高可用的Yugabyte数据库中 | 所有配置都在单个文件中 | ### Future of Curio ### Curio的未来 Curio的长期愿景是最终取代当前的lotus-miner和lotus-worker进程。这是简化和精简存储提供商设置和操作的持续努力的一部分。 \\ # 设计 | Design 本页面详细概述了构成 Curio 的核心概念和组件，包括 HarmonyDB、HarmonyTask 等。 ## Design ## 设计 ### Curio Cluster ### Curio 集群 Curio 的核心内部组件包括 HarmonyDB、HarmonyTask、ChainScheduler 以及配置和当前存储定义的数据库抽象。
Curio Node

Curio nodes

Curio 集群是由多个连接到 YugabyteDB 集群和市场节点的 Curio 节点组成的集群。单个 Curio 集群可以根据需要为多个矿工 ID 提供服务，并在它们之间共享计算资源。
Curio cluster

Curio cluster

### HarmonyDB HarmonyDB 是一个简单的 SQL 数据库抽象层，由 HarmonyTask 和 Curio 堆栈的其他组件使用，用于存储和检索 YugabyteDB 中的信息。 #### Key Features: #### 关键特性： * **弹性:** 如果主连接失败，自动切换到备用数据库。 * **安全性:** 防止 SQL 注入漏洞。 * **便利性:** 提供常见 Go + SQL 操作的辅助函数。 * **监控:** 通过 Prometheus 统计和错误日志提供数据库行为的洞察。 #### Basic Database Details #### 基本数据库详情 * Postgres 数据库模式称为 “curio”，所有的 harmony 数据库表都在这个模式下。 * 表 `harmony_task` 存储待处理任务列表。 * 表 `harmony_task_history` 存储已完成的任务、超过限制的重试任务，并作为触发后续任务（可能在不同机器上）的输入。 * 表 `harmony_task_machines` 由 lib/harmony/resources 管理。此表引用注册的机器用于任务分配。注册不意味着义务，但有助于发现。 ### HarmonyTask HarmonyTask 是纯粹的（无任务逻辑）分布式任务管理器。 #### Design Overview #### 设计概述 * 任务为中心：HarmonyTask 专注于将任务管理为小型工作单元，减轻开发人员的调度和管理负担。 * 分布式：任务分布在各个机器上以实现高效执行。 * 贪婪工人：工人主动认领他们可以处理的任务。 * 轮询分配：在 Curio 节点认领任务后，HarmonyDB 尝试将剩余工作分配给其他机器。
Curio Tasks

Harmony tasks

#### Model #### 模型 * **被阻止的任务:** 任务可能因以下原因被阻止： * 运行节点上的‘子系统’配置被禁用 * 达到指定的最大任务限制 * 资源耗尽 * CanAccept() 函数（任务特定）拒绝任务 * **任务启动:** 任务可以通过以下方式启动： * 定期数据库读取（每 3 秒） * 当前进程添加到数据库 * **任务添加方法：** * 异步监听任务（例如，用于区块链） * 由任务完成触发的后续任务（封装流水线） * **防止重复任务：** * 避免重复任务的机制由任务定义决定，最有可能使用唯一键。 ### Distributed Scheduling ### 分布式调度 Curio 实现了一种通过 HarmonyDB 协调的分布式调度机制。Curio 节点根据它们可以处理的任务类型和资源来选择任务。节点在接受任务后不会贪婪，即使它们有足够的资源。其他节点轮流认领任务。每隔 3 秒，如果有可用资源，则会接受额外的任务。这确保了任务的更均匀调度。 ### Chain Scheduler ### 链调度器 `CurioChainSched` 或链调度器在应用或移除新的 TipSet 时触发一些回调函数。这相当于在每个 epoch 获取最重的 TipSet。这些回调函数依次为每种依赖链变化的类型添加新任务。这些任务类型包括 WindowPost、WinningPost 和 MessageWatcher。 ### Poller ### 轮询器 轮询器是一个简单的循环，根据预定义的时间间隔（100 毫秒）定期获取待处理任务，或直到上下文发起优雅退出。一旦从数据库中获取到待处理任务，它会尝试在 Curio 节点上调度所有任务。此尝试将导致以下结果之一： * 任务被接受 * 任务未被调度，因为机器繁忙 * 任务未被接受，因为节点的 CanAccept（由任务定义）选择不处理指定任务 如果任务在轮询周期内被接受，则下一个周期前的等待时间为 100 毫秒。但如果任务因任何原因未被调度，轮询器将在 3 秒后重试。 ### Task Decision Logic ### 任务决策逻辑 对于机器可以处理的每种任务类型，它首先检查机器是否有足够的能力执行所述任务。然后它查询数据库中没有 `owner_id` 且与任务类型同名的任务。如果存在这样的任务，它会尝试接受它们的工作。如果任何工作被接受，它返回 true，否则返回 false。接受每个任务的决策逻辑如下： 1. 检查是否有任何任务要做。如果没有，返回 true。 2. 检查是否达到此类型任务的最大数量。如果运行任务的数量达到或超过最大限制，记录一条消息并返回 false。 3. 检查机器是否有足够的资源来处理任务。这包括检查 CPU、RAM、GPU 容量和可用存储。如果机器没有足够的资源，记录一条消息并返回 false。 4. 通过调用 `CanAccept` 方法检查任务是否可以被接受。如果不能接受，记录一条消息并返回 false。 5. 如果任务需要存储空间，机器尝试声明它。如果声明失败，记录一条消息并释放已声明的存储空间，然后返回 false。 6. 如果任务来源是 `recover`，即机器在关机前正在执行此任务，则任务计数增加一个，并在单独的 goroutine 中开始处理任务。 7. 如果任务来源是 `poller`，即新的待处理任务，尝试为当前主机名声明任务。如果不成功，释放已声明的存储并尝试考虑下一个任务。 8. 如果成功，任务计数增加一个，并在单独的 goroutine 中开始处理任务。 9. 这个 goroutine 还会更新任务历史中的任务状态，并根据任务是否成功，删除任务或更新任务表中的任务。 10. 返回 true，表示工作已被接受并将被处理。 ### GPU Management in Curio ### Curio 中的 GPU 管理 #### **Historical Issues with Lotus-Miner Scheduler** #### **Lotus-Miner 调度器的历史问题** 历史上，当 lotus-worker 进程有多个 GPU 可用时，Lotus-Miner 调度器在高效利用 GPU 方面遇到了困难。这些问题主要源于底层的 proofs 库，该库处理所有与 GPU 相关的任务并管理 GPU 分配。这导致了以下问题： * 单个任务被分配到多个 GPU。 * 多个任务被分配到单个 GPU。 #### **Solution with Curio: The GPU Picker Library "ffiselect"** #### **Curio 的解决方案：GPU 选择库 "ffiselect"** 为了解决 Curio 中的这些问题，我们实现了一个名为 "ffiselect" 的 GPU 选择库。该库确保每个需要 GPU 的任务都单独分配一个。过程如下： 1. **任务分配**：每个需要 GPU 的任务都分配一个特定的 GPU。 2. **子进程创建**：为每个任务生成一个新的子进程，并为其分配专用 GPU。 3. **Proofs 库调用**：子进程使用单个 GPU 和特定任务调用 Proofs 库。

Curio FFISelect in action

这种方法确保了高效且无冲突的 GPU 使用，每个任务都由专用 GPU 处理，从而解决了 `lotus-miner` 调度器观察到的历史问题。 # 密封 | Sealing 本页解释了Curio中密封管道的功能 ## Sealing ## 密封 ### Sealing Pipeline ### 密封管道 Curio的密封过程由HarmonyTasks驱动。密封扇区涉及的每个阶段都被分为更小的独立任务。这些单独的任务然后由Curio集群中的不同机器接管。这确保了任务在整个系统中有效分配，资源得到高效利用。
Curio密封管道概览

Curio密封管道

### SealPoller ### 密封轮询器 SealPoller结构设计用于跟踪密封操作的进度。密封工作流中的每个可能状态都由一个pollTask结构表示。这个结构通过在harmony数据库的`sectors_sdr_pipeline`表的单独列中设置布尔标志和保存任务ID来跟踪密封操作可能处于的每个步骤。 ```go type pollTask struct { SpID int64 `db:"sp_id"` SectorNumber int64 `db:"sector_number"` TaskSDR *int64 `db:"task_id_sdr"` AfterSDR bool `db:"after_sdr"` TaskTreeD *int64 `db:"task_id_tree_d"` AfterTreeD bool `db:"after_tree_d"` TaskTreeC *int64 `db:"task_id_tree_c"` AfterTreeC bool `db:"after_tree_c"` TaskTreeR *int64 `db:"task_id_tree_r"` AfterTreeR bool `db:"after_tree_r"` TaskPrecommitMsg *int64 `db:"task_id_precommit_msg"` AfterPrecommitMsg bool `db:"after_precommit_msg"` AfterPrecommitMsgSuccess bool `db:"after_precommit_msg_success"` SeedEpoch *int64 `db:"seed_epoch"` TaskPoRep *int64 `db:"task_id_porep"` PoRepProof []byte `db:"porep_proof"` AfterPoRep bool `db:"after_porep"` TaskFinalize *int64 `db:"task_id_finalize"` AfterFinalize bool `db:"after_finalize"` TaskMoveStorage *int64 `db:"task_id_move_storage"` AfterMoveStorage bool `db:"after_move_storage"` TaskCommitMsg *int64 `db:"task_id_commit_msg"` AfterCommitMsg bool `db:"after_commit_msg"` AfterCommitMsgSuccess bool `db:"after_commit_msg_success"` Failed bool `db:"failed"` FailedReason string `db:"failed_reason"` } ``` SealPoller从数据库中检索所有`after_commit_msg_success`或`after_move_storage`不为真的`pollTasks`，并尝试在可能的情况下推进它们的状态。当一个`pollTask`的依赖项（由"after\_"字段指示）完成，且任务本身尚未排队（其任务ID为nil）或完成（其"After"字段为false）时，就会推进该任务。每个pollTask的推进都会触发一个数据库事务，尝试用从HarmonyDB接收到的新任务ID更新任务ID。该事务确保在读取状态和更新任务ID之间，任务没有被其他人排队。这个轮询过程按顺序进行，每个阶段都有不同的条件，确保在继续之前满足所有先前的条件。如果一个任务由于其先前的依赖项未完成而无法继续，轮询器将在下一轮回来。大多数情况下，轮询器操作期间发生的错误会被记录，不会导致轮询器停止。但如果在数据库事务期间发生严重问题，它将被回滚，并给出详细的错误消息。通过这种方式组织工作，SealPoller确保密封程序中的每个步骤按正确的顺序发生，并在可能的情况下取得进展。它允许在考虑其他正在进行的任务的约束条件下，尽可能高效地密封扇区。
密封任务执行

Curio harmony任务执行

### Piece Park ### 数据块停放 传统上，数据需要在可以密封存储之前可用。然而，这可能导致效率低下。Curio通过引入"数据块停放"来解决这个问题。Curio的密封管道不要求数据一开始就随时可用。这允许我们在数据下载之前就开始密封过程。在密封过程进行时，数据被"停放"在存储位置内名为"piece"的指定目录中。这避免了长时间保持市场连接开放。本质上，本地数据块停放区作为数据的临时保存区，简化了密封过程并优化了资源使用。 Curio使用两个任务： ParkPiece：这个任务处理数据的下载并将其放置在"piece"目录中。 DropPiece：一旦不再需要数据，这个任务负责清理停放的数据。 将来，这个本地存储还可以允许Curio在原始扇区在密封过程中丢失的情况下，在新的扇区中重新密封数据。 ### LMRPCProvider ### LM远程过程调用提供者 LMRPCProvider提供了一组方法来与扇区和数据块相关的各种数据进行交互。这些方法是市场实现（Boost）所需要的，用于跟踪交易的密封进度。 ActorAddress：这个方法返回与LMRPCProvider相关的actor地址。换句话说，它返回矿工的地址。 WorkerJobs：这个函数返回一个以UUID为索引的工作任务映射。 SectorsStatus：这个方法根据给定的扇区标识符sid返回扇区的状态。这个函数包括有关扇区的详细信息，如密封状态、交易ID、日志、质押和到期时间等。 SectorsList：这个函数提供当前存储的扇区编号列表。 SectorsSummary：这个函数给出扇区的摘要，按其状态分类。它返回一个映射，将每个扇区状态映射到其数量。 SectorsListInStates：这个方法返回处于给定状态集合中的扇区编号列表。 ComputeDataCid：这个函数用于计算数据的CID。 AuthNew：这个函数为给定的权限创建一个新的授权令牌（JWT）。 ### Piece Ingester ### 数据块接收器 数据块接收器为给定的矿工地址分配一个数据块到一个扇区。它检查数据块大小是否与扇区大小匹配，确定首选的密封证明类型，检索矿工ID，分配扇区编号，将数据块和扇区管道条目插入数据库，并返回分配的数据块的扇区和偏移量。 # 和谐任务 | Harmony Tasks 本指南解释了Curio中可用的不同HarmonyTasks ## Harmony Tasks ## 和谐任务 Curio使用HarmonyTask作为通用任务容器，可以由轮询器定期安排执行。为了执行封装和证明的不同方面，Curio实现了以下任务类型。 #### SDR #### SDR（空间数据复制） SDR任务是复制证明过程的第一阶段，在这里进行数据的编码和复制。SDR任务主要使用单个CPU核心，并大量利用SHA256指令集。因此，建议使用具有SHA256指令集的CPU。所有11层计算按顺序逐层进行。每层大小为32GiB。当SDR过程完成时，您将生成384GiB的数据（32GiB未封装扇区 + (11层 x 32GiB)）。SDR任务需要commD作为输入参数之一。commD计算需要所有将成为扇区一部分的数据片段的大小和CID。在管道的这个阶段不需要数据片段本身（数据）。 #### SDRTrees #### SDR树 SDRTrees任务可以进一步分为3个按顺序完成的部分。 **TreeD** **树D** 构建TreeD需要访问要封装到扇区中的数据。它使用数据构建一个Merkle树，并将其写入指定的输出路径，以"tree-d.dat"结尾。它还返回生成树的根CID。 **TreeRC** **树RC** 在TreeRC任务中，基于PreCommit 1中生成的11层计算列哈希，并构建Merkle树。这与Lotus-miner上的PreCommit 2相同。这些任务生成未封装CID和封装CID。未封装CID应与TreeD输出的根CID匹配。在此阶段，除了封装的32GiB扇区外，还存储了一个额外的64GiB文件（32GiB扇区）表示Merkle树。这使得一个扇区所需的总存储量达到约500 GiB。 #### PreCommitSubmit #### 预提交提交 通过`PreCommitSector`消息，存储提供者为给定扇区的封装数据（通常称为SealedCID或复制承诺（commR））提交存款。消息被包含在链上后，扇区被注册到存储提供者，并进入WaitSeed状态，这是网络的安全等待要求。这种消息类型也可以批量处理，在一条消息中包含多个PreCommitSector消息，以节省支付给网络的gas费用。这些批量消息称为`PreCommitSectorBatch`。PreCommitSubmit任务本身不发送消息，而是将其交给`SendMessage`任务的队列。 #### PoRep #### 复制证明 PoRep任务结合了Lotus-Miner封装管道的Commit1和Commit2部分。 在等待种子状态结束时获得的随机性用于Commit 1阶段，从PreCommit 2阶段生成的Merkle树中选择叶节点的随机子集。从它检查的叶节点子集中，它生成一个比完整Merkle树小得多的文件。该文件大小约为16MiB。 在Commit 2阶段，Commit 1的文件使用zk-SNARKs压缩成更小的证明。在Commit 2结束时生成的证明可以非常快速地验证其正确性，并且足够小，适合区块链。最终证明的大小约为2kib，并发布在区块链上。 #### Finalize #### 完成 Finalize任务执行以下操作： 1. 它将TreeD文件的输出截断到扇区大小，然后将其移动到扇区的未封装文件位置。用户应注意，在封装管道的这一点之前不会存在未封装的扇区副本。只有当交易的"KeepUnsealed"为真时才会创建未封装的副本。 2. 在此阶段清理扇区的缓存文件。 3. 删除已添加到扇区的数据片段的本地副本。 #### MoveStorage #### 移动存储 MoveStorage任务将数据从封装存储移动到永久存储。 #### CommitSubmit #### 提交提交 在CommitSubmit任务中，我们为扇区创建`ProveCommitSector`消息，并将其交给`SendMessage`任务的队列。通过`ProveCommitSector`消息，存储提供者为在`PreCommitSector`消息中提交的扇区提供复制证明（PoRep）。这个证明必须在网络的安全等待要求（WaitSeed）之后，且在扇区的PreCommit过期之前提交。这种消息类型也可以聚合，在一条消息中包含多个ProveCommitSector消息。这些聚合消息称为`ProveCommitAggregate`。 #### WindowPost #### 时空证明 WindowPost允许存储提供者可验证地证明他们已经将承诺给网络的数据存储在磁盘上，以创建一个可验证的、公开的记录，证明存储提供者持续承诺存储数据，或者让网络奖励存储提供者的贡献。在Curio中，整个WindowPost过程被分解为3个独立的任务。当TipSet变化时，这些任务由`CurioChainScheduler`触发。 **WdPost** **时空证明** WindowPost任务负责为当前截止时间内的单个分区生成证明。Curio并行运行多个这样的任务，以加快每个截止时间的计算时间。 **WdPostRecover** **时空证明恢复** WdPostRecover任务也是针对每个截止时间的每个分区执行的。我们检查所有先前故障的扇区，并确定哪些扇区现在已经恢复。它为当前截止时间内的每个分区创建恢复消息，并将这些消息提交到`SendMessage`任务的队列。 **WdPostSubmit** **时空证明提交** WdPostSubmit为当前截止时间内的每个分区创建WindowPost消息，并将这些消息提交到`SendMessage`任务的队列。 #### WinPost #### 赢得时空证明 赢得时空证明（WinningPoSt）是Filecoin网络奖励存储提供者对网络贡献的机制。作为这样做的要求，每个存储提供者都被要求为指定的扇区提交压缩的时空证明。每个成功创建区块的当选存储提供者都会获得FIL奖励，以及向其他Filecoin参与者收取费用以在区块中包含消息的机会。未能在必要的时间窗口内完成此操作的存储提供者将失去挖掘区块的机会。WinPost任务在每个纪元变化时触发，如果矿工地址赢得选举，则创建新区块并提交到链上。 #### SendMessage #### 发送消息 SendMessage任务实现了一个消息队列，任何其他任务都可以向其添加消息。这些消息然后由`SendMessage`处理并单独处理。它通过HarmonyDB协调抽象了高可用性的消息发送。它确保以事务方式分配Nonce，并且消息正确广播到网络。它确保消息按顺序发送，并且发送失败不会导致nonce间隙。 #### ParkPiece #### 停放数据片段 Curio在存储子系统中实现了一个新的文件位置，称为"piece"。这个目录用于在数据片段被封装时临时停放它们。`parked_pieces`还包含下载数据的URL和头信息。ParkPiece任务每15秒扫描一次HarmonyDB中的`parked_pieces`表。如果找到任何数据片段，则在存储的"piece"目录下创建相应的文件，并从URL下载数据到文件中。当外部市场节点调用`SectorAddPieceToAny`方法时，它会创建ParkPiece任务。 #### DropPiece #### 删除数据片段 DropPiece任务负责从`Piece Park`中移除数据片段，并确保清理与该数据片段相关的所有文件和引用。此任务由扇区封装管道的Finalize任务触发。 #### UpdateEncode #### 更新编码 SnapDeal封装任务是一种特殊类型的封装任务，允许存储提供者将已提交的封装扇区中放入交易数据。UpdateEncode任务将传入的未封装数据（交易数据）编码到现有的封装扇区中。一旦编码完成，生成并验证普通证明，以检查和确认数据已正确编码到封装扇区文件中。 #### UpdateProve #### 更新证明 在UpdateProve阶段，UpdateEncode任务的输出使用zk-SNARKs压缩成更小的证明。UpdateProve后生成的zk-SNARK可以验证新数据是否编码在新的封装扇区中，并且足够小，适合区块链。zk-SNARK的生成可以由CPU完成，也可以使用GPU加速。 #### Resource requirements for each Task type in Curio #### Curio中每种任务类型的资源需求 默认情况下，每种类型允许的任务数量在任何Curio节点上都没有限制。分布式调度器确保没有Curio节点过度承诺资源。 | 任务名称 | 任务描述 | CPU | RAM(GiB) | GPU | 重试次数 | | --------------- | -------- | --- | -------- | --- | ---- | | SDR | 单数据复制 | 4 | 54 | 0 | 2 | | SDRTrees | 单数据复制树生成 | 1 | 8 | 1 | 3 | | PreCommitSubmit | 预提交提交 | 0 | 1 | 0 | 16 | | PoRep | 复制证明 | 1 | 50 | 1 | 5 | | Finalize | 完成封装 | 1 | 0.1 | 0 | 10 | | MoveStorage | 移动存储 | 1 | 0.128 | 0 | 10 | | CommitSubmit | 提交提交 | 0 | 0.001 | 0 | 16 | | WdPostSubmit | 窗口时空证明提交 | 0 | 0.010 | 0 | 10 | | WdPostRecover | 窗口时空证明恢复 | 1 | 0.128 | 0 | 10 | | WdPost | 窗口时空证明 | 1 | 待定 | 待定 | 3 | | WinPost | 赢得时空证明 | 1 | 待定 | 待定 | 3 | | SendMessage | 发送消息 | 0 | 0.001 | 0 | 1000 | | UpdateEncode | 更新编码 | 1 | 1 | 1 | 3 | | UpdateProve | 更新证明 | 1 | 50 | 1 | 3 | # 入门 | Getting Started 這是一個幫助新用戶熟悉 Curio 的逐步指南 ## Getting Started ## 入門指南 ### Curio Database and Distributed Architecture ### Curio 數據庫和分佈式架構 #### Familiarizing Yourself with Curio #### 熟悉 Curio 在深入設置和配置 Curio 之前，我們強烈建議您先熟悉 [Curio 的設計和基本原則](/zh/design)。這些基礎知識將極大地幫助您進行有效的管理和故障排除。 #### **HarmonyDB with YugabyteDB** #### **使用 YugabyteDB 的 HarmonyDB** Curio 利用 YugabyteDB 創建了一個稱為 HarmonyDB 的抽象層。這個 HarmonyDB 主要有兩個用途： 1. **元數據存儲**：它存儲所有與 Curio 相關的元數據。 2. **共識層**：它為 Curio 集群的分佈式架構建立了一個共識層。 {% hint style="danger" %} 我們建議使用至少 3 個節點的 YugabyteDB 集群以實現高可用性和可擴展性。數據庫的丟失將導致 Curio 無法運行。YugabyteDB 也應該定期備份。 {% endhint %} #### Key Features of HarmonyDB #### HarmonyDB 的主要特點 * **高可用性**：確保即使在節點故障的情況下，元數據和共識信息也始終可用。 * **可擴展性**：能夠處理不斷增加的數據量，並隨著 Curio 集群的增長而擴展。 * **一致性**：在 Curio 集群的分佈式節點之間保持數據一致性。 #### Benefits of Using YugabyteDB for HarmonyDB #### 使用 YugabyteDB 作為 HarmonyDB 的好處 * **分佈式 SQL**：結合了 SQL 的優點和分佈式數據庫的彈性和可擴展性。 * **容錯能力**：提供強大的容錯能力，確保 Curio 集群的可靠性。 * **多區域部署**：支持跨多個區域部署，以提高性能和冗餘。 ### Chain Node ### 鏈節點 Curio 需要訪問至少一個 Filecoin 鏈節點，如 [Lotus](https://lotus.filecoin.io/lotus/get-started/what-is-lotus/) 或 [Forest](https://docs.forest.chainsafe.io/)（正在進行整合）。Curio 使用這個鏈節點來獲取當前的鏈狀態並向鏈發送消息。Curio 支持使用多個鏈節點。 ### Network ### 網絡 每個 Curio 節點必須開放以下端口以進行 API 和 GUI 訪問 | 端口 | 詳情 | | ----- | -------------------------------- | | 12300 | 默認 API 端口 | | 4701 | 默認 GUI 端口。並非所有 Curio 節點都需要啟用 GUI | | 32100 | 市場端口。此端口由用戶在配置中啟用 Boost 訪問時確定。 | ### Boost Compatibility ### Boost 兼容性 Boost 與 Curio 完全兼容，可以用於進行交易和檢索數據，就像 `lotus-miner` 一樣。版本兼容性指南可以在 [Boost 文檔](https://boost.filecoin.io/getting-started#boost-and-lotus-compatibility-matrix) 中找到。 ### Installing Curio and creating a Curio cluster ### 安裝 Curio 並創建 Curio 集群 了解了 Curio 的內部機制後，您現在可以繼續 [安裝 Curio 二進制文件](/zh/installation)。我們建議使用 [Debian 包](/zh/installation#debian-package-installation) 進行安裝，因為它們可以方便地進行安裝、升級和進程管理。安裝完第一個 Curio 二進制文件後，您可以繼續 [設置 Curio](/zh/setup)，無論您是 [從 lotus-miner 遷移](/zh/setup#migrating-from-lotus-miner-to-curio) 還是 [初始化新的 minerID](/zh/setup#initiating-a-new-curio-cluster)。 ### Best Practices ### 最佳實踐 我們已經編制了 [一份最佳實踐列表](/zh/best-practices) 用於部署和維護 Curio 集群。我們鼓勵所有用戶遵循這些建議，以避免潛在的問題。 新用戶還應該熟悉 [Curio 附帶的兩個二進制文件](/zh/curio-cli) 和 [GUI 頁面](/zh/curio-gui)。 # 版本 | Versions ## Versions ## 版本 这是最新免费 Curio 版本的兼容性矩阵。 | Curio 版本 | Lotus 版本 | 网络 | Boost | Yugabyte | Forest | | ----------- | -------- | -- | ---------- | ----------- | --------- | | 1.22.1 / 自动 | v1.27.X | 主网 | v2.3.0-rc2 | 2.20.X / 自动 | 0.19 / 自动 | | 1.23.0 | >v1.28.1 | 主网 | v2.3.0 | 2.20.X / 自动 | 0.19 / 自动 | "X"表示无特定偏好。 配置和所需机器数量：A: Lotus、Curio（多个）、YugabyteDB（1或3个）、（可选Boost）B: Forest、Curio（多个）、Yugabyte（1或3个） ### Automatic Updates ### 自动更新 * Docker有Watchtower，可为YugabyteDB和Forest提供自动更新功能。 * Curio可以通过Ubuntu上的Debian更新过程在主网上自动更新。 * 目前，只有Lotus和Boost缺乏自动更新功能，必须手动构建和部署。 * Curio的DEB包包括curio-cuda（用于Nvidia）和curio-opencl（用于其他如ATI）。 * 这些可以在Curio集群中混合使用，因为它们只与机器上的硬件有关。 ### Notes ### 注意事项 * Forest（0.19+和Docker Watchtower）是Lotus客户端的轻量级替代品。它满足Curio的需求，但Boost兼容性仍在开发中。 ### Building for CalibrationNet ### 为校准网构建 * 参与校准网需要 * 使用仓库根目录的 `go.mod` 中指定的 Go 版本 * 可用的Curio分支名称格式为release/vVERSION，如：release/v1.23.4 * 校准网可能比主网领先一个网络版本。 * DEB包仅用于主网发布，将提前提供，以确保主网升级不会造成中断。 # 安装 | Installation 本指南将展示如何构建、安装和更新Curio二进制文件 ## Installation ## 安装 ### Debian package installation ### Debian包安装 Curio软件包可直接在Ubuntu / Debian系统上安装。 {% hint style="danger" %} 目前Debian包只适用于主网。对于其他网络如校准网络或开发网，必须从源代码构建二进制文件。 {% endhint %} ```` // Start of Selection 1. 安装先决条件 ```bash sudo apt install mesa-opencl-icd ocl-icd-opencl-dev gcc git jq pkg-config curl clang build-essential hwloc libhwloc-dev libarchive-dev libgmp-dev libconfig++-dev protobuf-compiler wget -y && sudo apt upgrade -y ```` 2. 启用Curio软件包仓库 ```bash sudo wget -O /usr/share/keyrings/curiostorage-archive-keyring.gpg https://filecoin-project.github.io/apt/KEY.gpg echo "deb [signed-by=/usr/share/keyrings/curiostorage-archive-keyring.gpg] https://filecoin-project.github.io/apt stable main" | sudo tee /etc/apt/sources.list.d/curiostorage.list sudo apt update ``` 3. 根据您的GPU安装Curio二进制文件。 对于NVIDIA GPU： sudo apt install curio-cuda 对于OpenCL GPU： ```bash sudo apt install curio-opencl ``` ### Linux Build from source ### Linux从源代码构建 您可以按照以下步骤从源代码构建Curio可执行文件。 #### Software dependencies #### 软件依赖 要安装和运行Curio，您需要安装以下软件。 **System-specific** **系统特定** 构建Curio需要一些系统依赖，通常由您的发行版提供。 {% hint style="warning" %} **注意（Linux 上默认会构建批量封装）：** Curio 的 Linux 构建流程现在会在常规的 `make deps/build` 中默认构建 `extern/supraseal`。因此在 Linux 上从源代码构建 Curio 需要批量封装的额外依赖，包括： * CUDA Toolkit **13.x 或更高版本**（需要 `nvcc`，即使你不打算在运行时使用批量封装） * GCC **13** 工具链（`gcc-13` / `g++-13`） * Python venv 工具（`python3-venv`）以及常见构建工具（`autoconf`、`automake`、`libtool`、`nasm`、`xxd` 等） 要检查机器是否支持 SnapDeals 的 **快速 TreeR** 路径，可运行： ```bash curio test supra system-info ``` {% endhint %} Arch: ```bash sudo pacman -Syu opencl-icd-loader gcc git bzr jq pkg-config opencl-headers hwloc libarchive nasm xxd python python-pip python-virtualenv aria2 time protobuf # 批量封装构建依赖（需要 nvcc） sudo pacman -Syu cuda # GCC 13 可能需要通过发行版/AUR 安装（取决于当前 supraseal 版本） ``` Ubuntu/Debian: ```bash sudo apt install -y \ mesa-opencl-icd ocl-icd-opencl-dev \ gcc-13 g++-13 \ gcc git jq pkg-config curl clang build-essential hwloc libhwloc-dev libarchive-dev wget \ python3 python3-dev python3-pip python3-venv \ autoconf automake libtool \ xxd nasm \ libssl-dev uuid-dev libfuse3-dev \ libnuma-dev libaio-dev libkeyutils-dev libncurses-dev \ libgmp-dev libconfig++-dev \ protobuf-compiler \ aria2 time \ && sudo apt upgrade -y # CUDA Toolkit（批量封装构建依赖；需要 nvcc） ``` Fedora: ```bash sudo dnf -y install gcc make git bzr jq pkgconfig mesa-libOpenCL mesa-libOpenCL-devel opencl-headers ocl-icd ocl-icd-devel clang llvm wget hwloc hwloc-devel libarchive-devel protobuf-compiler aria2 time ``` OpenSUSE: ```bash sudo zypper in gcc git jq make libOpenCL1 opencl-headers ocl-icd-devel clang llvm hwloc libarchive-devel protobuf-devel aria2 time && sudo ln -s /usr/lib64/libOpenCL.so.1 /usr/lib64/libOpenCL.so ``` Amazon Linux 2: ```bash sudo yum install -y https://dl.fedoraproject.org/pub/epel/epel-release-latest-7.noarch.rpm; sudo yum install -y git gcc bzr jq pkgconfig clang llvm mesa-libGL-devel opencl-headers ocl-icd ocl-icd-devel hwloc-devel libarchive-devel protobuf-compiler aria2 time ``` #### Rustup Curio需要[rustup](https://rustup.rs/)。最简单的安装方法是： ```bash curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh ``` #### Go 构建 Curio 需要安装 Go（最低版本以仓库根目录的 `go.mod` 为准）。 示例（当前仓库最小版本为 **1.26.2**）： ```bash wget -c https://go.dev/dl/go1.26.2.linux-amd64.tar.gz -O - | sudo tar -xz -C /usr/local ``` {% hint style="info" %} 您需要将`/usr/local/go/bin`添加到您的路径中。对于大多数Linux发行版，您可以运行类似以下的命令： echo 'export PATH=$PATH:/usr/local/go/bin' >> \~/.bashrc && source \~/.bashrc 如果遇到困难，请参阅[官方Golang安装说明](https://golang.org/doc/install)。 {% endhint %} #### System Configuration #### 系统配置 在继续安装之前，您应该增加UDP缓冲区。您可以通过运行以下命令来实现： ```bash sudo sysctl -w net.core.rmem_max=2097152 sudo sysctl -w net.core.rmem_default=2097152 ``` #### Build and install Curio #### 构建和安装Curio 一旦所有依赖项都安装完毕，您就可以构建和安装Curio了。 1. 克隆仓库： ```bash git clone https://github.com/filecoin-project/curio.git cd curio/ ``` 2. 切换到最新的稳定版本分支： ```bash git checkout ``` 3. 根据您的CPU型号，您需要导出额外的环境变量： 1. 如果您有**AMD Zen或Intel Ice Lake CPU（或更新版本）**，通过添加这两个环境变量来启用SHA扩展的使用： ```bash export RUSTFLAGS="-C target-cpu=native -g" export FFI_BUILD_FROM_SOURCE=1 ``` 有关此过程的更多详细信息，请参阅[Native Filecoin FFI部分](https://lotus.filecoin.io/storage-providers/curio/install/#native-filecoin-ffi)。 2. 一些不支持ADX指令的较旧Intel和AMD处理器可能会出现非法指令错误。要解决这个问题，请添加`CGO_CFLAGS`环境变量： export CGO\_CFLAGS\_ALLOW="-D\_\_BLST\_PORTABLE\_\_" export CGO\_CFLAGS="-D\_\_BLST\_PORTABLE\_\_" 3. 默认情况下，proofs库中使用"multicore-sdr"选项。除非明确禁用，否则FFI也会使用此功能。要禁用使用"multicore-sdr"依赖项构建，请将`FFI_USE_MULTICORE_SDR`设置为`0`： export FFI\_USE\_MULTICORE\_SDR=0 4. 构建和安装Curio： Curio被编译为在单个网络上运行。 选择您要加入的网络，然后运行相应的命令来构建Curio节点： ## 对于主网： make clean build ## 对于校准测试网： make clean calibnet 安装Curio： ```bash sudo make install ``` 这将把`curio`放在`/usr/local/bin`中。`curio`默认将使用`$HOME/.curio`文件夹。 运行`curio --version` curio version 1.27.0-dev+mainnet+git.78d9d9baa ## 或 curio version 1.27.0-dev+calibnet+git.78d9d9baa 5. 现在您应该已经安装了Curio。您现在可以[完成Curio节点的设置](https://lotus.filecoin.io/storage-providers/curio/setup/)。 #### Native Filecoin FFI #### 原生Filecoin FFI 一些较新的CPU架构，如AMD的Zen和Intel的Ice Lake，支持SHA扩展。启用这些扩展可以显著加速您的Curio节点。要充分利用处理器的功能，请确保在**从源代码构建之前**设置以下变量： ```bash export RUSTFLAGS="-C target-cpu=native -g" export FFI_BUILD_FROM_SOURCE=1 ``` 这种构建方法不会产生可移植的二进制文件。确保您在构建它的同一台计算机上运行二进制文件。 ### MacOS Build from source ### MacOS从源代码构建 您可以按照以下步骤从源代码构建Curio可执行文件。 #### Software dependencies #### 软件依赖 要从源代码构建Curio，您必须安装XCode和Homebrew。 **XCode Command Line Tools** **XCode命令行工具** 在构建Curio二进制文件之前，需要安装X-Code CLI工具。 通过CLI检查是否已安装XCode命令行工具，运行： ```bash xcode-select -p ``` 这应该输出类似以下内容： /Library/Developer/CommandLineTools 如果此命令返回一个路径，那么您已经安装了Xcode！您可以[继续使用Homebrew安装依赖项](https://lotus.filecoin.io/storage-providers/curio/install/#homebrew)。如果上述命令没有返回路径，请安装Xcode： ```bash xcode-select --install ``` 接下来是使用Homebrew安装Curio的依赖项。 #### **Homebrew** #### **Homebrew** 我们建议macOS用户使用[Homebrew](https://brew.sh/)安装每个必要的软件包。 使用命令`brew install`安装以下软件包： ```bash brew install go bzr jq pkg-config hwloc coreutils ``` 接下来是克隆Lotus仓库并构建可执行文件。 #### **Rust** Rustup是系统编程语言Rust的安装程序。运行安装程序并按照屏幕提示操作。除非您熟悉自定义，否则应选择默认安装选项： ```bash curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh ``` #### Build and install Curio #### 构建和安装Curio 安装说明因Mac中的CPU类型而异： * [基于ARM的CPU（M1、M2、M3）](#arm-based-cpus) * [Intel CPU](#intel-cpus) **基于ARM的CPU** 1. 克隆仓库： ```bash git clone https://github.com/filecoin-project/curio.git cd curio/ ``` 2. 切换到最新的稳定版本分支： ```bash git checkout ``` 3. 创建必要的环境变量以允许Curio在ARM架构上运行： ```bash export LIBRARY_PATH=/opt/homebrew/lib export FFI_BUILD_FROM_SOURCE=1 export PATH="$(brew --prefix coreutils)/libexec/gnubin:/usr/local/bin:$PATH" ``` 4. 构建`curio`二进制文件： ```bash make clean curio ``` 5. 运行最后的`make`命令将此`curio`可执行文件移动到`/usr/local/bin`。这允许您从任何目录运行`curio`。 ```bash sudo make install ``` 6. 运行`curio --version` curio version 1.27.0-dev+mainnet+git.78d9d9baa ## 或 curio version 1.27.0-dev+calibnet+git.78d9d9baa 7. 现在您应该已经安装了Curio。您现在可以[设置新的Curio集群或从Lotus-Miner迁移](https://lotus.filecoin.io/storage-providers/curio/setup/)。 **Intel CPU** ❗这些说明适用于在Intel Mac上安装Curio。如果您有基于ARM的CPU，请使用[基于ARM的CPU说明 ↑](https://lotus.filecoin.io/storage-providers/curio/install/#arm-based-cpus) 1. 克隆仓库： git clone cd curio/ 2. 切换到最新的稳定版本分支： ```bash git checkout ``` 3. 构建和安装Curio： ```bash make clean curio sudo make install ``` 4. 运行`curio --version` curio version 1.27.0-dev+mainnet+git.78d9d9baa ## 或 curio version 1.27.0-dev+calibnet+git.78d9d9baa 现在您可以[完成Curio节点的设置](https://lotus.filecoin.io/storage-providers/curio/setup/)。 # 设置 | Setup 本指南将向您展示如何设置新的Curio集群或从Lotus-Miner迁移到Curio ## Setup ## 设置 ### Setup YugabyteDB ### 设置YugabyteDB {% hint style="warning" %} 如果您已经为Boost设置了YugabyteDB，那么您可以为Curio重用相同的YugabyteDB实例。您必须确保YugabyteDB是多节点集群以实现高可用性。您可以直接跳到[从Lotus-Miner迁移到Curio](#migrating-from-lotus-miner-to-curio)或[初始化新的Curio矿工](#initiating-a-new-curio-cluster)。 {% endhint %} 在本指南中，我们将设置一个单节点YugaByteDB。但是，您必须在集群中设置多个YugaByteDB实例以实现高可用性。 在安装和设置YugabyteDB之前，请确保您具备以下条件： {% hint style="danger" %} **不要使用ZFS作为YugabyteDB的后备驱动器，因为目前无法使用高级文件系统命令。** {% endhint %} 1. 以下操作系统之一： * CentOS 7或更高版本 * Ubuntu 16.04或更高版本 对于其他操作系统，Docker或Kubernetes。请查看[YugabyteDB文档](https://docs.yugabyte.com/preview/quick-start/)。 2. **Python 3。** 要检查版本，请执行以下命令： ```bash python --version ``` Python 3.7.3 如果遇到`Command 'python' not found`错误，您可能没有未版本化的系统范围python命令。 * 从Ubuntu 20.04开始，python不再可用。要解决此问题，请运行`sudo apt install python-is-python3`。 * 对于CentOS 8，通过运行`sudo alternatives --set python /usr/bin/python3`将`python3`设置为python的替代方案。 安装这些依赖项后，我们可以运行安装脚本： ## YugabyteDB（单节点开发/测试示例） ## 注意：生产/HA 部署请参考 YugabyteDB 官方部署文档。 wget tar xvfz yugabyte-2.25.1.0-b381-linux-x86\_64.tar.gz && cd yugabyte-2.25.1.0/ ./bin/post\_install.sh ./bin/yugabyted start --advertise\_address 127.0.0.1 --master\_flags rpc\_bind\_addresses=127.0.0.1 --tserver\_flags rpc\_bind\_addresses=127.0.0.1 +----------------------------------------------------------------------------------------------------------+ | yugabyted | +----------------------------------------------------------------------------------------------------------+ | Status : | | Replication Factor : None | | YugabyteDB UI : | | JDBC : jdbc:postgresql://127.0.0.1:5433/yugabyte?user=yugabyte\&password=yugabyte | | YSQL : bin/ysqlsh -U yugabyte -d yugabyte | | YCQL : bin/ycqlsh -u cassandra | | Data Dir : /root/var/data | | Log Dir : /root/var/logs | | Universe UUID : 411422ee-4c17-4f33-996e-ced847d10f5c | +----------------------------------------------------------------------------------------------------------+ 您可以根据自己的配置和需求调整`--advertise_address`、`--rpc_bind_addresses`和`--tserver_flags`。 ### Migrating from Lotus-miner to Curio ### 从Lotus-miner迁移到Curio Curio为用户提供了快速上手的工具。请在您的`lotus-miner`节点上运行以下命令，并按照屏幕上的说明操作。它支持英语（en）、中文（zh）和韩语（ko）。 curio guided-setup 迁移完成后，您可以关闭所有工作节点和矿工进程。您可以启动`curio`进程，使用正确的[配置层](/zh/configuration#configuration-layers)来替换它们。 #### Testing the setup #### 测试设置 您可以通过运行WindowPoSt测试计算来确认`curio`进程能够调度和计算WindowPoSt： curio test window-post task 从输出中，我们可以确认WindowPoSt被插入到数据库中，并被运行\_wdpost\_配置层的Curio进程拾取。 测试成功后，请继续[curio服务配置](/zh/curio-service)。 ### Initiating a new Curio cluster ### 初始化新的Curio集群 要创建新的Curio集群，需要一个[Lotus守护节点](https://bafybeib7hujkpoqohpby6dqabdea2t6ehcysics3ejoh4jrgtuke4rmolu.on.fleek.co/lotus/install/prerequisites/)。 {% hint style="warning" %} Lotus守护节点必须与正在设置的Curio属于同一网络。 例如：`calibration`网络守护节点不能与`mainnet` Curio集群一起使用。 {% endhint %} #### Wallet setup #### 钱包设置 在Filecoin网络上初始化新的矿工ID需要所有者地址、工作者地址和发送者地址。这些地址可以相同或不同，取决于用户的选择。用户必须在运行Curio命令之前在Lotus节点上创建这些钱包。 ```bash lotus wallet new bls lotus wallet new bls ``` 创建新钱包后，我们必须向它们发送一些资金。 ```bash lotus send 5 lotus send 5 ``` #### Creating new miner ID #### 创建新的矿工ID Curio为用户提供了快速上手的工具。请在新的Curio节点上运行以下命令，选择"创建新矿工"选项，并按照屏幕上的说明操作。它支持英语（en）、中文（zh）和韩语（ko）。 1. 启动引导设置。 ```bash curio guided-setup ``` 2. 选择"创建新矿工"选项。 默认使用英语。如果您需要其他语言支持，请联系Curio团队。 使用箭头键导航：↓ ↑ → ← ? 我想要： 从现有的Lotus-Miner迁移 ▸ 创建新矿工 3. 输入您的YugabyteDB详细信息。 此过程部分是幂等的。一旦创建了新的矿工参与者并且后续步骤失败，用户需要运行'curio config new-cluster < miner ID >'来完成配置。 使用箭头键导航：↓ ↑ → ← ? 输入连接到您的Yugabyte数据库安装的信息（ ▸ 主机：127.0.0.1 端口：5433 用户名：yugabyte 密码：yugabyte 数据库：yugabyte 继续连接并更新架构。 ✔ 步骤完成：预初始化步骤完成 4. 输入用于"创建矿工"消息的钱包详细信息。 初始化新的矿工参与者。 使用箭头键导航：↓ ↑ → ← ? 输入创建新矿工的信息： ▸ 所有者地址：<空> <------ 在此处输入钱包1 工作者地址：<空> <------ 在此处输入钱包2 发送者地址：<空> <------ 在此处输入钱包1 扇区大小：0 <--------------- 扇区大小（32 G/GiB/GB） 置信纪元：0 继续验证地址并创建新的矿工参与者。 初始化新的矿工参与者。 ✔ 所有者地址：<空> 输入所有者地址：t3weiymrx3iyivzeuub5l232gb62ocu7zbjtztudiipm6wkkmsehdydrdddm6cdrflxir26cmrz4xui6t5gruq ✔ 工作者地址：<空> 输入工作者地址：t3xhmgfxurecrusgubzdgme4t2ecxbiyny5uanfzvcrrihzhia654f6gp2ynugpiyp5xe7ibg6fqly76kowfva ✔ 发送者地址：<空> 输入发送者地址：t3weiymrx3iyivzeuub5l232gb62ocu7zbjtztudiipm6wkkmsehdydrdddm6cdrflxir26cmrz4xui6t5gruq ✔ 扇区大小：0 输入扇区大小：8 MiB ✔ 置信纪元：0 置信纪元：0 推送CreateMiner消息：bafy2bzacebu3mhaj6chnz5frjo2sbxduebnh4e7e37fwm3jd7xhvhla7t6ylu 等待确认 5. 等待新的矿工参与者创建完成。 新矿工的地址是：t01004 (t2cmgqvicpcil5zlp6bqsffmjjfz7ix66k4zaojay) ✔ 步骤完成：矿工t01004创建成功 ✔ 步骤完成：配置'base'已更新以包含此矿工的地址 6. 我们请求您与我们分享有关您的矿工的基本数据，以帮助我们改进Curio。 Curio团队希望改进您使用的软件。告诉团队您正在使用`curio`。 使用箭头键导航：↓ ↑ → ← ? 选择您想与Curio团队分享的内容： ▸ 个人数据：矿工ID、Curio版本、链（主网或校准网）。已签名。 聚合匿名：版本、链和矿工算力（分桶）。 提示：我是在某个链上运行Curio的人。 什么都不分享。 7. 完成初始化。 ✔ 步骤完成：新矿工初始化完成。 尝试使用curio run --layers=gui运行Web界面，以获得进一步的引导改进。 8. 如果在步骤3中输入了非默认值，请在运行Curio命令之前导出相关详细信息。 | 环境变量 | 用途 | | ------------------- | ----------------- | | CURIO\_DB\_HOST | YugabyteDB SQL IP | | CURIO\_DB\_NAME | YugabyteDB名称 | | CURIO\_DB\_USER | 连接的DB用户 | | CURIO\_DB\_PASSWORD | 用户密码 | | CURIO\_DB\_PORT | YugabyteDB的SQL端口 | | CURIO\_REPO\_PATH | Curio的默认仓库路径 | 9. 首先尝试仅使用`GUI`运行Curio。 ```bash curio run --layers gui ``` 10. 如果`curio`进程成功启动，请继续使用GUI并验证您可以访问所有页面。验证完成后，请继续[curio服务配置](/zh/curio-service)。 # Curio服务 | Curio Service 本页面解释了如何为 Curio 设置 systemd 服务 Curio 服务 Curio 可以同时处理多个 GPU，而无需运行多个 Curio 进程实例。因此，Curio 可以作为单个 systemd 服务进行管理，而无需担心 GPU 分配问题。 Curio 可以同时处理多个 GPU，而无需运行多个 Curio 进程实例。因此，Curio 可以作为单个 systemd 服务进行管理，而无需担心 GPU 分配问题。 ## Systemd Service Configuration ## Systemd 服务配置 Curio 的服务文件包含在 Debian 包中，名为 `curio.service`。如果您是从源代码构建 Curio 的，可以按照下面的描述手动创建服务文件。 ### **Service File for Curio** ### **Curio 的服务文件** 要手动创建 `curio.service` 文件，请使用以下内容： ```yaml [Unit] Description=Curio After=network.target [Service] ExecStart=/usr/local/bin/curio run Environment=GOLOG_FILE="/var/log/curio/curio.log" Environment=GOLOG_LOG_FMT="json" LimitNOFILE=1000000 Restart=always RestartSec=10 EnvironmentFile=/etc/curio.env [Install] WantedBy=multi-user.target ``` ### Environment Variables Configuration ### 环境变量配置 服务文件需要存在一个 `/etc/curio.env` 文件。该文件包含连接数据库所需的所有环境变量。`env` 文件应在 Debian 包安装期间自动创建。如果您运行的是从源代码构建的 Curio，可以使用以下内容手动创建 `env` 文件： #### /etc/curio.env 文件 ```bash CURIO_LAYERS=gui,post CURIO_ALL_REMAINING_FIELDS_ARE_OPTIONAL=true CURIO_DB_HOST=yugabyte1,yugabyte2,yugabyte3 CURIO_DB_USER=yugabyte CURIO_DB_PASSWORD=yugabyte CURIO_DB_PORT=5433 CURIO_DB_NAME=yugabyte CURIO_REPO_PATH=~/.curio CURIO_NODE_NAME=ChangeMe FIL_PROOFS_USE_MULTICORE_SDR=1 ``` 确保所有变量根据您的环境正确设置。 ## Starting the Curio Service ## 启动 Curio 服务 一旦所有变量都正确更新，创建日志目录： ```bash mkdir -p /var/log/curio ``` 现在，您可以使用以下命令启动 systemd 服务： ```bash sudo systemctl start curio.service ``` 通过监控 `systemctl status curio.service` 验证进程是否成功启动 一旦 Curio 服务运行，您可以继续 [为 Curio 节点附加存储以进行封装或永久存储](/zh/storage-configuration) 或 [在集群中设置下一个 Curio 节点](/zh/scaling-curio-cluster)。 # 存储配置 | Storage Configuration 本指南描述了如何为Curio节点附加和配置密封和永久存储 ## Storage Configuration ## 存储配置 每个Curio节点在 `~/.curio/storage.json`（或 `$CURIO_REPO_PATH/storage.json`）中跟踪定义的存储位置，并使用 `~/.curio` 路径作为默认值。 初始化存储位置时，会创建一个 `/sectorstorage.json` 文件，其中包含分配给该位置的UUID，以及是否可用于密封或存储。 ### Adding sealing storage location ### 添加密封存储位置 在添加密封存储位置之前，您需要考虑密封任务将在哪里执行。此命令必须从您想要附加存储的Curio节点本地运行。 ```bash curio cli --machine storage attach --init --seal ``` ### Adding long-term storage location ### 添加长期存储位置 **自定义存储位置：** 密封过程完成后，密封的扇区会被移动到存储位置，可以按以下方式指定： ```bash curio cli --machine storage attach --init --store ``` 此命令必须从您想要附加存储的Curio节点本地运行。这个位置可以由大容量但较慢的旋转硬盘组成。 ### Attach existing storage to Curio ### 将现有存储附加到Curio `lotus-miner` 或 `lotus-worker` 使用的存储位置可以被Curio集群重复使用。一旦迁移的 `lotus-miner` 或 `lotus-worker` 已经运行Curio服务，就可以附加它。 ```bash curio cli --machine storage attach ``` ### Filter sector types ### 过滤扇区类型 您可以通过调整 `/sectorstorage.json` 中的配置文件来过滤每个密封路径中允许的扇区类型。 ```json { "ID": "1626519a-5e05-493b-aa7a-0af71612010b", "Weight": 10, "CanSeal": false, "CanStore": true, "MaxStorage": 0, "Groups": [], "AllowTo": [], "AllowTypes": null, "DenyTypes": null } ``` `AllowTypes` 和 `DenyTypes` 的有效值是： "unsealed" "sealed" "cache" "update" "update-cache" 这些值必须放在数组中才有效（例如 `"AllowTypes": ["unsealed", "update-cache"]`），任何其他值都会在 `Curio` 启动时生成错误。还需要重启附加了此存储的 `Curio` 节点，以使更改生效。 ### Separate sealed and unsealed ### 分离密封和未密封扇区 一个非常基本的设置，您可以通过以下方式分离未密封和密封的扇区： * 在您想要存储密封扇区的长期存储路径中添加 `"DenyTypes": ["unsealed"]`。 * 在您想要存储未密封扇区的长期存储路径中添加 `"AllowTypes": ["unsealed"]`。 仅为 `AllowTypes` 设置 `unsealed` 仍然允许 `cache` 和 `update-cache` 文件放置在此存储路径中。如果您想完全拒绝此路径中的所有其他类型的扇区，可以在 `"DenyTypes"` 字段中添加其他有效值。 {% hint style="info" %} 如果存储路径中存在不允许类型的现有文件，这些文件仍然可以用于PoSt/检索。因此，在存储路径配置错误的情况下，最坏的情况是密封任务会卡住，等待存储变得可用。 {% endhint %} ### Segregating long-term storage per miner ### 按矿工隔离长期存储 用户可以通过在 `sectorstore.json` 文件中指定矿工地址字符串来为特定矿工ID分配长期存储。这种配置允许精确控制哪些矿工可以使用存储。 * 要允许特定矿工，请在AllowMiners数组中包含他们的地址： ```yaml "AllowMiners": ["t01000", "t01002"] ``` 这种配置只允许列出的矿工（t01000和t01002）访问存储。 * 同样，要拒绝特定矿工访问存储，请在DenyMiners数组中包含他们的地址： ```yaml "DenyMiners": ["t01003", "t01004"] ``` 在这个例子中，地址为t01003和t01004的矿工被明确拒绝访问存储。 这种双重配置方法允许基于矿工ID灵活和安全地管理存储访问。 # 配置 | Configuration 如何编辑和管理Curio集群的配置 ## Configuration ## 配置 Curio的配置存储在HarmonyDB的一个名为`harmony_config`的表中。当启动Curio节点时，会提供一个或多个层名称以获取该节点所需的配置。 ### Configuration Layers ### 配置层 配置层提供了一组决定系统如何运行的配置参数。这些层可以在不同级别定义，意味着更高层可以覆盖较低层，系统将根据最终堆叠的输出进行行为。 配置层可以按层次结构排列，通常从`base`（最通用）到最具体。`base`层定义默认配置值。更具体的层会用更有针对性的配置覆盖这些默认值。 例如，在一个简单的两层配置系统中，层可以按以下顺序组织：基础层 - 这是最通用的层。它总是被包含，所以在这里添加任何对默认值的修改。如果你在这里包含所有的矿工ID（在地址部分定义），所有硬件将用于所有矿工需求。任务层 - 这一层启用SDR任务。考虑包含的层（如下）。 如果Curio节点以上述2层启动，那么它将为所有矿工ID执行SDR任务，并使用任何其他配置参数的默认值。 层的使用示例： ```bash curio run --layers=post ``` {% hint style="warning" %} 当Curio节点启动时，`base`层总是默认应用。 {% endhint %} #### Advantages of Configuration Layers #### 配置层的优势 灵活性：配置层允许系统的不同部分或不同用户根据预定义的设置表现不同。 可扩展性：通过分离关注点并允许特定配置，系统在扩展时变得更容易管理。 可维护性：可以在适当的层上进行配置更改，而不影响整个系统。 #### Layer Stacking #### 层堆叠 配置层按提供的顺序堆叠。`base`层总是默认应用，所以可以跳过。 例如，如果Curio节点以以下层启动： ``` --layers miner1,sdr,wdPost,pricing ``` 这些层将相互堆叠，创建节点的最终配置。堆叠顺序将是base > miner1 > sdr > wdPost > pricing。如果一个配置参数在多个层中定义，则将使用最终层的值。 #### Working with layers #### 使用层 Curio允许您使用层来管理节点配置。每个层可以独立应用或修改，其中'base'层在启动时是必需的。 **Print default configuration** **打印默认配置** 默认配置在base层中默认使用。 ```bash curio config default ``` **Adding a New Layer** **添加新层** 要添加新的配置层或更新现有的层，您可以提供文件名或通过stdin直接输入。 ```bash curio config set --title ``` **List all layers** **列出所有层** 列出数据库中存在的所有配置层。 ```bash curio config ls ``` **Editing a Layer** **编辑层** 直接编辑配置层。 * 使用`vim`编辑器编辑 ```bash curio config edit --editor vim ``` * 使用其他编辑器如nano编辑 ```bash curio config edit --editor nano ``` **Interpreting Stacked Layers** **解释堆叠的层** 解释并查看所有应用的配置层的组合效果，包括系统生成的注释。 ```bash curio config view --layers [layer1,layers2] ``` **Removing a Layer** **移除层** 通过名称移除特定的配置层。 ```bash curio config rm ``` #### Pre-built Layers #### 预构建层 当初始化第一个Curio矿工或将第一个Lotus-Miner迁移到Curio时，该过程默认为用户创建一些层。这些层主要定义特定任务是否应该被机器选择。 ```toml #### **post** [Subsystems] EnableWindowPost = true EnableWinningPost = true ``` ```bash #### **sdr** [Subsystems] EnableSealSDR = true ``` ```toml #### **seal** [Subsystems] EnableSealSDR = true EnableSealSDRTrees = true EnableSendPrecommitMsg = true EnablePoRepProof = true EnableSendCommitMsg = true EnableMoveStorage = true ``` ```toml #### **seal-gpu** [Subsystems] EnableSealSDRTrees = true EnableSendPrecommitMsg = true ``` ```toml #### **seal-snark** [Subsystems] EnablePoRepProof = true EnableSendCommitMsg = true ``` ```toml #### **gui** [Subsystems] EnableWebGui = true ``` ### Configuration management in UI ### UI中的配置管理 Curio GUI为管理配置提供了一个用户友好的界面。要访问此功能，请从UI菜单导航到"Configurations"页面。在此页面上，列出了数据库中所有可用的层。用户可以通过点击每个层来编辑它。
Configurations

Curio GUI配置页面

要更新配置字段，用户必须首先通过勾选相应的框来启用它。启用后，可以填充字段值。要注释掉该字段，只需取消勾选该框。
Configuration edit

Curio GUI配置编辑器

# 监听地址 | Listen Address 如何更新 Curio 服务的默认监听地址 ## Listen Address ## 监听地址 默认情况下，所有 Curio 节点都绑定到地址 "0.0.0.0" 和端口 "12300"，确保 Curio API 在所有接口上监听。您可以通过指定一个明确的 IP 地址和不同的端口来改变这种行为。 ```bash echo "CURIO_LISTEN=x.x.x.x:12301" >> /etc/curio.env ``` 然后重启 Curio 服务 ```bash systemctl restart curio.service ``` # 警报管理器 | Alert Manager Curio 警报管理器设置和配置 ## Alert Manager ## 警报管理器 Curio 有一个每小时运行一次的 AlertManager 任务，允许 Curio 集群向用户提醒集群中的某些问题。 目前，Curio 支持以下问题的警报： 1. 钱包余额低于 5 Fil。 2. 如果某个截止日期没有运行 WindowPost 任务。 3. 如果发现孤立块或者没有为任何 epoch 创建 WinningPost 任务。 4. 如果永久存储没有足够的空间来容纳当前正在封装的扇区。 AlertManager 是一个基于插件的模块，允许与任何插件集成。目前，Curio 有 2 个可用的插件。生成的警报可以同时发送到多个插件，以实现更强大的通知机制。 {% hint style="info" %} 欢迎对新的关键警报或与其他警报系统的集成做出贡献。 {% endhint %} #### PagerDuty Plugin #### PagerDuty 插件 Curio 默认集成了 [PagerDuty.com](https://www.pagerduty.com/)，允许向存储提供商发送关键警报。要配置您的 Curio 集群以发送警报，您必须设置一个 PagerDuty 账户。 {% hint style="danger" %} 与此软件开发相关的任何人都与 PagerDuty 没有任何业务关系。提供此集成是为了方便存储提供商选择的警报系统。 {% endhint %} 1. 在[这里](https://www.pagerduty.com/sign-up-free/?type=free)注册一个免费的 PagerDuty 账户。 2. 创建一个新的服务，用于处理来自 Curio 集群的警报。 3. 在创建服务过程中，在"Integration"页面上，选择"Events API V2"。 4. 服务创建完成后，从服务中复制"Integration Key"，并将其粘贴到"base"层配置中的"PagerDutyIntegrationKey"。 5. 在配置层中启用插件。 6. 重启其中一个节点，此后它会在每个整点生成关键警报。 \[Alerting] ## MinimumWalletBalance is the minimum balance all active wallets. If the balance is below this value, an ## alerts will be triggered for the wallet ## ## type: types.FIL \#MinimumWalletBalance = "5 FIL" \[Alerting.PagerDuty] # Enable is a flag to enable or disable the PagerDuty integration. # # type: bool Enable = true ``` # PagerDutyEventURL is URL for PagerDuty.com Events API v2 URL. Events sent to this API URL are ultimately # routed to a PagerDuty.com service and processed. # The default is sufficient for integration with the stock commercial PagerDuty.com company's service. # # type: string PagerDutyEventURL = "https://events.pagerduty.com/v2/enqueue" # PageDutyIntegrationKey is the integration key for a PagerDuty.com service. You can find this unique service # identifier in the integration page for the service. # # type: string PageDutyIntegrationKey = "" ``` #### Prometheus AlertManager #### Prometheus 警报管理器 1. 设置一个 [Prometheus AlertManager](https://prometheus.io/docs/alerting/latest/alertmanager/) 实例。 2. 通过将 `Enabled` 设置为 True 来启用插件。 3. 在配置中粘贴 `AlertManagerURL`。 4. 使用更新后的配置层重启节点。 \[Alerting.PrometheusAlertManager] # Enable is a flag to enable or disable the Prometheus AlertManager integration. # # type: bool Enable = true ``` # AlertManagerURL is the URL for the Prometheus AlertManager API v2 URL. # # type: string AlertManagerURL = "http://localhost:9093/api/v2/alerts" ``` # 默认Curio配置 | Default Curio Configuration The default curio configuration ```toml [Subsystems] # 启用窗口后证明在此 curio 实例上执行。集群中每台启用了窗口后证明的机器也将参与窗口后证明调度器。可以有多台启用了窗口后证明的机器，这将提供冗余，并且在每个截止日期有多个分区的情况下，将允许并行处理分区。 # # 可以有同时处理窗口后证明和获胜后证明的实例，这可以提供冗余而无需额外的机器。在这样的设置中，通常建议运行 partitionsPerDeadline+1 台机器。 # # 类型：bool #EnableWindowPost = false # 类型：int #WindowPostMaxTasks = 0 # 启用获胜后证明在此 curio 实例上执行。集群中每台启用了获胜后证明的机器也将参与获胜后证明调度器。 # 可以混合使用启用了窗口后证明和获胜后证明的机器，详情请参阅 EnableWindowPost 文档。 # # 类型：bool #EnableWinningPost = false # 类型：int #WinningPostMaxTasks = 0 # 启用"数据块停放"任务在此节点上运行。此任务负责从网络获取数据块并将其存储在存储子系统中，直到扇区被密封。此任务 # 仅适用于与 boost 集成时，并且应在将保存来自 boost 的交易数据的节点上启用，直到包含相关数据块的扇区构建了 TreeD/TreeR。 # 请注意，未来的 Curio 实现将有一个单独的任务类型用于从互联网获取数据块。 # # 类型：bool #EnableParkPiece = false # 类型：int #ParkPieceMaxTasks = 0 # 启用 SDR 任务运行。SDR 是在扇区缓存目录中创建 11 个层文件的长序列计算。 # # SDR 是密封管道中的第一个任务。它的输入仅仅是未密封数据的哈希（CommD）、扇区编号、矿工 ID 和密封证明类型。 # 它的输出是扇区缓存目录中的 11 个层文件。 # # 在 lotus-miner 中，这是作为 PreCommit1 的一部分运行的。 # # 类型：bool #EnableSealSDR = false # 可以同时运行的 SDR 任务的最大数量。请注意，最大任务数量也将受到机器上可用资源的限制。 # # 类型：int #SealSDRMaxTasks = 0 # 系统开始接受新任务之前需要排队的 SDR 任务的最大数量。 # 此设置的主要目的是允许积累足够的任务以进行批量密封。当集群中存在批量密封节点时，此值应设置为 batch_size+1，以允许批量密封节点填满批次。 # 此设置还可以用于通过在应该具有较低优先级的节点上设置较高的值来给予集群中的其他节点优先级。 # # 类型：int #SealSDRMinTasks = 0 # 启用 SDR 管道树构建任务运行。 # 此任务处理未密封数据编码到最后一个 SDR 层，并构建 TreeR、TreeC 和 TreeD。 # # 此任务在 SDR 之后运行 # 首先计算 TreeD，可选输入未密封数据 # TreeR 从副本计算，副本首先计算为最后一个 SDR 层和 TreeD 底层（即未密封数据）的字段加法 # TreeC 从 11 个 SDR 层计算 # 这 3 个树稍后将用于计算 PoRep 证明。 # # 在 SyntheticPoRep 的情况下，PoRep 的挑战将在此步骤预生成，树和层将被丢弃。SyntheticPoRep 通过预生成一个非常大的挑战集（磁盘上约 30GiB） # 然后使用其中的一小部分子集进行实际的 PoRep 计算。这允许在 PreCommit 和 PoRep 生成之间显著节省临时空间，代价是更多的计算（在此步骤生成挑战） # # 在 lotus-miner 中，这是作为 PreCommit2 的一部分运行的（TreeD 在 PreCommit1 中运行）。 # 请注意，启用了 SDRTrees 的节点也将响应 Finalize 任务， # 这只是在计算 PoRep 后删除不需要的树数据。 # # 类型：bool #EnableSealSDRTrees = false # 可以同时运行的 SealSDRTrees 任务的最大数量。请注意，最大任务数量也将受到机器上可用资源的限制。 # # 类型：int #SealSDRTreesMaxTasks = 0 # FinalizeMaxTasks 是可以同时运行的最大完成任务数量。 # 完成任务在所有处理 SDRTrees 任务的机器上都启用。完成任务始终在持有扇区缓存文件的机器上运行，因为它在计算 PoRep 后删除不需要的树数据。 # 完成任务将与 SubmitCommitMsg 任务并行运行。 # # 类型：int #FinalizeMaxTasks = 0 # EnableSendPrecommitMsg 启用从此 curio 实例向链发送预提交消息。 # 这在 SDRTrees 之后运行，并使用输出的 CommD / CommR（TreeD / TreeR 的根）作为消息内容 # # 类型：bool #EnableSendPrecommitMsg = false # EnablePoRepProof 启用 porep 证明的计算 # # 此任务在交互式 porep 种子可用后运行，这发生在预提交消息上链后 150 个纪元（75 分钟）。此任务应在具有 GPU 的机器上运行。普通 PoRep 证明 # 从持有扇区缓存文件的机器请求，该机器很可能是运行 SDRTrees 任务的机器。 # # 在 lotus-miner 中，这是 Commit1 / Commit2 # # 类型：bool #EnablePoRepProof = false # 可以同时运行的 PoRepProof 任务的最大数量。请注意，最大任务数量也将受到机器上可用资源的限制。 # # 类型：int #PoRepProofMaxTasks = 0 # EnableSendCommitMsg 启用从此 curio 实例向链发送提交消息。 # # 类型：bool #EnableSendCommitMsg = false # 是否在批处理中任何扇区激活失败时中止（仅适用于新密封的扇区，仅使用 ProveCommitSectors3）。 # # 类型：bool #RequireActivationSuccess = true # 是否在批处理中任何扇区激活失败时中止（更新扇区，仅使用 ProveReplicaUpdates3）。 # # 类型：bool #RequireNotificationSuccess = true # EnableMoveStorage 启用在此 curio 实例上运行移动到长期存储的任务。 # 此任务应仅在具有长期存储的节点上启用。 # # MoveStorage 任务是密封管道中的最后一个任务。它将密封的扇区数据从 SDRTrees 机器移动到长期存储中。此任务在完成任务之后运行。 # # 类型：bool #EnableMoveStorage = false # 可以同时运行的 MoveStorage 任务的最大数量。请注意，最大任务数量也将受到机器上可用资源的限制。建议将此值设置为一个能够充分利用机器上所有可用网络（或磁盘）带宽而不造成瓶颈的数字。 # # 类型：int #MoveStorageMaxTasks = 0 # EnableUpdateEncode 在此 curio 实例上启用 SnapDeal 过程的编码步骤。 # 此步骤涉及将数据编码到扇区中并计算更新的 TreeR（使用 gpu）。 # # 类型：bool #EnableUpdateEncode = false # EnableUpdateProve 在此 curio 实例上启用 SnapDeal 过程的证明步骤。 # 此步骤为更新的扇区生成 snark 证明。 # # 类型：bool #EnableUpdateProve = false # EnableUpdateSubmit 启用从此 curio 实例向区块链提交 SnapDeal 证明。 # 此步骤将生成的证明提交到链上。 # # 类型：bool #EnableUpdateSubmit = false # UpdateEncodeMaxTasks 设置此实例上可以运行的并发 SnapDeal 编码任务的最大数量。 # # 类型：int #UpdateEncodeMaxTasks = 0 # UpdateProveMaxTasks 设置此实例上可以运行的并发 SnapDeal 证明任务的最大数量。 # # 类型：int #UpdateProveMaxTasks = 0 # BoostAdapters 是矿工地址和端口/IP 的元组列表，用于监听市场（例如 boost）请求。 # 此接口与 lotus-miner RPC 兼容，实现了存储市场操作所需的子集。 # 字符串应采用 "actor:ip:port" 格式。IP 不能为 0.0.0.0。我们建议使用私有 IP。 # 示例："f0123:127.0.0.1:32100"。可以指定多个地址。 # # 当市场节点（如 boost）向 Curio 的市场 RPC 提供要放入扇区的交易时，Curio 首先将交易数据存储在临时位置 "Piece Park" 中，然后再将其分配给扇区。这要求集群中至少有一个节点启用了 EnableParkPiece 选项，并有足够的临时空间来存储交易数据。 # 这与 lotus-miner 不同，lotus-miner 在收到交易后立即将交易数据存储到 "未密封" 的扇区中。当计算扇区的 TreeD 和 TreeR 时会访问 PiecePark 中的交易数据，但在初始 SDR 层计算时不需要。在引用该数据的所有扇区都密封后，PiecePark 中的数据将被删除。 # # 要获取 boost 配置的 API 信息，请运行 'curio market rpc-info' # # 注意：所有交易数据都将通过此服务流动，因此应将其放置在运行 boost 的机器上或处理 ParkPiece 任务的机器上。 # # 类型：[]string #BoostAdapters = [] # EnableWebGui 在此 curio 实例上启用 Web GUI。UI 的本地开销很小，但通常只需要在集群中的一台机器上运行。 # # 类型：bool #EnableWebGui = false # 应该监听 Web GUI 请求的地址。 # # 类型：string #GuiAddress = "0.0.0.0:4701" # UseSyntheticPoRep 为所有新扇区启用合成 PoRep。设置为 true 时，将在 TreeRC 任务完成后将磁盘上保留的缓存数据量减少到 11GiB。 # # 类型：bool #UseSyntheticPoRep = false # 可以同时运行的 SyntheticPoRep 任务的最大数量。请注意，最大任务数量也将受到机器上可用资源的限制。 # # 类型：int #SyntheticPoRepMaxTasks = 0 # 批量密封 # # 类型：bool #EnableBatchSeal = false [Fees] # 类型：types.FIL #DefaultMaxFee = "0.07 FIL" # 类型：types.FIL #MaxPreCommitGasFee = "0.025 FIL" # 类型：types.FIL #MaxCommitGasFee = "0.05 FIL" # 类型：types.FIL #MaxTerminateGasFee = "0.5 FIL" # WindowPoSt 是一个高价值操作，因此默认费用应该较高。 # # 类型：types.FIL #MaxWindowPoStGasFee = "5 FIL" # 类型：types.FIL #MaxPublishDealsFee = "0.05 FIL" # 是否使用可用的矿工余额作为扇区抵押品，而不是随每条消息一起发送 # # type: bool #CollateralFromMinerBalance = false # 即使矿工参与者没有可用余额，也不要随消息发送抵押品 # # type: bool #DisableCollateralFallback = false [Fees.MaxPreCommitBatchGasFee] # 类型：types.FIL #Base = "0 FIL" # 类型：types.FIL #PerSector = "0.02 FIL" [Fees.MaxCommitBatchGasFee] # 类型：types.FIL #Base = "0 FIL" # 类型：types.FIL #PerSector = "0.03 FIL" [[Addresses]] #PreCommitControl = [] #CommitControl = [] #TerminateControl = [] #DisableOwnerFallback = false #DisableWorkerFallback = false #MinerAddresses = [] [Proving] # 并行运行的最大扇区检查数量。(0 = 无限制) # # 警告：将此值设置得太高可能会导致节点因耗尽堆栈而崩溃 # 警告：将此值设置得太低可能会使扇区挑战读取变得更慢，导致由于提交延迟而失败的 PoSt # # 更改此选项后，通过调用 'lotus-miner proving compute window-post 0' 确认新值在您的设置中是否有效 # # type: int #ParallelCheckLimit = 32 # 扇区证明预检可以花费的最长时间。如果检查超时，该扇区将被跳过 # # 警告：将此值设置得太低可能会导致扇区被跳过，即使它们是可访问的，只是读取测试挑战花费的时间超过了此超时时间 # 警告：将此值设置得太高可能会在与此扇区相关的 IO 操作被阻塞的情况下错过 PoSt 截止时间（例如，在 NFS 挂载断开连接的情况下） # # type: Duration #SingleCheckTimeout = "10m0s" # 整个分区的证明预检可以花费的最长时间。如果检查超时，未能及时检查的分区中的扇区将被跳过 # # 警告：将此值设置得太低可能会导致扇区被跳过，即使它们是可访问的，只是读取测试挑战花费的时间超过了此超时时间 # 警告：将此值设置得太高可能会在与此分区相关的 IO 操作被阻塞或缓慢的情况下错过 PoSt 截止时间 # # type: Duration #PartitionCheckTimeout = "20m0s" # 禁用 WindowPoSt 可证明扇区可读性检查。 # # 在正常操作中，当准备计算 WindowPoSt 时，lotus-miner 将执行一轮从所有扇区读取挑战的过程，以确认这些扇区可以被证明。在此过程中读取的挑战会被丢弃，因为我们只关心检查扇区数据是否可以被读取。 # # 当使用内置证明计算（没有 PoSt 工作者，且 DisableBuiltinWindowPoSt 设置为 false）时，如果某些扇区不可读，这个过程可以节省大量时间和计算资源 - 这是因为内置逻辑在需要跳过某些扇区时不会跳过 snark 计算。 # # 当使用 PoSt 工作者时，这个过程大部分是多余的，PoSt 工作者的挑战将只读取一次，如果某些扇区的挑战不可读，这些扇区将被跳过。 # # 禁用扇区预检将略微减少证明扇区时的 IO 负载，可能会缩短生成窗口 PoSt 的时间。在 IO 能力良好的设置中，此选项对证明时间的影响应该可以忽略不计。 # # 注意：在没有 PoSt 工作者的设置中禁用扇区预检可能是一个糟糕的主意。 # # 注意：即使启用此选项，在向链发送恢复声明消息之前，仍会检查恢复扇区 # # 更改此选项后，通过调用 'lotus-miner proving compute window-post 0' 确认新值在您的设置中是否有效 # # type: bool #DisableWDPoStPreChecks = false # 在单个 SubmitWindowPoSt 消息中证明的最大分区数。0 = 网络限制（nv21 中为 3） # # 单个分区可能包含最多 2349 个 32GiB 扇区，或 2300 个 64GiB 扇区。 # // # 注意，将此值设置得更低可能会导致气体使用效率降低 - 将发送更多消息来证明每个截止时间，导致总气体使用量增加（但每条消息的气体限制会更低） # # 将此值设置为高于网络限制没有效果 # # type: int #MaxPartitionsPerPoStMessage = 0 # 在某些情况下，提交 DeclareFaultsRecovered 消息时， # 可能有太多恢复无法适应 BlockGasLimit。 # 在这些情况下，可能需要将此值设置为较低的值（例如 1）； # 注意，将此值设置得更低可能会导致气体使用效率降低 - 将发送比需要更多的消息， # 导致总气体使用量增加（但每条消息的气体限制会更低） # # type: int #MaxPartitionsPerRecoveryMessage = 0 # 为包含恢复扇区的分区启用每个 PoSt 消息单个分区 # # 在提交包含恢复扇区的 PoSt 消息的情况下，默认网络限制可能仍然太高，无法适应区块气体限制。在这些情况下，将恢复扇区的单个分区放在 post 消息中变得有用 # # 注意，将此值设置得更低可能会导致气体使用效率降低 - 将发送更多消息来证明每个截止时间，导致总气体使用量增加（但每条消息的气体限制会更低） # # type: bool #SingleRecoveringPartitionPerPostMessage = false [Ingest] # 可以排队等待交易开始处理的最大扇区数量。 # 0 = 无限制 # 注意：此机制将延迟从市场获取交易数据，为市场子系统提供背压。 # DealSector 队列包括准备进入密封管道但尚未进入的交易 - # 此队列的大小也将影响可以同时运行的 ParkPiece 任务的最大数量。 # DealSector 队列是密封管道中的第一个队列，这意味着它应该用作主要的背压机制。 # # type: int #MaxQueueDealSector = 8 # 可以排队等待 SDR 开始处理的最大扇区数量。 # 0 = 无限制 # 注意：此机制将延迟从市场获取交易数据，为市场子系统提供背压。 # SDR 队列包括正在进入密封管道过程中的交易。对于 SDR 任务， # 可能会出现此队列增长超过此限制的情况（CC 扇区），背压仅应用于进入管道的扇区。 # # type: int #MaxQueueSDR = 8 # 可以排队等待 SDRTrees 开始处理的最大扇区数量。 # 0 = 无限制 # 注意：此机制将延迟从市场获取交易数据，为市场子系统提供背压。 # 对于树任务，可能会出现此队列增长超过此限制的情况，背压仅应用于进入管道的扇区。 # # type: int #MaxQueueTrees = 0 # 可以排队等待 PoRep 开始处理的最大扇区数量。 # 0 = 无限制 # 注意：此机制将延迟从市场获取交易数据，为市场子系统提供背压。 # 与树任务类似，可能会出现此队列增长超过此限制的情况，背压仅应用于进入管道的扇区。 # # type: int #MaxQueuePoRep = 0 # 开放的交易扇区在开始密封之前应等待更多交易的最长时间 # # type: Duration #MaxDealWaitTime = "1h0m0s" # DoSnap 启用此实例摄取的交易的快照交易处理。与 lotus-miner 不同，当没有可用于快照的扇区时，不会回退到 porep。启用后，所有交易都将是快照交易。 # # type: bool #DoSnap = false [Seal] # BatchSealSectorSize 允许设置批量密封任务支持的扇区大小。 # 可以是任何值，只要它是 "32GiB"。 # # type: string #BatchSealSectorSize = "32GiB" # 密封批次中的扇区数量。取决于硬件和批量封装（batch sealing）配置。 # # type: int #BatchSealBatchSize = 32 # 并行管道的数量。可以是 1 或 2。取决于可用的原始块存储 # # type: int #BatchSealPipelines = 2 # SingleHasherPerThread 是针对较旧 CPU 的兼容性标志。Zen3 及更高版本支持每个线程两个扇区。 # 对于较旧的 CPU（Zen 2 及之前），设置为 false。 # # type: bool #SingleHasherPerThread = false [Apis] # 存储子系统的 RPC 密钥。 # 如果与 lotus-miner 集成，这必须与以下命令的值匹配 # cat ~/.lotusminer/keystore/MF2XI2BNNJ3XILLQOJUXMYLUMU | jq -r .PrivateKey # # type: string #StorageRPCSecret = "" [Alerting] # MinimumWalletBalance 是所有活跃钱包的最低余额。如果余额低于此值，将为该钱包触发警报 # # type: types.FIL #MinimumWalletBalance = "5 FIL" [Alerting.PagerDuty] # Enable 是启用或禁用 PagerDuty 集成的标志。 # # type: bool #Enable = false # PagerDutyEventURL 是 PagerDuty.com Events API v2 URL。发送到此 API URL 的事件最终会路由到 PagerDuty.com 服务并进行处理。 # 默认值足以与商业 PagerDuty.com 公司的标准服务集成。 # # type: string #PagerDutyEventURL = "https://events.pagerduty.com/v2/enqueue" # PageDutyIntegrationKey 是 PagerDuty.com 服务的集成密钥。您可以在服务的集成页面中找到这个唯一的服务标识符。 # # type: string #PageDutyIntegrationKey = "" [Alerting.PrometheusAlertManager] # Enable 是启用或禁用 Prometheus AlertManager 集成的标志。 # # type: bool #Enable = false # AlertManagerURL 是 Prometheus AlertManager API v2 URL。 # # type: string #AlertManagerURL = "http://localhost:9093/api/v2/alerts" [Alerting.SlackWebhook] # Enable 是启用或禁用 Prometheus AlertManager 集成的标志。 # # type: bool #Enable = false # WebHookURL 是 Slack Webhook 的 URL。 # 示例：https://hooks.slack.com/services/T00000000/B00000000/XXXXXXXXXXXXXXXXXXXXXXXX # # type: string #WebHookURL = "" ``` # 启用市场 | Enabling market 如何啟用市場子模組並連接到 Boost ## Enabling market ## 啟用市場 ### Introduction ### 介紹 Curio 提供了一個市場子模組，可以無縫集成 Boost，而無需對 Boost 代碼庫進行任何更改。本文檔將指導您如何在 Curio 中啟用市場、配置 PiecePark，以及設置現有和新的 Boost 實例。 ### Enable Market adapter in Curio ### 在 Curio 中啟用市場適配器 編輯市場配置 ```shell curio config add --title mt01000 ``` 添加一個如下的條目： \[Subsystems] EnableParkPiece = true BoostAdapters = \["t10000:127.0.0.1:32100"] 按 `ctrl + D` 保存並退出。 {% hint style="info" %} 每個礦工 ID 應該只運行一個 Curio 市場適配器節點。Boost 一次只能與一個適配器通信。 {% endhint %} 編輯 `/etc/curio.env` 文件，並更新 `CURIO_LAYERS` 變量以包含新的市場層。 ```yaml CURIO_LAYERS=seal,mt01000 CURIO_ALL_REMAINING_FIELDS_ARE_OPTIONAL=true CURIO_DB_HOST=yugabyte1,yugabyte2,yugabyte3 CURIO_DB_USER=yugabyte CURIO_DB_PASSWORD=yugabyte ``` 重啟 Curio 服務以使更改生效。 {% hint style="info" %} 運行市場的節點是 Boost 將與之交互的節點。它代理交易數據，因此最好將其運行在 Boost 旁邊或與運行 TreeD（和 TreeRC）任務的同一節點上。TreeD 是我們將交易數據添加到扇區的地方。 {% endhint %} ### Connecting with Boost ### 連接 Boost #### Get Market RPC Info #### 獲取市場 RPC 信息 ```shell curio market rpc-info --layers mt01000 ``` {% hint style="info" %} 如果 rpc-info 的輸出為空，那麼您可能沒有包含包含 `BoostAdapters` 配置的正確層。請重新檢查您的 --layers 標誌。 {% endhint %} #### Connect with existing Boost after migration #### 遷移後連接現有的 Boost 使用市場 rpc-info 字符串替換 Boost 配置中的 `SealerApiInfo` 和 `SectorIndexApiInfo` 字符串，然後重啟 Boost。 #### Initialising New Boost #### 初始化新的 Boost 按照 [Boost 設置說明](https://boost.filecoin.io/new-boost-setup) 進行操作，並額外更改將 `MINER_API_INFO` 替換為市場 rpc-info 字符串。 #### Updating PeerID and On-Chain Address #### 更新 PeerID 和鏈上地址 確保在鏈上設置了您的 SP 的正確 *peer id* 和 *multiaddr*，因為 `boost init` 會生成一個新的身份。使用以下命令更新鏈上的值： 查找 PeerID ```shell boostd net id ``` ```shell boostd net listen ``` ```bash sptool --actor actor set-addrs sptool --actor actor set-peer-id ``` # 快速交易 | Snap Deals 本指南解释了如何在Curio中启用snap-deals。 ## Snap Deals ## 快照交易 ### Simplified explanation ### 简化解释 快照交易允许存储提供商接受用户的交易，并将用户的数据放入已经提交的存储块中。这听起来有点复杂，所以让我们这样想象一下。 想象有一个城镇，里面有一个很长的架子。这个城镇的任何人都可以在这个架子上存储任何东西。当一个镇民想要存储某样东西时，他们把那个"东西"交给存储提供商。存储提供商制作一个木箱，把镇民的东西放进箱子里，然后把箱子放在架子上。
代表Filecoin网络的架子。

扇区作为一个架子

一些箱子里装有有用的东西，比如照片、音乐或视频。但有时，存储提供商没有镇民排队要把有用的东西放进箱子里。所以他们就把包装花生放进箱子里，然后把它放在架子上。这意味着有很多箱子被制作出来只是为了装包装花生。制作箱子需要很长时间，也需要存储提供商付出大量的工作。
Filecoin扇区中的数据类型。

数据箱

与其每次有人想存储东西时都创建一个新箱子，不如我们直接用有用的东西替换包装花生！因为没有人在乎包装花生，所以把它们扔掉也不会有人不高兴。而且存储提供商可以在架子上放置有用的东西，而不必创建新的箱子！对镇民来说也更好，因为他们不必等待存储提供商创建新的箱子！
清空扇区中的虚拟数据，用真实数据填充。

替换数据

这是快照交易工作方式的简化视图。存储提供商不需要创建一个全新的扇区来存储客户的数据，而是可以将客户的数据放入已提交容量的扇区中。数据变得更快可用，对存储提供商来说成本更低，而且网络的存储容量得到了更多的利用！ ### How to enable snap-deals ### 如何启用快照交易 要在Curio集群中启用快照交易管道，用户需要在具有GPU资源的机器上启用特定于快照交易的任务。除此之外，还需要更新交易接收管道，以将交易传递给快照交易管道，而不是PoRep封装管道。 {% hint style="warning" %} 数据可以在任何给定时间使用快照交易管道或PoRep管道进行接收，但不能同时使用两者。 {% endhint %} ### FastSnap（SnapDeals UpdateEncode 加速） Curio 的 SnapDeals `UpdateEncode` 支持 **快速模式**（“fastsnap”）：使用批量封装 CUDA 工具链（`extern/supraseal`）加速 TreeR 生成，并使用 Curio 原生的 snap 编码实现。 * **能力检查**： ```bash curio test supra system-info ``` 查看 **“Can run fast TreeR: yes”**。 * **回退模式**：如果主机缺少 AVX-512（AMD64v4）或没有可用 CUDA GPU，会自动回退到 CPU 的 TreeR 生成路径。 * **故障排查 / 强制回退**： ```bash export DISABLE_SUPRA_TREE_R=1 ``` 这会强制使用 CPU 回退的 TreeR 路径（用于隔离批量封装/CUDA/工具链问题）。 #### Enable snap tasks #### 启用快照任务 1. 将Curio已经附带的`upgrade`层添加到具有GPU资源的Curio节点上的`/etc/curio.env`文件中。\\ ```bash CURIO_LAYERS=gui,seal,post,upgrade <----- 添加"upgrade"层 CURIO_ALL_REMAINING_FIELDS_ARE_OPTIONAL=true CURIO_DB_HOST=yugabyte1,yugabyte2,yugabyte3 CURIO_DB_USER=yugabyte CURIO_DB_PASSWORD=yugabyte CURIO_DB_PORT=5433 CURIO_DB_NAME=yugabyte CURIO_REPO_PATH=~/.curio CURIO_NODE_NAME=ChangeMe FIL_PROOFS_USE_MULTICORE_SDR=1 ``` \\ 2. 重启节点上的Curio服务。\\ ```bash systemctl restart curio ``` #### Update the Curio market adapter #### 更新Curio市场适配器 1. 为您希望使用快照交易管道的minerID创建或更新市场层（[如果已经创建了一个](/zh/enabling-market#enable-market-adapter-in-curio)）。\\ ```bash curio config add --title mt01000 ``` \ 添加一个类似这样的条目：\\ ```yaml [Subsystems] EnableParkPiece = true BoostAdapters = ["t10000:127.0.0.1:32100"] [Ingest] DoSnap = true ``` \ 按`ctrl + D`保存并退出。\ 或编辑现有层。\\ curio config edit mt01000 \ 为接收启用快照交易：\\ ```yaml [Subsystems] EnableParkPiece = true BoostAdapters = ["t10000:127.0.0.1:32100"] [Ingest] DoSnap = true ``` \ 保存层并退出。 2. 根据[最佳实践](/zh/best-practices)将新的市场配置层添加到适当的节点。 3. 重启Curio服务。 # 批量封装 | Batch Sealing 本页面解释了如何在 Curio 中设置批量封装（extern/supraseal） ## 批量封装 | Batch Sealing {% hint style="danger" %} **免责声明：** 批量封装目前处于 **测试阶段**。请谨慎使用，并预期在未来版本中可能出现潜在问题或变更。目前需要一些额外的手动系统配置。 {% endhint %} 批量封装是一种优化的封装方式，允许并行封装多个扇区，与逐个封装相比可显著提高吞吐量。 ### CC 调度器（仅用于批量封装） Curio 提供 **CC Scheduler**（CC 调度器）页面及其对应的数据库表（`sectors_cc_scheduler`），用于按 SP 配置要排队进行批量封装的 CC 扇区数量。 * **CC 调度器仅用于批量封装** * 不要将其用于 deals / SnapDeals 如果你在某个节点启用了批量封装，但集群未启用 SnapDeals，deal 可能会被路由到 CC/批量封装管道并最终封装空扇区（丢弃 deal 数据）。如需封装真实 deal，请确保集群启用 SnapDeals。 ### Key Features ### 主要特性 * 在单个批次中封装多个扇区（最多 128 个） * 核心利用效率提高高达 16 倍 * 优化以高效利用 CPU 和 GPU 资源 * 使用原始 NVMe 设备进行层存储，而不是 RAM ### Requirements ### 要求 * CPU 每个 CCX（AMD）或同等配置至少有 4 个核心 * 具有高 IOPS 的 NVMe 驱动器（建议总 IOPS 为 1000-2000 万） * 用于 PC2 阶段的 GPU（建议使用 NVIDIA RTX 3090 或更好的） * 配置 1GB 大页（最少 36 页） * Ubuntu 或兼容的 Linux 发行版（**需要 gcc-13**，不需要系统范围内安装） * 至少 256GB RAM，所有内存通道都已填满 * 如果没有填满**所有**内存通道，封装**性能将大幅下降** * NUMA-Per-Socket (NPS) 设置为 1 ### Storage Recommendations ### 存储建议 您需要两组 NVMe 驱动器： 1. 用于层的驱动器： * 总计 1000-2000 万 IOPS * 容量为 11 x 32G x 批次大小 x 管道数 * 原始未格式化的块设备（SPDK 将接管它们） * 每个驱动器应能持续 \~2GiB/s 的写入速度 * 这个要求目前还不太清楚，可能较低的写入速率也可以。需要更多测试。 2. 用于 P2 输出的驱动器： * 带有文件系统 * 快速且容量充足（\~70G x 批次大小 x 管道数） * 如果速度足够快（\~500MiB/s/GPU），可以是远程存储 ### Hardware Recommendations ### 硬件建议 目前，社区正在努力确定批量封装的最佳硬件配置。一些普遍观察如下： * 单插槽系统将更容易以全容量使用 * 您需要大量 NVMe 插槽，在 PCIe Gen4 平台上使用大批量大小时，可能会使用 20-24 个 3.84TB NVMe 驱动器 * 通常，您需要确保所有内存通道都已填满 * 您需要 4\~8 个物理核心（非线程）用于批处理范围的计算，然后在每个 CCX 上，您将失去 1 个核心作为"协调器" * 每个线程计算 2 个扇区 * 在 zen2 及更早版本上，哈希器每个线程只计算一个扇区 * 大型（多核心）CCX 通常更好 {% hint style="info" %} 请考虑为 [批量封装硬件示例](https://github.com/filecoin-project/curio/discussions/140) 做出贡献。 {% endhint %} #### Benchmark NVME IOPS #### 基准测试 NVME IOPS 在进行进一步配置之前，请确保对原始 NVME IOPS 进行基准测试，以验证是否满足 IOPS 要求。 ```bash cd extern/supraseal/deps/spdk-v24.05/ # repeat -b with all devices you plan to use with supraseal # 注意：您需要测试所有设备，以便查看系统中是否存在任何瓶颈 ./build/examples/perf -b 0000:85:00.0 -b 0000:86:00.0... -q 64 -o 4096 -w randread -t 10 ``` 输出应该如下所示 ## ======================================================== Latency(us) Device Information : IOPS MiB/s Average min max PCIE (0000:04:00.0) NSID 1 from core 0: 889422.78 3474.31 71.93 10.05 1040.94 PCIE (0000:41:00.0) NSID 1 from core 0: 890028.08 3476.67 71.88 10.69 1063.32 PCIE (0000:42:00.0) NSID 1 from core 0: 890035.08 3476.70 71.88 10.66 1001.86 PCIE (0000:86:00.0) NSID 1 from core 0: 889259.28 3473.67 71.95 10.62 1003.83 PCIE (0000:87:00.0) NSID 1 from core 0: 889179.58 3473.36 71.95 10.55 993.32 PCIE (0000:88:00.0) NSID 1 from core 0: 889272.18 3473.72 71.94 10.38 974.63 PCIE (0000:c1:00.0) NSID 1 from core 0: 889815.08 3475.84 71.90 10.97 1044.70 PCIE (0000:c2:00.0) NSID 1 from core 0: 889691.08 3475.36 71.91 11.04 1036.57 PCIE (0000:c3:00.0) NSID 1 from core 0: 890082.78 3476.89 71.88 10.44 1023.32 Total : 8006785.90 31276.51 71.91 10.05 1063.32 理想情况下，所有设备的总 IOPS 应大于 1000 万。 ### Setup ### 设置 #### Dependencies #### 依赖项 需要 CUDA 12.x，11.x 不能工作。构建过程依赖于系统范围内的 GCC 13.x 或本地安装的 `gcc-13`/`g++-13`。 * 在 Arch 上根据发行版/AUR 安装 GCC 13 * 在 Ubuntu/Debian 上安装 `gcc-13` 和 `g++-13` 包 #### Building #### 构建 构建支持批处理的 Curio 二进制文件： ```bash make curio ``` 对于 calibnet： ```bash make calibnet ``` {% hint style="warning" %} 构建应在目标机器上运行。由于不同的 AVX512 支持，二进制文件在 CPU 代之间不可移植。 {% endhint %} ### Configuration ### 配置 * 在目标机器上运行 `curio calc batch-cpu` 以确定您的 CPU 支持的批次大小
批量CPU输出示例 ``` # EPYC 7313 16-Core Processor root@udon:~# ./curio calc batch-cpu Basic CPU Information Processor count: 1 Core count: 16 Thread count: 32 Threads per core: 2 Cores per L3 cache (CCX): 4 L3 cache count (CCX count): 4 Hasher Threads per CCX: 6 Sectors per CCX: 12 --------- Batch Size: 16 sectors Required Threads: 8 Required CCX: 2 Required Cores: 6 hasher (+4 minimum for non-hashers) Enough cores available for hashers ✔ Non-hasher cores: 10 Enough cores for coordination ✔ pc1 writer: 1 pc1 reader: 2 pc1 orchestrator: 3 pc2 reader: 4 pc2 hasher: 5 pc2 hasher_cpu: 6 pc2 writer: 7 pc2 writer_cores: 3 c1 reader: 7 Unoccupied Cores: 0 { sectors = 16; coordinators = ( { core = 10; hashers = 2; }, { core = 12; hashers = 6; } ) } --------- Batch Size: 32 sectors Required Threads: 16 Required CCX: 3 Required Cores: 11 hasher (+4 minimum for non-hashers) Enough cores available for hashers ✔ Non-hasher cores: 5 Enough cores for coordination ✔ ! P2 hasher will share a core with P1 writer, performance may be impacted ! P2 hasher_cpu will share a core with P2 reader, performance may be impacted pc1 writer: 1 pc1 reader: 2 pc1 orchestrator: 3 pc2 reader: 0 pc2 hasher: 1 pc2 hasher_cpu: 0 pc2 writer: 4 pc2 writer_cores: 1 c1 reader: 0 Unoccupied Cores: 0 { sectors = 32; coordinators = ( { core = 5; hashers = 4; }, { core = 8; hashers = 6; }, { core = 12; hashers = 6; } ) } --------- Batch Size: 64 sectors Required Threads: 32 Required CCX: 6 Required Cores: 22 hasher (+4 minimum for non-hashers) Not enough cores available for hashers ✘ Batch Size: 128 sectors Required Threads: 64 Required CCX: 11 Required Cores: 43 hasher (+4 minimum for non-hashers) Not enough cores available for hashers ✘ ```
* 创建一个新的批量封装器层配置，例如：batch-machine1： ```yaml [Subsystems] EnableBatchSeal = true [Seal] LayerNVMEDevices = [ "0000:88:00.0", "0000:86:00.0", # 添加所有要使用的NVMe设备的PCIe地址 ] # 设置为您想要的批量大小（批量CPU命令所支持的CPU和您拥有的NVMe空间） BatchSealBatchSize = 32 # 管道可以是1或2；2个管道会使存储需求翻倍，但在正确平衡的系统中，使层哈希运行100%的时间，几乎使吞吐量翻倍 BatchSealPipelines = 2 # 对于Zen2或更旧的CPU设置为true以确保兼容性 SingleHasherPerThread = false ``` #### Environment Variables #### 环境变量 | 变量 | 描述 | | ---------------------- | ----------------------------------------------------------------------------------------------- | | `DISABLE_SPDK_SETUP=1` | 设置后，禁用 supraseal 初始化期间的自动 SPDK 设置（大页面配置和 NVMe 设备绑定）。适用于希望手动管理 SPDK 配置、映射驱动器或控制大页面 NUMA 分配的高级用户。 | #### Configure hugepages #### 配置大页面 这可以通过在 `/etc/default/grub` 中添加以下内容来完成。批量密封器需要36个1G的大页面。 ```bash GRUB_CMDLINE_LINUX_DEFAULT="hugepages=36 default_hugepagesz=1G hugepagesz=1G" ``` 然后运行 `sudo update-grub` 并重启机器。 或者在运行时： ```bash sudo sysctl -w vm.nr_hugepages=36 ``` 然后检查 /proc/meminfo 以验证大页面是否可用： ```bash cat /proc/meminfo | grep Huge ``` 预期输出如下： AnonHugePages: 0 kB ShmemHugePages: 0 kB FileHugePages: 0 kB HugePages\_Total: 36 HugePages\_Free: 36 HugePages\_Rsvd: 0 HugePages\_Surp: 0 Hugepagesize: 1048576 kB 检查 `HugePages_Free` 是否等于36，内核有时会将一些大页面用于其他目的。 #### Setup NVMe devices for SPDK: #### 为SPDK设置NVMe设备： {% hint style="success" %} 可以使用 Curio CLI 命令自动完成 SPDK 设置： {% endhint %} ```bash sudo curio batch setup ``` 此命令将： * 如果尚未可用，则下载 SPDK * 配置 1GB 大页面（默认 36 页） * 绑定 NVMe 设备以供批量封装使用 您可以自定义大页面数量： ```bash sudo curio batch setup --hugepages 36 --min-pages 36 ``` 或者，如果您需要手动检查 SPDK 状态或解绑设备，可以使用： ```bash cd extern/supraseal/deps/spdk-v24.05/ # 检查状态 sudo ./scripts/setup.sh status # 手动运行设置（通常不需要） sudo env NRHUGE=36 ./scripts/setup.sh ``` #### PC2 output storage #### PC2输出存储 附加临时存储空间用于PC2输出（批量密封器每个扇区需要约70GB - 32GiB用于密封扇区，36GiB用于包含TreeC/TreeR和辅助文件的缓存目录） ### Usage ### 使用方法 1. 启动带有批量密封层的Curio节点 ```bash curio run --layers batch-machine1 ``` 2. 添加一批CC扇区： ```bash curio seal start --now --cc --count 32 --actor f01234 --duration-days 365 ``` 3. 监控进度 - 你应该在[Curio GUI](/zh/curio-gui)中看到一个"Batch..."任务正在运行 4. PC1将花费3.5-5小时，之后是GPU上的PC2 5. 批处理完成后，存储将被释放用于下一批处理 ### Optimization ### 优化 * 平衡批处理大小、CPU核心和NVMe驱动器，以保持PC1持续运行 * 确保有足够的GPU容量在下一个PC1批处理完成之前完成PC2 * 监控CPU、GPU和NVMe利用率以识别瓶颈 * 监控哈希器核心利用率 ### Troubleshooting ### 故障排除 #### Node doesn't start / isn't visible in the UI #### 节点无法启动/在UI中不可见 * 确保正确配置了大页面 * 检查NVMe设备的IOPS和容量 * 如果spdk设置失败，尝试对NVMe设备执行 `wipefs -a`（这将擦除设备上的分区，请小心操作！） #### Performance issues #### 性能问题 你可以通过查看例如 `htop` 中的"hasher"核心利用率来监控性能。 要识别哈希器核心，调用 `curio calc supraseal-config --batch-size 128`（使用正确的批处理大小），并查找 `coordinators` ```go topology: ... { pc1: { writer = 1; ... hashers_per_core = 2; sector_configs: ( { sectors = 128; coordinators = ( { core = 59; hashers = 8; }, { core = 64; hashers = 14; }, { core = 72; hashers = 14; }, { core = 80; hashers = 14; }, { core = 88; hashers = 14; } ) } ) }, pc2: { ... } ``` 在此示例中，核心59、64、72、80和88是"协调器"，每个核心有两个哈希器，这意味着： * 在第一组中，核心59是协调器，核心60-63是哈希器（4个哈希器核心/8个哈希器线程） * 在第二组中，核心64是协调器，核心65-71是哈希器（7个哈希器核心/14个哈希器线程） * 以此类推 协调器核心通常会保持100%的利用率，哈希器线程**应该**保持100%的利用率，任何低于这个水平的情况都表明系统存在瓶颈，比如NVMe IOPS不足、内存带宽不足或NUMA设置不正确。 #### Performance issues #### 性能问题 ### Troubleshooting ### 故障排除 要进行故障排除： * 仔细阅读本页顶部的要求 * [对NVME IOPS进行基准测试](#benchmark-nvme-iops) * 如果PC2速度慢，验证GPU设置 * 检查批处理过程中的日志是否有任何错误 # 扩展Curio集群 | Scaling Curio cluster 本页描述如何向Curio集群添加额外的节点或矿工ID ## Scaling Curio cluster ## 扩展Curio集群 ### Migrating additional lotus-miner to Curio ### 将额外的lotus-miner迁移到Curio 要将第二个或之后的`lotus-miner`迁移到现有的Curio集群，您需要遵循与之前相同的步骤。唯一的例外是不需要安装新的YugabyteDB集群。在`lotus-miner`节点上安装`curio`二进制文件后，您可以运行`curio guided-setup`来开始迁移。 ### Initialising additional Miner IDs in existing Curio cluster ### 在现有Curio集群中初始化额外的矿工ID 在网络上启动新的矿工ID的过程与[使用新矿工ID初始化新的Curio集群](https://docs.curiostorage.org/zh/pages/vAxhqPrCSUAQa2P8Ind7#initiating-a-new-curio-cluster|使用新矿工ID初始化新的Curio集群)时相同。唯一的例外是不需要安装新的YugabyteDB集群。 ### Migrating lotus-worker to Curio cluster ### 将lotus-worker迁移到Curio集群 一旦您将矿工ID迁移到Curio集群，您需要将所有附加到已迁移矿工ID的`lotus-worker`节点重新用作Curio节点。 1. 在`lotus-worker`节点上[安装](https://github.com/filecoin-project/curio/blob/main/documentation/zh/installation.md%7C安装)`curio`二进制文件。 2. 使用正确的详细信息[配置服务ENV文件](https://docs.curiostorage.org/zh/pages/YvOCDAhtOGyemueNzzkm#environment-variables-configuration|配置服务ENV文件)。 3. 启动新的`curio`节点，并在GUI中验证新节点现在是集群的一部分。 4. [将现有存储附加](https://docs.curiostorage.org/zh/pages/krHYEH4j3tHrKLMjQy1h#attach-existing-storage-to-curio|将现有存储附加到Curio)到Curio节点。 5. 对其余的worker重复此过程。 ### Adding nodes to Curio cluster ### 向Curio集群添加节点 要向现有的Curio集群添加新节点，请按照以下流程操作。 * [安装](https://github.com/filecoin-project/curio/blob/main/documentation/zh/installation.md%7C安装)`curio`二进制文件。 * 使用正确的详细信息[配置服务ENV文件](https://docs.curiostorage.org/zh/pages/YvOCDAhtOGyemueNzzkm#environment-variables-configuration|配置服务ENV文件)。 * 启动新的`curio`节点，并在GUI中验证新节点现在是集群的一部分。 * 如果需要，[附加任何新的或现有的存储](https://github.com/filecoin-project/curio/blob/main/documentation/zh/storage-configuration.md%7C附加存储)。 * 对任何需要附加的额外节点重复此过程。 # Curio图形用户界面 | Curio GUI 本页面描述了如何访问 Curio GUI 以及可以在其中获取哪些信息。 ## Curio GUI ## Curio 图形用户界面 ### Accessing Curio GUI ### 访问 Curio 图形用户界面 默认端口是 `4701` 以访问 Curio 图形用户界面。要在 Curio 节点上启用图形用户界面，用户必须启动带有 `gui` 层的 Curio 节点。这是一个随 Curio 二进制文件一起提供的预构建层。 建议使用网页翻译扩展程序，因为网页界面是英文的。 如果这不合理，请在 Curio 的 GitHub 仓库中提交问题。 #### Changing default GUI port #### 更改默认图形用户界面端口 您可以通过在配置的“base”层中设置不同的 IP 地址和端口来更改默认的图形用户界面端口。我们强烈建议不要在其他层中指定图形用户界面地址，以避免混淆。 ```bash curio config edit base ``` 这将在您的默认文本编辑器中打开“base”层。 应更改为以下内容 应该监听 Web 图形用户界面请求的地址。 ```yaml GuiAddress = "127.0.0.1:4702" ``` 保存配置并重新启动运行图形用户界面层的 Curio 服务，以访问新地址和端口上的图形用户界面。 ### 图形用户界面菜单和仪表板 {% hint style="danger" %} Curio Web 用户界面目前正在开发中。某些用户界面页面可能会随着时间的推移而更改，并且可能与下面的截图和描述不同。 {% endhint %} #### Home Page 主页

Curio 主页

链连接性：所有可用的 Lotus 守护进程节点的链同步状态 集群机器：集群中所有 Curio 节点的快速列表 PoRep 管道：集群范围内封装扇区的快速摘要 参与者摘要：此 Curio 集群服务的 minerID 摘要 #### Configuration page #### 配置页面 所有配置层都可以在图形用户界面的 `configuration` 页面中找到并编辑。它还允许通过图形用户界面添加新的配置层。

配置页面

编辑配置层

编辑配置层

#### Sectors #### 扇区 Curio 图形用户界面可用于浏览 Curio 集群服务的所有 minerID 的所有扇区列表。 这是 `lotus-miner` 中 `lotus-miner sectors list` 的图形用户界面替代品。

Curio 扇区列表

#### PoRep 管道 此页面可用于浏览当前由 Curio 集群封装和历史封装的扇区。它详细说明了扇区在各个封装阶段的移动情况以及每个阶段的状态。

PoRep 管道

用户可以点击“DETAILS”并获取有关扇区的更多详细信息。此页面将告诉您有关碎片、存储等信息。

扇区详情

#### Node details #### 节点详情 在主页的“Cluster Machines”列表中，用户可以点击机器名称以获取每个节点的更详细视图。它将列出附加存储和该特定机器处理的任务状态。

节点详情

# 垃圾回收 | Garbage Collection Curio 中的垃圾回收和清理過程 ## Garbage Collection ## 垃圾回收 ### Sealing Pipeline cleanup ### 密封管道清理 **SDRPipelineGC** 是系統中的一個定期任務，確保存儲密封過程的效率和有效性。它負責清理密封管道中已完成的條目。 #### Process #### 過程 GC 通過移除已完成密封過程的密封管道條目來運作。這些條目的元數據已存儲在長期扇區元數據表（也稱為 `sectors_meta` 表）中。此操作有助於保持管道流暢和整潔，提高整體系統性能。 #### Handling failed sector #### 處理失敗的扇區 如果一個扇區在密封過程中失敗，其相應的條目可以通過網頁用戶界面（WebUI）手動移除。此功能允許主動管理管道條目，確保失敗的條目不會阻塞管道。 ### Storage Cleanup ### 存儲清理 `StorageGCMark` 組件負責掃描系統中的所有扇區文件。 在以下條件下，扇區文件將在 `storage_removal_marks` 表中被標記： * 該扇區在 `storage_gc_pins` 表中沒有被"固定"。（扇區固定表示即使扇區已過期，也不應被移除。） * 該扇區不存在於名為 `sectors_sdr_pipeline` 的密封表中。 * 該扇區被標記為"失敗"扇區。（注意，"失敗"的扇區必須首先從管道表中移除，然後才能對其數據進行垃圾回收。） * 該扇區不存在於礦工參與者預提交扇區集中。 * 該扇區不存在於 `Live` 或 `Unproven` 扇區集中。 #### Approval and Removal #### 批准和移除 來自 `StorageGCMark` 過程的移除標記需要單獨批准。目前，此批准僅通過 WebUI 提供。未來可能會擴展以允許自動批准，由自定義的選擇策略支持。

存儲 GC 批准

一旦移除標記獲得批准，定期的 `StorageGCSweep` 任務將審查所有已批准的移除標記。然後，此任務將繼續刪除已批准移除的文件。這最後階段確保系統中只保留必要的數據，優化存儲並改善整體系統功能。 #### Removing a failed sector #### 移除失敗的扇區 要移除在密封過程中失敗的扇區，用戶應前往 WebUI 的"PoRep"頁面，選擇相應扇區的"DETAILS"鏈接。此操作將引導他們到一個有"Remove"按鈕的頁面。點擊此按鈕後，失敗的扇區將從 SDR 管道表中移除，使其可供 StorageGCMark 過程標記為垃圾回收。 然而，扇區的移除只會在獲得必要的批准後進行。此批准可以在"Storage GC Info"頁面上提供。在收到批准後，StorageGCSweep 任務將審查標記並繼續刪除扇區文件，有效地從系統中移除失敗的扇區。

如何 GC 失敗的扇區

# 最佳实践 | Best Practices Curio 最佳实践 ## Best Practices ## 最佳实践 1. 支持 Curio 集群的 YugybteDB 应该是多节点的，以避免单点故障。 2. 所有矿工 ID 都应该是基础层的一部分。我们强烈建议不要为不同的矿工 ID 创建单独的层，但如果需要，可以为控制地址使用不同的层。 3. 不应将任何工作者专用于特定的矿工 ID。所有 Curio 节点都应设置为允许任何矿工 ID 的作业。 4. 应该使用 `--post` 层启动多个工作者，以允许快速的 wdPost 和 winPost 周转时间。 5. 我们建议运行一个启用 GUI 层的节点。可以通过这个节点访问集群范围的 GUI，而不会对数据库的读取操作造成额外的压力。 6. 未密封和已密封的副本不应存储在同一存储位置。如果其中一个丢失，Curio 将允许在未来自动重新生成已密封和未密封的副本。 7. 建议为每个市场适配器创建一个独特的层，对应每个矿工 ID。此配置允许精确控制，允许将特定的矿工 ID 分配给 Snap Deals 管道或 PoRep 管道。 8. 建议在将执行 PoRep 管道的 TreeD 任务或 Snap Deals 管道的 Encode 任务的同一节点上运行市场适配器。 # 日志记录 | Logging 本指南描述如何在Curio中更新日志记录首选项。 ## Logging ## 日志记录 ### Log file configuration ### 日志文件配置 每个Curio节点生成Go日志，如果您以systemd服务的方式运行Curio，默认情况下这些日志会被定向到`/var/log/curio/curio.log`文件。 #### Redirect Go logs to a file #### 将Go日志重定向到文件 默认情况下，如果不作为systemd服务运行，Curio会将所有日志重定向到标准输出。要更改此行为，请将以下变量添加到`.bashrc`文件中，并重启`curio`进程，以开始将所有日志重定向到文件。 ```bash export GOLOG_OUTPUT=FILE >> ~/.bashrc export GOLOG_FILE="$HOME/curio.log" >> ~/.bashrc && source ~/.bashrc ``` #### Redirect Rust logs to a standard output #### 将Rust日志重定向到标准输出 默认情况下，`rust-fil-proof`使用的`fil_logger`库不会记录任何内容。您可以通过将RUST\_LOG环境变量设置为另一个级别来更改此设置。这将在stderr上显示日志输出，可以通过systemd或在手动启动`curio`进程时在shell中将其重定向到文件。 使用systemd服务文件： export RUST\_LOG=info >> /etc/curio.env systemctl restart curio.service 手动运行Curio： export RUST\_LOG=info >> \~/.bashrc && source \~/.bashrc 日志级别可以在5个选项之间选择： * trace * debug * info * warn * error #### Change logging verbosity #### 更改日志记录详细程度 可以在不重启服务或进程的情况下更改`curio`日志的详细程度。可以使用以下命令列出`curio`进程中的不同子系统，并更改单个子系统的详细程度，以获得更多/更少的详细日志。 curio cli --machine log list 要更改详细程度，请运行： curio cli --machine log set-level --system chain debug 日志级别可以在4个选项之间选择： * debug * info * warn * error 您可以指定多个子系统，以一次更改多个子系统的日志级别。 curio cli --machine log set-level --system chain --system chainxchg debug # Curio命令行界面 | Curio CLI Curio命令行界面 ## Curio CLI ## Curio 命令行界面 Curio默认附带两个二进制文件，分别称为`curio`和`sptool`。 ### Curio Binary ### Curio 二进制文件 Curio的命令行界面（CLI）的运行方式与典型软件略有不同。一些命令，如与存储相关的命令，在后端进行API调用，而其他命令则直接与数据库交互以执行所需操作。 需要进行API调用的命令需要认证，因为Curio API是经过权限控制的。这种认证涉及在API调用中传递令牌和地址。Curio CLI无缝处理这种认证，无需为令牌和地址设置环境变量。它从数据库中检索认证和执行者详细信息，并相应地生成API调用。 这种设计允许您在不直接访问远程节点的情况下对其进行更改。例如，您可以从节点1中分离节点2的存储。所有这些命令都嵌套在`cli`子命令下。 要在远程机器上执行操作，您必须使用`--machine`标志以`--machine=10.0.0.1:12300`的格式提供正确的IP地址和端口。 本文档解释了Curio CLI的独特方面，包括其认证过程以及如何与远程节点交互。 `curio` CLI参考可以在[这里](/zh/curio-cli/curio)找到。 ### Sptool Binary ### Sptool 二进制文件 某些管理和监控操作需要更新或从链上获取有关minerID的信息。这些操作不需要访问数据库，因此不包含在Curio二进制文件中。相反，这些命令托管在`sptool`二进制文件下。`sptool`二进制文件为存储提供商所需的Filecoin区块链操作提供了一个接口。 `sptool` CLI参考可以在[这里](/zh/curio-cli/sptool)找到。 # Curio NAME: curio - Filecoin 去中心化存储网络提供商 USAGE: curio \[全局选项] 命令 \[命令选项] \[参数...] VERSION: 1.23.0 COMMANDS: cli 执行 CLI 命令 run 启动 Curio 进程 config 按层管理节点配置。'base' 层将始终在 Curio 启动时应用。 test 测试的实用功能 web 启动 Curio 网页界面 guided-setup 运行引导式设置，用于从 lotus-miner 迁移到 Curio 或创建新的 Curio 矿工 seal 管理封装流程 market\ fetch-params 获取证明参数 calc 数学工具 help, h 显示命令列表或某个命令的帮助 GLOBAL OPTIONS: --color 在显示输出中使用颜色（默认：取决于输出是否为 TTY） --db-host value Yugabyte 集群的主机名列表，用逗号分隔（默认："127.0.0.1"）\[$CURIO\_DB\_HOST, $CURIO\_HARMONYDB\_HOSTS] --db-name value （默认："yugabyte"）\[$CURIO\_DB\_NAME, $CURIO\_HARMONYDB\_NAME] --db-user value （默认："yugabyte"）\[$CURIO\_DB\_USER, $CURIO\_HARMONYDB\_USERNAME] --db-password value （默认："yugabyte"）\[$CURIO\_DB\_PASSWORD, $CURIO\_HARMONYDB\_PASSWORD] --db-port value （默认："5433"）\[$CURIO\_DB\_PORT, $CURIO\_HARMONYDB\_PORT] --repo-path value （默认："\~/.curio"）\[$CURIO\_REPO\_PATH] --vv 启用非常详细的模式，用于调试 CLI（默认：false） --help, -h 显示帮助 --version, -v 打印版本 ## curio cli NAME: curio cli - 执行 CLI 命令 USAGE: curio cli 命令 \[命令选项] \[参数...] COMMANDS: storage 管理扇区存储 log 管理日志 wait-api 等待 Curio API 上线 stop 停止正在运行的 Curio 进程 help, h 显示命令列表或某个命令的帮助 OPTIONS: --machine value 机器主机:端口（curio run --listen 地址） --help, -h 显示帮助 ### curio cli storage NAME: curio cli storage - 管理扇区存储 USAGE: curio cli storage 命令 \[命令选项] \[参数...] DESCRIPTION: 扇区可以存储在多个文件系统路径中。这些 命令提供了管理 Curio 节点用于长期存储扇区以进行证明的存储（称为 'store'） 以及扇区在封装流程中如何存储（称为 'seal'）的方法。 COMMANDS: attach 附加本地存储路径 detach 分离本地存储路径 list 列出本地存储路径 find 在存储系统中查找扇区 help, h 显示命令列表或某个命令的帮助 OPTIONS: --help, -h 显示帮助 #### curio cli storage attach NAME: curio cli storage attach - 附加本地存储路径 USAGE: curio cli storage attach \[命令选项] \[路径] DESCRIPTION: 可以使用此命令将存储附加到 Curio 节点。存储卷 列表存储在 curio run 中设置的 storage.json 中，位于 Curio 节点本地。我们不 建议在不进一步了解存储系统的情况下手动修改此值。 每个存储卷都包含一个描述卷 功能的配置文件。当提供 '--init' 标志时，将使用 附加标志创建此文件。 权重 较高的权重值意味着数据更有可能存储在此路径中 封装 封装过程的数据将存储在这里 存储 最终确定的扇区将被移动到这里进行长期存储，并随时间 进行证明 OPTIONS: --init 首先初始化路径（默认：false） --weight value （用于初始化）路径权重（默认：10） --seal （用于初始化）将路径用于封装（默认：false） --store （用于初始化）将路径用于长期存储（默认：false） --max-storage value （用于初始化）限制扇区的存储空间（对于非常大的路径来说代价很高！） --groups value \[ --groups value ] 路径组名称 --allow-to value \[ --allow-to value ] 允许从此路径拉取数据的路径组（如果未指定则允许所有） --help, -h 显示帮助 #### curio cli storage detach NAME: curio cli storage detach - 分离本地存储路径 USAGE: curio cli storage detach \[命令选项] \[路径] OPTIONS: --really-do-it （默认：false） --help, -h 显示帮助 #### curio cli storage list NAME: curio cli storage list - 列出本地存储路径 USAGE: curio cli storage list \[命令选项] \[参数...] OPTIONS: --local 仅列出本地存储路径（默认：false） --help, -h 显示帮助 #### curio cli storage find NAME: curio cli storage find - 在存储系统中查找扇区 USAGE: curio cli storage find \[命令选项] \[矿工地址] \[扇区编号] OPTIONS: --help, -h 显示帮助 ### curio cli log NAME: curio cli log - 管理日志 USAGE: curio cli log 命令 \[命令选项] \[参数...] COMMANDS: list 列出日志系统 set-level 设置日志级别 help, h 显示命令列表或某个命令的帮助 OPTIONS: --help, -h 显示帮助 #### curio cli log list NAME: curio cli log list - 列出日志系统 USAGE: curio cli log list \[命令选项] \[参数...] OPTIONS: --help, -h 显示帮助 #### curio cli log set-level NAME: curio cli log set-level - 设置日志级别 USAGE: curio cli log set-level \[命令选项] \[级别] DESCRIPTION: 为日志系统设置日志级别： ``` 系统标志可以多次指定。 例如）log set-level --system chain --system chainxchg debug 可用级别： debug info warn error 环境变量： GOLOG_LOG_LEVEL - 所有日志系统的默认日志级别 GOLOG_LOG_FMT - 更改输出日志格式（json，nocolor） GOLOG_FILE - 将日志写入文件 GOLOG_OUTPUT - 指定是否输出到文件、stderr、stdout 或组合，例如 file+stderr ``` OPTIONS: --system value \[ --system value ] 限制到日志系统 --help, -h 显示帮助 ### curio cli wait-api NAME: curio cli wait-api - 等待 Curio API 上线 USAGE: curio cli wait-api \[command options] \[arguments...] OPTIONS: --timeout value 等待失败的持续时间（默认：30s） --help, -h 显示帮助 ### curio cli stop NAME: curio cli stop - 停止正在运行的 Curio 进程 USAGE: curio cli stop \[command options] \[arguments...] OPTIONS: --help, -h 显示帮助 ## curio run NAME: curio run - 启动 Curio 进程 USAGE: curio run \[command options] \[arguments...] OPTIONS: --listen value 工作者 API 将监听的主机地址和端口（默认："0.0.0.0:12300"）\[$CURIO\_LISTEN] --nosync 不检查全节点同步状态（默认：false） --manage-fdlimit 管理打开文件限制（默认：true） --layers value, -l value, --layer value \[ --layers value, -l value, --layer value ] 要解释的层列表（在默认值之上）。默认：base \[$CURIO\_LAYERS] --name value 自定义节点名称 \[$CURIO\_NODE\_NAME] --help, -h 显示帮助 ## curio config NAME: curio config - 通过层管理节点配置。'base' 层将始终在 Curio 启动时应用。 USAGE: curio config command \[command options] \[arguments...] COMMANDS: default, defaults 打印默认节点配置 set, add, update, create 通过提供文件名或标准输入来设置配置层或基础层。 get, cat, show 按名称获取配置层。您可能想将输出管道到文件，或使用 'less' list, ls 列出数据库中存在的配置层。 interpret, view, stacked, stack 通过此版本的 curio 解释堆叠的配置层，并带有系统生成的注释。 remove, rm, del, delete 删除指定的配置层。 edit 编辑配置层 new-cluster 为新集群创建新配置 help, h 显示命令列表或某个命令的帮助 OPTIONS: --help, -h 显示帮助 ### curio config default NAME: curio config default - 打印默认节点配置 USAGE: curio config default \[command options] \[arguments...] OPTIONS: --no-comment 不注释默认值（默认：false） --help, -h 显示帮助 ### curio config set NAME: curio config set - 通过提供文件名或标准输入来设置配置层或基础层。 USAGE: curio config set \[command options] 层的文件名 OPTIONS: --title value 配置层的标题（对于标准输入是必需的） --help, -h 显示帮助 ### curio config get NAME: curio config get - 按名称获取配置层。您可能想将输出管道到文件，或使用 'less' USAGE: curio config get \[command options] 层名称 OPTIONS: --help, -h 显示帮助 ### curio config list NAME: curio config list - 列出数据库中存在的配置层。 USAGE: curio config list \[command options] \[arguments...] OPTIONS: --help, -h 显示帮助 ### curio config interpret NAME: curio config interpret - 通过此版本的 curio 解释堆叠的配置层，并带有系统生成的注释。 USAGE: curio config interpret \[command options] 要解释为最终配置的层列表 OPTIONS: --layers value \[ --layers value ] 要解释的层的逗号或空格分隔列表（base 总是应用） --help, -h 显示帮助 ### curio config remove NAME: curio config remove - 删除指定的配置层。 USAGE: curio config remove \[command options] \[arguments...] OPTIONS: --help, -h 显示帮助 ### curio config edit NAME: curio config edit - 编辑配置层 USAGE: curio config edit \[command options] \[层名称] OPTIONS: --editor value 要使用的编辑器（默认："vim"）\[$EDITOR] --source value 源配置层（默认：<编辑的层>） --allow-overwrite 如果源是不同的层，允许覆盖现有层（默认：false） --no-source-diff 将整个配置保存到层中，而不仅仅是差异（默认：false） --no-interpret-source 不解释源层（如果设置了 --source，默认为 true） --help, -h 显示帮助 ### curio config new-cluster NAME: curio config new-cluster - 为新集群创建新配置 USAGE: curio config new-cluster \[command options] \[SP actor 地址...] OPTIONS: --help, -h 显示帮助 ## curio test NAME: curio test - 用于测试的实用功能 USAGE: curio test command \[command options] \[arguments...] COMMANDS: window-post, wd, windowpost, wdpost 为扇区计算时空证明（需要预先密封扇区）。这些不会发送到链上。 help, h 显示命令列表或某个命令的帮助 OPTIONS: --help, -h 显示帮助 ### curio test window-post ```yaml NAME: curio test window-post - Compute a proof-of-spacetime for a sector (requires the sector to be pre-sealed). These will not send to the chain. USAGE: curio test window-post command [command options] [arguments...] COMMANDS: here, cli Compute WindowPoSt for performance and configuration testing. task, scheduled, schedule, async, asynchronous Test the windowpost scheduler by running it on the next available curio. If tasks fail all retries, you will need to ctrl+c to exit. help, h Shows a list of commands or help for one command OPTIONS: --help, -h show help ``` #### curio test window-post here ```yaml NAME: curio test window-post here - Compute WindowPoSt for performance and configuration testing. USAGE: curio test window-post here [command options] [deadline index] DESCRIPTION: Note: This command is intended to be used to verify PoSt compute performance. It will not send any messages to the chain. Since it can compute any deadline, output may be incorrectly timed for the chain. OPTIONS: --deadline value deadline to compute WindowPoSt for (default: 0) --layers value [ --layers value ] list of layers to be interpreted (atop defaults). Default: base --storage-json value path to json file containing storage config (default: "~/.curio/storage.json") --partition value partition to compute WindowPoSt for (default: 0) --help, -h show help ``` #### curio test window-post task ```yaml NAME: curio test window-post task - Test the windowpost scheduler by running it on the next available curio. If tasks fail all retries, you will need to ctrl+c to exit. USAGE: curio test window-post task [command options] [arguments...] OPTIONS: --deadline value deadline to compute WindowPoSt for (default: 0) --layers value [ --layers value ] list of layers to be interpreted (atop defaults). Default: base --help, -h show help ``` ## curio web ```yaml NAME: curio web - Start Curio web interface USAGE: curio web [command options] [arguments...] DESCRIPTION: Start an instance of Curio web interface. This creates the 'web' layer if it does not exist, then calls run with that layer. OPTIONS: --gui-listen value Address to listen for the GUI on (default: "0.0.0.0:4701") --nosync don't check full-node sync status (default: false) --layers value [ --layers value ] list of layers to be interpreted (atop defaults). Default: base --help, -h show help ``` ## curio guided-setup ```yaml NAME: curio guided-setup - Run the guided setup for migrating from lotus-miner to Curio or Creating a new Curio miner USAGE: curio guided-setup [command options] [arguments...] OPTIONS: --help, -h show help ``` ## curio seal ```yaml NAME: curio seal - Manage the sealing pipeline USAGE: curio seal command [command options] [arguments...] COMMANDS: start Start new sealing operations manually help, h Shows a list of commands or help for one command OPTIONS: --help, -h show help ``` ### curio seal start ```yaml NAME: curio seal start - Start new sealing operations manually USAGE: curio seal start [command options] [arguments...] OPTIONS: --actor value Specify actor address to start sealing sectors for --now Start sealing sectors for all actors now (not on schedule) (default: false) --cc Start sealing new CC sectors (default: false) --count value Number of sectors to start (default: 1) --synthetic Use synthetic PoRep (default: false) --layers value [ --layers value ] list of layers to be interpreted (atop defaults). Default: base --duration-days value, -d value How long to commit sectors for (default: 1278 (3.5 years)) --help, -h show help ``` ## curio market ```yaml NAME: curio market USAGE: curio market command [command options] [arguments...] COMMANDS: rpc-info seal start sealing a deal sector early help, h Shows a list of commands or help for one command OPTIONS: --help, -h show help ``` ### curio market rpc-info ```yaml NAME: curio market rpc-info USAGE: curio market rpc-info [command options] [arguments...] OPTIONS: --layers value [ --layers value ] list of layers to be interpreted (atop defaults). Default: base --help, -h show help ``` ### curio market seal ```yaml NAME: curio market seal - start sealing a deal sector early USAGE: curio market seal [command options] OPTIONS: --actor value Specify actor address to start sealing sectors for --synthetic Use synthetic PoRep (default: false) --help, -h show help ``` ## curio fetch-params ```yaml NAME: curio fetch-params - Fetch proving parameters USAGE: curio fetch-params [command options] [sectorSize] OPTIONS: --help, -h show help ``` ## curio calc ```yaml NAME: curio calc - Math Utils USAGE: curio calc command [command options] [arguments...] COMMANDS: batch-cpu Analyze and display the layout of batch sealer threads supraseal-config Generate a supra_seal configuration help, h Shows a list of commands or help for one command OPTIONS: --actor value --help, -h show help ``` ### curio calc batch-cpu ```yaml NAME: curio calc batch-cpu - Analyze and display the layout of batch sealer threads USAGE: curio calc batch-cpu [command options] [arguments...] DESCRIPTION: Analyze and display the layout of batch sealer threads on your CPU. It provides detailed information about CPU utilization for batch sealing operations, including core allocation, thread distribution for different batch sizes. OPTIONS: --dual-hashers (default: true) --help, -h show help ``` ### curio calc supraseal-config ```yaml NAME: curio calc supraseal-config - Generate a supra_seal configuration USAGE: curio calc supraseal-config [command options] [arguments...] DESCRIPTION: Generate a supra_seal configuration for a given batch size. This command outputs a configuration expected by SupraSeal. Main purpose of this command is for debugging and testing. The config can be used directly with SupraSeal binaries to test it without involving Curio. OPTIONS: --dual-hashers Zen3 and later supports two sectors per thread, set to false for older CPUs (default: true) --batch-size value, -b value (default: 0) --help, -h show help ``` # Sptool 名称: sptool - 管理 Filecoin 矿工参与者 用法: sptool \[全局选项] 命令 \[命令选项] \[参数...] 版本: 1.23.0 命令: actor 管理 Filecoin 矿工参与者元数据 info 打印矿工参与者信息 sectors 与扇区存储交互 proving 查看证明信息 help, h 显示命令列表或某个命令的帮助 全局选项: --log-level value (默认: "info") --actor value 要管理的矿工参与者 \[$SP\_ADDRESS] --help, -h 显示帮助 --version, -v 打印版本 ## sptool actor 参与者 名称: sptool actor - 管理Filecoin矿工Actor元数据 用法: sptool actor 命令 \[命令选项] \[参数...] 命令: set-addresses, set-addrs 设置您的矿工可以公开拨号的地址 withdraw 将可用余额提取到受益人 repay-debt 偿还矿工的债务 set-peer-id 设置您矿工的对等ID set-owner 设置所有者地址（此命令应该被调用两次，首先使用旧所有者作为发送者地址，然后使用新所有者） control 管理控制地址 propose-change-worker 提议更改工作者地址 confirm-change-worker 确认工作者地址更改 compact-allocated 压缩已分配的扇区位域 propose-change-beneficiary 提议更改受益人地址 confirm-change-beneficiary 确认受益人地址更改 new-miner 初始化新的矿工actor help, h 显示命令列表或某个命令的帮助 选项: --help, -h 显示帮助 ### sptool actor set-addresses 设置地址 名称: sptool actor set-addresses - 设置您的矿工可以公开拨号的地址 用法: sptool actor set-addresses \[命令选项] <多地址> 选项: --from value 可选择指定发送消息的账户 --gas-limit value 设置燃气限制 (默认: 0) --unset 取消设置地址 (默认: false) --help, -h 显示帮助 ### sptool actor withdraw 提现 名称: sptool actor withdraw - 将可用余额提取到受益人 用法: sptool actor withdraw \[命令选项] \[金额 (FIL)] 选项: --confidence value 等待的区块确认数 (默认: 5) --beneficiary 从受益人地址发送提现消息 (默认: false) --help, -h 显示帮助 ### sptool actor repay-debt 偿还债务 名称: sptool actor repay-debt - 偿还矿工的债务 用法: sptool actor repay-debt \[命令选项] \[金额 (FIL)] 选项: --from value 可选择指定发送资金的账户 --help, -h 显示帮助 ### sptool actor set-peer-id 设置对等ID 名称: sptool actor set-peer-id - 设置您矿工的对等ID 用法: sptool actor set-peer-id \[命令选项] <对等ID> 选项: --gas-limit value 设置燃气限制 (默认: 0) --help, -h 显示帮助 ### sptool actor set-owner 设置所有者 名称: sptool actor set-owner - 设置所有者地址（此命令应该被调用两次，首先使用旧所有者作为发送者地址，然后使用新所有者） 用法: sptool actor set-owner \[命令选项] \[新所有者地址 发送者地址] 选项: --really-do-it 实际发送执行操作的交易 (默认: false) --help, -h 显示帮助 ### sptool actor control 管理控制 名称: sptool actor control - 管理控制地址 用法: sptool actor control 命令 \[命令选项] \[参数...] 命令: list 获取当前设置的控制地址。注意：这不包括大多数角色，因为它们在即时链状态中是未知的。 set 设置控制地址 help, h 显示命令列表或某个命令的帮助 选项: --help, -h 显示帮助 #### sptool actor control list 获取控制地址 名称: sptool actor control list - 获取当前设置的控制地址。注意：这不包括大多数角色，因为它们在即时链状态中是未知的。 用法: sptool actor control list \[命令选项] \[参数...] 选项: --verbose (默认: false) --help, -h 显示帮助 #### sptool actor control set 设置控制地址 名称: sptool actor control set - 设置控制地址 用法: sptool actor control set \[命令选项] \[...地址] 选项: --really-do-it 实际发送执行操作的交易 (默认: false) --help, -h 显示帮助 ### sptool actor propose-change-worker 提议更改工作者 名称: sptool actor propose-change-worker - 提议更改工作者地址 用法: sptool actor propose-change-worker \[命令选项] \[地址] 选项: --really-do-it 实际发送执行操作的交易 (默认: false) --help, -h 显示帮助 ### sptool actor confirm-change-worker 确认更改工作者 名称: sptool actor confirm-change-worker - 确认工作者地址更改 用法: sptool actor confirm-change-worker \[命令选项] \[地址] 选项: --really-do-it 实际发送执行操作的交易 (默认: false) --help, -h 显示帮助 ### sptool actor compact-allocated 压缩已分配 名称: sptool actor compact-allocated - 压缩已分配的扇区位域 用法: sptool actor compact-allocated \[命令选项] \[参数...] 选项: --mask-last-offset value 从0到'最高分配 - 偏移量'掩蔽扇区ID (默认: 0) --mask-upto-n value 从0到'n'掩蔽扇区ID (默认: 0) --really-do-it 实际发送执行操作的交易 (默认: false) --help, -h 显示帮助 ### sptool actor propose-change-beneficiary 提议更改受益人 名称: sptool actor propose-change-beneficiary - 提议更改受益人地址 用法: sptool actor propose-change-beneficiary \[命令选项] \[受益人地址 配额 过期时间] 选项: --really-do-it 实际发送执行操作的交易 (默认: false) --overwrite-pending-change 覆盖当前的受益人更改提议 (默认: false) --actor value 指定矿工角色的地址 --help, -h 显示帮助 ### sptool actor confirm-change-beneficiary 确认更改受益人 名称: sptool actor confirm-change-beneficiary - 确认受益人地址变更 用法: sptool actor confirm-change-beneficiary \[命令选项] \[矿工ID] 选项: --really-do-it 实际发送执行操作的交易 (默认: false) --existing-beneficiary 从现有受益人地址发送确认 (默认: false) --new-beneficiary 从新受益人地址发送确认 (默认: false) --help, -h 显示帮助 ### sptool actor new-miner 初始化新的矿工 名称: sptool actor new-miner - 初始化新的矿工actor 用法: sptool actor new-miner \[命令选项] \[参数...] 选项: --worker value, -w value 用于新矿工初始化的worker密钥 --owner value, -o value 用于新矿工初始化的owner密钥 --from value, -f value 发送actor(矿工)创建消息的地址 --sector-size value 指定用于新矿工初始化的扇区大小 --confidence value 等待的区块确认数 (默认: 5) --help, -h 显示帮助 ## sptool info 打印矿工信息 名称: sptool info - 打印矿工actor信息 用法: sptool info \[命令选项] \[参数...] 选项: --help, -h 显示帮助 ## sptool sectors 与扇区存储交互 名称: sptool sectors - 与扇区存储交互 用法: sptool sectors 命令 \[命令选项] \[参数...] 命令: status 通过扇区编号获取扇区的密封状态 list 列出扇区 precommits 打印链上预提交信息 check-expire 检查即将到期的扇区 expired 获取或清理已过期的扇区 extend 延长即将到期的扇区，但不超过每个扇区的最大生命周期 terminate 强制终止扇区（警告：这意味着失去算力并为终止的扇区支付一次性终止罚金（包括抵押品）） compact-partitions 从分区中移除死亡扇区，并尽可能减少使用的分区数量 help, h 显示命令列表或某个命令的帮助 选项: --help, -h 显示帮助 ### sptool sectors status 获取扇区状态 名称: sptool sectors status - 通过扇区编号获取扇区的密封状态 用法: sptool sectors status \[命令选项] <扇区编号> 选项: --log, -l 显示事件日志 (默认: false) --on-chain-info, -c 显示扇区的链上信息 (默认: false) --partition-info, -p 显示分区相关信息 (默认: false) --proof 以十六进制打印snark证明字节 (默认: false) --help, -h 显示帮助 ### sptool sectors list 列出扇区 名称: sptool sectors list - 列出扇区 用法: sptool sectors list \[命令选项] \[参数...] 选项: --help, -h 显示帮助 ### sptool sectors precommits 打印预提交信息 名称: sptool sectors precommits - 打印链上预提交信息 用法: sptool sectors precommits \[命令选项] \[参数...] 选项: --help, -h 显示帮助 ### sptool sectors check-expire 检查扇区到期 名称: sptool sectors check-expire - 检查即将到期的扇区 用法: sptool sectors check-expire \[命令选项] \[参数...] 选项: --cutoff value 跳过当前到期时间距离现在超过个纪元的扇区，默认为60天 (默认: 172800) --help, -h 显示帮助 ### sptool sectors expired 获取或清理已过期扇区 名称: sptool sectors expired - 获取或清理已过期的扇区 用法: sptool sectors expired \[命令选项] \[参数...] 选项: --expired-epoch value 检查扇区到期的纪元 (默认: WinningPoSt回溯纪元) --help, -h 显示帮助 ### sptool sectors extend 延长扇区 名称: sptool sectors extend - 延长即将到期的扇区，但不超过每个扇区的最大生命周期 用法: sptool sectors extend \[命令选项] <扇区编号...(可选)> 选项: --from value 仅考虑当前到期纪元在\[from, to]范围内的扇区，默认为：现在 + 120 (1小时) (默认: 0) --to value 仅考虑当前到期纪元在\[from, to]范围内的扇区，默认为：现在 + 92160 (32天) (默认: 0) --sector-file value 提供一个文件，每行包含一个扇区编号，忽略上述选择标准 --exclude value 可选提供一个包含要排除的扇区的文件 --extension value 尝试将选定的扇区延长这个纪元数，默认为540天 (默认: 1555200) --new-expiration value 尝试将选定的扇区延长到这个纪元，忽略extension (默认: 0) --only-cc 仅延长CC扇区（对于准备扇区进行快照升级很有用） (默认: false) --drop-claims 为可以延长但只能通过放弃一些验证算力声明的扇区放弃声明 (默认: false) --tolerance value 不尝试延长少于这个纪元数的扇区，默认为7天 (默认: 20160) --max-fee value 为一条消息最多使用这么多FIL。传递此标志以避免消息拥堵。 (默认: "0") --max-sectors value 每条消息包含的最大扇区数 (默认: 0) --really-do-it 传递此标志以真正延长扇区，否则只会打印参数的json表示 (默认: false) --help, -h 显示帮助 ### sptool sectors terminate 强制终止扇区 名称: sptool sectors terminate - 强制终止扇区（警告：这意味着失去算力并为终止的扇区支付一次性终止罚金（包括抵押品）） 用法: sptool sectors terminate \[命令选项] \[扇区编号1 扇区编号2 ...] 选项: --actor value 指定矿工actor的地址 --really-do-it 如果你知道你在做什么，请传递此标志 (默认: false) --from value 指定发送终止消息的地址 --help, -h 显示帮助 ### sptool sectors compact-partitions 压缩分区 名称: sptool sectors compact-partitions - 从分区中移除死亡扇区，并尽可能减少使用的分区数量 用法: sptool sectors compact-partitions \[命令选项] \[参数...] 选项: --deadline value 要压缩分区的截止时间 (默认: 0) --partitions value \[ --partitions value ] 要压缩扇区的分区列表 --really-do-it 实际发送执行操作的交易 (默认: false) --help, -h 显示帮助 ## sptool proving 查看证明信息 名称: sptool proving - 查看证明信息 用法: sptool proving 命令 \[命令选项] \[参数...] 命令: info 查看当前状态信息 deadlines 查看当前证明期限的截止时间信息 deadline 通过索引查看当前证明期限的截止时间信息 faults 查看当前已知的证明故障扇区信息 help, h 显示命令列表或某个命令的帮助 选项: --help, -h 显示帮助 ### sptool proving info 查看状态信息 名称: sptool proving info - 查看当前状态信息 用法: sptool proving info \[命令选项] \[参数...] 选项: --help, -h 显示帮助 ### sptool proving deadlines 查看证明期限 名称: sptool proving deadlines - 查看当前证明期限的截止时间信息 用法: sptool proving deadlines \[命令选项] \[参数...] 选项: --all, -a 计算所有扇区（默认只计算活跃扇区） (默认: false) --help, -h 显示帮助 ### sptool proving deadline 查看证明期限索引 名称: sptool proving deadline - 通过索引查看当前证明期限的截止时间信息 用法: sptool proving deadline \[命令选项] <截止时间索引> 选项: --sector-nums, -n 打印属于此截止时间的扇区/故障编号 (默认: false) --bitfield, -b 打印分区位域统计信息 (默认: false) --help, -h 显示帮助 ### sptool proving faults 查看证明故障 名称: sptool proving faults - 查看当前已知的证明故障扇区信息 用法: sptool proving faults \[命令选项] \[参数...] 选项: --help, -h 显示帮助 # Experimental Features | 实验性功能 本节介绍 Curio 当前提供的实验性功能。 ## Experimental Features ## 实验性功能 Curio 会持续开发新功能。本节介绍 Curio 当前发布的实验性功能，以及它们的用途和使用方式。 不建议在生产环境中直接使用实验性功能。请根据自身需求先进行测试，如遇到问题或有改进建议，请通过 GitHub 或 Slack 反馈给团队。 当这些新功能经过充分测试并稳定后，相关文档会移动到本站更合适的章节中。 当前实验性功能如下。 {% content-ref url="/pages/UZMq7lzBWLiEYkfDg8gQ" %} [Snark Market | Snark 市场（提供方）](/zh/experimental-features/snark-market) {% endcontent-ref %} {% content-ref url="/pages/OeuC9YtpAc5Uf6g17l9Q" %} [Snark Market (Consumer) | Snark 市场（消费方）](/zh/experimental-features/snark-market-consumer) {% endcontent-ref %} # Snark Market | Snark 市场（提供方） 本页面介绍如何将 Curio Snark Market 配置为提供方（实验性功能）。 ## Snark Market ## Snark 市场（提供方） > ⚠️ **实验性功能，正在测试中**\ > 此功能目前仍处于实验和活跃测试阶段，界面、行为和要求**都可能在没有通知的情况下发生变化**。 *** ### 什么是 Snark Market？ Snark Market 允许 Curio 节点出售或购买证明计算能力，以换取 FIL。对于拥有空闲封装或 GPU 资源的存储提供商来说，它提供了一个去中心化的证明计算市场。 存储提供商也可以作为**消费方**使用 Snark Market，从市场中购买证明计算以节省本地 GPU 资源。相关设置请参见 [Snark Market（消费方）](/zh/experimental-features/snark-market-consumer)。 本指南将介绍如何： * 在 GPU 节点上启用证明出售 * 配置价格和钱包 * 在界面中查看活动与结算信息 *** ### 前置条件 在节点上启用 Snark Market 之前，请确保： * 你运行的是具备 GPU 能力的 Curio 节点。 * 你可以正常访问 Curio Web UI。 * 你已经安装 Lotus，并且 Filecoin 主网已同步完成。\ 可参考 Lotus 文档：\ * 你已经安装 **YugabyteDB**。\ 👉 可参考官方文档：\ *** ### 系统要求 * 现代 **NVIDIA GPU**（建议 12GB 及以上显存） * 基础系统内存 70GB，加上每张 GPU 约 220GB * 内存更低也能运行，但无法使用更快的 CUDA C2 批量封装工具链，单次证明时间可能从约 2 分钟增加到约 10 分钟 * Curio **v1.27.0 或更新版本** * 主网上有可用的 FIL 余额 *** ### 设置步骤 #### 1. 在配置层中启用 Market 请确保你**不是在 WindowPoSt 节点上启用该功能**。该功能仅适用于基于 GPU 的 PoRep 或 Snap 节点。 在 Web UI 中： 1. 进入 `Overview` → `Configuration`，选择你的 `snark-provider` 配置层 2. 找到 **Subsystems** 部分 3. 将 `EnableProofShare` 设置为 `true` 4. 保存并重启节点 *** #### 2. 配置 Provider Settings 在侧边栏进入 `Snark Market`，在 **Provider Settings** 中： * 勾选启用 * **创建一个新的 `f1` 钱包**（不要复用已有钱包）\ 可在命令行使用 `lotus wallet new secp256k1`\ ⚠️ 后续虽然可以修改此钱包，但操作会比较麻烦 * 将 **Price (FIL/P)** 设置为合适值，例如测试时可先用 `0.005` * 点击 **Update Settings** *** #### 3. 验证节点状态 完成配置后： * 节点会自动开始排队获取证明任务 * 仪表盘中会看到： * Active Asks * SNARK Queue * Payment Summaries * Recent Settlements * 你也可以查看公共仪表盘： *** ### 钱包说明 * 请创建一个**新的 `f1` 地址**并为其充值（例如 `0.1 FIL`） * 该钱包用于接收 SNARK 证明收益 * 请确保该钱包保持**已解锁**状态 *** ### 价格与结算 * 价格按约 **130M constraints** 为一个基准单位设置 * 测试时可先从 `0.005 FIL` 起步 * 当结算 gas 费用低于待结算余额的 0.2% 时，系统会自动结算 *** ### 说明 * 提供方需要先完成 **50 个 challenge proofs** 才能获得一个工作槽位 * 每个工作槽位可用于在市场中挂出一个 ask，或处理一个已分配的证明任务 * 工作槽位数量 = 已完成的 challenge proofs / 50 * 若未能在截止时间前完成任务，会损失 **50 个** challenge proofs * 如果为了调价而撤销 ask，会损失 **1 个** challenge proof * challenge proofs 只会通过完成未付费的 challenge 工作获得 * 证明任务必须在 **45 分钟** 内完成，否则需要重新积累信誉 * 系统具备故障重试能力，可自动重试失败任务 * 你可以通过增加 GPU 工作节点来进行水平扩展 *** 如果你正在测试，欢迎在 Slack 的 `#fil-curio-help` 频道反馈，我们会持续关注。 # Snark Market (Consumer) | Snark 市场（消费方） 本页面介绍如何将 Curio Snark Market 配置为消费方（节省 GPU，实验性功能）。 ## Snark Market (Consumer) ## Snark 市场（消费方） > ⚠️ **实验性功能，正在测试中**\ > 此功能目前仍处于实验和活跃测试阶段，界面、行为和要求**都可能在没有通知的情况下发生变化**。 *** ### 什么是 Snark Market（消费方）？ Snark Market 允许存储提供商从市场中**购买**证明计算，而不是在本地运行 GPU。你可以把 PoRep 和 Snap 的证明工作外包给市场中的提供方，并用 FIL 支付费用。这对于想节省 GPU 资源、或者根本没有本地 GPU 的集群尤其有用。 如果你想出售证明算力，请参见 [Snark Market（提供方）](/zh/experimental-features/snark-market)。 *** ### 前置条件 在节点上启用 Snark Market 消费方之前，请确保： * 你运行的是带有**封装流水线**的 Curio 节点（PoRep 或 Snap 任务）。 * 你可以正常访问 Curio Web UI。 * 你已经安装 Lotus，并且 Filecoin 主网已同步完成。\ 可参考 Lotus 文档：\ * 你已经安装 **YugabyteDB**。\ 👉 可参考官方文档：\ *** ### 系统要求 * **不需要 GPU**，因为证明是从市场购买的 * 常规封装节点所需的内存和存储 * Curio **v1.27.0 或更新版本** * 主网上有可用的 FIL 余额（用于支付证明费用） *** ### 设置步骤 #### 1. 在配置中启用 Remote Proofs 1. 进入 `Overview` → `Configuration`，选择你的矿工层或封装层 2. 找到 **Subsystems** 部分 3. 将 `EnableRemoteProofs` 设置为 `true` 4. 保存并重启节点 *** #### 2. 添加并充值 Client Wallets 在侧边栏进入 `Snark Market`，然后查看 **Client Wallets**： * **Add Wallet**：点击 **Add Wallet**，输入一个你控制的 `f1` 地址。你可以直接使用已有钱包（例如 worker、collateral 等），也可以专门新建一个独立钱包。多数 SP 已经有可用的钱包，因此通常不需要专门再创建一个。 * **Deposit**：点击 **Deposit**，把该钱包链上的 FIL 转入支付路由器。只有路由器里有可用余额，系统才能购买证明。 *** #### 3. 配置 Client Settings 在 **Client Settings**（页面右侧）中： 1. 根据提示接受 **Client Terms of Service** 2. 点击 **Add SP**，输入你的 SP 地址以及用于付款的客户端钱包地址（该钱包应当已经在 Client Wallets 中添加） 3. 对每一行 SP 设置： * 勾选 **Enabled** * 将 **Wallet** 设为用于付款的 `f1` 地址 * 设置 **buy\_delay\_secs**，表示在把任务外包出去前等待多长时间，以便本地 GPU（如果有）先尝试处理 * 根据需要启用 **do\_porep** 和/或 **do\_snap** * 设置 **FIL/P**，表示你愿意接受的最高价格 4. 点击 **Save** *** ### 价格说明 **FIL/P** 表示你愿意为一个 **P** 支付的最高价格。一个 **P** 可以理解为一个“证明价格单位”，大致对应 **一个 32 GiB C2（PoRep）证明** 的价格基准。 | 证明类型 | 倍数 | 费用公式 | 当 `FIL/P = 0.005` 时的示例 | | ---------------------------------- | ---- | ------------- | ---------------------- | | **32 GiB C2**（PoRep） | 1× | `1 × FIL/P` | **0.005 FIL** | | **32 GiB Snap**（UpdateEncode/更新证明） | 1.6× | `1.6 × FIL/P` | **0.008 FIL** | 例如，如果你把 **FIL/P** 设为 `0.005`，并且当前市场价格不高于这个值，那么一个 32 GiB C2 大约会花费 `0.005 FIL`，一个 32 GiB Snap 大约会花费 `0.008 FIL`。 你的 **FIL/P** 是你愿意支付的**上限**。只有当当前市场价格低于或等于该上限时，系统才会购买证明。 你可以查看公共仪表盘 中的 **Min price**，并据此设置自己的 `FIL/P`。 *** ### Balance Manager（可选） 如果你希望自动补充客户端钱包余额： 1. 进入 **Wallet** → **Balance Manager** 2. 点击 **Add SnarkMarket Client Rule** 3. 将 **Subject** 设为你的客户端钱包地址 4. 设置 **Low** 和 **High** 水位（单位：FIL） 5. 保存 当支付路由器中的可用余额低于低水位时，Balance Manager 会自动把链上余额补充到路由器中。 *** ### 验证 * **View Requests**：可查看某个 SP 当前和历史的证明请求 * **Client Messages**：可查看充值、提现等客户端消息状态 *** ### 说明 * `buy_delay_secs` 可以让本地 GPU（如果有）先尝试接手任务；若设为 `0`，则会更快地把任务送往市场 * 只有当当前市场价格低于或等于你设置的 `FIL/P` 时，任务才会被发送到市场 * 请确保客户端钱包持续有余额，否则无法继续购买证明 *** 如果你正在测试，欢迎在 Slack 的 `#fil-curio-help` 频道反馈，我们会持续关注。 --- [Next Page](/llms-full.txt/1)