# What is Curio? What is Curio and how is it different from Lotus-Miner? ## Overview Curio is the new implementation of Filecoin storage protocol. It aims to simplify the setup and operation of storage providers. {% hint style="danger" %} Please note that Curio cluster cannot be shared across different networks. Example: A single Curio cluster cannot host miners IDs coming from Mainnet and Calibnet together. {% endhint %} ## Key Features #### High Availability Curio is designed for high availability. You can run multiple instances of Curio nodes to handle similar type of tasks. The distributed scheduler and greedy worker design will ensure that tasks are completed on time despite most partial outages. You can safely update one of your Curio machines without disrupting the operation of the others. #### Node Heartbeat Each Curio node in a cluster must post a heartbeat message every 10 minutes in HarmonyDB updating its status. If a heartbeat is missed, the node is considered lost and all tasks can now be scheduled on remaining nodes. #### Task Retry Each task in Curio has a limit on how many times it should be tried before being declared lost. This ensures that Curio does not keep retrying bad tasks indefinitely. This safeguards against lost computation time and storage. #### Polling Curio avoids overloading nodes with a polling system. Nodes check for tasks they can handle, prioritizing idle nodes for even workload distribution. #### Simple Configuration Management The configuration is stored in the database in the forms of layers. These layers can be stacked on top of each other create a final configuration. Users can reuse these layers to control the behaviour of multiple machines without needing to maintain the configuration of each node. Start the binary with the appropriate flags to connect with YugabyteDB and specify which configuration layers to use to get desired behaviour. #### Running Curio with Multiple GPUs Curio can handle multiple GPUs simultaneously without needing to run multiple instances of the Curio process. Therefore, Curio can be managed as a single systemd service without concerns about GPU allocations. ## Curio vs Lotus Miner | Feature | Curio | Lotus-Miner | | ------------------------------------ | -------------------------------------------------------------- | --------------------------------------------------- | | Scheduling | Collaborative (Prioritized Greedy) | Single point of failure | | High Availability | Available | Single control process | | Redundant Post | Available | Not Available | | Task Retry Control | Task retry with a cutoff limit (per task) | Unlimited retry leading to resource exhaustion | | Multiple Miner IDs | Curio cluster can support multiple Miner IDs | Single Miner ID per Lotus-Miner | | Shared Task nodes | Curio nodes can handle task for multiple Miner IDS | Attached workers handle tasks for a single Miner ID | | Distributed Configuration Management | Configuration stored in the highly-available Yugabyte Database | All configuration in a single File | ## Future of Curio The long-term vision for Curio is to eventually replace the current lotus-miner and lotus-worker processes. This is part of an ongoing effort to simplify and streamline the setup and operation of storage providers.
# Filecoin Storage Provider This pages covers some basic concepts useful to be a Filecoin storage providers ## 1. Overview of Filecoin Storage Providers A Storage Provider in the Filecoin network: * **Stores client data** in exchange for fees and potential block rewards. * Must **continuously prove** that stored data is available (via cryptographic proofs). * Must **pledge collateral** in the form of FIL tokens to ensure honest behavior. * Is subject to **penalties** if data is lost or if proofs are missed. Becoming a Storage Provider blends **technical** and **economic** considerations. It requires: 1. **Sufficient hardware** to store, seal, and prove data reliably. 2. **Continuous on-chain proofs** to show you are storing data as promised. 3. **Economic planning** to handle locked collateral, fees, and eventual block rewards. *** ## 2. Key Concepts in Filecoin Below are the foundational ideas you need to understand thoroughly before running a Storage Provider. #### 2.1 Sectors * **Definition**: The basic unit of storage in Filecoin. You commit user data to the network by packaging it into *sectors*. * **Sizes**: Commonly 32 GiB (the most popular) or 64 GiB. Once you choose a sector size for your provider setup, you typically **cannot change** it without a new identity. * **Committed Capacity (CC)**: You can commit “empty” sectors to the network to build storage power without actual client data initially. These sectors can be upgraded with real data later via a feature called **SnapDeals**. **Sealed vs. Unsealed Sectors** 1. **Unsealed sector**: Contains the original (plaintext) data. Clients sometimes request that you keep an unsealed copy for faster retrieval. 2. **Sealed sector**: Cryptographically processed data ready for continuous proofs (Proof-of-Spacetime). Sealed data cannot be directly read until it’s unsealed (if needed for retrieval). #### 2.2 Proof-of-Spacetime (PoSt) Two key proof mechanisms ensure a provider’s data is verifiably present: 1. **WindowPoSt**: Happens in \~24-hour windows; you must prove that each sector is still stored. If you miss this proof, you incur faults and penalties. 2. **WinningPoSt**: A small subset of providers is elected every epoch (\~30 seconds) to produce a new block. If chosen, you submit a short proof showing you hold your pledged data. In return, you can earn a **block reward**. #### 2.3 Deals and Retrieval * **Storage Deals**: Contracts made with clients who pay you to store their data for a specified duration. * **Retrieval Deals**: When clients pull their data, they pay you retrieval fees, typically via payment channels. * **Verified Clients**: Some clients store “verified” data. Providers who store verified data get higher *quality-adjusted power* (and thus a higher chance for block rewards). #### 2.4 Epoch Filecoin time is measured in **epochs** (\~30-second intervals). * Each epoch triggers new tasks like verifying proofs, adding blocks, distributing block rewards, and so on. *** ## 3. Economics & Rewards #### 3.1 Rewards 1. **Storage fees**: Paid over time by clients whose data you store. 2. **Block rewards**: Awarded by the network when you produce a block. You typically need **at least 10 TiB** of raw power to be eligible for these. A portion of each block reward is locked and vests linearly over \~180 days. #### 3.2 Collateral and Locked Funds 1. **PreCommit Deposits**: You lock up some FIL when you “pre-commit” a sector. If the sector never proceeds to full commitment, the deposit is lost. 2. **Initial Pledge**: More FIL is locked when you fully commit a sector (prove-commit). This collateral is held to ensure you keep data online. 3. **Locked Rewards**: 75% of any block reward vests over \~180 days; 25% is immediately available. #### 3.3 Penalties & Slashing * **Fault Fees**: If you fail to prove a sector on time (WindowPoSt), you pay a daily fault fee until you fix or terminate the sector. * **Sector Penalties**: If you do not declare a fault before a scheduled proof check, you pay an immediate penalty. * **Termination Fees**: For sectors that are terminated early (voluntarily or involuntarily). * **Consensus Fault Slashing**: Severe penalty for malicious consensus-level actions. *** ## 4. Theory of Sector Lifecycle Although exact commands differ, every Filecoin SP software follows a similar lifecycle for sector creation and maintenance: 1. **Sector Allocation** * The provider decides to add capacity, either in the form of empty (CC) or client-filled sectors. 2. **PreCommit** * You produce a sector with partial sealing steps (encoding the data) and record this “precommit” on-chain with the required deposit. 3. **ProveCommit** * Final sealing steps generate a proof of replication (PoRep). * You submit this proof on-chain to finalize the commitment, locking your initial pledge collateral. 4. **Continuous Proof (WindowPoSt)** * Regularly (every \~24 hours), you must prove that each committed sector remains intact. * Missing this proof for a sector causes it to become **faulty**. 5. **Maintenance** * You can recover faulty sectors by re-running proofs. * You can terminate sectors early if needed, but you pay a termination penalty. 6. **Expiration or Upgrades** * Deals eventually expire. You can extend deals or reuse the sector for new deals (SnapDeals). * Once the entire sector’s lifetime ends, you can choose to remove it from the network or keep it sealed if you plan to extend. *** ## 5. Daily Operations & Tasks (General) #### 5.1 Monitoring Proofs * **WindowPoSt** schedules are strict. Know your deadlines and ensure your system has enough CPU/GPU resources to generate proofs on time. * Watch for I/O bottlenecks that could slow proof generation. #### 5.2 Accepting and Managing Deals * **Negotiation**: Clients discover your provider (through a market or index) and propose deals. * **Data Transfer**: You receive client data. This data is prepared for sealing. * **Sealing**: The data is integrated into new or partially filled sectors. * **Payments**: Storage fees accumulate over time, typically locked and then released to your available balance after each deal cycle. #### 5.3 Retrievals * **Unsealing on Demand**: If you do not maintain an unsealed copy of the data, you need to unseal it when a client requests retrieval (this can be slow). * **Incremental Payments**: Retrieval fees are often paid via payment channels, chunk by chunk. #### 5.4 Maintenance * **Fault Recovery**: If a sector goes faulty, you can declare it in advance (to reduce penalties) and later recover it. * **Hardware Upgrades**: As you add more capacity or want to speed up sealing, you may add more RAM, GPUs, or worker machines. * **Software Updates**: Regularly update your chosen node/miner software to get protocol improvements and security fixes. *** ## 6. SnapDeals and Batching (Advanced Concepts) #### 6.1 SnapDeals * **Upgrade CC Sectors**: SnapDeals lets you add real client data to a sector that was previously committed as empty capacity. * **Advantages**: * Saves time and resources: no need to run a full seal again. * Increases your effective power if the data is from verified clients (quality-adjusted power boost). #### 6.2 Batching * **On-Chain Efficiency**: Sending many single-sector proofs can be expensive in network fees (gas). Batching helps you combine multiple proofs into a single message. * **Collateral Savings**: Batching can also help group sector pledges, reducing overhead. *** ## 7. Handling Balances & Collateral 1. **Ensure Adequate FIL** in your provider or worker wallets for day-to-day operations. 2. **Monitor** locked vs. unlocked balances. Collateral must be readily available to avoid interruptions in sealing and deal-making. 3. **Reward Vesting**: Remember that block rewards vest linearly over \~180 days, so you can’t immediately withdraw 100% of your earnings. *** ## 8. Penalties and How to Avoid Them 1. **Declare Faults in Time**: If you know a sector or machine is failing, declare a fault early. This is cheaper than letting a proof window pass without submission. 2. **Maintain Redundancy**: Keep backups or have spares for crucial components (power supplies, GPUs, or entire machines). 3. **Stay Online**: A stable network connection is key. If your system is offline at your WindowPoSt deadline, you’ll incur faults. *** ## 9. Scaling Your Operation As you gain experience, you may want to scale: * **Multiple Storage Locations**: Distribute sealed data across multiple racks or data centers. * **Parallel Sealing**: Invest in more CPU cores, more RAM, and multiple GPUs to process many sectors at once. * **Distributed Workers**: In some implementations, you can run separate “worker” processes on different machines to handle sealing tasks more efficiently. *** ## 10. Best Practices and Tips 1. **Monitor Everything**: Use dashboards and logs to watch for unusual CPU, RAM, or disk usage. Keep track of system health. 2. **Time Synchronization**: Keep your system clock accurate (via NTP or Chrony). Mismatched clocks can cause missed proofs. 3. **Hardware Stress Testing**: Sealing is CPU/GPU heavy. Make sure your hardware can handle sustained loads. 4. **Diversify Deals**: Seek out verified clients if possible, since that can boost your block reward chances (due to higher quality-adjusted power). 5. **Community Interaction**: Join Filecoin Slack or other community channels to stay updated on protocol changes and to troubleshoot issues with other providers. *** ## 11. Summary Becoming a successful Filecoin Storage Provider requires: 1. **Solid Understanding** of Filecoin’s protocol and concepts (sectors, PoSt, deals, collateral, etc.). 2. **Robust Hardware** sized for your planned sealing rate and total storage. 3. **Continuous Maintenance** to ensure you never miss your proofs or run out of collateral. 4. **Economic Planning** to manage rewards, locked funds, and the cost of potential faults. 5. **Attention to Detail** with advanced features like SnapDeals, batching, and retrieval market optimizations. *** ## Next Steps * **Deep-Dive Documentation**: Explore the [official Filecoin Specification](https://spec.filecoin.io/) for in-depth details of each proof, state machine, and economic mechanism. * **Pilot Deployment**: Start small to gain hands-on experience with sealing, proof submission, and deal flow. * **Expand** after building confidence in your setup, hardware reliability, and economic strategy. By following these principles, you can build and manage a reliable Filecoin storage operation, confidently handle client deals, avoid excessive faults, and steadily grow your provider business over time. # Design This page provides a detailed overview of the core concepts and components that make up Curio, including HarmonyDB, HarmonyTask, and more. ## Design ### Curio Cluster The core internal components of Curio are HarmonyDB, HarmonyTask, ChainScheduler and a database abstraction of configuration & today’s storage definitions.
Curio Node

Curio nodes

A Curio cluster is a cluster of multiple Curio nodes connected to a YugabyteDB cluster and market nodes. A single Curio cluster can serve multiple miner ID and share the computation resources between them as required.
Curio cluster

Curio cluster

### HarmonyDB HarmonyDB is a simple SQL database abstraction layer used by HarmonyTask and other components of the Curio stack to store and retrieve information from YugabyteDB. #### Key Features: * **Resilience:** Automatically fails over to secondary databases if the primary connection fails. * **Security:** Protects against SQL injection vulnerabilities. * **Convenience:** Offers helper functions for common Go + SQL operations. * **Monitoring:** Provides insights into database behavior through Prometheus stats and error logging. See [Prometheus Metrics](/configuration/prometheus-metrics) for setup and available metrics. #### Basic Database Details * The Postgres DB schema is called “curio” and all the harmony DB tables reside under this schema. * Table `harmony_task` stores a list of pending tasks. * Table `harmony_task_history` stores completed tasks, retried tasks exceeding limits, and serves as input for triggering follower tasks (potentially on different machines). * Table `harmony_task_machines` is managed by lib/harmony/resources. This table references registered machines for task distribution. Registration does not imply obligation, but facilitates discovery. ### HarmonyTask The HarmonyTask is pure (no task logic) distributed task manager. #### Design Overview * Task-Centric: HarmonyTask focuses on managing tasks as small units of work, relieving developers from scheduling and management concerns. * Distributed: Tasks are distributed across machines for efficient execution. * Greedy Workers: Workers actively claim tasks they can handle. * Round Robin Assignment: After a Curio node claims a task, HarmonyDB attempts to distribute remaining work among other machines.
Curio Tasks

Harmony tasks

#### Model * **Blocked Tasks:** Tasks can be blocked due to: * Configuration under ‘subsystems’ disabled on the running node * Reaching specified maximum task limits * Resource exhaustion * CanAccept() function (task-specific) rejecting the task * **Task Initiation:** Tasks can be initiated through: * Periodic database reads (every 3 seconds) * Addition to the database by the current process * **Task Addition Methods:** * Asynchronous listener tasks (e.g., for blockchains) * Follower tasks triggered by task completion (sealing pipeline) * **Duplicate Task Prevention:** * The mechanism for avoiding duplicate tasks is left to the task definition, most likely using a unique key. ### Distributed Scheduling Curio implements a distributed scheduling mechanism co-ordinated via the HarmonyDB. The tasks are picked by the Curio nodes based on what they can handle (type and resources). Nodes are not greedy after taking a task, even if they have enough resources. Other nodes get a turn to claim a task. On 3 second intervals, if resources are available then an additional task will be taken. This ensures a more even scheduling of the tasks. ### Chain Scheduler The `CurioChainSched` or the chain scheduler trigger some call back functions when a new TipSet is applied or removed. This is equivalent to getting the heaviest TipSet on each epoch. These callback function in turn add the new tasks for each type that depends on the changes in the chain. These task types are WindowPost, WinningPost and MessageWatcher. ### Poller Poller is a simple loop that fetches pending tasks periodically according to predefined durations (100ms), or until a graceful exit is initiated by the context. Once the pending tasks are fetched from the database, it attempts to schedule all the tasks on the Curio node. This attempt will result in one of the following results: * Task is accepted * Task is not scheduled as machine is busy * Task is not accepted as the node’s CanAccept (defined by the task) elects not to handle the specified Task If the task is accepted during a polling cycle, the wait time before the next cycle is equal to 100ms. But if the task is not scheduled for any reason the poller will retry after 3 seconds. ### Task Decision Logic For each task type a machine can handle, it first checks if the machine has enough capacity to execute the said task. Then it queries the database for tasks with no `owner_id` and the same name as the task type. If such tasks are present, it attempts to accept their work. It returns true if any work has been accepted and false otherwise. The decision-making logic to accept each task is following: 1. Checks if there are any tasks to do. If there are none, returns true. 2. Checks if the maximum number of tasks of this type is reached. If the number of running tasks meets or exceeds the maximum limit, log a message and returns false. 3. Checks if the machine has enough resources to handle the task. This includes checking the CPU, RAM, GPU capacity, and available storage. If the machine does not have enough resources, log a message and return false. 4. Checks if the task can be accepted by calling the `CanAccept` method. If it cannot be accepted, log a message and return false. 5. If the task requires storage space, the machine attempts to claim it. If the claim fails, log a message and releases the claimed storage space, then return false. 6. If the task source is `recover` i.e. the machine was performing this task before shutdown then increase the task count by one and begin processing the task in a separate goroutine. 7. If the task source is `poller` i.e. new pending task, attempt to claim the task for the current hostname. If unsuccessful, release the claimed storage and attempt to consider the next task. 8. If successful, increase the task count by one and begin processing the task in a separate goroutine. 9. This goroutine also updates the task status in the task history and depending on whether the task was successful or not, either deletes the task or updates the task in the tasks table. 10. Returns true, indicating that the work was accepted and will be processed. ### GPU Management in Curio #### **Historical Issues with Lotus-Miner Scheduler** Historically, the Lotus-Miner scheduler has encountered difficulties efficiently utilizing GPUs when more than one GPU is available to the lotus-worker process. These issues primarily arise from the underlying proofs library, which handles all GPU-related tasks and manages GPU assignments. This has led to problems such as: * A single task being assigned to multiple GPUs. * Multiple tasks being assigned to a single GPU. #### **Solution with Curio: The GPU Picker Library "ffiselect"** To address these issues in Curio, we have implemented a GPU picker library called "ffiselect". This library ensures that each task requiring a GPU is assigned one individually. The process works as follows: 1. **Task Assignment**: Each GPU-requiring task is assigned a specific GPU. 2. **Subprocess Creation**: A new subprocess is spawned for each task, with the dedicated GPU allocated to it. 3. **Proofs Library Call**: The subprocess calls the Proofs library with a single GPU and the specific task.

Curio FFISelect in action

This approach ensures efficient and conflict-free GPU usage, with each task being handled by a dedicated GPU, thus resolving the historical issues observed with the `lotus-miner` scheduler. ## Security Boundary This is what Curio expects an SP to secure in order to have a safe experience. Curio is cluster software which coordinates directly and through the database. It also communicates to the public through chain providers (Lotus) and the market node. To secure this properly, ensure that only trusted people & services have access to: * logs: (these include inputs to failing processes) * physical machines, * virtual machine access (ssh) for Curio, Lotus, or Yugabyte * Curio or Lotus' or Yugabyte's open ports (with exceptions noted by Lotus, and the Curio market node) -- This includes the admin web ui for Curio which exposes numerous capabilities beyond viewing. Safe to share with untrusted parties: (will not receive private information) * Prometheus output * alerts can be sent to untrusted receivers * CuView (at your own risk) has modes for light investigation. Curio team recommends a network (VPN) containing all the pieces to have limited access. Logs are mostly clean except for errors which try to be as specific as possible, so partial redaction may be best here if sharing with untrusted parties. # Sealing This page explains how the sealing pipeline functions in Curio ## Sealing Pipeline Curio’s sealing process is powered by HarmonyTasks. Each stage involved in sealing a sector is divided into smaller, independent tasks. These individual tasks are then picked up by different machines within the Curio cluster. This ensures tasks are distributed effectively and resources are used efficiently across the entire system.
Overview of Curio sealing pipeline

Curio sealing pipeline

## SealPoller The SealPoller struct is designed to track the progress of sealing operations. Each possible state in the sealing workflow is represented by a pollTask struct. This struct tracks each step a sealing operation might be in by setting boolean flags and saving task ids in individual columns of the `sectors_sdr_pipeline` table within the harmony database. ``` type pollTask struct { SpID int64 `db:"sp_id"` SectorNumber int64 `db:"sector_number"` TaskSDR *int64 `db:"task_id_sdr"` AfterSDR bool `db:"after_sdr"` TaskTreeD *int64 `db:"task_id_tree_d"` AfterTreeD bool `db:"after_tree_d"` TaskTreeC *int64 `db:"task_id_tree_c"` AfterTreeC bool `db:"after_tree_c"` TaskTreeR *int64 `db:"task_id_tree_r"` AfterTreeR bool `db:"after_tree_r"` TaskPrecommitMsg *int64 `db:"task_id_precommit_msg"` AfterPrecommitMsg bool `db:"after_precommit_msg"` AfterPrecommitMsgSuccess bool `db:"after_precommit_msg_success"` SeedEpoch *int64 `db:"seed_epoch"` TaskPoRep *int64 `db:"task_id_porep"` PoRepProof []byte `db:"porep_proof"` AfterPoRep bool `db:"after_porep"` TaskFinalize *int64 `db:"task_id_finalize"` AfterFinalize bool `db:"after_finalize"` TaskMoveStorage *int64 `db:"task_id_move_storage"` AfterMoveStorage bool `db:"after_move_storage"` TaskCommitMsg *int64 `db:"task_id_commit_msg"` AfterCommitMsg bool `db:"after_commit_msg"` AfterCommitMsgSuccess bool `db:"after_commit_msg_success"` Failed bool `db:"failed"` FailedReason string `db:"failed_reason"` } ``` The SealPoller retrieves all the `pollTasks` from the database, for which `after_commit_msg_success` or `after_move_storage` is not true, and tries to advance their state if possible. A `pollTask` is advanced when its dependencies, indicated by the “after\_” fields, are completed and the task itself is not yet queued (its task id is nil) or completed (its “After” field is false). Each pollTask’s advancement will trigger a database transaction attempting to update the task id with the new task id received from the HarmonyDB. The transaction makes sure that the task hasn’t been queued by others between reading the state and updating the task id. This polling process happens sequentially with different conditions for each stage, ensuring that all previous conditions are fulfilled before proceeding. If a task cannot proceed due to its previous dependencies not being completed, the poller will come back in the next round. Mostly, errors occurring during the poller operation are logged and don’t cause the poller to stop. But if something serious happens during a database transaction, it will be rolled back, with an error message giving details. By organizing work in this way, SealPoller ensures that each step in the sealing procedure occurs in the correct order, and that progress is made whenever it is possible to do so. It allows sectors to be sealed as efficiently as possible given the constraints of other tasks in progress.
Sealing task execution

Curio harmony task execution

## Piece Park Traditionally, data needs to be available before it can be sealed for storage. However, this can lead to inefficiencies. Curio addresses this by introducing a “Piece Park.” Curio’s sealing pipeline does not require the data to be readily available upfront. This allows us to initiate the sealing process even before the data is downloaded. While the sealing process progresses, the data is “parked” in a designated directory called “piece” within the storage location. This avoids keeping market connections open for extended periods. In essence, the local piece park acts as a temporary holding area for data, streamlining the sealing process and optimizing resource usage. Curio utilizes two tasks: ParkPiece: This task handles downloading the data and placing it in the “piece” directory. DropPiece: Once the data is no longer needed, this task takes care of cleaning up the parked data. In the future, this local storage can also allow Curio to reseal data in a new sector if the original sector gets lost during the sealing process. ## LMRPCProvider The LMRPCProvider provides a set of methods to interact with various data related to sectors and pieces. These methods are required by market implementation (Boost) to track the sealing progress of a deal. ``` ActorAddress: This method returns the actor's address associated with the LMRPCProvider. In other words, it returns the miner's address. WorkerJobs: This function returns a map of worker jobs, indexed by UUID. SectorsStatus: This method returns the status of a sector given the sector identifier sid. This function includes detailed information about the sector such as the state of sealing, deal ids, log, pledge, and expiration etc. SectorsList: This function provides a list of sector numbers currently stored. SectorsSummary: This function gives a summary of the sectors, categorized by their state. It returns a map that maps each sector state to its count. SectorsListInStates: This method returns a list of sector numbers that are in a given set of states. ComputeDataCid: This function is used to compute the CID of the data. AuthNew: This function creates a new authorization token (JWT) for the given permissions. ``` ## Piece Ingester The Piece Ingestor allocates a piece to a sector for a given miner address. It checks if the piece size matches the sector size, determines the preferred seal proof type, retrieves the miner ID, allocates a sector number, inserts the piece and sector pipeline entries into the database, and returns the sector and offset of the allocated piece.
# Harmony Tasks This guide explains the different HarmonyTasks available in Curio Curio uses HarmonyTask as a generic task container which can be scheduled by the poller in regular basis for the execution. To perform the different aspects of sealing and proving, Curio implements the following task types. ### SDR The SDR task is the first phase of the Proof-of-Replication process and is where encoding and replication of the data takes place. The SDR task is predominantly using a single CPU core, and is heavily utilizing the SHA256 instruction set. Using a CPU that has the SHA256 instruction set is therefore recommended. All 11 layers of calculation, layer by layer, are calculated sequentially. Each layer is 32GiB in size. When the SDR process is finished you will have generated data to the amount of 384GiB (A 32GiB unsealed sector + (11 layers x 32GiB)). SDR task requires commD as one of the input parameters. The commD calculation requires piece size and CID for all the pieces that will be part of the sector. The pieces themselves (data) is not required at this stage of the pipeline. ### SDRTrees The SDRTrees task can be further divided into 3 parts which are completed sequentially. #### TreeD Building the TreeD requires the access to the data to be sealed into the sector. It builds a Merkle tree using the data and writes it to the specified output path ending with “tree-d.dat”. It also returns the root CID of the generated tree. #### TreeRC In the TreeRC task, a column hash computation based on the 11 layers generated in PreCommit 1 is calculated, and a merkle tree gets constructed. This is same as PreCommit 2 on Lotus-miner. These tasks generates unsealed CID and sealed CID. The unsealed CID should match the root CID of the TreeD output. During this phase an additional 64GiB file (32GiB sectors) that represents the merkle tree is stored, in addition to the sealed 32GiB sector. Bringing the total amount of storage needed to approximately 500 GiB for one sector. ### PreCommitSubmit Through the `PreCommitSector` message a storage provider submits a deposit for a given sector’s sealed data, often referred to as the SealedCID, or commitment to replica (commR). After the message is included on-chain, the sector is registered to the storage provider and the sector enters the WaitSeed state, which is a security wait requirement by the network. This message type can also be batched to include multiple PreCommitSector messages in a single message to save gas fees paid to the network. These batched messages are called `PreCommitSectorBatch`. The message is not sent by the PreCommitSubmit task itself but is handed over to the queue of `SendMessage` task. ### PoRep The PoRep task combines the Commit1 and Commit2 parts of the Lotus-Miner sealing pipeline. The randomness acquired at the end of the wait seed state is used in the Commit 1 phase to select a random subset of leaf nodes from the merkle tree generated in the PreCommit 2 phase. From the subset of leaf nodes it checks, it generates a much smaller file than the full merkle tree. That file is approximately 16MiB in size. In the Commit 2 phase, the file from the Commit 1 gets compressed into a much smaller proof using zk-SNARKs. The proof generated at the end of Commit 2 can be verified that is correct very fast, and is small enough to be suitable for a blockchain. The final size of the proof is approximately 2kib, and gets published on the blockchain. ### Finalize The Finalize task performs the following operations: 1. It truncates the output of TreeD file to the sector size and them moves it to the unsealed file location of the sector. User should not that unsealed sector copy will not exist till point in the sealing pipeline. The unsealed copy is created on if “KeepUnsealed” is true for the deal. 2. The cache files for the sector are cleaned up at this stage. 3. Delete the local copy of the pieces that have been added to the sector. ### MoveStorage The MoveStorage task moves the data from sealing storage to permanent storage. ### CommitSubmit In CommitSubmit task, we create the `ProveCommitSector` message for the sector and hand it over to the queue of `SendMessage` task. Through the `ProveCommitSector` message the storage provider provides a Proof of Replication (PoRep) for the sector committed in the `PreCommitSector` message. This proof must be submitted AFTER the security wait requirement by the network (WaitSeed), and before the PreCommit expiration of the sector. This message type can also be aggregated to include multiple ProveCommitSector messages in a single message. These aggregated messages are called `ProveCommitAggregate`. ### WindowPost The WindowPost allow storage providers to verifiably prove they have the data they have committed to the network on disk to create a verifiable, and public record attesting to the storage providers continued commitment of storing the data, or for the network to reward storage providers for their contributions. The overall WindowPost process has been broken into 3 independent tasks in Curio. Each of these tasks are triggered by the `CurioChainScheduler` when the TipSet changes. #### WdPost WindowPost task is responsible for generating the proof for an individual partition in the current deadline. Curio runs multiple such tasks in parallel to speed up the calculation time for each deadline. #### WdPostRecover The WdPostRecover task is also executed on per partition basis for each deadline. We check all the previously faulty sectors and determine which sectors have now recovered since then. it creates the recovery message for each partition in the current deadline and submits these messages to the queue of `SendMessage` task. #### WdPostSubmit WdPostSubmit creates the WindowPost messages for each partition in the current deadline and submits these messages to the queue of `SendMessage` task. ### WinPost Winning Proof-of-SpaceTime (WinningPoSt) is the mechanism by which storage providers are rewarded by the Filecoin network for their contributions to it. As a requirement for doing so, each storage provider is tasked with submitting a compressed Proof-of-Spacetime for a specified sector. Each elected storage provider who successfully creates a block is granted FIL, as well as the opportunity to charge other Filecoin participants fees to include messages in the block. Storage providers who fail to do this in the necessary window will forfeit their opportunity to mine a block. The WinPost task is triggered on each epoch change and if Miner address wins the election then a new block is created and submitted to the chain. ### SendMessage The SendMessage task implements a message queue where message can be added by any other task. These messages are then processed by the `SendMessage` and processed individually. It abstracts away highly-available message sending with coordination through HarmonyDB. It makes sure that Nonce are assigned in transactional manner, and that messages are correctly broadcast to the network. It ensures that messages are sent serially, and that failures to send doesn’t cause a nonce gap. ### ParkPiece Curio has implemented a new file location within the storage subsystem called “piece”. This directory is used to temporarily park the pieces while they are being sealed. The `parked_pieces` also contains the URL and headers to download the data. The ParkPiece task scans the `parked_pieces` table in HarmonyDB every 15 seconds. If any pieces are found, a corresponding file is created in under “piece” directory of the storage and data is downloaded to the file from the URL. When `SectorAddPieceToAny` method is called by an external market node, it creates a ParkPiece tasks. ### DropPiece The DropPiece tasks are responsible for removing a piece from `Piece Park` and ensuring all the files and reference related to the piece are cleaned up. This task is triggered by the Finalize task of a sector is sealing pipeline. ### UpdateEncode SnapDeal sealing tasks are a special type of sealing tasks which allows a storage provider to takes already committed sealed sectors and place deal data into them. The UpdateEncode task encodes the incoming unsealed data (deal data), into an existing sealed sector. Once the encoding is complete, vanilla proofs are generated and verified to check and confirm that the data has been encoded correctly in the sealed sector file. ### UpdateProve In the UpdateProve phase, the output from the UpdateEncode task gets compressed into a smaller proof using zk-SNARKs. The zk-SNARK generated after the UpdateProve can verify that the new data is encoded in the new sealed sector, and is small enough to be suitable for a blockchain. The generation of the zk-SNARK can be done by the CPU or accelerated by using a GPU. ### Resource requirements for each Task type in Curio By default, the number of tasks allowed for each type are not limited on any Curio node. The distributed scheduler ensures that no Curio node over-commits the resources. | Task Name | CPU | RAM(GiB) | GPU | Retry | | --------------- | --- | -------- | --- | ----- | | SDR | 4 | 64 | 0 | 2 | | SDRTreeD | 1 | 1 | 0 | 3 | | SDRTreeRC | 1 | 8 | 1 | 3 | | SyntheticProofs | 1 | 8 | 0 | 5 | | PreCommitSubmit | 0 | 1 | 0 | 16 | | PoRep | 1 | 50 | 1 | 5 | | Finalize | 1 | 0.1 | 0 | 10 | | MoveStorage | 1 | 0.128 | 0 | 10 | | CommitSubmit | 0 | 0.001 | 0 | 16 | | WdPostSubmit | 0 | 0.010 | 0 | 10 | | WdPostRecover | 1 | 0.128 | 0 | 10 | | WdPost | 1 | 32 | 1 | 3 | | WinPost | 1 | 1 | 1 | 3 | | SendMessage | 0 | 0.001 | 0 | 1000 | | UpdateEncode | 1 | 1 | 1 | 3 | | UpdateProve | 1 | 50 | 1 | 3 | # Getting Started This is a step by step guide for new users to get onboarded with Curio ## Curio Database and Distributed Architecture ### Familiarizing Yourself with Curio Before diving into the setup and configuration of Curio, we highly recommend becoming familiar with [Curio's design and fundamental principles](/design). This foundational knowledge will greatly assist in effective administration and troubleshooting. ### **HarmonyDB with YugabyteDB** Curio utilizes YugabyteDB to create an abstraction layer known as HarmonyDB. This HarmonyDB serves two primary purposes: 1. **Metadata Storage**: It stores all Curio-related metadata. 2. **Consensus Layer**: It establishes a consensus layer for the distributed architecture of a Curio cluster. {% hint style="danger" %} We recommend using at least 3 node YugabyteDB cluster for HA and scalability. Loss of the DB will render Curio dead. YugabyteDB should also be backed up regularly. {% endhint %} ### Key Features of HarmonyDB * **High Availability**: Ensures that the metadata and consensus information is always available, even in the event of node failures. * **Scalability**: Capable of handling increasing amounts of data and expanding as the Curio cluster grows. * **Consistency**: Maintains data consistency across the distributed nodes of the Curio cluster. ### Benefits of Using YugabyteDB for HarmonyDB * **Distributed SQL**: Combines the benefits of SQL with the resilience and scalability of a distributed database. * **Fault Tolerance**: Provides strong fault tolerance, ensuring the reliability of the Curio cluster. * **Multi-Region Deployment**: Supports deployment across multiple regions for improved performance and redundancy. ## Chain Node Curio requires access to at least one Filecoin chain node like [Lotus](https://lotus.filecoin.io/lotus/get-started/what-is-lotus/) or [Forest](https://docs.forest.chainsafe.io/) (integration in progress). This chain node is used by Curio to get the current chain state and send messages to the chain. Curio support using multiple chain nodes. ## Network Following port must be opened on each Curio node for API and GUI access | Port | Details | | ----- | ---------------------------------------------------------------- | | 12300 | Default API port | | 4701 | Default GUI port. Not all Curio nodes are required to enable GUI | | 12310 | HTTP server port | ## Boost Compatibility (Deprecated) Boost is no longer compatible with latest Curio releases. Boost adapter is no longer shipped with our main branch and we recommend users to migrate to Curio markets. ## Installing Curio and creating a Curio cluster With an understanding of Curio's internal mechanisms, you can now proceed to [install the Curio binaries](/installation). We recommend using [Debian packages](/installation#debian-package-installation) for the installation, as they facilitate easy installation, upgrades, and process management. After installing your first Curio binary, you can move on to [setting up Curio](/setup), whether you are [migrating from lotus-miner](/setup#migrating-from-lotus-miner-to-curio) or [initializing a new minerID](/setup#initiating-a-new-curio-cluster). ## Best Practices We have compiled [a list of best practices](/best-practices) for deploying and maintaining a Curio cluster. All users are encouraged to follow these recommendations to avoid potential issues. New users should also familiarize themselves with [both binaries shipped with Curio](/curio-cli) and the [GUI pages](/curio-gui). # Versions This is the compatibility matrix for the latest free Curio releases. | Curio Version | Lotus Version | Net | Boost | Yugabyte | Forest | | ------------------------------------------------------------ | ------------- | ------- | ---------- | ------------------- | ---------------- | | 1.22.1 / Automatic | v1.27.X | MainNet | v2.3.0-rc2 | 2.20.X / Automatic | 0.19 / Automatic | | 1.23.0 | >v1.28.1 | MainNet | v2.3.0 | 2.20.X / Automatic | 0.19 / Automatic | | v1.23.1 | >v1.28.1 | MainNet | v2.3.0 | 2.20.x / Automatic | 0.19 / Automatic | | v1.24.0 | v1.30.0-rcX | MainNet | v2.4.0-rc1 | 2.20.x / Automatic | 0.21 / Automatic | | v1.24.1 | v1.30.0-rcX | MainNet | v2.4.0-rc1 | 2.20.x / Automatic | 0.21 / Automatic | | v1.24.2 | v1.30.0 | MainNet | v2.4.0 | 2.20.x / Automatic | 0.21 / Automatic | | v1.24.3 | v1.32.0-rcX | MainNet | v2.4.1 | 2.20.x / Automatic | 0.21 / Automatic | | v1.24.4 | v1.32.0-rcX | MainNet | v2.4.1 | 2.20.x / Automatic | 0.23 / Automatic | | v1.24.5 | v1.32.0-rcX | Mainnet | v2.4.1 | v2024.2 / Automatic | 0.23 / Automatic | | v1.25.0 | v1.32.2 | Mainnet | NA | v2024.2 / Automatic | 0.25 / Automatic | | v1.25.1 | v1.33.0 | Mainnet | NA | v2024.2 / Automatic | 0.26 / Automatic | | v1.26.0 | v1.33.1 | Mainnet | NA | v2024.2 / Automatic | 0.26 / Automatic | | v1.27.0 | v1.34.0 | Mainnet | NA | v2025.1 / Automatic | 0.30 / Automatic | | v1.27.1 | v1.34.1 | Mainnet | NA | v2025.1 / Automatic | 0.30 / Automatic | | v1.27.2 | v1.34.1 | Mainnet | NA | v2025.1 / Automatic | 0.30 / Automatic | | v1.27.3 | v1.34.1 | Mainnet | NA | v2025.1 / Automatic | 0.30 / Automatic | | v1.27.4 | v1.35.1 | Mainnet | NA | v2025.1 / Automatic | 0.33 / Automatic | | v1.28.0 | v1.36.0 | Mainnet | NA | v2025.1 / Automatic | 0.33 / Automatic | | v1.28.1 | v1.36.0 | Mainnet | NA | v2025.1 / Automatic | 0.33 / Automatic | | v1.28.2 | v1.36.1 | Mainnet | NA | v2025.1 / Automatic | 0.34 / Automatic | | v1.28.3 | v1.36.2 | Mainnet | NA | v2025.1 / Automatic | 0.35 / Automatic | | v1.28.4 | v1.36.2 | Mainnet | NA | v2025.1 / Automatic | 0.35 / Automatic | {% hint style="danger" %} Releases in red color are **not recommended**. Please proceed with the next stable release. {% endhint %} No preference is denoted by "X". Configurations and the number of machines needed: A: Lotus, Curio (numerous), YugabyteDB (1 or 3), (optional Boost) B: Forest, Curio (numerous), Yugabyte (1 or 3) ## Automatic Updates * Docker has Watchtower which offers automatic updates which work for YugabyteDB and Forest. * Curio can automatically be updated on MainNet through the Debian update process on Ubuntu. * Today, only Lotus & Boost lacks automatic updates and must be built and deployed. * Curio's DEBs include curio-cuda (for Nvidia) and curio-opencl (others like ATI). * These can be mixed in a Curio cluster as they only relate to the hardware on the box. ## Database Schema Versions * When the latest Curio starts-up, it applies any upgrades & migrations to Yugabyte's schema. * This may cause errors on other nodes in your cluster that run the old version (low likelihood), which has the simple solution of completing the upgrade. * If, however, the upgrade has a serious bug and you need to downgrade, "curio toolbox downgrade --last\_good\_date=20250515" ## Notes * Forest (0.19+ & Docker Watchtower) is a light alternative to Lotus Client. It meets Curio's needs, but Boost compatibility is in development. ## Building for CalibrationNet * Required for CalibrationNet participation * Use the Go version specified in curio/GO\_VERSION\_MIN * The available Curio branches are named as release/vVERSION like: release/v1.23.4 * CalibrationNet may be a network-version ahead of MainNet. * DEBs are only for MainNet releases and will be available early so MainNet upgrades cause no interruption. # Installation This guide will show how to build, install and update Curio binaries ## Debian package installation Curio packages are available to be installed directly on Ubuntu / Debian systems. {% hint style="danger" %} Debian packages are only available for mainnet right now. For any other network like calibration network or devnet, binaries must be built from source. {% endhint %} 1. Install prerequisites ```shell sudo apt install mesa-opencl-icd ocl-icd-opencl-dev gcc git jq pkg-config curl clang build-essential hwloc libhwloc-dev wget libarchive-dev libgmp-dev libconfig++-dev protobuf-compiler -y && sudo apt upgrade -y ``` 2. Enable Curio package repo ```bash sudo wget -O /usr/share/keyrings/curiostorage-archive-keyring.gpg https://filecoin-project.github.io/apt/KEY.gpg echo "deb [signed-by=/usr/share/keyrings/curiostorage-archive-keyring.gpg] https://filecoin-project.github.io/apt stable main" | sudo tee /etc/apt/sources.list.d/curiostorage.list sudo apt update ``` 3. Install Curio binaries based on your GPU. For NVIDIA GPUs: ```bash sudo apt install curio-cuda ``` For OpenCL GPUs: ```bash sudo apt install curio-opencl ``` ## Linux Build from source You can build the Curio executables from source by following these steps. ### Software dependencies You will need the following software installed to install and run Curio. #### System-specific Building Curio requires some system dependencies, usually provided by your distribution. {% hint style="warning" %} **Note (batch sealing now builds by default on Linux):** Curio's Linux build now compiles `extern/supraseal` as part of the standard `make build` flow (needed for SnapDeals fast TreeR and batch sealing). This adds the following build requirements: * **CUDA Toolkit 12.x or later** (`nvcc` must be in PATH) * **GCC 12 or 13** — pick the version your CUDA toolkit supports: * CUDA 12.0–12.5 → `gcc-12`/`g++-12` * CUDA 12.6+ or 13+ → `gcc-13`/`g++-13` * Python venv tooling (`python3-venv`) and build tools (`autoconf`, `automake`, `libtool`, `nasm`, `xxd`) To skip the supraseal build (e.g. if you don't have CUDA), use: `make build FFI_USE_OPENCL=1` or `make build DISABLE_SUPRASEAL=1` {% endhint %} Arch: ```shell sudo pacman -Syu opencl-icd-loader gcc git jq pkg-config opencl-headers hwloc libarchive nasm xxd python python-pip python-virtualenv aria2 time protobuf # For batch sealing builds (SnapDeals fast TreeR / batch sealing toolchain): sudo pacman -Syu cuda # GCC 12 or 13 required — pick based on your CUDA version (see note above). ``` Ubuntu/Debian: ```shell sudo apt install -y \ mesa-opencl-icd ocl-icd-opencl-dev \ git jq pkg-config curl clang build-essential hwloc libhwloc-dev wget \ python3 python3-dev python3-pip python3-venv \ autoconf automake libtool \ xxd nasm \ libarchive-dev libssl-dev uuid-dev libfuse3-dev \ libnuma-dev libaio-dev libkeyutils-dev libncurses-dev \ libgmp-dev libconfig++-dev \ protobuf-compiler \ aria2 time \ && sudo apt upgrade -y # GCC 12 or 13 — pick based on your CUDA version: # CUDA 12.0–12.5: sudo apt install gcc-12 g++-12 # CUDA 12.6+/13+: sudo apt install gcc-13 g++-13 # On older Ubuntu (e.g. 20.04) you may need the toolchain PPA: # sudo add-apt-repository ppa:ubuntu-toolchain-r/test -y && sudo apt update # CUDA Toolkit (needed for GPU proving and supraseal build; nvcc must be in PATH) # Install via NVIDIA's CUDA repository for your distro. ``` Fedora: ```shell sudo dnf -y install gcc make git jq pkgconfig mesa-libOpenCL mesa-libOpenCL-devel opencl-headers ocl-icd ocl-icd-devel clang llvm wget hwloc hwloc-devel libarchive-devel protobuf-compiler aria2 time ``` OpenSUSE: ```shell sudo zypper in gcc git jq make libOpenCL1 opencl-headers ocl-icd-devel clang llvm hwloc libarchive-devel protobuf-devel aria2 time && sudo ln -s /usr/lib64/libOpenCL.so.1 /usr/lib64/libOpenCL.so ``` Amazon Linux 2: ```shell sudo yum install -y https://dl.fedoraproject.org/pub/epel/epel-release-latest-7.noarch.rpm; sudo yum install -y git gcc jq pkgconfig clang llvm mesa-libGL-devel opencl-headers ocl-icd ocl-icd-devel hwloc-devel libarchive-devel protobuf-compiler aria2 time ``` ### Rustup Curio needs [rustup](https://rustup.rs/). The easiest way to install it is: ```shell curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh ``` ### Go To build Curio, you need a working installation of [Go](https://golang.org/dl/): It needs to be at-least the version specified in go.mod. Example (match `go.mod`; current repo min is **1.26.2**): ```shell wget -c https://go.dev/dl/go1.26.2.linux-amd64.tar.gz -O - | sudo tar -xz -C /usr/local ``` {% hint style="info" %} You'll need to add `/usr/local/go/bin` to your path. For most Linux distributions you can run something like: ```shell echo 'export PATH=$PATH:/usr/local/go/bin' >> ~/.bashrc && source ~/.bashrc ``` See the [official Golang installation instructions](https://golang.org/doc/install) if you get stuck. {% endhint %} ### System Configuration Before you proceed with the installation, you should increase the UDP buffer. You can do this by running the following commands: ```shell sudo sysctl -w net.core.rmem_max=2097152 sudo sysctl -w net.core.rmem_default=2097152 ``` To persist the UDP buffer size across the reboot, update the `/etc/sysctl.conf` file. ```bash echo 'net.core.rmem_max=2097152' | sudo tee -a /etc/sysctl.conf echo 'net.core.rmem_default=2097152' | sudo tee -a /etc/sysctl.conf ``` ### Build and install Curio Once all the dependencies are installed, you can build and install Curio. 1. Clone the repository:\\ ```bash git clone https://github.com/filecoin-project/curio.git cd curio/ ``` 2. Switch to the latest stable release branch:\\ ```bash git checkout ``` 3. Enable the use of SHA extensions by adding these two environment variables: 
export RUSTFLAGS="-C target-cpu=native -g"
   export FFI_BUILD_FROM_SOURCE=1

   echo 'export RUSTFLAGS="-C target-cpu=native -g"' >> ~/.bashrc
   echo 'export FFI_BUILD_FROM_SOURCE=1' >> ~/.bashrc
   source ~/.bashrc
4. If you are using a **Nvidia GPU**, please set the below environment variables.\\ ```bash export FFI_USE_CUDA=1 export FFI_USE_CUDA_SUPRASEAL=1 echo 'export FFI_USE_CUDA=1' >> ~/.bashrc echo 'export FFI_USE_CUDA_SUPRASEAL=1' >> ~/.bashrc source ~/.bashrc ```

On Linux, the Curio build requires CUDA by default and will fail if nvcc is not found in your PATH. This ensures you don't accidentally build without proper GPU support.

If you want to use OpenCL instead of CUDA (e.g., for AMD GPUs or systems without CUDA), build with:

FFI_USE_OPENCL=1 make clean build
5. Curio is compiled to operate on a single network. Choose the network you want to join, then run the corresponding command to build the Curio node:\\ ```shell # For Mainnet: make clean build # For Calibration Testnet: make clean calibnet ``` 6. Install Curio. This will put `curio` in `/usr/local/bin`. `curio` will use the `$HOME/.curio` folder by default. ```shell sudo make install ``` 7. Run `curio --version` ```md curio version 1.24.5+mainnet+git_214226e7_2025-02-19T17:02:54+04:00 # or curio version 1.24.5+calibnet+git_214226e7_2025-02-19T17:02:54+04:00 ``` You should now have Curio installed. You can now [finish setting up the Curio node](/setup). ## MacOS Build from source You can build the Curio executables from source by following these steps. ### Software dependencies You must have XCode and Homebrew installed to build Curio from source. #### **XCode Command Line Tools** Curio requires that X-Code CLI tools be installed before building the Curio binaries. Check if you already have the XCode Command Line Tools installed via the CLI, run: ```shell xcode-select -p ``` This should output something like: ```plaintext /Library/Developer/CommandLineTools ``` If this command returns a path, then you have Xcode already installed! You can [move on to installing dependencies with Homebrew](#homebrew). If the above command doesn't return a path, install Xcode: ```shell xcode-select --install ``` Next up is installing Curio's dependencies using Homebrew. ### **Homebrew** We recommend that macOS users use [Homebrew](https://brew.sh/) to install each of the necessary packages. Use the command `brew install` to install the following packages: ```shell brew install jq pkg-config hwloc coreutils brew install go@1.24 ``` Next up is cloning the Lotus repository and building the executables. ### **Rust** Rustup is an installer for the systems programming language Rust. Run the installer and follow the onscreen prompts. The default installation option should be chosen unless you are familiar with customisation: ```shell curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh ``` ### Build and install Curio The installation instructions are different depending on which CPU is in your Mac: * [ARM-based CPUs (M1, M2, M3)](#arm-based-cpus) * [Intel CPUs](#intel-cpus) #### **Arm based CPUs** 1. Clone the repository: ```shell git clone https://github.com/filecoin-project/curio.git cd curio/ ``` 2. Switch to the latest stable release branch: ```shell git checkout ``` 3. Create the necessary environment variables to allow Curio to run on Arm architecture: ```shell export LIBRARY_PATH=/opt/homebrew/lib export FFI_BUILD_FROM_SOURCE=1 export PATH="$(brew --prefix coreutils)/libexec/gnubin:/usr/local/bin:$PATH" ``` 4. Build the `curio` binary: ```shell make clean curio ``` 5. Run the final `make` command to move this `curio` executable to `/usr/local/bin`. This allows you to run `curio` from any directory. ```shell sudo make install ``` 6. Run `curio --version` ```md curio version 1.24.5+mainnet+git_214226e7_2025-02-19T17:02:54+04:00 # or curio version 1.24.5+calibnet+git_214226e7_2025-02-19T17:02:54+04:00 ``` 7. You should now have Curio installed. You can now [finish setting up the Curio node](/setup). #### **Intel CPUs** ❗These instructions are for installing Curio on an Intel Mac. If you have an Arm-based CPU, use the [Arm-based CPU instructions ↑](#arm-based-cpus) 1. Clone the repository: ```shell git clone https://github.com/filecoin-project/curio.git cd curio/ ``` 2. Switch to the latest stable release branch: ```shell git checkout ``` 3. Build and install Curio: ```shell make clean curio sudo make install ``` 4. Run `curio --version` ```md curio version 1.23.0+mainnet+git_ae625a5_2024-08-21T15:21:45+04:00 # or curio version 1.23.0+calibnet+git_ae625a5_2024-08-21T15:21:45+04:00 ``` You can now [finish setting up the Curio node](/setup). # Setup This guide will show you setup a new Curio cluster or migrate to Curio from Lotus-Miner ## Setup YugabyteDB {% hint style="warning" %} If you have already set up a YugabyteDB for Boost then you can reuse the same YugabyteDB instance for Curio. You must ensure that YugabyteDB is multiple node cluster for HA. You can skip directly to [migrating from Lotus-Miner to Curio](#migrating-from-lotus-miner-to-curio) or [Initializing new Curio Miner.](#initiating-a-new-curio-cluster) {% endhint %} For this guide, we’re setting up a single node YugaByteDB. However, you must set up multiple YugaByteDB instances in a cluster to enable high availability. {% hint style="danger" %} **Note:** We do **not recommend** using a single-node YugabyteDB setup in production. The instructions below are intended **only for testing and development purposes**. For production deployments, please refer to the official [YugabyteDB documentation](https://docs.yugabyte.com/preview/deploy/manual-deployment/) for setting up a highly available (HA) cluster. {% endhint %} Ensure that you have the following available before we install and set up YugabyteDB: {% hint style="danger" %} **Do not use ZFS as the backing drive for YugabyteDB because of advanced filesystem commands so-far is unavailable.** {% endhint %} 1. One of the following operating systems: * CentOS 7 or later * Ubuntu 16.04 or later For other operating systems, Docker or Kubernetes. Please check out the [YugabyteDB documentation](https://docs.yugabyte.com/preview/quick-start/). 2. **Python 3.** To check the version, execute the following command:
```shell python --version ``` ```undefined Python 3.7.3 ``` \ If you encounter a `Command 'python' not found` error, you might not have an unversioned system-wide python command. * Starting from Ubuntu 20.04, python is no longer available. To fix this, run `sudo apt install python-is-python3`. * For CentOS 8, set `python3` as the alternative for python by running `sudo alternatives --set python /usr/bin/python3` Once these dependencies have been installed, we can run the install script:
```shell # YugabyteDB (single-node dev/test example) # NOTE: For production/HA, follow YugabyteDB’s official deployment docs. wget https://software.yugabyte.com/releases/2.25.1.0/yugabyte-2.25.1.0-b381-linux-x86_64.tar.gz tar xvfz yugabyte-2.25.1.0-b381-linux-x86_64.tar.gz && cd yugabyte-2.25.1.0/ ./bin/post_install.sh ./bin/yugabyted start --advertise_address 127.0.0.1 --master_flags rpc_bind_addresses=127.0.0.1 --tserver_flags rpc_bind_addresses=127.0.0.1 ``` ```yaml +----------------------------------------------------------------------------------------------------------+ | yugabyted | +----------------------------------------------------------------------------------------------------------+ | Status : | | Replication Factor : None | | YugabyteDB UI : http://127.0.0.1:15433 | | JDBC : jdbc:postgresql://127.0.0.1:5433/yugabyte?user=yugabyte&password=yugabyte | | YSQL : bin/ysqlsh -U yugabyte -d yugabyte | | YCQL : bin/ycqlsh -u cassandra | | Data Dir : /root/var/data | | Log Dir : /root/var/logs | | Universe UUID : 411422ee-4c17-4f33-996e-ced847d10f5c | +----------------------------------------------------------------------------------------------------------+ ``` You can adjust the `--advertise_address`, `--rpc_bind_addresses` and `--tserver_flags` according to your own configuration and needs. ## Migrating from Lotus-miner to Curio Curio provides a utility to users onboard quickly. Please run the below command on your `lotus-miner` node and follow the on-screen instructions. ```shell curio guided-setup ``` Once the migration is complete, you can shut down all of your workers and miner processes. You can start `curio` process to replace them with correct [configuration layer](/configuration#configuration-layers). If you entered non-default in step 3 then please export the relevant details in your `~/.bashrc` file as Curio command needs access to the Database. This step is different from the
| Env Variable | UseCase | | ------------------- | ------------------------- | | CURIO\_DB\_HOST | YugabyteDB SQL IP | | CURIO\_DB\_NAME | YugabyteDB Name | | CURIO\_DB\_USER | DB user for connection | | CURIO\_DB\_PASSWORD | User’s password | | CURIO\_DB\_PORT | YugabyteDB’s SQL port | | CURIO\_REPO\_PATH | Curio’s default repo path | Please proceed to [curio service configuration](/curio-service) for the first node. Once you have completed the service and storage configuration, you can come back to this page and proceed with testing the setup. ### Testing the setup You can confirm that the `curio` process is able to schedule and compute WindowPoSt by running a WindowPoSt test computation: ```shell curio test window-post task ``` From the output we can confirm that a WindowPoSt gets inserted to the database, and is being picked up by the Curio process running with the *wdpost* configuration layer. ## Initiating a new Curio cluster To create a new Curio cluster, a [Lotus daemon node](https://bafybeib7hujkpoqohpby6dqabdea2t6ehcysics3ejoh4jrgtuke4rmolu.on.fleek.co/lotus/install/prerequisites/) is required. {% hint style="warning" %} The Lotus daemon node must be part of the same network as Curio being setup. Example: A `calibration` network daemon cannot be used with `mainnet` Curio cluster. {% endhint %} ### Wallet setup Initiating a new miner ID on the Filecoin network requires an owner, worker and sender address. These address can be same or different depending on the user’s choice. Users must create these wallet on Lotus node before running the Curio commands. ```shell lotus wallet new bls lotus wallet new bls ``` Once new wallet are created, we must send some funds to them. ```shell lotus send 5 lotus send 5 ``` ### Creating new miner ID Curio provides a utility for users to onboard quickly. Please run the below command on your new Curio node, choose `Create a new miner` option and follow the on-screen instructions. 1. Start the guided setup.
```shell curio guided-setup ``` 2. Choose “Create a new miner” option.
``` Defaulting to English. Please reach out to the Curio team if you would like to have additional language support. Use the arrow keys to navigate: ↓ ↑ → ← ? I want to:: Migrate from existing Lotus-Miner ▸ Create a new miner ``` 3. Enter your YugabyteDB details.
``` This process is partially idempotent. Once a new miner actor has been created and subsequent steps fail, the user need to run 'curio config new-cluster < miner ID >' to finish the configuration. Use the arrow keys to navigate: ↓ ↑ → ← ? Enter the info to connect to your Yugabyte database installation (https://download.yugabyte.com/): ▸ Host: 127.0.0.1 Port: 5433 Username: yugabyte Password: yugabyte Database: yugabyte Continue to connect and update schema. ✔ Step Complete: Pre-initialization steps complete ``` 4. Enter the wallet details be used for “create miner” message.
``` Initializing a new miner actor. Use the arrow keys to navigate: ↓ ↑ → ← ? Enter the info to create a new miner: ▸ Owner Address: <------ Enter wallet 1 here Worker Address: <------ Enter wallet 2 here Sender Address: <------ Enter wallet 1 here Sector Size: 0 <--------------- Sector Size (32 G/GiB/GB) Confidence epochs: 0 Continue to verify the addresses and create a new miner actor. ``` ``` Initializing a new miner actor. ✔ Owner Address: Enter the owner address: t3weiymrx3iyivzeuub5l232gb62ocu7zbjtztudiipm6wkkmsehdydrdddm6cdrflxir26cmrz4xui6t5gruq ✔ Worker Address: Enter worker address: t3xhmgfxurecrusgubzdgme4t2ecxbiyny5uanfzvcrrihzhia654f6gp2ynugpiyp5xe7ibg6fqly76kowfva ✔ Sender Address: Enter sender address: t3weiymrx3iyivzeuub5l232gb62ocu7zbjtztudiipm6wkkmsehdydrdddm6cdrflxir26cmrz4xui6t5gruq ✔ Sector Size: 0 Enter the sector size: 8 MiB ✔ Confidence epochs: 0 Confidence epochs: 0 Pushed CreateMiner message: bafy2bzacebu3mhaj6chnz5frjo2sbxduebnh4e7e37fwm3jd7xhvhla7t6ylu Waiting for confirmation ``` 5. Wait for new miner actor to get created.
``` New miners address is: t01004 (t2cmgqvicpcil5zlp6bqsffmjjfz7ix66k4zaojay) ✔ Step Complete: Miner t01004 created successfully ✔ Step Complete: Configuration 'base' was updated to include this miner's address ``` 6. We request you to please share the basic data about your miner with us to help us improve Curio.
``` The Curio team wants to improve the software you use. Tell the team you're using `curio`. Use the arrow keys to navigate: ↓ ↑ → ← ? Select what you want to share with the Curio team.: ▸ Individual Data: Miner ID, Curio version, chain (mainnet or calibration). Signed. Aggregate-Anonymous: version, chain, and Miner power (bucketed). Hint: I am someone running Curio on whichever chain. Nothing. ``` 7. Finish the initialisation.
``` ✔ Step Complete: New Miner initialization complete. Try the web interface with curio run --layers=gui for further guided improvements. ``` 1. If you entered non-default in step 3 then please export the relevant details via your `~/.bashrc` file before running the Curio command.
| Env Variable | UseCase | | ------------------- | ------------------------- | | CURIO\_DB\_HOST | YugabyteDB SQL IP | | CURIO\_DB\_NAME | YugabyteDB Name | | CURIO\_DB\_USER | DB user for connection | | CURIO\_DB\_PASSWORD | User’s password | | CURIO\_DB\_PORT | YugabyteDB’s SQL port | | CURIO\_REPO\_PATH | Curio’s default repo path | 8. Try running Curio with only `GUI` first.
```shell curio run --layers gui ``` 9. If the `curio` process starts successfully, please proceed with GUI and verify that you can access all the pages. Once, verified, please proceed to [curio service configuration](/curio-service). # Troubleshooting Troubleshooting and operational playbooks for common Curio issues. This section is built from real operator issues seen in `#fil-curio-help`, with notes tied back to the actual Curio code paths. Start here if you are: * stuck during setup/migration, * seeing task failures in the UI, * seeing repeated errors/warnings in logs, * unsure what information to collect before asking for help. ## A simple mental model (helps you debug faster) Most Curio incidents fall into one of these buckets: 1. Chain / Lotus connectivity * Curio can’t talk to the full node, or the full node is unhealthy/out of sync. 2. Control plane (DB + scheduler) * Yugabyte is slow/unhealthy, schema upgrades failed, or tasks are not being scheduled/claimed. 3. Data plane (storage + retrieval) * the bytes aren’t where the system thinks they are (paths, indexes, URLs, permissions). If you don’t know where to start: check **DB health** first. A sick DB creates “random” symptoms everywhere. ## Quick links * [Collect debug info](/troubleshooting/collect-debug-info) (copy/paste checklist) * [Common errors](/troubleshooting/common-errors) (error string → meaning → next step) * [Market / indexing troubleshooting](/curio-market/troubleshooting) * [YugabyteDB troubleshooting](/administration/yugabyte-troubleshooting) # Collect debug info What to collect before reporting a Curio issue. When asking for help, posting the right info up front saves *hours*. Copy/paste this checklist and fill it in. ## Environment * Curio version: ```bash curio --version ``` * Deployment type: * systemd / docker / k8s / other * Enabled layers (and where you set them): * `/etc/curio.env` (`CURIO_LAYERS=...`) or docker env / service unit ## Config snapshot If you use layered config, include the layer list and the relevant sections. * If available in your build: ```bash curio config view --layers ``` * Otherwise: paste the relevant layer TOML sections (Subsystems/Fees/Market/Ingest/HTTP/etc.). ## Logs (include context) * systemd: ```bash journalctl -u curio -n 2000 --no-pager ``` * docker: ```bash docker logs --tail 2000 ``` Include: * the **exact error line** * \~30–100 lines before/after ## DB connectivity and health Confirm the Curio host can reach Yugabyte (YSQL): ```bash ysqlsh -h "$CURIO_DB_HOST" -p "${CURIO_DB_PORT:-5433}" -U "$CURIO_DB_USER" -d "${CURIO_DB_NAME:-yugabyte}" -c "select 1;" ``` If you have multiple DB hosts, include the host list and whether load-balancing is enabled. ## UI / task context * Screenshot of failing task(s) * The task ID(s) from the UI ### Sealing issues Include: * `sp_id` (miner id) * sector number(s) * which stage (SDR/TreeD/PC1/PC2/C2/PreCommit/Commit/WdPoSt) ### Market / deal ingestion issues Include: * deal UUID * piece CID * whether offline/online * the URL you provided (if any) and headers (redact secrets) * indexing status (IPNI enabled? CheckIndex errors?) ### PDP issues Include: * PDP endpoint URL you are testing * the command used (e.g. `pdptool ping ...`) * whether you are testing locally or from outside the network ## Redaction Before posting publicly: * redact API tokens, passwords, and private IPs (if needed) * keep message CIDs / actor IDs (they’re useful for debugging) # Common errors Common Curio errors seen in support, with meaning and next steps. This page maps frequent error strings to: * what they usually mean (in plain language), * whether they are benign vs urgent, * what to do next. If your error isn’t listed, start with [Collect debug info](/troubleshooting/collect-debug-info) and include logs + context. Tip: many errors are symptoms, not causes. If the same task keeps failing, scroll **up** to find the first error in the chain. *** ### API / CLI #### `ERROR: could not get API info for Curio: could not determine API endpoint ... Try setting environment variable: CURIO_API_INFO` What it means: * Your CLI environment can’t find the Curio API endpoint/token. What to do: * If running via systemd, env vars may exist in the service unit but not in your shell. * Export `CURIO_API_INFO` in the shell where you run `curio cli ...`, or run the command on the node/container where Curio is running. *** ### Chain / Lotus connectivity #### `Not able to establish connection to node with addr: ws://.../rpc/v1` What it means: * Curio can’t reach the configured full node endpoint. What to do: * Verify connectivity from the Curio host (DNS, firewall, port). * Confirm Lotus is synced and serving RPC. * Check the configured endpoint in your layers and restart Curio after changes. *** ### Sealing / PoSt #### `invalid partIdx 0 (deadline has 0 partitions)` Plain-English meaning: * Curio asked Lotus for “deadline partitions” (groups of sectors) and got **zero** partitions back. Then it tried to run a proof against partition index `0`, which doesn’t exist. When it’s benign: * Fresh miner / devnet / early migration: you simply don’t have proving sectors yet, so there are no partitions. When it’s not: * You *expect* proving sectors, but Lotus reports none (possible chain sync issue, wrong miner address, wrong network, or a migration/config mismatch). What to do: 1. Confirm your miner actually has proving sectors (via Lotus commands / UI). 2. Confirm Curio is pointing at the right Lotus node and the right network. 3. If this is a `curio test window-post` run, remember it is a *diagnostic*; it will fail if there’s nothing meaningful to prove. Code reference (for maintainers): `tasks/window/compute_do.go`. #### `Error computing WindowPoSt ... no sectors to prove` What it means: * There are no sectors in that deadline/partition that require proofs. Next steps: * Confirm there are proving sectors and deadlines populated. * If you expect proofs: look earlier in the pipeline for commit/precommit failures. *** ### Storage / paths #### `sector {..} redeclared in ... paths/db_index.go` What it suggests: * The same sector appears declared in multiple storage locations, or your local store index contains conflicting entries. What to do: * Verify storage attach/mount configuration. * Confirm there aren’t duplicated mount paths or reused repo paths between nodes. * If you recently moved storage, ensure the old path is detached/removed. *** ### Market / indexing #### `duplicate key value violates unique constraint ... (SQLSTATE 23505)` during indexing / CheckIndex What it means: * Two workers attempted to insert the same identity row (often a retry/concurrency artifact). What to do: * Determine if the pipeline is *progressing* despite the error (harmless noise) or if tasks are stuck. * Collect: deal UUID, piece CID, task ID, and the full log span. * If stuck: see [curio-market troubleshooting](/curio-market/troubleshooting). #### `no suitable data URL found for piece_id ` (ParkPiece) Plain-English meaning: * Curio has a “parked piece” that it needs to read, but it can’t find any usable location for the bytes. * In most setups, that location is an HTTP(S) **data URL** (and optional headers) stored in the DB. Where it happens: * The ParkPiece workflow reads `parked_piece_refs` rows for the piece. Common causes: * The data URL was never added. * The data URL exists but is malformed (bad scheme) or requires headers that weren’t stored. * The task is running on a node that is *not* actually responsible for ingestion (layers mismatch). What to do (operator workflow): 1. Find how the deal was supposed to be ingested: * online (URL) vs offline (local file / piece locator) 2. Verify you actually added a data URL for the deal (see Storage Market docs). 3. Confirm the node running ParkPiece has the **market/ingest** layers enabled. 4. Collect: deal UUID, piece CID, and the ParkPiece task ID. Docs: * See `curio-market/storage-market.md` (“Add data URL for offline deals”). Code reference (for maintainers): `tasks/piece/task_park_piece.go`. *** ### Pipeline #### Sectors stuck mid-pipeline after cordoning a node What it means: * Cordoning blocks all new task scheduling, including location-bound pipeline stages (TreeD, TreeRC, SyntheticProofs, Finalize). If sector data lives on the cordoned node's local storage, no other node can pick up those tasks. What to do: * Uncordon the node to let pipelines resume, or attach storage to another node. * See: [Node Maintenance & Cordoning](/administration/node-maintenance) for recommended workflows. *** ### Batch sealing #### `panic: SupraSealInit: supraseal build tag not enabled` What it means: * The binary you’re running does not include batch sealing support. What to do: * Prefer official release binaries. * If building from source, follow the batch sealing build instructions and ensure required build flags/features are enabled. ### Batch commit failures: `all-or-nothing ... Batch successes 67/68 ... idx=N` What it means: * Batch submission/claiming is atomic: one failing item can cause the batch to fail. What to do: * Identify the failing sector/task corresponding to `idx=N` (UI + logs). * Check wallet balance / message errors. * Consider temporarily reducing batch size/concurrency to isolate the failing entry. (Expanded workflow will live in the Batch Sealing page’s troubleshooting section.) *** ### YugabyteDB #### `Could not upgrade! ...` / `Rewriting of YB table is not yet implemented (SQLSTATE 0A000)` What it means: * You hit a schema migration or table rewrite Yugabyte can’t perform in-place for your version. What to do: * Verify Yugabyte version compatibility with your Curio release. * Always back up before upgrades/downgrades. * Collect the migration filename and the error. See: [YugabyteDB troubleshooting](/administration/yugabyte-troubleshooting) # Curio Service This page explains how to setup a systemd service for Curio Curio can handle multiple GPUs simultaneously without needing to run multiple instances of the Curio process. Therefore, Curio can be managed as a single systemd service without concerns about GPU allocations. ## Systemd Service Configuration The service file for Curio is included in the Debian package and is named `curio.service`. If you have built Curio from source, you can create the service file manually as described below. ### **Service File for Curio** To create the `curio.service` file manually, use the following content: ```ini [Unit] Description=Curio After=network.target [Service] ExecStart=/usr/local/bin/curio run Environment=GOLOG_FILE="/var/log/curio/curio.log" Environment=GOLOG_LOG_FMT="json" LimitNOFILE=1000000 Restart=always RestartSec=10 EnvironmentFile=/etc/curio.env RestartForceExitStatus=100 [Install] WantedBy=multi-user.target ``` ### Environment Variables Configuration The service file requires an `/etc/curio.env` file to be present. This file contains all the necessary environment variables to connect to the database. The `env` file should be created automatically during the Debian package installation. If you are running Curio built from source, you can create the `env` file manually with the following content: **/etc/curio.env File** ```sh CURIO_LAYERS=gui,post CURIO_ALL_REMAINING_FIELDS_ARE_OPTIONAL=true CURIO_DB_HOST=yugabyte1,yugabyte2,yugabyte3 CURIO_DB_USER=yugabyte CURIO_DB_PASSWORD=yugabyte CURIO_DB_PORT=5433 CURIO_DB_NAME=yugabyte CURIO_DB_CASSANDRA_PORT=9042 CURIO_REPO_PATH=~/.curio CURIO_NODE_NAME=ChangeMe FIL_PROOFS_USE_MULTICORE_SDR=1 ``` | Variable | Description | Example Value | | ----------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------- | | `CURIO_LAYERS` | Config layers to be stacked and used for this Curio node | `gui,post` | | `CURIO_ALL_REMAINING_FIELDS_ARE_OPTIONAL` | Allows optional fields to use defaults. This ensures that missing configurations are not treated as errors, instead, they will be assigned defaults | `true` | | `CURIO_DB_HOST` | Database hosts (comma-separated) for the YugabyteDB cluster | `yugabyte1,yugabyte2,yugabyte3` | | `CURIO_DB_USER` | Username for authenticating with YugabyteDB's Postgres Database defined by `CURIO_DB_NAME` | `yugabyte` | | `CURIO_DB_PASSWORD` | Password for the specified CURIO\_DB\_USER | `yugabyte` | | `CURIO_DB_PORT` | Port for YugabyteDB Postgres connection | `5433` | | `CURIO_DB_NAME` | Name of the PostgreSQL database in YugabyteDB | `yugabyte` | | `CURIO_DB_CASSANDRA_PORT` | Port for YugabyteDB Cassandra connection | `9042` | | `CURIO_REPO_PATH` | Directory for Curio storage configuration `json` file | `~/.curio` | | `CURIO_NODE_NAME` | Name of the Curio node | `ChangeMe` | | `FIL_PROOFS_USE_MULTICORE_SDR` | Enables multi-core SDR in Filecoin | `1` | | `FIL_PROOFS_MULTICORE_SDR_PRODUCERS` | Optional. Controls the number of multicore SDR producer threads. Valid values are `1` to `3`; SDR CPU cost is derived from this value when set. | `3` | Ensure all variables are correctly set according to your environment. Additionally, you can also export the following variable for cache location. ```sh FIL_PROOFS_PARAMETER_CACHE=/path/to/folder/in/fast/disk FIL_PROOFS_PARENT_CACHE=/path/to/folder/in/fast/disk2 ``` ## Starting the Curio Service Once all the variables are correctly updated, create the log directory: ```sh mkdir -p /var/log/curio ``` Now, you can start the systemd service with the following command: ```sh sudo systemctl start curio.service ``` Verify that process started successfully by monitoring `systemctl status curio.service` Once Curio service is running, you can proceed to [attaching storage for sealing or permanent storage to the Curio node](/storage-configuration) or [setting up next Curio node in the cluster](/scaling-curio-cluster). # Storage Configuration This guide describes how to attach and configure sealing and permanent storage for Curio nodes Each Curio node keeps track of defined storage locations in `~/.curio/storage.json` (or `$CURIO_REPO_PATH/storage.json`) and uses `~/.curio` path as default. Upon initialization of a storage location, a `/sectorstorage.json` file is created that contains the UUID assigned to this location, along with whether it can be used for sealing or storing. ## Adding sealing storage location Before adding your sealing storage location you will need to consider where the sealing tasks are going to be performed. This command must be run locally from the Curio node where you want to attach the storage. ```sh curio cli storage attach --init --seal OR curio cli --machine storage attach --init --seal ``` {% hint style="info" %} The `--machine` flag accepts an `` input, specifying the machine on which the command should be executed. The **IP** refers to the address of the machine where the Curio process is running, and the **Port** indicates the port on which the Curio process is listening. By default, Curio operates on **port 12300**. This flag enables Curio commands to be executed on remote nodes, facilitating centralized administration from a single node. {% endhint %} ## Adding long-term storage location **Custom location for storing:** After the *sealing* process is completed, sealed sectors are moved to the *store* location, which can be specified as follows: ```sh curio cli storage attach --init --store OR curio cli --machine storage attach --init --store ``` This command must be run locally from the Curio node where you want to attach the storage. This location can be made of large capacity, albeit slower, spinning-disks. ## Attach existing storage to Curio The storage location used by `lotus-miner` or `lotus-worker` can be reused by the Curio cluster. It can be attached once the migrated `lotus-miner` or `lotus-worker` is already running a Curio service. ## Common errors (from support) ### `permission denied` when attaching storage Example error: * `mkdir '/key': permission denied` Plain-English meaning: * Curio writes small metadata into the storage path when attaching/initializing it (for example `sectorstorage.json` and a `key/` directory). If Curio can’t write there, attach will fail. Fix checklist: 1. Confirm the filesystem is mounted **read-write**. 2. Ensure the Curio service user owns (or can write to) the directory. 3. If you use `curio cli --machine ... storage attach ...`, the operation is executed on the *remote machine*, so permissions must be correct there. ### Stale/ghost storage endpoints in `storage list` Symptom: * `curio cli storage list` shows IDs/paths with errors like “all endpoints failed for remote storage ”. Meaning: * Curio still remembers a storage entry, but the endpoint is no longer reachable (retired node, IP change, firewall). Safe cleanup options: * Preferred: run `curio cli storage detach --really-do-it ` on the machine where that path was attached. * If the path is gone and detach can’t work: last resort is editing `~/.curio/storage.json` (or `$CURIO_REPO_PATH/storage.json`). Warning: * Removing the wrong entry can make sectors look “missing” until you re-attach the correct storage path. ```sh curio cli storage attach OR curio cli --machine storage attach ``` ## Filter sector types You can filter for what sectors types are allowed in each sealing path by adjusting the configuration file in: `/sectorstorage.json`. ```json { "ID": "1626519a-5e05-493b-aa7a-0af71612010b", "Weight": 10, "CanSeal": false, "CanStore": true, "MaxStorage": 0, "Groups": [], "AllowTo": [], "AllowTypes": null, "DenyTypes": null } ``` Valid values for `AllowTypes` and `DenyTypes` are: ```javascript "unsealed" "sealed" "cache" "update" "update-cache" ``` These values must be put in an array to be valid (e.g `"AllowTypes": ["unsealed", "update-cache"]`), any other values will generate an error on startup of the `Curio`. A restart of the `Curio` node where this storage is attached is also needed for changes to take effect. ## Separate sealed and unsealed A very basic setup where you want to separate unsealed and sealed sectors could be achieved by: * Add `"DenyTypes": ["unsealed"]` to long-term storage path(s) where you want to store the sealed sectors. * Add `"AllowTypes": ["unsealed"]` to long-term storage path(s) where you want to store the unsealed sectors. Setting only `unsealed` for `AllowTypes` will still allow `cache` and `update-cache` files to be placed in this storage path. If you want to completely deny all other types of sectors in this path, you can add additional valid values to the `"DenyTypes"` field. {% hint style="info" %} If there are existing files with disallowed types in a storage path, those files will remain readable for PoSt/Retrieval. So the worst that can happen in case of misconfiguration in the storage path is that sealing tasks will get stuck waiting for storage to become available. {% endhint %} ## Segregating long-term storage per miner Users can allocate long-term storage to specific miner IDs by specifying miner address strings in the `sectorstore.json` file. This configuration allows precise control over which miners can use the storage. * To allow specific miners, include their addresses in the AllowMiners array: ``` "AllowMiners": ["t01000", "t01002"] ``` This configuration permits only the listed miners (t01000 and t01002) to access the storage. * Similarly, to deny specific miners access to the storage, include their addresses in the DenyMiners array: ``` "DenyMiners": ["t01003", "t01004"] ``` In this example, miners with addresses t01003 and t01004 are explicitly denied access to the storage. This dual configuration approach allows for flexible and secure management of storage access based on miner IDs.
# Configuration How to edit and manage configuration for a Curio cluster ## Configuration The configuration for Curio is stored in the HarmonyDB in a table called `harmony_config`. When a Curio node is started, one or more layer names are supplied to get the desired configuration for the node. ### Configuration Layers Configuration layers provide a set of configuration parameters that determine how a system will operate. These layers can be defined at different levels, meaning that higher layer can override the lower layer, and the system will behave based on the final stacked output. Configuration layers can be arranged in a hierarchy, often from `base`(most general) to most specific. The `base` layer defines default configuration values. More specific layers override these defaults with more targeted configurations. For example, in a simple two-layer configuration system, layers could be organized in the following order: Base Layer - This is the most general layer. It is always included, so add any modifications to defaults here. If you include all your miner IDs here (defined in the addresses section) all hardware will be used for all miner needs. Task Layer - This layer enables SDR tasks. Consider the included layers (below). If a Curio node is started with above 2 layers then it will perform SDR tasks for all miner IDs and will use default values of any other configuration parameter. ``` Example Use of Layers: curio run --layers=post ``` {% hint style="warning" %} `base` layer is always applied by default when a Curio node is started. {% endhint %} #### Advantages of Configuration Layers Flexibility: Configuration layers allow different parts of the system or different users to behave differently according to predefined settings. Scalability: By separating concerns and allowing for specific configuration, the systems become easier to manage as they scale. Maintainability: Changes to configuration can be made on an appropriate layer without affecting the entire system. #### Layer Stacking The configuration layers are stack in the supplied order. The `base` layer is always applied by default so you don't need to specify it. For example, if a Curio node is started with the following layers: ``` --layers miner1,sdr,wdPost,pricing ``` These layer will be stacked on top of each other to create the final configuration for the node. The order of stacking will base > miner1 > sdr > wdPost > pricing. If a configuration parameter is defined in multiple layers then the final layer value will be used. #### Working with layers Curio allows you to manage node configurations using layers. Each layer can be applied or modified independently, with the ‘base’ layer being essential at startup. **Print default configuration** The default configuration is used in base layer by default. ```shell curio config default ``` **Adding a New Layer** To add a new configuration layer or update an existing one, you can provide a filename or input directly via stdin. ```shell curio config set --title ``` **List all layers** List all configuration layers present in the database. ```shell curio config ls ``` **Editing a Layer** Directly edit a configuration layer. * Edit with `vim` editor\\ ```shell curio config edit --editor vim ``` * Edit with a different editor like nano\\ ```shell curio config edit --editor nano ``` **Interpreting Stacked Layers** Interpret and view the combined effect of all applied configuration layers, including system-generated comments. ```shell curio config view --layers [layer1,layers2] ``` **Removing a Layer** Remove a specific configuration layer by name. ```shell curio config rm ``` #### Pre-built Layers When the first Curio miner is initialized or when the first Lotus-Miner is migrated to Curio, the process creates some layers by default for the users. These layers mostly define if a particular task should be picked by the machine or not. **post** ```toml [Subsystems] EnableWindowPost = true EnableWinningPost = true ``` **sdr** ```toml [Subsystems] EnableSealSDR = true ``` **seal** ```toml [Subsystems] EnableSealSDR = true EnableSealSDRTrees = true EnableSendPrecommitMsg = true EnablePoRepProof = true EnableSendCommitMsg = true EnableMoveStorage = true ``` **seal-gpu** ```toml [Subsystems] EnableSealSDRTrees = true EnableSendPrecommitMsg = true ``` **seal-snark** ```toml [Subsystems] EnablePoRepProof = true EnableSendCommitMsg = true ``` **gui** ```toml [Subsystems] EnableWebGui = true ``` ### Configuration management in UI The Curio GUI provides a user-friendly interface for managing configurations. To access this feature, navigate to the “Configurations” page from the UI menu. On this page, all available layers in the database are listed. Users can edit each layer by clicking on it.
Configurations

Curio GUI configuration page

To update a configuration field, users must first enable it by checking the corresponding box. After enabling, the field value can be populated. To comment out the field, simply uncheck the box.
Configuration edit

Curio GUI configuration editor

### Dynamic Configuration Updates Historically, **all configuration changes required a full server restart**, impacting uptime and SLA.\ This is no longer the case — **a growing number of settings can now be updated on the fly**, without restarting any node. More dynamic options will be added over time. **Identifying Hot-Reloadable Settings** In the UI and documentation, these settings are clearly marked with: > **“Updates will affect running instances”** When updated (via UI, CLI, or direct SQL), they are applied automatically within **\~30 seconds**. #### **Behaviour & Caveats** * **Invalid values apply immediately** (e.g., unparsable or out-of-range values) and may cause system-wide disruption.\ They can be corrected by pushing a valid update, but there is **no safety delay or rollback**. * A few configuration changes still **require a restart** — typically structural or first-time operations\ (e.g., adding the first miner address while Market 2 is active).\ In these cases the system will **clearly return an error stating that a restart is required**. #### **Quick Reference** | Type of Change | Restart Needed? | Notes | | ------------------------------------------------------- | ---------------- | ------------------------------------------- | | Marked with **“Updates will affect running instances”** | ❌ No | Takes effect in \~30s | | Bad or invalid value | ❌ No, but unsafe | Applies instantly, may break services | | Structural / unsupported change (rare) | ✅ Yes | System explicitly errors and refuses update | This gradual shift to dynamic configuration reduces downtime, shortens maintenance windows, and improves operational flexibility. # Listen Address How to update default listen address for Curio service By default, all Curio nodes bind to the address "0.0.0.0" on port "12300," ensuring that the Curio API listens on all interfaces. You can change this behavior by specifying an explicit IP address and a different port for Curio to use. ```bash echo "CURIO_LISTEN=x.x.x.x:12301" >> /etc/curio.env ``` and restart the Curio service ```bash systemctl restart curio.service ``` # Prometheus Metrics How to configure Prometheus to scrape metrics from Curio Curio exposes a comprehensive set of metrics via a built-in Prometheus endpoint. These metrics provide visibility into database performance, task execution, resource utilization, and more. ## Metrics Endpoint Curio exposes metrics at the `/debug/metrics` endpoint on the RPC server: ``` http:///debug/metrics ``` The endpoint is automatically enabled on all Curio nodes—no configuration is required to expose basic metrics. All metrics are prefixed with the `curio_` namespace. ## Prometheus Configuration To scrape metrics from Curio, add a job to your Prometheus configuration: ```yaml scrape_configs: - job_name: 'curio' static_configs: - targets: - 'curio-node-1:12300' - 'curio-node-2:12300' - 'curio-node-3:12300' # Optional: adjust scrape interval scrape_interval: 15s ``` Replace the targets with your actual Curio node addresses and ports. ### Service Discovery Curio also provides a service discovery endpoint at `/debug/service-discovery` which can be used with Prometheus file-based service discovery for dynamic node discovery. ## Available Metrics For a complete list of all exported metrics, see the [**Metrics Reference**](/configuration/metrics-reference). Key metric categories include: * **Database (HarmonyDB)**: Query latency, connection pool, errors, failover tracking * **Tasks (HarmonyTask)**: Task counts, durations, success/failure rates, resource usage * **Proof Service**: Proofshare queue, durations, retry counts * **Storage**: NVMe health, cache stats, slot management * **HTTP/Retrieval**: Request counts, bytes served, response codes ## Wallet Exporter (Optional) The Wallet Exporter provides additional metrics about wallet balances, miner power, and message activity. {% hint style="warning" %} **Enable the Wallet Exporter on exactly ONE Curio node in the cluster.** Enabling it on multiple nodes causes duplicate metrics and incorrect aggregations. {% endhint %} ### Enabling the Wallet Exporter Add to a layer active on exactly one node: ```toml [CurioSubsystems] EnableWalletExporter = true ``` Restart the node. Metrics appear within \~30 seconds. See [Wallet Exporter](/experimental-features/wallet-exporter) for detailed usage and the full list of wallet metrics. ## Example Prometheus Queries ### Task Performance ```promql # Average task duration over last hour rate(curio_harmonytask_task_duration_seconds_sum[1h]) / rate(curio_harmonytask_task_duration_seconds_count[1h]) # Task success rate sum(rate(curio_harmonytask_tasks_completed[5m])) / sum(rate(curio_harmonytask_tasks_started[5m])) # Active tasks by type curio_harmonytask_active_tasks ``` ### Database Health ```promql # Average query latency in milliseconds rate(curio_db_total_wait[5m]) / rate(curio_db_hits[5m]) # Database error rate rate(curio_db_errors[5m]) # Connection pool usage curio_db_open_connections ``` ### Resource Utilization ```promql # Node resource usage curio_harmonytask_cpu_usage curio_harmonytask_gpu_usage curio_harmonytask_ram_usage # Node uptime curio_harmonytask_uptime ``` ## Grafana Integration These Prometheus metrics can be visualized using Grafana. Create dashboards to monitor: * **Cluster Overview**: Node uptime, task throughput, error rates * **Task Performance**: Duration histograms, success/failure rates by task type * **Resource Utilization**: CPU, GPU, RAM usage across nodes * **Database Health**: Query latency, connection pool, failover events * **Wallet Activity**: Balances, gas usage, message throughput (with Wallet Exporter) ## AlertManager Integration Curio can send alerts to Prometheus AlertManager. See [Alert Manager](/configuration/alert-manager) for configuration details. ## Troubleshooting ### Metrics not appearing 1. Verify the Curio node is running and accessible 2. Check that you can reach the metrics endpoint: `curl http://:/debug/metrics` 3. Verify your Prometheus configuration targets are correct ### Duplicate metrics If you see duplicate metric series, ensure the Wallet Exporter is enabled on only one node. ### Missing wallet metrics Wallet Exporter metrics require: * `EnableWalletExporter = true` in configuration * Wallets registered in the `wallet_names` table (via the Wallets page in UI) * At least one miner ID configured on the node # Metrics Reference This document lists Prometheus metrics exported by Curio using the exact metric names exposed on the scrape endpoint. > **Note**: This file is auto-generated from source code. Run `make docsgen-metrics` to update. ## Node Metrics | Metric | Type | Labels | Description | | ----------------- | ----- | ----------------- | --------------------------------------------------- | | `curio_node_info` | gauge | `node`, `version` | Curio node identity and version. Value is always 1. | ## Task Metrics (HarmonyTask) | Metric | Type | Labels | Description | | ----------------------------------------------- | --------- | --------------------- | ------------------------------------------------------------------------------------------------------ | | `curio_harmonytask_active_tasks` | gauge | `task_name` | Current number of active tasks. | | `curio_harmonytask_added_tasks` | counter | `task_name` | Total number of tasks added. | | `curio_harmonytask_cpu_usage` | gauge | — | Percentage of CPU in use. | | `curio_harmonytask_gpu_usage` | gauge | — | Percentage of GPU in use. | | `curio_harmonytask_poller_iterations` | counter | — | Total number of poller iterations. | | `curio_harmonytask_ram_usage` | gauge | — | Percentage of RAM in use. | | `curio_harmonytask_task_duration_seconds` | histogram | `task_name` | The histogram of task durations in seconds. | | `curio_harmonytask_task_scheduled_wait_seconds` | histogram | `task_name` | The histogram of task wait times from posting or previous attempt completion to work start in seconds. | | `curio_harmonytask_tasks_completed` | counter | `task_name` | Total number of tasks completed successfully. | | `curio_harmonytask_tasks_failed` | counter | `task_name` | Total number of tasks that failed. | | `curio_harmonytask_tasks_started` | counter | `task_name`, `source` | Total number of tasks started. | | `curio_harmonytask_uptime` | gauge | `version` | Total uptime of the node in seconds. | ## Wallet Exporter Metrics (Optional) | Metric | Type | Labels | Description | | -------------------------------------------- | --------- | -------------------------------------------------------------------------- | --------------------------------------------------- | | `curio_wallet_balance_nfil` | gauge | `address`, `type`, `name` | Balance in NanoFIL | | `curio_wallet_gas_paid_nfil` | counter | `from`, `from_name`, `to`, `to_name`, `method`, `send_reason`, `exit_code` | Gas paid NanoFIL | | `curio_wallet_gas_units_requested` | counter | `from`, `from_name`, `to`, `to_name`, `method`, `send_reason` | Gas units requested | | `curio_wallet_gas_units_used` | counter | `from`, `from_name`, `to`, `to_name`, `method`, `send_reason`, `exit_code` | Gas units used | | `curio_wallet_message_land_duration_seconds` | histogram | — | The histogram of message land durations in seconds. | | `curio_wallet_message_landed` | counter | `from`, `from_name`, `to`, `to_name`, `method`, `send_reason`, `exit_code` | Message landed | | `curio_wallet_message_sent` | counter | `from`, `from_name`, `to`, `to_name`, `method`, `send_reason` | Message sent | | `curio_wallet_power` | gauge | `address`, `type` | Power in Bytes | | `curio_wallet_sent_nfil` | counter | `from`, `from_name`, `to`, `to_name`, `method`, `send_reason`, `exit_code` | Sent NanoFIL | ## Proof Share Metrics | Metric | Type | Labels | Description | | -------------------------------------------------- | --------- | ------ | ------------------------------------------------------------ | | `curio_psvc_proofshare_adder_commits_total` | counter | — | Total number of successful task additions scheduled by Adder | | `curio_psvc_proofshare_adder_hold_decisions_total` | counter | `hold` | Total number of hold decisions made by Adder | | `curio_psvc_proofshare_create_asks_seconds` | histogram | — | Duration of create asks inner loop | | `curio_psvc_proofshare_duration_seconds` | histogram | `call` | Duration of proofshare client\_common operations | | `curio_psvc_proofshare_need_asks` | gauge | — | Number of asks still needed in current Do loop iteration | | `curio_psvc_proofshare_newly_added_total` | counter | — | Total number of new work requests inserted locally | | `curio_psvc_proofshare_queue_count` | gauge | — | Current proofshare request queue count | | `curio_psvc_proofshare_retry_count` | histogram | `call` | Retry count per call in proofshare client\_common operations | | `curio_psvc_proofshare_to_request_remaining` | gauge | — | Remaining requests to fulfill for high-water mark | ## Proof Service Metrics | Metric | Type | Labels | Description | | --------------------------------------- | --------- | ------ | ------------------------------------------------ | | `curio_psvc_clientctl_duration_seconds` | histogram | `call` | Duration of proofsvc clientctl operations | | `curio_psvc_l1ops_duration_seconds` | histogram | `call` | Duration of L1 operations | | `curio_psvc_provictl_duration_seconds` | histogram | `call` | Duration of proofsvc provider control operations | ## Sealing Metrics | Metric | Type | Labels | Description | | ---------------------------------------- | ------- | ------- | ---------------------------------- | | `curio_seal_commit_submitted_total` | counter | `miner` | Commit messages submitted. | | `curio_seal_finalize_completed_total` | counter | `miner` | Finalizations completed. | | `curio_seal_movestorage_completed_total` | counter | `miner` | Move storage operations completed. | | `curio_seal_porep_completed_total` | counter | `miner` | PoRep computations completed. | | `curio_seal_precommit_submitted_total` | counter | `miner` | Precommit messages submitted. | | `curio_seal_sdr_completed_total` | counter | `miner` | SDR computations completed. | | `curio_seal_synth_completed_total` | counter | `miner` | Synthetic proofs completed. | | `curio_seal_treed_completed_total` | counter | `miner` | Tree D computations completed. | | `curio_seal_treerc_completed_total` | counter | `miner` | Tree R/C computations completed. | ## Snap Pipeline Metrics | Metric | Type | Labels | Description | | ---------------------------------------- | ------- | ------- | --------------------------------------- | | `curio_snap_encode_completed_total` | counter | `miner` | Snap encodes completed. | | `curio_snap_movestorage_completed_total` | counter | `miner` | Snap move storage operations completed. | | `curio_snap_prove_completed_total` | counter | `miner` | Snap proves completed. | | `curio_snap_submit_completed_total` | counter | `miner` | Snap submissions completed. | ## Mining Metrics | Metric | Type | Labels | Description | | ------------------------------------- | --------- | ------- | --------------------------------------------------- | | `curio_mining_blocks_included_total` | counter | `miner` | Blocks included in the chain. | | `curio_mining_blocks_submitted_total` | counter | `miner` | Blocks submitted to the network. | | `curio_mining_compute_time_seconds` | histogram | — | Histogram of winning post compute times in seconds. | | `curio_mining_wins_total` | counter | `miner` | Blocks won (election success). | ## Storage Metrics | Metric | Type | Labels | Description | | ---------------------------------------------------------------- | --------- | --------------------------------- | ------------------------------------------------------ | | `curio_stor_available_bytes` | gauge | `id`, `can_seal`, `can_store` | Available storage capacity in bytes | | `curio_stor_capacity_bytes` | gauge | `id`, `can_seal`, `can_store` | Total storage capacity in bytes | | `curio_stor_find_sector_cache_hits` | counter | — | Number of findSectorCache hits | | `curio_stor_find_sector_cache_misses` | counter | — | Number of findSectorCache misses | | `curio_stor_find_sector_uncached` | counter | — | Number of findSector uncached calls | | `curio_stor_generate_single_vanilla_proof_calls` | counter | `update`, `cache_id`, `sealed_id` | Number of calls to GenerateSingleVanillaProof | | `curio_stor_generate_single_vanilla_proof_duration_milliseconds` | histogram | `update`, `cache_id`, `sealed_id` | Duration of GenerateSingleVanillaProof in milliseconds | | `curio_stor_generate_single_vanilla_proof_errors` | counter | `update`, `cache_id`, `sealed_id` | Number of errors in GenerateSingleVanillaProof | | `curio_stor_used_bytes` | gauge | `id`, `can_seal`, `can_store` | Used storage capacity in bytes | ## Cache Metrics | Metric | Type | Labels | Description | | ------------------------------------- | ------- | ---------------------- | -------------------------------------------------- | | `curio_cachedreader_cache_evictions` | counter | `cache_type`, `reason` | Number of cache evictions. | | `curio_cachedreader_cache_hits` | counter | `cache_type` | Number of cache hits. | | `curio_cachedreader_cache_misses` | counter | `cache_type` | Number of cache misses. | | `curio_cachedreader_cache_refs` | gauge | — | Number of active references to cached readers. | | `curio_cachedreader_cache_size` | gauge | `cache_type` | Current number of entries in cache. | | `curio_cachedreader_reader_errors` | counter | `reason` | Total number of piece reader errors. | | `curio_cachedreader_reader_successes` | counter | — | Total number of successful piece reader creations. | ## Network Metrics | Metric | Type | Labels | Description | | ----------------------------------- | ------- | ------ | -------------------------------------------------------- | | `curio_dealdata_data_read` | gauge | `kind` | Number of bytes read from data URLs | | `curio_robusthttp_active_transfers` | gauge | — | Current number of active robusthttp transfers | | `curio_robusthttp_bytes_read` | counter | — | Total bytes delivered by robusthttp readers | | `curio_robusthttp_read_errors` | counter | — | Total number of robusthttp read/request errors (non-EOF) | | `curio_robusthttp_read_failures` | counter | — | Number of robusthttp requests that failed after retries | | `curio_robusthttp_requests_started` | counter | — | Number of robusthttp logical requests started | | `curio_robusthttp_retries` | counter | — | Total number of robusthttp retries across requests | ## FFI Metrics | Metric | Type | Labels | Description | | ----------------------------- | ----- | ------- | ----------------------------- | | `curio_cuffi_snap_enc_active` | gauge | `phase` | Number of tasks in each phase | ## HTTP/Retrieval Metrics | Metric | Type | Labels | Description | | ------------------------------------------------- | --------- | ------------------------------- | ------------------------------------------------------------------------------------------------------ | | `curio_dagstore_pr_at_cache_fill_count` | counter | `network` | PieceReader ReadAt full cache fill count | | `curio_dagstore_pr_at_hit_bytes` | counter | `network` | PieceReader ReadAt bytes from cache | | `curio_dagstore_pr_at_hit_count` | counter | `network` | PieceReader ReadAt from cache hits | | `curio_dagstore_pr_at_read_bytes` | counter | `pr_size`, `network` | PieceReader ReadAt bytes read from source | | `curio_dagstore_pr_at_read_count` | counter | `pr_size`, `network` | PieceReader ReadAt reads from source | | `curio_dagstore_pr_discard_count` | counter | `network` | PieceReader discard count | | `curio_dagstore_pr_discarded_bytes` | counter | `network` | PieceReader discarded bytes | | `curio_dagstore_pr_init_count` | counter | `network` | PieceReader init count | | `curio_dagstore_pr_requested_bytes` | counter | `pr_type`, `network` | PieceReader requested bytes | | `curio_dagstore_pr_seek_back_bytes` | counter | `network` | PieceReader seek back bytes | | `curio_dagstore_pr_seek_back_count` | counter | `network` | PieceReader seek back count | | `curio_dagstore_pr_seek_forward_bytes` | counter | `network` | PieceReader seek forward bytes | | `curio_dagstore_pr_seek_forward_count` | counter | `network` | PieceReader seek forward count | | `curio_http_active_requests` | gauge | `path`, `method` | Number of active/in-flight HTTP requests | | `curio_http_blockstore_cache_hits` | counter | — | Counter of blockstore cache hits | | `curio_http_blockstore_cache_misses` | counter | — | Counter of blockstore cache misses | | `curio_http_piece_by_cid_200_response_count` | counter | — | Counter of /piece/ 200 responses | | `curio_http_piece_by_cid_400_response_count` | counter | — | Counter of /piece/ 400 responses | | `curio_http_piece_by_cid_404_response_count` | counter | — | Counter of /piece/ 404 responses | | `curio_http_piece_by_cid_500_response_count` | counter | — | Counter of /piece/ 500 responses | | `curio_http_piece_by_cid_request_count` | counter | — | Counter of /piece/ requests | | `curio_http_piece_by_cid_request_duration_ms` | histogram | — | Time spent retrieving a piece by cid | | `curio_http_rbls_bytes_sent_count` | counter | — | Counter of the number of bytes sent by bitswap since startup | | `curio_http_rbls_get_fail_response_count` | counter | — | Counter of failed RemoteBlockstore Get responses | | `curio_http_rbls_get_request_count` | counter | — | Counter of RemoteBlockstore Get requests | | `curio_http_rbls_get_success_response_count` | counter | — | Counter of successful RemoteBlockstore Get responses | | `curio_http_rbls_getsize_fail_response_count` | counter | — | Counter of failed RemoteBlockstore GetSize responses | | `curio_http_rbls_getsize_request_count` | counter | — | Counter of RemoteBlockstore GetSize requests | | `curio_http_rbls_getsize_success_response_count` | counter | — | Counter of successful RemoteBlockstore GetSize responses | | `curio_http_rbls_has_fail_response_count` | counter | — | Counter of failed RemoteBlockstore Has responses | | `curio_http_rbls_has_request_count` | counter | — | Counter of RemoteBlockstore Has requests | | `curio_http_rbls_has_success_response_count` | counter | — | Counter of successful RemoteBlockstore Has responses | | `curio_http_request_count` | counter | `path`, `method` | Counter of HTTP requests | | `curio_http_response_bytes_count` | counter | `status_code`, `path` | Sum of HTTP response content-length | | `curio_http_response_status_count` | counter | `status_code`, `path`, `method` | Counter of HTTP response status codes | | `curio_ipni_announce_attempts_total` | counter | `provider`, `result` | Total number of IPNI direct announce attempts. | | `curio_ipni_announce_http_roundtrip_milliseconds` | histogram | `provider`, `status` | Duration of outbound IPNI announce HTTP round trips in milliseconds. | | `curio_ipni_entry_cache_hit_wait_milliseconds` | histogram | `request`, `origin`, `state` | Duration callers wait after hitting the IPNI entry cache. | | `curio_ipni_entry_cache_hits_total` | counter | `request`, `origin`, `state` | Total number of IPNI entry cache hits by request type, cache origin, and readiness state. | | `curio_ipni_entry_cache_lookups_total` | counter | `request`, `result` | Total number of IPNI entry cache lookups. | | `curio_ipni_entry_reconstruction_milliseconds` | histogram | `request`, `source`, `result` | Duration of IPNI entry reconstruction after cache miss. | | `curio_ipni_entry_requests_total` | counter | `request` | Total number of IPNI entry requests handled by the serve chunker. | | `curio_ipni_entry_speculative_unused_total` | counter | — | Total number of speculative IPNI entry cache fills evicted without being consumed by a demand request. | | `curio_ipni_provider_http_request_milliseconds` | histogram | `provider`, `content`, `status` | Duration of inbound IPNI provider HTTP requests in milliseconds. | | `curio_ipni_provider_http_requests_total` | counter | `provider`, `content`, `status` | Total number of inbound IPNI provider HTTP requests. | | `curio_pdp_piece_by_cid_200_response_count` | counter | — | Counter of /piece/ 200 responses for PDP | | `curio_pdp_piece_by_cid_request_count` | counter | — | Counter of /piece/ requests for PDP | | `curio_pdp_piece_by_cid_request_duration_ms` | histogram | — | Time spent retrieving a piece by cid for PDP | | `curio_pdp_piece_bytes_served_count` | counter | — | Counter of the number of bytes served by PDP since startup | ## GC Metrics | Metric | Type | Labels | Description | | ------------------------------- | ------- | ------------------- | ---------------------- | | `curio_gc_sectors_marked_total` | counter | `miner`, `filetype` | Sectors marked for GC. | ## Batching Metrics | Metric | Type | Labels | Description | | ------------------------------------------ | ------- | ------------- | ------------------------------------------------------------ | | `curio_sealsupra_nvme_available_spare` | gauge | `nvme_device` | NVMe Available Spare | | `curio_sealsupra_nvme_bytes_read` | counter | `nvme_device` | NVMe Bytes Read | | `curio_sealsupra_nvme_bytes_written` | counter | `nvme_device` | NVMe Bytes Written | | `curio_sealsupra_nvme_critical_warning` | gauge | `nvme_device` | NVMe Critical Warning Flags | | `curio_sealsupra_nvme_error_log_entries` | gauge | `nvme_device` | NVMe Error Log Entries | | `curio_sealsupra_nvme_media_errors` | gauge | `nvme_device` | NVMe Media Errors | | `curio_sealsupra_nvme_percentage_used` | gauge | `nvme_device` | NVMe Percentage Used | | `curio_sealsupra_nvme_power_cycles` | gauge | `nvme_device` | NVMe Power Cycles | | `curio_sealsupra_nvme_power_on_hours` | gauge | `nvme_device` | NVMe Power On Hours | | `curio_sealsupra_nvme_read_io` | counter | `nvme_device` | NVMe Read IOs | | `curio_sealsupra_nvme_temperature_celsius` | gauge | `nvme_device` | NVMe Temperature in Celsius | | `curio_sealsupra_nvme_unsafe_shutdowns` | gauge | `nvme_device` | NVMe Unsafe Shutdowns | | `curio_sealsupra_nvme_write_io` | counter | `nvme_device` | NVMe Write IOs | | `curio_sealsupra_phase_avg_duration` | gauge | `phase` | Average duration of each phase in seconds | | `curio_sealsupra_phase_duration_so_far` | gauge | `phase` | Duration of the phase so far in seconds | | `curio_sealsupra_phase_lock_count` | gauge | `phase` | Number of active locks in each phase | | `curio_sealsupra_phase_waiting_count` | gauge | `phase` | Number of goroutines waiting for a phase lock | | `curio_slotmgr_slot_errors` | counter | — | Total number of slot errors (e.g., failed to put). | | `curio_slotmgr_slot_in_use` | gauge | `slot_offset` | Slot actively in use (batch sealing). 1=in use, 0=not in use | | `curio_slotmgr_slot_sector_count` | gauge | `slot_offset` | Number of sectors in the slot | | `curio_slotmgr_slots_acquired` | counter | — | Total number of slots acquired. | | `curio_slotmgr_slots_available` | gauge | — | Number of available slots. | | `curio_slotmgr_slots_released` | counter | — | Total number of slots released. | *** *Generated from source files. See* [*Prometheus Metrics*](/configuration/prometheus-metrics) *for setup instructions.* # Alert Manager Curio alert manager setup and configuration Curio has an AlertManager task which runs every 1 hour and allows Curio cluster to alert users about certain issues in the cluster. Currently, Curio supports the alert for following issues: 1. Wallet balance is below 5 Fil. 2. If not WindowPost task is run for a deadline. 3. If an orphan block is found or if no WinningPost task is not created for any epoch. 4. If permanent storage does not have enough space to accommodate the sectors currently being sealed. The AlertManager is a plugin based module and allows integration with any plugin. As of now, Curio has 2 plugins available. Alerts generated can be send to multiple plugins at the same time to allow for a more robust notification mechanism. {% hint style="info" %} Contributions to new critical alerts or integrations with other alerting systems are welcome. {% endhint %} ### PagerDuty Plugin Curio comes with a default integration with [PagerDuty.com](https://www.pagerduty.com/), allowing the sending of critical alerts to storage providers. To configure your Curio cluster to send alerts, you must set up a PagerDuty account. {% hint style="danger" %} Nobody associated with the development of this software has any business relationship with PagerDuty. This integration is provided as a convenient gateway to the storage provider’s alert system of choice. {% endhint %} 1. Sign up for a free PagerDuty account [here](https://www.pagerduty.com/sign-up-free/?type=free). 2. Create a new service that will handle the alerts from the Curio cluster. 3. During the service creation, on the “Integration” page, choose “Events API V2”. 4. Once the service creation is complete, copy the “Integration Key” from the service and paste it in the “base” layer configuration for “PagerDutyIntegrationKey”. 5. Enable the plugin in config layer. 6. Restart one of the nodes, and it will now generate critical alerts at the top of every hour. ``` [Alerting] # MinimumWalletBalance is the minimum balance all active wallets. If the balance is below this value, an # alerts will be triggered for the wallet # # type: types.FIL #MinimumWalletBalance = "5 FIL" [Alerting.PagerDuty] # Enable is a flag to enable or disable the PagerDuty integration. # # type: bool Enable = true # PagerDutyEventURL is URL for PagerDuty.com Events API v2 URL. Events sent to this API URL are ultimately # routed to a PagerDuty.com service and processed. # The default is sufficient for integration with the stock commercial PagerDuty.com company's service. # # type: string PagerDutyEventURL = "https://events.pagerduty.com/v2/enqueue" # PageDutyIntegrationKey is the integration key for a PagerDuty.com service. You can find this unique service # identifier in the integration page for the service. # # type: string PageDutyIntegrationKey = "" ``` ### Prometheus AlertManager 1. Setup a [Prometheus AlertManager](https://prometheus.io/docs/alerting/latest/alertmanager/) instance. 2. Enable the plugin but setting the `Enabled` to True. 3. Paste the `AlertManagerURL` in the configguration. 4. Restart the node with the updated config layer. ``` [Alerting.PrometheusAlertManager] # Enable is a flag to enable or disable the Prometheus AlertManager integration. # # type: bool Enable = true # AlertManagerURL is the URL for the Prometheus AlertManager API v2 URL. # # type: string AlertManagerURL = "http://localhost:9093/api/v2/alerts" ``` ### Slack Webhook Integration This setup ensures that alerts generated by Curio can be forwarded to a designated Slack channel for real-time notifications. 1. Set up a Slack Webhook integration for alert notifications. 2. Enable the plugin by setting `Enable` to `true`, and provide the **WebHookURL** in the configuration. This URL should point to the Slack Webhook endpoint where alerts will be sent. 3. Once configured, restart the node to apply the updated configuration. ``` [Alerting.SlackWebhook] # Enable is a flag to enable or disable the Prometheus AlertManager integration. # # type: bool #Enable = false # WebHookURL is the URL for the URL for slack Webhook. # Example: https://hooks.slack.com/services/T00000000/B00000000/XXXXXXXXXXXXXXXXXXXXXXXX # # type: string #WebHookURL = "" ``` # Balance Manager How to configure balance manager in Curio cluster ## Overview The **Balance Manager** is a subsystem within Curio responsible for **automatically managing balances** for certain actors. Currently, it is used to **top up deal collateral** when the available balance falls below a defined threshold. In future iterations, the Balance Manager will be extended to **handle control addresses and other balance-related operations.** ## Functionality The Balance Manager operates on a periodic task cycle and ensures that a miner's **market balance** remains within an acceptable range by: 1. **Monitoring market balance**: Checks the miner's market escrow balance at regular intervals. 2. **Comparing against thresholds**: * If the balance drops **below** `CollateralLowThreshold`, additional funds are added. * The balance is topped **up to** `CollateralHighThreshold`. 3. **Sending transactions**: Issues an `AddBalance` message to the **market actor (`f05`)** from the **configured wallet**. 4. **Ensuring sufficient funds**: Verifies that the deal collateral wallet has enough balance before sending funds. 5. In the future, this functionality will be expanded to include maintaining control address and other operational balances. The Balance Manager runs every **5 minutes** by default (`BalanceCheckInterval`). ## Configuration The **Balance Manager configuration** should be defined at the **base layer** within Curio. It is part of the **`BalanceManagerConfig`** section inside the `CurioConfig`. ### **Enabling Balance Manager** To enable the Balance Manager, modify the **`EnableBalanceManager`** setting under `Subsystems`: ```toml [Subsystems] EnableBalanceManager = true # Set to true to activate Balance Manager ``` ### **Balance Manager Settings** The Balance Manager's configuration is located under `Addresses`: ```toml [Addresses.BalanceManager.MK12Collateral] DealCollateralWallet = "t3xyz..." # The wallet used to fund market balance CollateralLowThreshold = "5 FIL" # If market balance drops below this, a top-up is triggered CollateralHighThreshold = "20 FIL" # The target balance after a top-up ``` #### Configuration Parameters | Parameter | Description | | ------------------------- | ----------------------------------------------------------------------------------- | | `DealCollateralWallet` | The wallet used to top up the miner's market balance. | | `CollateralLowThreshold` | When the balance falls below this threshold, the Balance Manager triggers a top-up. | | `CollateralHighThreshold` | The target balance after a top-up operation is performed. | ### Example Configuration ```toml [Subsystems] EnableBalanceManager = true [Addresses] [Addresses.BalanceManager] [Addresses.BalanceManager.MK12Collateral] DealCollateralWallet = "t3xyz..." CollateralLowThreshold = "5 FIL" CollateralHighThreshold = "20 FIL" ``` ## Future Enhancements Currently, the Balance Manager is only used for deal collateral top-ups, but it will be extended to: * Manage control addresses balances. * Monitor and maintain additional operation-specific balances. By enabling the Balance Manager, miners can **automate their collateral management**, reducing the risk of **deal failures due to insufficient balance** while optimizing operational efficiency. # Default Curio Configuration The default curio configuration ```toml # Subsystems defines configuration settings for various subsystems within the Curio node. # # type: CurioSubsystemsConfig [Subsystems] # EnableWindowPost enables window post to be executed on this curio instance. Each machine in the cluster # with WindowPoSt enabled will also participate in the window post scheduler. It is possible to have multiple # machines with WindowPoSt enabled which will provide redundancy, and in case of multiple partitions per deadline, # will allow for parallel processing of partitions. # # It is possible to have instances handling both WindowPoSt and WinningPoSt, which can provide redundancy without # the need for additional machines. In setups like this it is generally recommended to run # partitionsPerDeadline+1 machines. (Default: false) # # type: bool #EnableWindowPost = false # The maximum amount of WindowPostMaxTasks tasks that can run simultaneously. Note that the maximum number of tasks will # also be bounded by resources available on the machine. We do not recommend setting this value and let system resources determine # the maximum tasks (Default: 0 - unlimited) # # type: int #WindowPostMaxTasks = 0 # EnableWinningPost enables winning post to be executed on this curio instance. # Each machine in the cluster with WinningPoSt enabled will also participate in the winning post scheduler. # It is possible to mix machines with WindowPoSt and WinningPoSt enabled, for details see the EnableWindowPost # documentation. (Default: false) # # type: bool #EnableWinningPost = false # The maximum amount of WinningPostMaxTasks tasks that can run simultaneously. Note that the maximum number of tasks will # also be bounded by resources available on the machine. We do not recommend setting this value and let system resources determine # the maximum tasks (Default: 0 - unlimited) # # type: int #WinningPostMaxTasks = 0 # EnableParkPiece enables the "piece parking" task to run on this node. This task is responsible for fetching # pieces from the network and storing them in the storage subsystem until sectors are sealed. This task is # only applicable when integrating with boost, and should be enabled on nodes which will hold deal data # from boost until sectors containing the related pieces have the TreeD/TreeR constructed. # Note that future Curio implementations will have a separate task type for fetching pieces from the internet. (Default: false) # # type: bool #EnableParkPiece = false # The maximum amount of ParkPieceMaxTasks tasks that can run simultaneously. Note that the maximum number of tasks will # also be bounded by resources available on the machine (Default: 0 - unlimited) # # type: int #ParkPieceMaxTasks = 0 # The maximum number of pieces that should be in storage + active tasks writing to storage on this node (Default: 0 - unlimited) # # type: int #ParkPieceMaxInPark = 0 # The minimum free storage percentage required for the ParkPiece task to run. (Default: 5) # # type: float64 #ParkPieceMinFreeStoragePercent = 5.0 # EnableSealSDR enables SDR tasks to run. SDR is the long sequential computation # creating 11 layer files in sector cache directory. # # SDR is the first task in the sealing pipeline. It's inputs are just the hash of the # unsealed data (CommD), sector number, miner id, and the seal proof type. # It's outputs are the 11 layer files in the sector cache directory. # # In lotus-miner this was run as part of PreCommit1. (Default: false) # # type: bool #EnableSealSDR = false # The maximum amount of SDR tasks that can run simultaneously. Note that the maximum number of tasks will # also be bounded by resources available on the machine. (Default: 0 - unlimited) # # type: int #SealSDRMaxTasks = 0 # The maximum amount of SDR tasks that need to be queued before the system will start accepting new tasks. # The main purpose of this setting is to allow for enough tasks to accumulate for batch sealing. When batch sealing # nodes are present in the cluster, this value should be set to batch_size+1 to allow for the batch sealing node to # fill up the batch. # This setting can also be used to give priority to other nodes in the cluster by setting this value to a higher # value on the nodes which should have less priority. (Default: 0 - unlimited) # # type: int #SealSDRMinTasks = 0 # EnableSealSDRTrees enables the SDR pipeline tree-building task to run. # This task handles encoding of unsealed data into last sdr layer and building # of TreeR, TreeC and TreeD. # # This task runs after SDR # TreeD is first computed with optional input of unsealed data # TreeR is computed from replica, which is first computed as field # addition of the last SDR layer and the bottom layer of TreeD (which is the unsealed data) # TreeC is computed from the 11 SDR layers # The 3 trees will later be used to compute the PoRep proof. # # In case of SyntheticPoRep challenges for PoRep will be pre-generated at this step, and trees and layers # will be dropped. SyntheticPoRep works by pre-generating a very large set of challenges (~30GiB on disk) # then using a small subset of them for the actual PoRep computation. This allows for significant scratch space # saving between PreCommit and PoRep generation at the expense of more computation (generating challenges in this step) # # In lotus-miner this was run as part of PreCommit2 (TreeD was run in PreCommit1). # Note that nodes with SDRTrees enabled will also answer to Finalize tasks, # which just remove unneeded tree data after PoRep is computed. (Default: false) # # type: bool #EnableSealSDRTrees = false # The maximum amount of SealSDRTrees tasks that can run simultaneously. Note that the maximum number of tasks will # also be bounded by resources available on the machine. (Default: 0 - unlimited) # # type: int #SealSDRTreesMaxTasks = 0 # FinalizeMaxTasks is the maximum amount of finalize tasks that can run simultaneously. # The finalize task is enabled on all machines which also handle SDRTrees tasks. Finalize ALWAYS runs on whichever # machine holds sector cache files, as it removes unneeded tree data after PoRep is computed. # Finalize will run in parallel with the SubmitCommitMsg task. (Default: 0 - unlimited) # # type: int #FinalizeMaxTasks = 0 # EnableSendPrecommitMsg enables the sending of precommit messages to the chain # from this curio instance. # This runs after SDRTrees and uses the output CommD / CommR (roots of TreeD / TreeR) for the message (Default: false) # # type: bool #EnableSendPrecommitMsg = false # EnablePoRepProof enables the computation of the porep proof # # This task runs after interactive-porep seed becomes available, which happens 150 epochs (75min) after the # precommit message lands on chain. This task should run on a machine with a GPU. Vanilla PoRep proofs are # requested from the machine which holds sector cache files which most likely is the machine which ran the SDRTrees # task. # # In lotus-miner this was Commit1 / Commit2 (Default: false) # # type: bool #EnablePoRepProof = false # The maximum amount of PoRepProof tasks that can run simultaneously. Note that the maximum number of tasks will # also be bounded by resources available on the machine. (Default: 0 - unlimited) # # type: int #PoRepProofMaxTasks = 0 # EnableSendCommitMsg enables the sending of commit messages to the chain # from this curio instance. (Default: false) # # type: bool #EnableSendCommitMsg = false # Whether to abort if any sector activation in a batch fails (newly sealed sectors, only with ProveCommitSectors3). (Default: true) # # type: bool #RequireActivationSuccess = true # Whether to abort if any sector activation in a batch fails (updating sectors, only with ProveReplicaUpdates3). (Default: true) # # type: bool #RequireNotificationSuccess = true # EnableMoveStorage enables the move-into-long-term-storage task to run on this curio instance. # This tasks should only be enabled on nodes with long-term storage. # # The MoveStorage task is the last task in the sealing pipeline. It moves the sealed sector data from the # SDRTrees machine into long-term storage. This task runs after the Finalize task. (Default: false) # # type: bool #EnableMoveStorage = false # NoUnsealedDecode disables the decoding sector data on this node. Normally data encoding is enabled by default on # storage nodes with the MoveStorage task enabled. Setting this option to true means that unsealed data for sectors # will not be stored on this node (Default: false) # # type: bool #NoUnsealedDecode = false # The maximum amount of MoveStorage tasks that can run simultaneously. Note that the maximum number of tasks will # also be bounded by resources available on the machine. It is recommended that this value is set to a number which # uses all available network (or disk) bandwidth on the machine without causing bottlenecks. NOTE: unlike most other # tasks, when this value is set the maximum number of concurrent tasks will not be bounded by CPU core count (Default: 0 - unlimited) # # type: int #MoveStorageMaxTasks = 0 # EnableUpdateEncode enables the encoding step of the SnapDeal process on this curio instance. # This step involves encoding the data into the sector and computing updated TreeR (uses gpu). (Default: false) # # type: bool #EnableUpdateEncode = false # EnableUpdateProve enables the proving step of the SnapDeal process on this curio instance. # This step generates the snark proof for the updated sector. (Default: false) # # type: bool #EnableUpdateProve = false # EnableUpdateSubmit enables the submission of SnapDeal proofs to the blockchain from this curio instance. # This step submits the generated proofs to the chain. (Default: false) # # type: bool #EnableUpdateSubmit = false # UpdateEncodeMaxTasks sets the maximum number of concurrent SnapDeal encoding tasks that can run on this instance. (Default: 0 - unlimited) # # type: int #UpdateEncodeMaxTasks = 0 # BindEncodeToData forces the Encode task to be executed on the same node where the data was parked. # Please ensure that ParkPiece task is enabled and relevant resources are available before enabling this option. # (Default: false) # # type: bool #BindEncodeToData = false # AllowEncodeGPUOverprovision allows the Encode task to run on regardress of declared GPU usage. (Default: false) # NOTE: This definitely is not safe on PoSt nodes. # # type: bool #AllowEncodeGPUOverprovision = false # UpdateProveMaxTasks sets the maximum number of concurrent SnapDeal proving tasks that can run on this instance. (Default: 0 - unlimited) # # type: int #UpdateProveMaxTasks = 0 # EnableWebGui enables the web GUI on this curio instance. The UI has minimal local overhead, but it should # only need to be run on a single machine in the cluster. (Default: false) # # type: bool #EnableWebGui = false # The address that should listen for Web GUI requests. It should be in form "x.x.x.x:1234" (Default: 0.0.0.0:4701) # # type: string #GuiAddress = "0.0.0.0:4701" # UseSyntheticPoRep enables the synthetic PoRep for all new sectors. When set to true, will reduce the amount of # cache data held on disk after the completion of TreeRC task to 11GiB. (Default: false) # # type: bool #UseSyntheticPoRep = false # The maximum amount of SyntheticPoRep tasks that can run simultaneously. Note that the maximum number of tasks will # also be bounded by resources available on the machine. (Default: 0 - unlimited) # # type: int #SyntheticPoRepMaxTasks = 0 # EnableBatchSeal enabled SupraSeal batch sealing on the node. (Default: false) # # type: bool #EnableBatchSeal = false # EnableDealMarket enabled the deal market on the node. This would also enable libp2p on the node, if configured. (Default: false) # # type: bool #EnableDealMarket = false # Enable handling for PDP (proof-of-data possession) deals / proving on this node. # PDP deals allow the node to directly store and prove unsealed data with "PDP Services" like Storacha. # This feature is BETA and should only be enabled on nodes which are part of a PDP network. # # type: bool #EnablePDP = false # DataPath is the root directory Curio-PDP scans for writable storage locations. # The node treats this directory and every subdirectory as a candidate store path. # Overridden by the DATA_STORAGE env var and the --data CLI flag. (Default: /data) # # type: string #DataPath = "" # PDPPullPieceMaxTasks is the maximum number of PDPv0 pull-piece download tasks that can run simultaneously. # Set 0 for unlimited. (Default: 20) # # type: int #PDPPullPieceMaxTasks = 20 # PDPUnclaimedUploadKeepHours is how many hours to keep unclaimed PDP piece uploads (orphaned pdp_piecerefs # with data_set_refcount = 0) before PieceGC deletes them. Must be >= 1. (Default: 2) # Updates will affect running instances. # # type: int #PDPUnclaimedUploadKeepHours = 2 # EnableCommP enables the commP task on te node. CommP is calculated before sending PublishDealMessage for a Mk12 deal # Must have EnableDealMarket = True (Default: false) # # type: bool #EnableCommP = false # The maximum amount of CommP tasks that can run simultaneously. Note that the maximum number of tasks will # also be bounded by resources available on the machine. (Default: 0 - unlimited) # # type: int #CommPMaxTasks = 0 # BindCommPToData forces the CommP task to be executed on the same node where the data was parked. # Please ensure that ParkPiece task is enabled and relevant resources are available before enabling this option. # (Default: false) # # type: bool #BindCommPToData = false # The maximum amount of indexing and IPNI tasks that can run simultaneously. Note that the maximum number of tasks will # also be bounded by resources available on the machine. (Default: 8) # # type: int #IndexingMaxTasks = 8 # BindSDRTreeToNode forces the TreeD and TreeRC tasks to be executed on the same node where SDR task was executed # for the sector. Please ensure that TreeD and TreeRC task are enabled and relevant resources are available before # enabling this option. (Default: false) # # type: bool #BindSDRTreeToNode = false # EnableProofShare enables the ProofShare tasks on the node. This subsystem will request proof work from a marketplace # whenever local machine can take on more Snark work. ProofShare tasks have priority over local snark tasks, but new # ProofShare work will only be requested if there is no local work to do. # # This feature is currently experimental and may change in the future. (Default: false) # # type: bool #EnableProofShare = false # The maximum amount of ProofShare tasks that can run simultaneously. Note that the maximum number of tasks will # also be bounded by resources available on the machine. (Default: 0 - unlimited) # # type: int #ProofShareMaxTasks = 0 # EnableRemoteProofs enables the remote proof tasks on the node. Local snark tasks will be transformed into remote # proving tasks when this option is enabled. Details on which SP IDs are allowed to request remote proofs are managed # via Client Settings on the Proofshare webui page. Buy delay can also be set in the Client Settings page. (Default: false) # # type: bool #EnableRemoteProofs = false # The maximum number of remote proofs that can be uploaded simultaneously by each node. (Default: 15) # # type: int #RemoteProofMaxUploads = 15 # EnableWalletExporter enables the wallet exporter on the node. This will export wallet stats to prometheus. # NOTE: THIS MUST BE ENABLED ONLY ON A SINGLE NODE IN THE CLUSTER TO BE USEFUL (Default: false) # # type: bool #EnableWalletExporter = false # EnableDBAnalyze enables the cluster-wide DBAnalyze singleton task to speed up SQL queries. # It periodically runs ANALYZE on tables whose write churn (pg_stat_user_tables) has grown # by 10% since the last analyze. # Disable this if you manage table statistics outside Curio. (Default: true) # # type: bool #EnableDBAnalyze = true # Fees holds the fee-related configuration parameters for various operations in the Curio node. # # type: CurioFees [Fees] # WindowPoSt is a high-value operation, so the default fee should be high. # Accepts a decimal string (e.g., "123.45") with optional "fil" or "attofil" suffix. (Default: "5 fil") # # type: types.FIL #MaxWindowPoStGasFee = "5 FIL" # Whether to use available miner balance for sector collateral instead of sending it with each message (Default: false) # # type: bool #CollateralFromMinerBalance = false # Don't send collateral with messages even if there is no available balance in the miner actor (Default: false) # # type: bool #DisableCollateralFallback = false # MaximizeFeeCap makes the sender set maximum allowed FeeCap on all sent messages. # This generally doesn't increase message cost, but in highly congested network messages # are much less likely to get stuck in mempool. (Default: true) # # type: bool #MaximizeFeeCap = true # maxBatchFee = maxBase + maxPerSector * nSectors # (Default: #Base = "0 FIL" and #PerSector = "0.02 FIL") # # type: BatchFeeConfig [Fees.MaxPreCommitBatchGasFee] # Accepts a decimal string (e.g., "123.45") with optional "fil" or "attofil" suffix. # # type: types.FIL #Base = "0 FIL" # Accepts a decimal string (e.g., "123.45") with optional "fil" or "attofil" suffix. # # type: types.FIL #PerSector = "0.02 FIL" # maxBatchFee = maxBase + maxPerSector * nSectors # (Default: #Base = "0 FIL" and #PerSector = "0.03 FIL") # # type: BatchFeeConfig [Fees.MaxCommitBatchGasFee] # Accepts a decimal string (e.g., "123.45") with optional "fil" or "attofil" suffix. # # type: types.FIL #Base = "0 FIL" # Accepts a decimal string (e.g., "123.45") with optional "fil" or "attofil" suffix. # # type: types.FIL #PerSector = "0.03 FIL" # Accepts a decimal string (e.g., "123.45") with optional "fil" or "attofil" suffix. # (Default: #Base = "0 FIL" and #PerSector = "0.03 FIL") # # type: BatchFeeConfig [Fees.MaxUpdateBatchGasFee] # Accepts a decimal string (e.g., "123.45") with optional "fil" or "attofil" suffix. # # type: types.FIL #Base = "0 FIL" # Accepts a decimal string (e.g., "123.45") with optional "fil" or "attofil" suffix. # # type: types.FIL #PerSector = "0.03 FIL" # Addresses specifies the list of miner addresses and their related wallet addresses. # Updates will affect running instances. # # type: []CurioAddresses [[Addresses]] # PreCommitControl is an array of Addresses to send PreCommit messages from # # type: []string #PreCommitControl = [] # CommitControl is an array of Addresses to send Commit messages from # # type: []string #CommitControl = [] # DealPublishControl is an array of Address to send the deal collateral from with PublishStorageDeal Message # # type: []string #DealPublishControl = [] # TerminateControl is a list of addresses used to send Terminate messages. # # type: []string #TerminateControl = [] # DisableOwnerFallback disables usage of the owner address for messages # sent automatically # # type: bool #DisableOwnerFallback = false # DisableWorkerFallback disables usage of the worker address for messages # sent automatically, if control addresses are configured. # A control address that doesn't have enough funds will still be chosen # over the worker address if this flag is set. # # type: bool #DisableWorkerFallback = false # MinerAddresses are the addresses of the miner actors # # type: []string #MinerAddresses = [] # BalanceManagerConfig specifies the configuration parameters for managing wallet balances and actor-related funds, # including collateral and other operational resources. # # type: BalanceManagerConfig [Addresses.BalanceManager] # MK12Collateral defines the configuration for managing collateral and related balance thresholds in the miner's market. # # type: MK12CollateralConfig [Addresses.BalanceManager.MK12Collateral] # DealCollateralWallet is the wallet used to add balance to Miner's market balance. This balance is # utilized for deal collateral in market (f05) deals. # # type: string #DealCollateralWallet = "" # CollateralLowThreshold is the balance below which more balance will be added to miner's market balance # Accepts a decimal string (e.g., "123.45" or "123 fil") with optional "fil" or "attofil" suffix. (Default: "5 FIL") # # type: types.FIL #CollateralLowThreshold = "5 FIL" # CollateralHighThreshold is the target balance to which the miner's market balance will be topped up # when it drops below CollateralLowThreshold. # Accepts a decimal string (e.g., "123.45" or "123 fil") with optional "fil" or "attofil" suffix. (Default: "20 FIL") # # type: types.FIL #CollateralHighThreshold = "20 FIL" # Proving defines the configuration settings related to proving functionality within the Curio node. # # type: CurioProvingConfig [Proving] # Maximum number of sector checks to run in parallel. (0 = unlimited) # # WARNING: Setting this value too high may make the node crash by running out of stack # WARNING: Setting this value too low may make sector challenge reading much slower, resulting in failed PoSt due # to late submission. # # After changing this option, confirm that the new value works in your setup by invoking # 'curio test wd task 0' (Default: 32) # # type: int #ParallelCheckLimit = 32 # Maximum amount of time a proving pre-check can take for a sector. If the check times out the sector will be skipped # # WARNING: Setting this value too low risks in sectors being skipped even though they are accessible, just reading the # test challenge took longer than this timeout # WARNING: Setting this value too high risks missing PoSt deadline in case IO operations related to this sector are # blocked (e.g. in case of disconnected NFS mount) # Time duration string (e.g., "1h2m3s") in TOML format. (Default: "10m0s") # # type: time.Duration #SingleCheckTimeout = "10m0s" # Maximum amount of time a proving pre-check can take for an entire partition. If the check times out, sectors in # the partition which didn't get checked on time will be skipped # # WARNING: Setting this value too low risks in sectors being skipped even though they are accessible, just reading the # test challenge took longer than this timeout # WARNING: Setting this value too high risks missing PoSt deadline in case IO operations related to this partition are # blocked or slow. Time duration string (e.g., "1h2m3s") in TOML format. (Default: "20m0s") # # type: time.Duration #PartitionCheckTimeout = "20m0s" # HTTP represents the configuration for the HTTP server settings in the Curio node. # # type: HTTPConfig [HTTP] # Enable the HTTP server on the node # # type: bool #Enable = false # DomainName specifies the domain name that the server uses to serve HTTP requests. DomainName cannot be empty and cannot be # an IP address # # type: string #DomainName = "" # ListenAddress is the address that the server listens for HTTP requests. It should be in form "x.x.x.x:1234" (Default: 0.0.0.0:12310) # # type: string #ListenAddress = "0.0.0.0:12310" # DelegateTLS allows the server to delegate TLS to a reverse proxy. When enabled the listen address will serve # HTTP and the reverse proxy will handle TLS termination. # # type: bool #DelegateTLS = false # ReadTimeout is the maximum duration for reading the entire or next request, including body, from the client. # Time duration string (e.g., "1h2m3s") in TOML format. (Default: "30m0s") # # type: time.Duration #ReadTimeout = "30m0s" # IdleTimeout is the maximum duration of an idle session. If set, idle connections are closed after this duration. # Time duration string (e.g., "1h2m3s") in TOML format. (Default: "30m0s") # # type: time.Duration #IdleTimeout = "30m0s" # ReadHeaderTimeout is amount of time allowed to read request headers # Time duration string (e.g., "1h2m3s") in TOML format. (Default: "0m5s") # # type: time.Duration #ReadHeaderTimeout = "5s" # CORSOrigins specifies the allowed origins for CORS requests to the Curio admin UI. If empty, CORS is disabled. # If not empty, only the specified origins will be allowed for CORS requests. # This is required for third-party UI servers. # "*" allows everyone, it's best to specify the UI servers' hostname. # # type: []string #CORSOrigins = [] # CSP sets the Content Security Policy for content served via the /piece/ retrieval endpoint. # Valid values: "off", "self", "inline" (Default: "inline") # # Since storage providers serve user-uploaded content on their domain, CSP helps control # what these files can do when rendered in browsers. Choose based on your use case: # # - "off": No CSP headers. Content can load any external resources and execute any scripts. # Use only if you fully trust all stored content or need maximum compatibility. # # - "self": Restricts content to only load resources from your domain. Prevents external # resource loading but allows stored HTML/JS/CSS to interact with each other. # Good for semi-trusted content that needs internal functionality. # # - "inline": (Default) Allows inline scripts/styles and same-origin resources. Provides # basic protection while maintaining compatibility with most web content. # Suitable for general-purpose content hosting. # # Note: Stricter policies may prevent some HTML content from displaying as intended. # Consider the trust level of your users and whether you need to support interactive content. # # type: string #CSP = "inline" # DenylistServers is a list of URLs pointing to denylist.json files. # Each URL should serve a JSON array of objects with an "anchor" field containing a SHA256 hash. # Denylisted CIDs will be rejected with HTTP 451. Requests arriving before denylists are loaded # will receive HTTP 503. (Default: ["https://badbits.dwebops.pub/denylist.json"]) # Updates will affect running instances. # # type: []string #DenylistServers = ["https://badbits.dwebops.pub/denylist.json"] # CompressionLevels hold the compression level for various compression methods supported by the server # # type: CompressionConfig [HTTP.CompressionLevels] # type: int #GzipLevel = 6 # type: int #BrotliLevel = 4 # type: int #DeflateLevel = 6 # Market specifies configuration options for the Market subsystem within the Curio node. # # type: MarketConfig [Market] # StorageMarketConfig houses all the deal related market configuration # # type: StorageMarketConfig [Market.StorageMarketConfig] # PieceLocator is a list of HTTP url and headers combination to query for a piece for offline deals # User can run a remote file server which can host all the pieces over the HTTP and supply a reader when requested. # The server must support "HEAD" request and "GET" request. # 1. ?id=pieceCID with "HEAD" request responds with 200 if found or 404 if not. Must send header "Content-Length" with file size as value # 2. ?id=pieceCID must provide a reader for the requested piece along with header "Content-Length" with file size as value # # type: []PieceLocatorConfig #PieceLocator = [] # MK12 encompasses all configuration related to deal protocol mk1.2.0 and mk1.2.1 (i.e. Boost deals) # # type: MK12Config [Market.StorageMarketConfig.MK12] # When a deal is ready to publish, the amount of time to wait for more # deals to be ready to publish before publishing them all as a batch # Time duration string (e.g., "1h2m3s") in TOML format. (Default: "5m0s") # # type: time.Duration #PublishMsgPeriod = "5m0s" # The maximum number of deals to include in a single PublishStorageDeals # message (Default: 8) # # type: uint64 #MaxDealsPerPublishMsg = 8 # The maximum fee to pay per deal when sending the PublishStorageDeals message # Accepts a decimal string (e.g., "123.45" or "123 fil") with optional "fil" or "attofil" suffix. (Default: "0.5 FIL") # # type: types.FIL #MaxPublishDealFee = "0.5 FIL" # ExpectedPoRepSealDuration is the expected time it would take to seal the deal sector # This will be used to fail the deals which cannot be sealed on time. # Time duration string (e.g., "1h2m3s") in TOML format. (Default: "8h0m0s") # # type: time.Duration #ExpectedPoRepSealDuration = "8h0m0s" # ExpectedSnapSealDuration is the expected time it would take to snap the deal sector # This will be used to fail the deals which cannot be sealed on time. # Time duration string (e.g., "1h2m3s") in TOML format. (Default: "2h0m0s") # # type: time.Duration #ExpectedSnapSealDuration = "2h0m0s" # SkipCommP can be used to skip doing a commP check before PublishDealMessage is sent on chain # Warning: If this check is skipped and there is a commP mismatch, all deals in the # sector will need to be sent again (Default: false) # # type: bool #SkipCommP = false # MaxConcurrentDealSizeGiB is a sum of all size of all deals which are waiting to be added to a sector # When the cumulative size of all deals in process reaches this number, new deals will be rejected. # (Default: 0 = unlimited) # # type: int64 #MaxConcurrentDealSizeGiB = 0 # DenyUnknownClients determines the default behaviour for the deal of clients which are not in allow/deny list # If True then all deals coming from unknown clients will be rejected. (Default: false) # # type: bool #DenyUnknownClients = false # DenyOnlineDeals determines if the storage provider will accept online deals (Default: false) # # type: bool #DenyOnlineDeals = false # DenyOfflineDeals determines if the storage provider will accept offline deals (Default: false) # # type: bool #DenyOfflineDeals = false # CIDGravityTokens is the list of authorization token to use for CIDGravity filters. These should be in format # "minerID1:Token1", "minerID2:Token2". If a token for a minerID within the cluster is not provided, # CIDGravity filters will not be applied to deals associated with that miner ID. # # type: []string #CIDGravityTokens = [] # DefaultCIDGravityAccept when set to true till accept deals when CIDGravity service is not available. # Default behaviors is to reject the deals (Default: false) # # type: bool #DefaultCIDGravityAccept = false # MK20 encompasses all configuration related to deal protocol mk2.0 i.e. market 2.0 # # type: MK20Config [Market.StorageMarketConfig.MK20] # ExpectedPoRepSealDuration is the expected time it would take to seal the deal sector # This will be used to fail the deals which cannot be sealed on time. # Time duration string (e.g., "1h2m3s") in TOML format. (Default: "8h0m0s") # # type: time.Duration #ExpectedPoRepSealDuration = "8h0m0s" # ExpectedSnapSealDuration is the expected time it would take to snap the deal sector # This will be used to fail the deals which cannot be sealed on time. # Time duration string (e.g., "1h2m3s") in TOML format. (Default: "2h0m0s") # # type: time.Duration #ExpectedSnapSealDuration = "2h0m0s" # SkipCommP can be used to skip doing a commP check before PublishDealMessage is sent on chain # Warning: If this check is skipped and there is a commP mismatch, all deals in the # sector will need to be sent again (Default: false) # # type: bool #SkipCommP = false # MaxConcurrentDealSizeGiB is a sum of all size of all deals which are waiting to be added to a sector # When the cumulative size of all deals in process reaches this number, new deals will be rejected. # (Default: 0 = unlimited) # # type: int64 #MaxConcurrentDealSizeGiB = 0 # DenyUnknownClients determines the default behaviour for the deal of clients which are not in allow/deny list # If True then all deals coming from unknown clients will be rejected. (Default: false) # # type: bool #DenyUnknownClients = false # MaxParallelChunkUploads defines the maximum number of upload operations that can run in parallel. (Default: 512) # # type: int #MaxParallelChunkUploads = 512 # MinimumChunkSize defines the smallest size of a chunk allowed for processing, expressed in bytes. Must be a power of 2. (Default: 16 MiB) # # type: int64 #MinimumChunkSize = 16777216 # MaximumChunkSize defines the maximum size of a chunk allowed for processing, expressed in bytes. Must be a power of 2. (Default: 256 MiB) # # type: int64 #MaximumChunkSize = 268435456 # IPNI configuration for ipni-provider # # type: IPNIConfig [Market.StorageMarketConfig.IPNI] # Disable set whether to disable indexing announcement to the network and expose endpoints that # allow indexer nodes to process announcements. Default: False # # type: bool #Disable = false # The network indexer web UI URL for viewing published announcements # # type: []string #ServiceURL = ["https://cid.contact", "https://filecoinpin.contact"] # The list of URLs of indexing nodes to announce to. This is a list of hosts we talk to tell them about new # heads. # # type: []string #DirectAnnounceURLs = ["https://cid.contact/ingest/announce", "https://filecoinpin.contact/announce"] # Indexing configuration for deal indexing # # type: IndexingConfig [Market.StorageMarketConfig.Indexing] # Number of records per insert batch # # type: int #InsertBatchSize = 1000 # Number of concurrent inserts to split AddIndex calls to # # type: int #InsertConcurrency = 10 # Ingest defines configuration parameters for handling and limiting deal ingestion pipelines within the Curio node. # # type: CurioIngestConfig [Ingest] # MaxMarketRunningPipelines is the maximum number of market pipelines that can be actively running tasks. # A "running" pipeline is one that has at least one task currently assigned to a machine (owner_id is not null). # If this limit is exceeded, the system will apply backpressure to delay processing of new deals. # 0 means unlimited. (Default: 64) # Updates will affect running instances. # # type: int #MaxMarketRunningPipelines = 64 # MaxQueueDownload is the maximum number of pipelines that can be queued at the downloading stage, # waiting for a machine to pick up their task (owner_id is null). # If this limit is exceeded, the system will apply backpressure to slow the ingestion of new deals. # 0 means unlimited. (Default: 8) # Updates will affect running instances. # # type: int #MaxQueueDownload = 8 # MaxQueueCommP is the maximum number of pipelines that can be queued at the CommP (verify) stage, # waiting for a machine to pick up their verification task (owner_id is null). # If this limit is exceeded, the system will apply backpressure, delaying new deal processing. # 0 means unlimited. (Default: 8) # Updates will affect running instances. # # type: int #MaxQueueCommP = 8 # Maximum number of sectors that can be queued waiting for deals to start processing. # 0 = unlimited # Note: This mechanism will delay taking deal data from markets, providing backpressure to the market subsystem. # The DealSector queue includes deals that are ready to enter the sealing pipeline but are not yet part of it. # DealSector queue is the first queue in the sealing pipeline, making it the primary backpressure mechanism. (Default: 8) # Updates will affect running instances. # # type: int #MaxQueueDealSector = 8 # Maximum number of sectors that can be queued waiting for SDR to start processing. # 0 = unlimited # Note: This mechanism will delay taking deal data from markets, providing backpressure to the market subsystem. # The SDR queue includes deals which are in the process of entering the sealing pipeline. In case of the SDR tasks it is # possible that this queue grows more than this limit(CC sectors), the backpressure is only applied to sectors # entering the pipeline. # Only applies to PoRep pipeline (DoSnap = false) (Default: 8) # Updates will affect running instances. # # type: int #MaxQueueSDR = 8 # Maximum number of sectors that can be queued waiting for SDRTrees to start processing. # 0 = unlimited # Note: This mechanism will delay taking deal data from markets, providing backpressure to the market subsystem. # In case of the trees tasks it is possible that this queue grows more than this limit, the backpressure is only # applied to sectors entering the pipeline. # Only applies to PoRep pipeline (DoSnap = false) (Default: 0) # Updates will affect running instances. # # type: int #MaxQueueTrees = 0 # Maximum number of sectors that can be queued waiting for PoRep to start processing. # 0 = unlimited # Note: This mechanism will delay taking deal data from markets, providing backpressure to the market subsystem. # Like with the trees tasks, it is possible that this queue grows more than this limit, the backpressure is only # applied to sectors entering the pipeline. # Only applies to PoRep pipeline (DoSnap = false) (Default: 0) # Updates will affect running instances. # # type: int #MaxQueuePoRep = 0 # MaxQueueSnapEncode is the maximum number of sectors that can be queued waiting for UpdateEncode tasks to start. # 0 means unlimited. # This applies backpressure to the market subsystem by delaying the ingestion of deal data. # Only applies to the Snap Deals pipeline (DoSnap = true). (Default: 16) # Updates will affect running instances. # # type: int #MaxQueueSnapEncode = 16 # MaxQueueSnapProve is the maximum number of sectors that can be queued waiting for UpdateProve to start processing. # 0 means unlimited. # This applies backpressure in the Snap Deals pipeline (DoSnap = true) by delaying new deal ingestion. (Default: 0) # Updates will affect running instances. # # type: int #MaxQueueSnapProve = 0 # Maximum time an open deal sector should wait for more deals before it starts sealing. # This ensures that sectors don't remain open indefinitely, consuming resources. # Time duration string (e.g., "1h2m3s") in TOML format. (Default: "1h0m0s") # Updates will affect running instances. # # type: time.Duration #MaxDealWaitTime = "1h0m0s" # DoSnap, when set to true, enables snap deal processing for deals ingested by this instance. # Unlike lotus-miner, there is no fallback to PoRep when no snap sectors are available. # When enabled, all deals will be processed as snap deals. (Default: false) # # type: bool #DoSnap = false # DisableSSRFProtection disables all SSRF (Server-Side Request Forgery) protection # on deal data URL fetching. When true, URLs pointing to private IPs, loopback # addresses, and local hostnames are allowed. URL structure validation (scheme, # control characters) is still enforced. # WARNING: Only enable in development or testing environments. (Default: false) # Updates will affect running instances. # # type: bool #DisableSSRFProtection = false # SSRFAllowedHosts is a list of hosts or host:port pairs that are allowed through # SSRF protection. Matched hosts bypass IP and hostname restrictions while other # safety checks (URL scheme, headers) remain active. # Entries without a port match any port for that host. # Example: ["192.168.1.100:8080", "my-dev-server.local"] # (Default: []) # Updates will affect running instances. # # type: []string #SSRFAllowedHosts = [] # Seal defines the configuration related to the sealing process in Curio. # # type: CurioSealConfig [Seal] # BatchSealSectorSize Allows setting the sector size supported by the batch seal task. # Can be any value as long as it is "32GiB". (Default: "32GiB") # # type: string #BatchSealSectorSize = "32GiB" # Number of sectors in a seal batch. Depends on hardware and supraseal configuration. (Default: 32) # # type: int #BatchSealBatchSize = 32 # Number of parallel pipelines. Can be 1 or 2. Depends on available raw block storage (Default: 2) # # type: int #BatchSealPipelines = 2 # SingleHasherPerThread is a compatibility flag for older CPUs. Zen3 and later supports two sectors per thread. # Set to false for older CPUs (Zen 2 and before). (Default: false) # # type: bool #SingleHasherPerThread = false # Apis defines the configuration for API-related settings in the Curio system. # # type: ApisConfig [Apis] # API auth secret for the Curio nodes to use. This value should only be set on the bade layer. # # type: string #StorageRPCSecret = "" # Alerting specifies configuration settings for alerting mechanisms, including thresholds and external integrations. # # type: CurioAlertingConfig [Alerting] # ClusterName identifies the Curio cluster in external alerts. When empty, the hostname of the node sending the alert is used. # # type: string #ClusterName = "" # MinimumWalletBalance is the minimum balance all active wallets. If the balance is below this value, an # alerts will be triggered for the wallet # Accepts a decimal string (e.g., "123.45" or "123 fil") with optional "fil" or "attofil" suffix. (Default: "5 FIL") # # type: types.FIL #MinimumWalletBalance = "5 FIL" # PagerDutyConfig is the configuration for the PagerDuty alerting integration. # # type: PagerDutyConfig [Alerting.PagerDuty] # Enable is a flag to enable or disable the PagerDuty integration. # # type: bool #Enable = false # PagerDutyEventURL is URL for PagerDuty.com Events API v2 URL. Events sent to this API URL are ultimately # routed to a PagerDuty.com service and processed. # The default is sufficient for integration with the stock commercial PagerDuty.com company's service. # # type: string #PagerDutyEventURL = "https://events.pagerduty.com/v2/enqueue" # PageDutyIntegrationKey is the integration key for a PagerDuty.com service. You can find this unique service # identifier in the integration page for the service. # # type: string #PageDutyIntegrationKey = "" # PrometheusAlertManagerConfig is the configuration for the Prometheus AlertManager alerting integration. # # type: PrometheusAlertManagerConfig [Alerting.PrometheusAlertManager] # Enable is a flag to enable or disable the Prometheus AlertManager integration. # # type: bool #Enable = false # AlertManagerURL is the URL for the Prometheus AlertManager API v2 URL. # # type: string #AlertManagerURL = "http://localhost:9093/api/v2/alerts" # SlackWebhookConfig is a configuration type for Slack webhook integration. # # type: SlackWebhookConfig [Alerting.SlackWebhook] # Enable is a flag to enable or disable the Prometheus AlertManager integration. # # type: bool #Enable = false # WebHookURL is the URL for the URL for slack Webhook. # Example: https://hooks.slack.com/services/T00000000/B00000000/XXXXXXXXXXXXXXXXXXXXXXXX # # type: string #WebHookURL = "" # AppriseConfig is the configuration for the Apprise (https://github.com/caronc/apprise-api) integration. # # type: AppriseConfig [Alerting.Apprise] # URL is the notify endpoint of a running Apprise API server (https://github.com/caronc/apprise-api). # Either its stateless endpoint (e.g. "http://127.0.0.1:8000/notify", use with NotifyURLs) or a # stateful, pre-configured endpoint (e.g. "http://127.0.0.1:8000/notify/curio", leave NotifyURLs empty). # Leave empty to disable the Apprise integration. # # type: string #URL = "" # Tag restricts delivery to Apprise URLs carrying this tag. Only applies to stateful configs. OPTIONAL. # # type: string #Tag = "" # Batching represents the batching configuration for pre-commit, commit, and update operations. # # type: CurioBatchingConfig [Batching] # Precommit Batching configuration # # type: PreCommitBatchingConfig [Batching.PreCommit] # Base fee value below which we should try to send Precommit messages immediately # Accepts a decimal string (e.g., "123.45" or "123 fil") with optional "fil" or "attofil" suffix. (Default: "0.005 FIL") # # type: types.FIL #BaseFeeThreshold = "0.005 FIL" # Maximum amount of time any given sector in the batch can wait for the batch to accumulate # Time duration string (e.g., "1h2m3s") in TOML format. (Default: "4h0m0s") # # type: time.Duration #Timeout = "4h0m0s" # Time buffer for forceful batch submission before sectors/deal in batch would start expiring # Time duration string (e.g., "1h2m3s") in TOML format. (Default: "6h0m0s") # # type: time.Duration #Slack = "6h0m0s" # Maximum number of sectors per precommit batch message. The batch will be submitted # immediately when this many sectors are ready, without waiting for the timeout. # 0 = use the protocol maximum. (Default: 0) # # type: int #MaxBatch = 0 # Commit batching configuration # # type: CommitBatchingConfig [Batching.Commit] # Base fee value below which we should try to send Commit messages immediately # Accepts a decimal string (e.g., "123.45" or "123 fil") with optional "fil" or "attofil" suffix. (Default: "0.005 FIL") # # type: types.FIL #BaseFeeThreshold = "0.005 FIL" # Maximum amount of time any given sector in the batch can wait for the batch to accumulate # Time duration string (e.g., "1h2m3s") in TOML format. (Default: "1h0m0s") # # type: time.Duration #Timeout = "1h0m0s" # Time buffer for forceful batch submission before sectors/deals in batch would start expiring # Time duration string (e.g., "1h2m3s") in TOML format. (Default: "1h0m0s") # # type: time.Duration #Slack = "1h0m0s" # Maximum number of sectors per commit batch message. The batch will be submitted # immediately when this many sectors are ready, without waiting for the timeout. # 0 = use the protocol maximum. (Default: 0) # # type: int #MaxBatch = 0 # Snap Deals batching configuration # # type: UpdateBatchingConfig [Batching.Update] # Base fee value below which we should try to send Commit messages immediately # Accepts a decimal string (e.g., "123.45" or "123 fil") with optional "fil" or "attofil" suffix. (Default: "0.005 FIL") # # type: types.FIL #BaseFeeThreshold = "0.005 FIL" # Maximum amount of time any given sector in the batch can wait for the batch to accumulate # Time duration string (e.g., "1h2m3s") in TOML format. (Default: "1h0m0s") # # type: time.Duration #Timeout = "1h0m0s" # Time buffer for forceful batch submission before sectors/deals in batch would start expiring # Time duration string (e.g., "1h2m3s") in TOML format. (Default: "1h0m0s") # # type: time.Duration #Slack = "1h0m0s" # Cuzk configures integration with the cuzk proving daemon. # When enabled, SNARK proving tasks (PoRep C2, SnapDeals prove, and PSProve) are delegated # to an external cuzk daemon over gRPC instead of using local GPU resources. # # type: CuzkConfig [Cuzk] # Address of the cuzk daemon gRPC endpoint. # Supports unix socket (e.g., "unix:///run/curio/cuzk.sock") or TCP (e.g., "127.0.0.1:9820"). # Empty string disables cuzk integration. (Default: "") # # type: string #Address = "" # MaxPending is the maximum number of proof jobs that may be pending in the cuzk daemon queue # before Curio stops accepting new proving tasks (backpressure). When the daemon's pending # queue reaches this level, CanAccept will reject new tasks until capacity frees up. # (Default: 10) # # type: int #MaxPending = 10 # ProveTimeout is the maximum time to wait for a proof result from the cuzk daemon. # If the proof is not completed within this duration, the task will be retried. # Time duration string (e.g., "30m", "1h"). (Default: "30m") # # type: time.Duration #ProveTimeout = "30m0s" ``` # Curio Market An overview of the Curio market module The Curio Storage Market is a comprehensive framework designed to manage Filecoin storage deals using a flexible and scalable architecture that supports multiple protocols, currently including MK1.2, with future protocols planned for seamless integration. The market handles deal-making through both online and offline processes, providing efficient tools for each step of the deal lifecycle. * **Protocols:** The market is designed to support multiple deal protocols. Currently, MK1.2 is implemented, but the system can easily expand to accommodate future protocols without major reconfiguration. * **Deal Flow:** The market manages both online and offline deals, ensuring smooth operations for various storage needs. Offline deals can utilize a PieceLocator configuration with HTTP servers to retrieve pieces when requested. * **libp2p and Networking:** Curio’s market integrates with libp2p for peer-to-peer communication, allowing for decentralized and secure data transmission between storage clients and providers. * **HTTP Server:** The HTTP server plays a vital role in managing tasks like offline deal data retrieval through URLs and handling client requests for deal processing. * **Retrieval and Indexing:** Deals are efficiently indexed and can be retrieved for future use, ensuring that stored data remains accessible and manageable throughout its lifecycle. The Curio Storage Market provides a highly configurable and extensible system for deal management, optimized for current and future decentralized storage protocols. ## Curio vs Boost How is Curio market different from Boost? ### Deal Processing In Curio’s deal processing, the task-based approach introduces modularity and flexibility in handling different stages of a deal's lifecycle. Each step, from data preparation to sealing, is managed as an individual task. This approach breaks down the complexity of the process by distributing specific responsibilities, such as data validation, publishing deals, and commP calculation, across multiple independent tasks. Each task is monitored, retried on failure, and orchestrated through the Harmony task system, ensuring better resource utilization and scalability. This structure allows for parallel processing of deals and ensures that failures in one part do not impact the entire deal flow. Additionally, it provides flexibility for incorporating new protocols as they are introduced, seamlessly integrating them into the task orchestration. This method ensures that Curio’s deal processing remains future-proof, adaptable to different protocols, and scalable across large volumes of data and interactions. ### Offline Deals In Boost, offline deals were initiated by manually importing data on the storage provider's side. However, Curio simplifies this process by allowing users to [add a URL for offline deal data directly into the database](/curio-market/storage-market#add-data-url-for-offline-deals) or use the [`PieceLocator` configuration](/curio-market/storage-market#piecelocator-configuration) to point to a remote server that can serve the deal’s piece. Since Curio operates as a cluster rather than a single node, data might be needed on different nodes depending on task scheduling and execution. The remote read design in Curio provides flexibility by enabling any node to fetch the required data dynamically during task execution, ensuring efficient processing across the cluster. ### Optional CommP In Boost, the CommP (Commitment to Piece) calculation was a mandatory step in the deal processing workflow. However, in Curio, CommP is optional and can be skipped if desired. This flexibility allows users to streamline deal processing by bypassing the CommP check when necessary, while still ensuring that deals can progress smoothly without compromising efficiency in specific use cases. ### HTTP only retrievals Curio’s retrieval mechanism is HTTP-only, allowing serving deal data from multiple nodes, each running an HTTP server. These nodes can operate under different domain names, providing flexibility and scalability. This distributed retrieval design ensures that data can be fetched from any node within the cluster, based on availability and proximity, optimizing retrieval speed. By supporting multiple nodes and domains for HTTP-based retrieval, Curio enhances resilience and performance, allowing clients to access deal data efficiently without relying on a single centralized source, all within the HTTP protocol. ### IPNI Sync Curio creates dedicated peerID for each miner ID which are used to identify the provider on IPNI. This peer ID is different from the peer ID on chain. Curio announces per piece instead of per deal to IPNI nodes. This removes unnecessary overhead of mapping deals to ads and instead takes a simpler approach of advertising only for the piece we have regardless of how many times they are onboarded. The chunking in Curio process leverages a database to ensure fast and efficient chunk creation and reconstruction. By storing the first CID of each chunk along with metadata like offsets and chunk numbers, the system can quickly locate and retrieve chunks. Sorted entries minimize redundancy by eliminating duplicates and allow for efficient querying and retrieval. Additionally, caching mechanisms and database indexing further enhance speed, making the system optimized for rapid chunking and reconstruction. This makes IPNI sync much faster in Curio compared to Boost. # Storage Market A comprehensive guide to configuring and managing storage deals in Curio ## Overview The Curio Storage Market provides a comprehensive framework for managing deals, data retrieval, and storage through decentralized protocols. This page details the different configurations, commands, and workflows available for managing storage deals, both online and offline. The storage market in Curio is built around several key concepts: * **Deal Protocols**: Protocols and workflows for deal-making, sealing, and data transfers. * **Tasks**: Various tasks managed by the storage market, such as commP, PSD, indexing deals, IPNI advertisement etc. * **Deal Flows**: The workflows for processing online and offline deals, each of which has specific tasks and checks to ensure the deal is successfully completed. ## Configuration The Curio storage market is configurable through the `StorageMarketConfig` structure. This section outlines the main configuration parameters and the implications of setting them. ```go type MarketConfig struct { StorageMarketConfig StorageMarketConfig } type StorageMarketConfig struct { MK12 MK12Config IPNI IPNIConfig Indexing IndexingConfig PieceLocator []PieceLocatorConfig } ``` ### **MK12 Configuration** The `MK12` configuration encompasses all deal-related settings for the **MK1.2.0** and **MK1.2.1** deal protocols (commonly referred to as **Boost deals**). This configuration controls key parameters like batching, sealing time, and the number of deals that can be published at once. ```go type MK12Config struct { PublishMsgPeriod Duration MaxDealsPerPublishMsg uint64 MaxPublishDealFee types.FIL ExpectedPoRepSealDuration Duration ExpectedSnapSealDuration Duration SkipCommP bool DisabledMiners []string MaxConcurrentDealSizeGiB int64 DenyUnknownClients bool DenyOnlineDeals bool DenyOfflineDeals bool CIDGravityToken string DefaultCIDGravityAccept bool } ``` **Key Parameters** 1. **PublishMsgPeriod**:\ Specifies the time to wait before publishing deals as a batch. Increasing this period allows more deals to be included in a single message but delays the publishing of deals. Lowering this period will result in faster publishing but fewer deals being batched together, increasing chain overhead. 2. **MaxDealsPerPublishMsg**:\ Controls the maximum number of deals to include in one batch. If set too high, the publish message may become too large and expensive. Setting it too low might reduce efficiency, as the node sends more messages. 3. **MaxPublishDealFee**:\ This defines the maximum fee you’re willing to pay per deal when sending the `PublishStorageDeals` message. The consequence of setting a low fee is that deal publishing may be delayed or fail if network congestion raises gas costs. 4. **ExpectedPoRepSealDuration**:\ This value controls how long you expect the Proof of Replication (PoRep) sealing process to take. Deals that cannot be sealed within this time will fail. 5. **ExpectedSnapSealDuration**:\ Similar to PoRep, this defines the expected time for snap sealing. The duration should account for hardware speed and network delays. 6. **SkipCommP**:\ If set to `true`, the CommP (Commitment Proof) check is skipped before the `PublishDealMessage` is sent on-chain. Skipping this step is risky because if there’s a mismatch, all deals in the sector may need to be resent. 7. **DisabledMiners**:\ A list of miner addresses excluded from participating in deal-making. Use this option to prevent specific miners from handling deals if needed. 8. **MaxConcurrentDealSizeGiB:**\ MaxConcurrentDealSizeGiB is a sum of all size of all deals which are waiting to be added to a sector when the cumulative size of all deals in process reaches this number, new deals will be rejected. (Default: 0 = unlimited) 9. **DenyUnknownClients:**\ DenyUnknownClients determines the default behaviour for the deal of clients which are not in allow/deny list. If True then all deals coming from unknown clients will be rejected. 10. **DenyOnlineDeals**: Determines whether the storage provider **accepts online deals**. 11. **DenyOfflineDeals**: Determines whether the storage provider **accepts offline deals**. 12. **CIDGravityToken**:\ The authorization token used for **CIDGravity filters**, a service that filters deal proposals based on custom policies. If empty (`""`), **CIDGravity filtering is disabled**. If set, the miner will **query CIDGravity** for each deal proposal before accepting it. 13. **DefaultCIDGravityAccept**:\ Defines what happens if the **CIDGravity service is unavailable**. If`true`: **Accepts deals** even if CIDGravity is unreachable. If`false`: **Rejects deals** when CIDGravity is unavailable (**default**). ### **PieceLocator Configuration** This configuration allows you to set up remote HTTP servers that provide piece data for offline deals. A `PieceLocator` config is a combination of a URL and headers for fetching pieces when requested by the miner. This is crucial for handling offline deals where data is not available immediately and must be retrieved during the commP and sealing phase. ```go type PieceLocatorConfig struct { URL string Headers http.Header } ``` * **URL**: The endpoint where the piece data can be located. * **Headers**: Any custom headers needed for the HTTP request, such as authorization tokens. {% hint style="warning" %} PieceLocator service will allow Curio to lookup details of a piece automatically for an offline deal. The `add-url` command should not be used for deal which are expected to fetch the data from PieceLocator services. {% endhint %} Consequences: If the piece data is not available at the specified URL, the offline deal will fail. Make sure that the remote server is properly configured and available. ## Enabling Storage Market To enable the Curio market on a Curio node, the following configuration changes are required: 1. **Enable the Deal Market**: * Set `EnableDealMarket` to `true` in the `CurioSubsystemsConfig` for at least one node. This enables deal-making capabilities on the node. 2. **Enable CommP**: * On one of the nodes where `EnableDealMarket` is set to `true`, ensure that `EnableCommP` is also set to `true`. This allows the node to compute piece commitments (CommP) before publishing storage deal messages. 3. **Enable HTTP**: * At least one node must have HTTP enabled to support: * Retrievals. * IPNI sync. * Handling storage deals. * To enable HTTP, set the `Enable` flag in the `HTTPConfig` to `true` and specify the `ListenAddress` for the HTTP server. 4. **Set a Domain Name**: * Ensure that a valid `DomainName` is specified in the `HTTPConfig`. This is mandatory for proper HTTP server functionality and essential for enabling TLS. The domain name cannot be an IP address. * In case `DelegateTLS` is `False` , the domain name must point to the public IP address your curio node is listening on. The purpose of setting this field is to allow lets encrypt ACME protocol to automatically issue a certificate to use TLS for encrypting access to the curio api. For let's encrypt policy reasons this will only work if curio listens on port 443. * Domain name should be specified in the base layer 5. **HTTP Configuration Details**: * If TLS is managed by a reverse proxy, enable `DelegateTLS` in the `HTTPConfig` to allow the HTTP server to run without handling TLS directly. * Configure additional parameters such as `ReadTimeout`, `IdleTimeout`, and `CompressionLevels` to ensure the server operates efficiently. 6. **Libp2p Activation**: * The `libp2p` service will automatically start on one of the servers running the HTTP server where `EnableDealMarket` is set to `true`. If more than 1 node satsifies the condition and the node running libp2p goes down then it will switch over to another node after 5 minutes. 7. **Other Considerations**: * Ensure the `MK12Config` settings under `StorageMarketConfig` are properly configured for deal publishing. Key parameters include: * `PublishMsgPeriod` for deal batching frequency. * `MaxDealsPerPublishMsg` for the maximum number of deals per message. * `MaxPublishDealFee` to set the fee limit for publishing deals. * If handling offline deals, configure `PieceLocator` to specify the endpoints for piece retrieval. 8. Verify that HTTP server is working: * Curl to your domain name and verify that server is reachable from outside\\ ```shell curl https:// Hello, World! -Curio ``` {% hint style="warning" %} If you do not get above output then something went wrong with configuration and you should not proceed with migration from Boost or Deal making. {% endhint %} By applying these changes, the Curio market subsystem will be activated on the specified node(s), enabling storage deals, IPNI synchronization, and retrieval functionality. ## MK12 Deals (Boost Deals) The **MK12** protocol governs the entire deal process, from proposing and publishing deals to sealing and validating them. It's designed to work efficiently for both online and offline deals. Key tasks within MK12 include: * **Piece Commitment**: Ensuring that the piece has been added to a sector. * **Publish Storage Deals (PSD)**: Sending the on-chain message to register the deal. * **Finding Deals**: Identifying the deal ID on-chain and adding it to the sector. ### MK12 Tasks Tasks refer to operations that the system performs on deals to ensure their success. Tasks include: * **CommP tasks**: Commitment proof tasks, ensuring the piece information is correct by calculating the commitment locally. * **PSD tasks**: Tasks for sending and validating the PublishStorageDeals message. * **Find Deal tasks**: These tasks poll the blockchain to identify the deal’s status and retrieve its ID after the deal has been published successfully with PSD task. ### Online Deal Flow 1. **Deal Proposal**:\ A client proposes a deal, specifying the amount of data and terms. 2. **Data Transfer**:\ Data is transferred immediately. The system checks that the entire piece has been received. 3. **CommP Task**:\ Once data is received, a commitment proof is generated. 4. **PSD Task**:\ The deal is published on-chain. 5. **Sector Assignment**:\ The deal is assigned to a sector and sealed. ### Offline Deal Flow 1. **Deal Proposal**:\ The client proposes a deal for data that is not yet available on the miner’s node. 2. **PieceLocator**:\ The miner is provided with a URL where the data can be fetched later. 3. **Data Fetch**:\ The miner fetches the piece from the provided URL using the `PieceLocator` configuration or [local database](#add-data-url-for-offline-deals). 4. **CommP Task**:\ Once the piece is retrieved, a commitment proof is generated. 5. **PSD Task**:\ The deal is published on-chain. 6. **Sector Assignment**:\ The deal is added to a sector and sealed. ### Add data URL for offline deals Why this exists (plain language): * Curio needs a way to *fetch the bytes* for the piece when it’s time to compute CommP / index / snap. * For offline deals, there isn’t necessarily a live HTTP endpoint by default—so you must provide one (or otherwise ensure the bytes are accessible via your chosen ingestion method). If you skip this step, you will typically see ParkPiece failures such as: * `no suitable data URL found for piece_id ...` The `add-url` command allows you to specify a URL from which the miner can fetch piece data for offline deals. This is essential for deals where the client does not transfer the data immediately upon deal acceptance. {% hint style="warning" %} The `add-url` command should not be used for deal which are expected to fetch the data from PieceLocator services. {% endhint %} ```bash curio market add-url [command options] ``` **Example Usage**: ```bash curio market add-url --url "https://data.server/pieces?id=pieceCID" --header "Authorization: Bearer token" OR curio market add-url --url "https://data.server/filename" --header "Authorization: Bearer token" OR curio market add-url --url "https://data.server/filename" ``` **Options**: * **--url**: The URL where the piece data can be fetched. * **--header**: Custom headers to include in the HTTP request. Consequences: If the URL is not accessible or the headers are incorrect, the deal will fail to retrieve the data and will not be able to complete. ### Move funds to escrow The `move-to-escrow` command moves funds from the deal collateral wallet to the escrow account with the storage market actor. This is necessary to lock in collateral for a deal. ```bash curio market move-to-escrow [command options] ``` **Example Usage**: ```bash curio market move-to-escrow --actor --max-fee 3 ``` **Options**: * **--actor**: Specifies the actor address that should start sealing sectors for the deal. * **--max-fee**: Maximum fee in FIL you’re willing to pay for this message. Consequences: If insufficient funds are moved to escrow, the deal may not be processed, and the collateral may not be secured. ### Start Sealing Early The `seal` command allows you to start sealing a deal's sector early, before all the deals have been batched. ```bash curio market seal [command options] ``` **Example Usage**: ```bash curio market seal --actor ``` **Options**: * **--actor**: Specifies the actor address. Consequences: Sealing early can speed up the process, but it may result in inefficiencies if all deals are not batched correctly. ## Offline Verified DDO deals Curio only supports offline verified DDO deals as of now. The allocation must be created by the client for the piece and handed over to the SP alongside the data. ### How to create allocation Clients can create allocation using the `sptool toolbox` or other methods. ```shell sptool --actor t01000 toolbox mk12-client allocate -p --piece-cid --piece-size ``` ### Start a DDO deal Storage providers can onboard the DDO deal using the below command. ```shell curio market ddo --actor ``` This command does **not** fetch any data. It only: 1. Validates the allocation against the chain and the actor. 2. Generates a fresh deal UUID. 3. Inserts a row into `market_direct_deals` and a corresponding offline-deal row into `market_mk12_deal_pipeline` (with `offline=true` and `started=false`). 4. Prints the deal UUID to stdout, for example: ``` Direct deals inserted successfully: 0f1b2c3d-... ``` Because the pipeline row is created with `started=false`, the commP task will not run until Curio has resolved a source URL for the piece. There are two supported ways to provide that URL: #### Option A: PieceLocator (preferred for bulk / automated ingestion) If `[Market.StorageMarketConfig.PieceLocator]` is configured, Curio will automatically discover the piece by issuing a `HEAD ?id=` against each configured locator. The first locator that returns `200 OK` with a valid `Content-Length` wins, and Curio writes that URL and `raw_size` into the pipeline row, flips `started=true`, and the commP task picks it up on its next tick. This is the only path you need if your locator already has the piece. You do not need to (and should not) also call `add-url` for the same deal; see the warning in [Add data URL for offline deals](#add-data-url-for-offline-deals). #### Option B: `curio market add-url` (manual / one-off) If you do not run a PieceLocator (or the locator does not have this particular piece), pass the UUID printed by `curio market ddo` to `add-url`: ```shell curio market add-url \ --url "https://data.server/pieces?id=" \ \ ``` This inserts a row into `market_offline_urls` keyed on the UUID. On its next pass, the storage-market poller calls `findURLForOfflineDeals`, joins `market_offline_urls` into `market_mk12_deal_pipeline`, flips `started=true`, and the commP task proceeds. #### Troubleshooting CommP mismatches on DDO deals If the commP task finishes much faster than expected (for example, a few seconds for a 32 GiB piece) and reports `commP mismatch calculated and supplied `, the most common causes are: * **PieceLocator returned a misleading response.** A locator that answers `200 OK` with a `Content-Length` on HEAD but serves wrong, partial, or unrelated bytes on GET will look healthy to Curio's URL resolver and only fail later during commP. This shows up as a subset of deals failing while most succeed, because the locator happens to have valid data for some PieceCIDs and not others. Verify each locator with: ```bash curl -I '?id=' curl -s -o /tmp/piece.bin '?id=' && wc -c /tmp/piece.bin ``` Confirm the body length matches the expected raw size and that `Content-Type` is what you expect. Bad or stale `PieceLocator` entries are a frequent culprit. * **Wrong `raw size` passed to `add-url`.** The commP task pads the fetched stream up to the declared raw size using `padreader.New(reader, rawSize)`. If `rawSize` is smaller than the real piece payload, the padder will truncate; if it is larger, the padder will append zeros. Either way the final CommP will not match the allocation's PieceCID. Always use the unpadded CAR / raw byte size from the same source that produced the PieceCID. * **Both `add-url` and a PieceLocator hit on the same deal.** Only one source is used per deal; if you call `add-url` for a deal that the PieceLocator can already serve, the two URL writers can race depending on which poller pass runs first. Pick one method per deal. You can inspect the resolved state directly: ```sql SELECT uuid, started, after_commp, url, raw_size, piece_cid, piece_size FROM market_mk12_deal_pipeline WHERE uuid = ''; ``` If `url` is `NULL` after several poller cycles, neither PieceLocator nor `add-url` has produced a source yet, and the deal will not advance. # Deal filters How to setup and use storage deal filters ## Deal filters ### Overview Curio provides a flexible filtering system to manage storage deals effectively. User have an option to choose from external filter like [CIDGravity](#cidgravity-filter) and built-in filters. The built-in filters allow you to: * Set pricing rules based on deal duration, size, and whether the data is verified. * Define client-specific rules, including rate limits and acceptable wallets or peers. * Explicitly allow or deny deals from certain wallets, overriding the default acceptance behavior. By configuring these filters, you can optimize your storage provisioning according to your business needs and network policies.

Deal filters

### Pricing Filters Pricing Filters determine the price at which you are willing to accept storage deals based on various criteria such as duration, size, and verification status. ### Adding a Pricing Filter 1. **Navigate to the Pricing Filters section** in the Curio UI. 2. **Click on the "Add Pricing Filter"** button. 3. **Fill in the form fields**: * **Min Duration (Days)**: The minimum deal duration in days. * **Max Duration (Days)**: The maximum deal duration in days. * **Min Size (Bytes)**: The minimum deal size in bytes. * **Max Size (Bytes)**: The maximum deal size in bytes. * **Price (FIL/TiB/Month)**: The price per TiB per month in FIL. * **Verified Deal**: Check this box if the filter applies to verified deals. 4. **Submit the form** by clicking the "Add" button. #### Editing a Pricing Filter 1. **Locate the pricing filter** you wish to edit in the table. 2. **Click the "Edit" button** next to it. 3. **Modify the desired fields** in the form. 4. **Submit the changes** by clicking the "Update" button. #### Removing a Pricing Filter 1. **Find the pricing filter** you want to remove in the table. 2. **Click the "Remove" button** next to it. 3. **Confirm the deletion** when prompted. ### Client Filters Client Filters allow you to define rules for specific clients, including rate limits and acceptable wallets or peers. #### Adding a Client Filter 1. **Go to the Client Filters section** in the Curio UI. 2. **Click on "Add Client Filter"**. 3. **Complete the form**: * **Name**: A unique identifier for the client filter. * **Active**: Check to activate the filter. * **Wallets**: Comma-separated list of client wallet addresses. * **Peers**: Comma-separated list of peer IDs. * **Pricing Filters**: Comma-separated list of pricing filter numbers to apply. * **Max Deals per Hour**: Maximum number of deals allowed per hour. * **Max Deal Size per Hour (GiB)**: Maximum total deal size per hour in GiB. * **Additional Info**: Any extra information or notes. 4. **Click "Add"** to save the filter. #### Editing a Client Filter 1. **Identify the client filter** you want to edit in the list. 2. **Click the "Edit" button** next to it. 3. **Update the fields** as needed. 4. **Click "Update"** to apply the changes. #### Removing a Client Filter 1. **Find the client filter** in the list. 2. **Click the "Remove" button**. 3. **Confirm the deletion** when asked. ### Allow/Deny List The Allow/Deny List lets you explicitly permit or reject deals from specific wallets, overriding the default client acceptance behavior. **Note:** Specify wallets to explicitly allow or deny deals from, overriding the default client acceptance behavior. #### Adding an Allow/Deny Entry 1. **Navigate to the Allow/Deny List section**. 2. **Click "Add Allow/Deny Entry"**. 3. **Fill in the details**: * **Wallet**: The wallet address to allow or deny. * **Allow**: Check to allow, uncheck to deny. 4. **Click "Add"** to save the entry. #### Editing an Allow/Deny Entry 1. **Locate the entry** in the list. 2. **Click the "Edit" button**. 3. **Modify the fields** as necessary. 4. **Click "Update"** to save changes. #### Removing an Allow/Deny Entry 1. **Find the entry** you wish to remove. 2. **Click the "Remove" button**. 3. **Confirm the action** when prompted. ### Default Allow Behaviour The default allow behaviour determines whether deals from unknown clients are accepted or denied by default. This can be modified in Curio configuration. ``` // DenyUnknownClients determines the default behaviour for the deal of clients which are not in allow/deny list // If True then all deals coming from unknown clients will be rejected. DenyUnknownClients bool ``` * **False**: Deals from unknown clients are accepted unless explicitly denied in the Allow/Deny List. * **True**: Deals from unknown clients are rejected unless explicitly allowed in the Allow/Deny List. **To view the current default behaviour**: * The Curio UI displays the default behaviour in the relevant section, indicating whether deals from unknown clients are allowed or denied. ### Usage Examples **Example 1: Accepting Large Verified Deals Only** * **Pricing Filter**: * Min Size: 1 TiB * Max Size: 10 TiB * Verified Deal: Yes * Price: 0 FIL/TiB/Month (or your desired price) * **Client Filter**: * Active: Yes * Max Deals per Hour: 5 * Max Deal Size per Hour: 50 GiB **Example 2: Denying Deals from a Specific Wallet** * **Allow/Deny Entry**: * Wallet: `t01234...` * Allow: Unchecked (deny) **Example 3: Rate Limiting a Client** * **Client Filter**: * Name: `HighVolumeClient` * Active: Yes * Wallets: `t05678...` * Max Deals per Hour: 10 * Max Deal Size per Hour: 100 GiB #### Best Practices * **Use Specific Filters**: Create filters that closely match your desired deal parameters to avoid unwanted deals. * **Regularly Review Filters**: Update your filters as your storage provisioning strategy evolves. * **Test New Filters**: Before deploying filters widely, test them with a small subset to ensure they behave as expected. * **Monitor Deal Flow**: Keep an eye on incoming deals to adjust filters proactively. #### Troubleshooting * **No Deals Being Accepted**: * Check if the default allow behaviour is set to deny. * Ensure your filters are not overly restrictive. * **Error Messages When Saving Filters**: * Review the error displayed in the UI. * Ensure all required fields are filled and values are within acceptable ranges. * Check for duplicates in names or wallet addresses. * **Unexpected Deals Being Accepted or Rejected**: * Verify the order and specificity of your filters. * Check the Allow/Deny List for conflicting entries. * Review client filters to ensure they are active and correctly configured. ## CIDGravity Filter ### What is CIDGravity? [CIDGravity](https://www.cidgravity.com/) is a powerful pricing and client management tool designed for Filecoin storage providers. It enables storage providers to efficiently filter storage and retrieval deals through a user-friendly interface. With CIDGravity, providers can set rules and policies for accepting or rejecting deals based on their business preferences, ensuring better control over their storage operations. For more details, refer to the [CIDGravity documentation](https://docs.cidgravity.com/). ### How to Enable CIDGravity in Curio CIDGravity integration in Curio is controlled through Curio configuration. To enable CIDGravity, you need to set the below parameters in the configuration. We highly recommend setting these values in "base" layer to control the market behaviour and correct UI rendering. ```toml # CIDGravityTokens is the list of authorization tokens to use for CIDGravity filters. # These should be in the format "minerID1:Token1", "minerID2:Token2". # If a token for a minerID within the cluster is not provided, CIDGravity filters will not be applied to deals associated with that miner ID. # # type: []string #CIDGravityTokens = [] # DefaultCIDGravityAccept when set to true will accept deals when CIDGravity service is not available. # Default behavior is to reject the deals (Default: false) # # type: bool #DefaultCIDGravityAccept = false ``` #### Configuration Options: **1. `CIDGravityTokens`** * **Description**: A list of authorization tokens used for CIDGravity filters. Each entry should be formatted as `"minerID:Token"`. * **Default Behavior**: If no token is provided for a `minerID`, CIDGravity filters will **not** be applied to deals associated with that miner ID. * **Type**: `[]string` **Example Configuration:** ```toml CIDGravityTokens = ["t01234:your-auth-token1", "t05678:your-auth-token2"] ``` {% hint style="info" %} To generate a CIDGravity token [claim your miner](https://docs.cidgravity.com/storage-providers/get-started/claim-a-miner/) in CIDGravity. If you already have an existing miner then you can use the same token. {% endhint %} **2. `DefaultCIDGravityAccept`** * **Description**: Defines the default behavior when the CIDGravity service is unavailable. * **Default Behavior**: If set to `false`, deals will be rejected when CIDGravity is not reachable. If set to `true`, deals will be accepted even if CIDGravity is not available. * **Type**: `bool` **Example Configuration:** ```toml DefaultCIDGravityAccept = false ``` #### Steps to Enable CIDGravity in Curio: 1. Obtain a **CIDGravityToken** from the [CIDGravity platform](https://app.cidgravity.com/). 2. Add the token(s) to the Curio configuration file under `CIDGravityTokens` in the format `"minerID:Token"`.

CID Gravity Disabled

3. Set `DefaultCIDGravityAccept` based on your preference: * `true` to accept deals when CIDGravity is unreachable. * `false` to reject deals when CIDGravity is unreachable.

Reject Deals when CID Gravity is unreachable

4. Restart Curio for the changes to take effect. 5. Verify that CIDGravity is enabled via UI Market Settings page.

CID Gravity Enabled

Once enabled, Curio will automatically interact with CIDGravity to apply deal filtering and pricing rules according to the policies set in your CIDGravity account. # Curio HTTP Server This page provides an overview of the Curio HTTP Server's key features, including HTTPS support, security, middleware, routing capabilities, and instructions for attaching custom service routes. The Curio HTTP Server is a secure, flexible, and high-performance HTTP server designed for use with Curio cluster. It comes with built-in support for HTTPS using Let's Encrypt certificates, advanced middleware features like logging and compression, and the ability to integrate various Curio services, including IPNI, retrieval providers, and LibP2P. ## Key Features ### 1. **HTTPS with Let's Encrypt** * Automatic certificate management using Let's Encrypt, ensuring that all traffic is encrypted. * Supports automatic renewal of certificates and domain validation through the `autocert.Manager`. ### 2. **Security-First Design** * **Strict Security Headers**: Adds essential security headers like Strict-Transport-Security, Content-Security-Policy, X-Frame-Options, and XSS protection to mitigate common vulnerabilities. * Protects from clickjacking, content-type sniffing attacks, and more by enforcing best practices via HTTP headers. ### 3. **Flexible Routing with Chi** * Uses the `chi` router for lightweight, flexible routing. Easily extend routes or handle custom paths by attaching new service modules. * Out-of-the-box support for path handling and easy extension for future service needs. ### 4. **Built-in Middleware** * **Compression**: Utilizes the `httpcompression` package to support GZIP, Brotli, and Deflate, optimizing bandwidth usage based on configurable compression levels. * **Logging**: Logs every incoming request with details such as request method, path, and duration, aiding in easier debugging and monitoring. * **CORS Support**: Conditional CORS support based on configuration for handling cross-origin requests securely. ### 5. **WebSocket and LibP2P Support** * Specialized handling for WebSocket upgrade requests, with support for forwarding them to the `/libp2p` endpoint. * Facilitates smooth communication between nodes and services in distributed Curio cluster running a single LibP2P. ### 6. **Health Check Endpoint** * A built-in `/health` endpoint provides an easy way to check the status of the HTTP server, ensuring it is running smoothly. ## Performance and Scalability * The server is optimized for handling large numbers of concurrent requests efficiently with appropriate timeouts (read, write, idle) to prevent overload. * Integrated compression ensures minimal bandwidth usage, even for large data exchanges. ## Database Integration for TLS Certificate Cache The server stores Let's Encrypt certificates and cache information in `Harmonydb`. This ensures persistence and fast access to TLS certificates in a Curio cluster with multiple HTTP servers. ### Database Operations * **Get**: Retrieves TLS certificates from the `autocert_cache` table. * **Put**: Inserts or updates certificates in the database. * **Delete**: Removes expired or invalid certificates from the cache. ## Attaching Routes to Extend Server Functionality Curio HTTP Server is designed to allow easy integration of various services by attaching custom routes. The `attachRouters` function enables the attachment of specific service routes to the Chi router. Examples include: 1. **Retrieval Provider**: Attaches routes to handle data retrieval using the `retrieval` module. * Creates a `RetrievalProvider` and registers the necessary HTTP endpoints. 2. **IPNI (Interplanetary Network Indexer)**: Integrates IPNI-specific routes. * The IPNI provider is instantiated, and routes are attached for IPNI services to handle data advertisement publishing. 3. **LibP2P Redirector**: Handles WebSocket connections for LibP2P communication. * Redirects WebSocket upgrade requests from `/` to `/libp2p` for seamless peer-to-peer communication. Here’s how the routes are attached: ```go attachRouters(ctx context.Context, r *chi.Mux, d *deps.Deps) (*chi.Mux, error) { // Attach retrievals rp := retrieval.NewRetrievalProvider(ctx, d.DB, d.IndexStore, d.CachedPieceReader) retrieval.Router(r, rp) // Attach IPNI ipp, err := ipni_provider.NewProvider(d) if err != nil { return nil, xerrors.Errorf("failed to create new IPNI provider: %w", err) } ipni_provider.Routes(r, ipp) go ipp.StartPublishing(ctx) // Attach LibP2P redirector rd := libp2p.NewRedirector(d.DB) libp2p.Router(r, rd) return r, nil } ``` This flexibility allows the server to be easily extended with new services without modifying the core server logic. ## Configuration The Curio HTTP Server can be customized using the `HTTPConfig` structure, which allows you to configure timeouts, compression levels, and security settings to suit your application. Below are the key configuration options along with their default values and explanations of their impact. ### **HTTPConfig** The list below is aligned with the actual `HTTPConfig` struct in `deps/config/types.go`. * **Enable**: Enables/disables the HTTP server on this node. * **DomainName**: DNS name used for requests and (when Curio terminates TLS) certificate issuance. Must be a real domain (not an IP). Default: `""`. * **ListenAddress**: IP:port to bind. Default: `"0.0.0.0:12310"`. * **DelegateTLS**: When `true`, Curio serves **plain HTTP** on `ListenAddress` and expects a reverse proxy to terminate TLS. * **ReadTimeout**: Max time to read request body. Default: `10s`. * **IdleTimeout**: Max keep-alive idle time. Default: `1h`. * **ReadHeaderTimeout**: Max time to read headers. Default: `5s`. * **CORSOrigins**: Allowed origins; empty disables CORS. Default: `[]`. * **CSP**: Content Security Policy mode for `/piece/` content. Values: `off`, `self`, `inline`. Default: `inline`. * **CompressionLevels**: Response compression tuning. Defaults: gzip=6, brotli=4, deflate=6. ### **Impact of Compression Levels** The compression levels directly affect server performance and bandwidth usage: * Higher compression levels (e.g., `GzipLevel 9`, `BrotliLevel 11`) reduce the size of responses, which can save bandwidth but require more CPU processing time, especially for large responses. * Lower levels (e.g., `GzipLevel 1`) are faster but provide less compression, meaning higher bandwidth usage but reduced server load. For most applications, the default values of `6` for GZIP and Deflate, and `4` for Brotli provide a good trade-off between compression efficiency and CPU load, especially for responses that contain text or JSON. *** ## HTTPS setup (Let’s Encrypt / autocert) — operational guide Curio can terminate TLS itself (autocert / Let’s Encrypt) or you can terminate TLS in a reverse proxy. ### Prerequisites (both modes) * You must use a real **domain name** (not an IP) in `HTTPConfig.DomainName`. * DNS must point your domain to the host that will receive traffic. * Inbound access must be allowed for: * **80/tcp** (ACME HTTP-01) and * **443/tcp** (HTTPS) If ports 80/443 are blocked (cloud firewall, NAT, or corporate network), Let’s Encrypt cannot validate your domain. ### Mode A: Curio terminates TLS (DelegateTLS = false) Use this when you want Curio to handle certificates directly. Checklist: * `DomainName` is set and resolvable publicly. * Curio must be reachable from the internet on 80/443. Operational notes: * If your OS restricts binding to privileged ports, you may need one of: * run the Curio HTTP service with permissions to bind 80/443, or * terminate TLS in a reverse proxy (Mode B), or * forward 80/443 to Curio’s listen port via firewall/NAT. ### Mode B: Reverse proxy terminates TLS (DelegateTLS = true) Use this when you already run Nginx/Caddy/Traefik or cannot (or do not want to) expose Curio directly. In this mode: * Curio serves **plain HTTP** internally. * Your reverse proxy handles TLS + Let’s Encrypt. * When the proxy connects over loopback, Curio treats the rightmost `X-Forwarded-For` entry as the client IP for rate limiting and abuse controls. * The proxy must overwrite or append `X-Forwarded-For` on every request. Forwarding headers from non-loopback peers are ignored. Minimal Nginx example: ```nginx server { listen 443 ssl; server_name example.com; # TLS config here (certbot/caddy/managed) location / { proxy_pass http://127.0.0.1:12310; proxy_set_header Host $host; proxy_set_header X-Forwarded-Proto https; proxy_set_header X-Forwarded-For $remote_addr; } } ``` ### Troubleshooting HTTPS If `curl https://` fails: * Confirm DNS A/AAAA is correct. * Confirm ports 80/443 are reachable externally. * Confirm `DomainName` matches the hostname you’re curling. * Confirm you didn’t enable `DelegateTLS=true` without actually configuring a reverse proxy. * Check Curio logs around certificate issuance / autocert. # Curio Market troubleshooting Troubleshooting Curio Market ingestion, indexing, retrieval, and IPNI. This page targets the common “deal is stuck” / “indexing is failing” cases seen in support. Before proceeding, collect the basics: * [Collect debug info](/troubleshooting/collect-debug-info) * deal UUID + piece CID * failing task IDs from the UI *** ## 1) Deal stuck at indexing / “CheckIndex” errors What CheckIndex is: * Curio runs task-based indexing and health checks; a recurring task checks for missing indexes / announcements and schedules follow-up work. * Implementation reference: `tasks/indexing/task_check_indexes.go` (task `CheckIndex`). If you see errors in CheckIndex: * Determine whether indexing is still progressing (tasks completing) vs stalled. * Many “indexing failures” are actually DB health/latency problems. Verify Yugabyte health first. ### Common error: `duplicate key value violates unique constraint ... (SQLSTATE 23505)` What it often means: * Concurrent retries inserted the same identity row. What to do: 1. Collect: deal UUID, piece CID, task ID(s), and the full error span. 2. Confirm whether the failing rows keep reappearing or are transient. 3. If stuck, restart only the market/indexing layer process (not the whole cluster). *** ## 2) IPNI failures Symptoms: * Indexing seems OK locally, but IPNI publishing is failing. Checklist: * Outbound HTTPS works from Curio host. * The “public URL” for your provider is correct (reachable from the internet). * DNS + firewall allow inbound connectivity to your HTTP server. Temporary mitigation: * If you need onboarding to proceed while IPNI is unstable, you may temporarily disable IPNI publishing (only if supported by your release/config). Document and keep track of the change so it isn’t forgotten. *** ## 3) ParkPiece: `no data URL found for piece_id` What it means: * Curio cannot find the configured URL+headers for fetching the piece data. Most common causes: * The URL was never registered, or it was registered on a different node than the one running ParkPiece. * Ingest layer is not enabled on the node you think. What to collect: * deal UUID, piece CID * whether offline/online deal * the exact command used to add the URL (if online) *** ## 4) Retrieval looks broken, but indexing is fine Checklist: * Ensure unsealed data is available on at least one node that can serve retrieval. * Confirm the Curio HTTP server is enabled and reachable. * If behind a reverse proxy, confirm TLS delegation mode matches your config. See also: * [Curio HTTP Server](/curio-market/curio-http-server) # Curio libp2p Server This page outlines the main features of the Libp2p server, including its configuration and components. ## Overview The Curio Libp2p server facilitates network communication between nodes and clients using the Libp2p protocol. It primarily handles deal proposals, deal status updates, and miner information in the Filecoin network. This server is tightly integrated with Curio's HTTP server and ensures the network operates smoothly for Filecoin deal processing. ### Key Components: * **Host Setup**: The libp2p host is initialized with necessary configuration details, including listening addresses, identity keys, and a peerstore for managing peer connections. * **Deal Handling**: The libp2p server manages deal proposals and responses using the MK12 market protocol. It processes incoming streams for deal proposals, status checks, and query requests. * **WebSocket Proxy**: WebSocket connections from clients are proxied through the Curio HTTP server at the `/libp2p` path, directing traffic to the libp2p node's listening address for secure P2P communication. * **Miner Information Updates**: The libp2p provider also ensures that miner information, such as peer IDs and multi-addresses, is kept up-to-date on-chain, allowing for smooth communication during deal negotiations. ## libp2p Host A **Libp2p host** is initialized with randomized ports for listening to incoming connections. Unlike other networks where the listen address might be broadcasted, in Curio's implementation, the Libp2p node listens on a random port and does not advertise its address to other peers. The setup involves creating a peer identity, configuring listen addresses, and ensuring that the node is ready to handle connections and requests. Once the node is operational, its local listen address is stored in the database, which the WebSocket proxy uses for connecting clients. ## Network Deal Management Curio libp2p server plays a critical role in handling incoming deal proposals over the network. Deals are initiated using the `DealProtocolv120ID` and `DealProtocolv121ID` for handling storage market deals, and the `DealStatusV12ProtocolID` for tracking deal statuses. The deal provider continuously listens for deal requests, processes them, and sends responses back to the clients. ### Deal Proposal Handling When a deal proposal is received, the libp2p handler send the request to MK12(INSERT LINK HERE) provider to validates the request, checks miner permissions, and processes the deal. If the deal is valid, MK12 provider moves forward with execution, handling the storage and sealing process on the storage provider's side. ### Deal Status Tracking The libp2p server supports querying the status of a deal using the `DealStatusV12ProtocolID`. This allows clients to monitor the current state of their proposals, such as whether the deal is sealed, indexed, or still in progress. ## WebSocket Proxy The Curio libp2p server leverages the HTTP server's `/libp2p` path to proxy WebSocket connections between the client and the libp2p node. This mechanism ensures secure and direct P2P communication through WebSocket channels. By connecting via the WebSocket proxy, peers can securely exchange deal information, status updates, and more. This proxy system enables seamless data flow between Curio nodes and external clients or peers, facilitating decentralized communication across the network without exposing local node details directly to clients. ### Failover and Node Coordination To ensure proper coordination between multiple nodes, the libp2p server periodically updates the database with the node's status, such as the `running_on` field, and performs keep-alive checks. If the node fails to update its status in time, another node may take over, ensuring that there is no interruption in the decentralized network. # libp2p Protocols Curio exposes libp2p protocols so that clients can initiate storage deals with the SP The client makes a deal proposal over `v1.2.0` or `v1.2.1` of the Propose Storage Deal Protocol: - `/fil/storage/mk/1.2.0` or\ \- `/fil/storage/mk/1.2.1` It is a request / response protocol, where the request and response are CBOR-marshalled. There are two new fields in the request of `v1.2.1` of the protocol, described in the table below. ### Request | Field | Type | Description | | --------------------------- | ------------------ | -------------------------------------------------------------------------------------------------- | | DealUUID | uuid | A uuid for the deal specified by the client | | IsOffline | boolean | Indicates whether the deal is online or offline | | ClientDealProposal | ClientDealProposal | Same as `.DealProposal` | | DealDataRoot | cid | The root cid of the CAR file. Same as `.Piece.Root` | | Transfer.Type | string | eg "http" | | Transfer.ClientID | string | Any id the client wants (useful for matching logs between client and server) | | Transfer.Params | byte array | Interpreted according to `Type`. eg for "http" `Transfer.Params` contains the http headers as JSON | | Transfer.Size | integer | The size of the data that is sent across the network | | SkipIPNIAnnounce (v1.2.1) | boolean | Whether the provider should announce the deal to IPNI or not (default: false) | | RemoveUnsealedCopy (v1.2.1) | boolean | Whether the provider should keep an unsealed copy of the deal (default: false) | ### Response | Field | Type | Description | | -------- | ------- | -------------------------------------------------- | | Accepted | boolean | Indicates whether the deal proposal was accepted | | Message | string | A message about why the deal proposal was rejected | ## Storage Deal Status Protocol The client requests the status of a deal over `v1.2.0` of the Storage Deal Status Protocol: `/fil/storage/status/1.2.0` It is a request / response protocol, where the request and response are CBOR-marshalled. ### Request | Field | Type | Description | | --------- | ----------------------------------------------------------------------------------------------------------------------------------------- | -------------------------------------------------- | | DealUUID | uuid | The uuid of the deal | | Signature | [Signature](https://github.com/filecoin-project/go-state-types/blob/057cdfb837f7a0309c1607c7c4640f315e51d7af/crypto/signature.go#L36-L39) | A signature over the uuid with the client's wallet | ### Response | Field | Type | Description | | ------------------- | ------------ | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | DealUUID | uuid | The uuid of the deal | | Error | string | Non-empty if there's an error getting the deal status | | IsOffline | boolean | Indicates whether the deal is online or offline | | TransferSize | integer | The total size of the transfer in bytes | | NBytesReceived | integer | The number of bytes that have been downloaded | | DealStatus.Error | string | Non-empty if the deal has failed | | DealStatus.Status | string | The [checkpoint](https://github.com/filecoin-project/boost/blob/4fb17ba117784479e09db4012a3abf9862b8afd9/storagemarket/types/dealcheckpoints/checkpoints.go#L7-L15) that the deal has reached | | DealStatus.Proposal | DealProposal | | | SignedProposalCid | cid | cid of the client deal proposal + signature | | PublishCid | cid | The cid of the publish message, if the deal has been published | | ChainDealID | integer | The ID of the deal on chain, if it's been published | # Indexing This page covers the local indexing process in Curio, detailing how data is stored, indexed, and managed for efficient retrieval and processing. Curio uses a local index store to manage the mapping of multihashes to content pieces. This system allows efficient retrieval of content across multiple nodes in the Curio cluster. The index store does not interact with IPNI directly, but Curio may use index data to create and serve advertisements for the IPNI network when enabled. ## Key Components * **Local Index Store**: A Cassandra-based local store that maps multihashes → piece CIDs + offsets/sizes. * **Concurrency & Batching**: Configurable concurrency and batch sizing for index inserts. * **Integration with Retrievals**: Retrieval performance depends on correct, timely indexing. * **Task-Based Indexing**: Indexing work is performed asynchronously via tasks. ## Index Store Curio’s indexing mechanism relies on a Cassandra-based storage system. The local copy of the indexes allows efficient lookups for retrieval operations. The IndexStore component handles all interactions with the underlying Cassandra database, including creating, adding, and removing index entries, and querying pieces by multihashes. ### Concurrency and Batching The IndexStore allows configurable concurrency and batching for operations. It uses multiple workers to process index entries, enabling parallel indexing to scale with the size of the dataset. ### Configuration The `IndexingConfig` struct defines parameters that control how indexing operations are performed: * **InsertBatchSize**: number of records per batch insert. * **InsertConcurrency**: number of concurrent insert workers. ## Task-Based Indexing Indexing tasks are created and executed asynchronously, allowing Curio to manage indexing without blocking other subsystems. Tasks are assigned to nodes within the Curio cluster, and each node executes indexing tasks based on the available system resources and the availability of the local copy of unsealed sector containing the deal. *** ## How “CheckIndex” works (and why it fails) Operators frequently see errors on a task called **CheckIndex** and aren’t sure what it does. What it does (high level): * Periodically scans for indexing and announcement work that should exist (or be retried). * Schedules follow-up tasks when it finds missing/failed items. Code reference: * `tasks/indexing/task_check_indexes.go` implements the `CheckIndex` task. ### Common failure mode: DB health/latency If Yugabyte is slow/unhealthy, downstream tasks (including market/indexing workflows) can appear broken. Before chasing indexing logic: * run the DB connectivity check from the Curio host * check tserver logs for FATALs See: [Yugabyte troubleshooting](/administration/yugabyte-troubleshooting) ### Common failure mode: duplicate key (SQLSTATE 23505) You may see: * `duplicate key value violates unique constraint ... (SQLSTATE 23505)` This often indicates concurrency/retries inserting the same identity row. What to do: * Determine whether the pipeline is still progressing. * If tasks are stuck, collect deal UUID + piece CID + failing task IDs and consult: * [Curio Market troubleshooting](/curio-market/troubleshooting) # IPNI (Interplanetary Network Indexer) Provider This page details the IPNI provider's sync process over HTTP, covering proactive announcements, indexer polling, and advertisement chain retrieval. The IPNI provider in Curio is designed to manage HTTP-based content announcements and indexing for decentralized discovery through external indexing nodes. It facilitates content chunking, advertisement generation, and HTTP-based announcements to indexers. The following sections explain the provider design, configuration, tasks, advertisements, and the synchronization process based on the [IPNI HTTP Provider specification](https://github.com/ipni/specs/blob/main/IPNI_HTTP_PROVIDER.md). ## IPNI Provider Identification **Curio's Approach to PeerID Management** In Curio, a **single PeerID is used across all MinerIDs** and is maintained on-chain. This ensures a consistent identity for all interactions and prevents unauthorized modifications. If the PeerID is changed manually on-chain, Curio will revert it to the expected value upon restart, ensuring stability and automation. This peerID is used only for mk1.2 deals which use Graphsync protocol. For **IPNI integration**, Curio assigns **unique PeerIDs per MinerID**, but these are **not stored on-chain**. These unique PeerIDs are essential for retrieval verification. Spark uses them to correctly identify miners and validate retrieval operations. This dual approach—using a **single on-chain PeerID for general operations** and **unique off-chain PeerIDs for IPNI**—ensures compatibility across different systems while maintaining efficiency. **Implementation Details** To support this architecture, the **MinerPeerIDMapping** smart contract has been deployed at **0x40721e8Ef366375492ee63c54e881068b15C8633**. This contract provides a decentralized and secure way to manage MinerID-to-PeerID mappings, ensuring that only authorized entities can modify these associations. * **Public State Mapping**: The contract maintains a **gas-free lookup table** mapping Filecoin MinerIDs (`uint64`) to PeerIDs. * **Secure Update Mechanism**: Mappings can be modified through three operations using one of the control addresses: * **Add**: Associates a new PeerID with a MinerID. * **Update**: Replaces an existing PeerID with a new one. * **Delete**: Removes the binding between a MinerID and a PeerID. * **Signed Message Verification**: Updates require a JSON-encoded signed message, ensuring authenticity. The message is signed using the on-chain PeerID. * **Automated Integration with Curio**: Curio ensures that the on-chain PeerID remains updated automatically, preventing unauthorized changes. This allows Spark to correctly associate Ads with MinerIDs, even when different PeerIDs are used in other systems. ## Provider Design The Curio IPNI provider operates over HTTP, managing content updates through advertisement creation and announcement. It does not use libp2p; instead, it interacts with indexing nodes by sending HTTP requests to announce content and expose the advertisement chain for indexing. ### Key Components * **Chunking**: Large datasets are divided into smaller chunks using the `Chunker` to create manageable entries for advertisement. Each chunk contains a subset of the data for efficient indexing by external nodes. * **Advertisement Creation**: After chunking the content, the provider generates IPLD-based advertisements. These advertisements contain essential metadata, including the content's addresses, the provider's identity, and a chain linking the advertisement to previous content updates. * **HTTP Announcements**: Once an advertisement is created, HTTP requests are sent to specific indexing nodes via the configured `DirectAnnounceURLs`, informing them about new advertisements. This proactive method ensures that indexers are notified of content changes quickly. * **Sync Mechanism**: Indexers receive announcements about new advertisements via HTTP. In cases where no new announcements are received for a while, the indexers poll the provider’s `/ipni/v1/ad/head` endpoint to retrieve the latest advertisement head CID. From there, they traverse the advertisement chain, fetching the missing updates. This ensures that indexers stay up-to-date with the provider's latest content, even if no explicit announcements are sent for a period of time. ## Configuration The behaviour of the Curio IPNI provider is controlled through the `IPNIConfig` structure, which defines how content announcements and synchronization with indexing nodes are handled. ### Default Configuration ```go [Market.StorageMarketConfig.IPNI] # Disable set whether to disable indexing announcement to the network and expose endpoints that # allow indexer nodes to process announcements. Default: False # # type: bool #Disable = false # The network indexer web UI URL for viewing published announcements # TODO: should we use this for checking published heads before publishing? Later commit # # type: []string #ServiceURL = ["https://cid.contact"] # The list of URLs of indexing nodes to announce to. This is a list of hosts we talk to tell them about new # heads. # # type: []string #DirectAnnounceURLs = ["https://cid.contact/ingest/announce"] ``` ### **IPNIConfig Fields** * **Disable**: Disables indexing announcements if set to `true`. Default: `false`. IPNI should be disabled on base layer. * **ServiceURL**: URLs for accessing the indexer web UI to view published advertisements. * **DirectAnnounceURLs**: URLs of indexing nodes where the provider sends HTTP announcements of new advertisements. ## IPNI Task The **IPNI task** is responsible for reading content, chunking it, creating advertisements, and sending HTTP announcements to indexing nodes. #### Task Lifecycle 1. **Content Reading**: The task reads content (sectors) from storage. 2. **Chunking**: The content is divided into smaller chunks using a`chunker`, organizing it for efficient indexing. 3. **Advertisement Creation**: A signed IPLD-based advertisement is generated for the chunked content, linking it to previous advertisements in the chain. 4. **Database Update**: The task marks itself as complete in the database once the content is advertised and announced to the indexer. The task ensures that content is indexed and made available to indexing nodes by creating manageable advertisement entries and sending updates via HTTP. ## Chunker The **chunker** generates entries for advertisements by leveraging the [**index store**](/curio-market/indexing#index-store) 1. **First CID for Speed Optimization**: * The chunking process begins with multihashes (`multihash.Multihash`) stored in the database. * For each chunk, the **first CID** (Content Identifier) is crucial as it is used to quickly locate and reconstruct the chunk when needed. This CID is often stored as the `first_cid` in the database along with other metadata like chunk number, piece CID, and offsets. 2. **Sorted Entries in Database**: * During chunk creation, the entries (multihashes) are first sorted in ascending order based on their binary value for efficient processing and retrieval. * Sorting ensures that duplicates are identified and removed, minimizing redundancy. This process involves: * Sorting the multihashes. * Removing duplicates. * After sorting, the entries are split into chunks of predefined size (`EntriesChunkSize`, e.g., 16,384 multihashes per chunk). 3. **Chunk Metadata Storage**: * Each chunk is associated with metadata stored in the database, including: * `cid`: Unique identifier for the chunk. * `piece_cid`: Identifier linking the chunk to its corresponding data piece. * `chunk_num`: Chunk sequence number. * `first_cid`: First multihash in the chunk for quick lookup. * `start_offset`: Byte offset (if applicable). * `num_blocks`: Number of entries in the chunk. 4. **Chunk Retrieval and Reconstruction**: * To reconstruct a chunk: * The database is queried using the `cid` to fetch metadata, including the `first_cid`. * The system then either reads data directly from the database (sorted entries). * For each chunk, a linked structure (`schema.EntryChunk`) is created, linking to the next chunk via an IPLD link. 5. **Linked Chunks**: * Chunks are linked to each other using the `Next` field in the IPLD node structure, forming a chain that can be navigated from the head. 6. **Efficient Querying**: * The system leverages the database for rapid querying of metadata and uses caching (e.g., LRU caches) to store recent chunk data for reuse and speculative pre-fetching. This design ensures fast chunking and reconstruction by leveraging the sorted entries and storing the first CID, allowing efficient access and reusability. ## Advertisements Advertisements describe the content available for indexing and discovery. Each advertisement is represented as an IPLD node containing the following fields: * **PreviousID**: The CID of the previous advertisement, forming a chain of advertisements. It’s empty for the genesis (first) advertisement. * **Provider**: The unique identifier of the content provider (peerID). This peerID is unique for IPNI per miner ID. * **Addresses**: Multiaddresses where clients can access the provider’s content i.e. retrievals. * **Entries**: A link to the multihashes of the advertised content. * **ContextID**: An identifier used to track updates or removals associated with the advertisement. * **Metadata**: Additional protocol-specific data used for retrieval. * **IsRm**: A flag indicating whether the advertisement removes previously published content. ### Entries Structure A linked list of multihashes in an advertisement, where `Next` links to the next chunk. Each chunk is kept under 4MB, allowing up to 16384 multihashes per chunk. ## Announcement The provider sends HTTP announcements to notify indexing nodes of new advertisements. This is a proactive method for updating indexers about content changes without relying solely on polling. ### Announcement Workflow 1. **Ad Creation**: The provider creates IPLD-based advertisements containing content metadata and multiaddresses. 2. **HTTP Announcements**: These advertisements are announced to the indexer nodes specified in the `DirectAnnounceURLs` via HTTP requests. 3. **Sync:** Indexer nodes query the IPNI provider for new advertisements based on the announcements and sync the available context indexes. 4. **Client Access**: Clients query the indexer nodes to discover content based on the newly announced advertisements. Announcements ensure that indexers are aware of updates quickly, reducing the time it takes to ingest new content. ## Serving IPNI Ads and Entries To serve IPNI advertisements and entries in Curio, the HTTP server exposes specific route. This route allow the retrieval of advertisement data and entry chunks through the following paths: 1. **Head Request Path:** * **Endpoint:** `/ipni-provider/{providerId}/ipni/v1/ad/head` * **Description:** This endpoint allows indexers to fetch the latest advertisement from a provider by requesting the head of the advertisement chain. 2. **Advertisement and Entry Request Path:** * **Endpoint:** `/ipni-provider/{providerId}/ipni/v1/ad/{cid}` * **Description:** This endpoint serves both advertisements and entry chunks based on the requested CID. The type of content returned (advertisement or entry) is determined by the CID and schema provided in the request headers. If the schema is not specified, the server defaults to checking for an advertisement and, if not found, falls back to serving the entry chunk associated with the given CID. These routes are registered in the Curio HTTP server as part of the IPNI integration, enabling smooth and efficient data sharing between providers and indexers. The server also periodically publishes the latest advertisement head for each provider. ## IPNI Sync The synchronization process ensures that indexing nodes stay updated with the latest content announcements from the provider. #### Sync Process: 1. **Proactive Announcements**: Indexing nodes are notified via HTTP announcements whenever the provider has new advertisements. This is the primary mechanism for keeping indexers in sync with the provider. 2. **Head Resource for Polling**: If no announcements are received for a period of time, indexers may poll the provider’s `/ipni/v1/ad/head` endpoint to check for new content. This head CID represents the latest advertisement in the chain. 3. **Chain Retrieval**: When indexers receive a new head CID, they traverse the advertisement chain backward, starting from the head, fetching each advertisement via HTTP (`/ipni/v1/ad/{CID}`). The indexer processes the advertisement chain in order from the oldest unseen advertisement to the newest. 4. **Continuous Updates**: Through a combination of announcements and periodic polling, indexers ensure they have the most up-to-date content from the provider, even if an explicit announcement was not sent. This process ensures that indexers remain synchronized with the provider’s content, maintaining up-to-date knowledge of available advertisements and content retrieval addresses. By using this mechanism, indexers can stay synchronized with the provider’s content in real-time or through periodic polling, ensuring continuous updates. ### Serving Advertisements: * **Advertisement Fetching**: Indexers fetch advertisements and entries directly from the provider via HTTP, making the advertised IPLD objects available for ingestion. * **Head Requests**: The provider exposes the latest advertisement through the `/ipni/v1/ad/head` endpoint, allowing indexers to know the most recent state of the advertisement chain. * **Serving Entries:** When indexers need to fetch advertised entries, they request specific entry chunks through their corresponding CID by making a GET request to the `/ipni/v1/ad/{CID}` endpoint. The provider serves these entries by reading either from the CAR file or from the stored index data, depending on how the entries were chunked during advertisement creation. This ensures efficient retrieval of multihashes for large datasets. This synchronization process ensures that indexers can efficiently track the latest updates from the provider, enabling quicker content discovery across the network. # Market UI Curio market related UI pages and their content description The "storage market" page is the main page for storage deal market. The page provides quick summary of market balances, piece status, storage ask and deal pipeline status.

Storage market UI page

You can set the storage ask for each miner ID in the Curio cluster individually from the UI.

Storage ask UI

By clocking on the UUID column of the deal pipeline in the storage market page, you can find more detailed information about the deal.

Deal detail page

The "Storage Deals" page list outs all the details in the Curio market. This can be used to get the summary of latest deals or to lookup a specific deal with unique identifier (UUID) using the search function.

Storage Deals page

The "Piece Info" contains all the deals about a piece onboarded by the Curio market. This is a one stop shop for all details about a piece and the deal containing the piece. It provides information about the piece itself, indexing, IPNI announcement status of the piece along with all the deals which contains this piece with their processing status. It also lists out the sectors which contain this piece. This is the single most important page for debugging issues with data onboarding. This page can be opened by clicking on the "Piece Cid" from multiple pages including the deal detail and deal list pages.

Piece Info page

Piece Info deal details

Piece Info storage pipeline status

All the IPNI provider and advertisement related details can be found on the "IPNI" page. The current status of IPNI provider (Curio) can be found on this page. The status is listed for each miner ID for each IPNI (ex: cid.contact) node.

IPNI page

The page also allows searching IPNI advertisements using the Cid. Users can scan the original piece to rebuild the entry list to debug issues with advertisement sync.

IPNI advertisement search page

# Retrievals This page explains Curio's HTTP-based content retrieval system. ## Overview The Curio HTTP server serves as the primary interface for handling content retrieval requests. In a Curio cluster, multiple nodes within the Curio cluster can host HTTP servers. Users can retrieve data using any of the available URLs, allowing for redundancy and ensuring high availability. ### Key Components: * **Piece-Based Retrievals**: Data can be fetched by a piece CID, enabling users to retrieve content directly from specific pieces stored within the Curio ecosystem. * **IPFS-Based Retrievals**: The Curio HTTP server also integrates with IPFS, allowing users to request content using an IPFS CID via the `/ipfs` route. * **Caching and Immutable Data**: Retrieved content is served with caching headers, ensuring immutability and efficiency for repeat requests. ## Piece-Based Retrievals The retrieval server allows users to fetch content by providing a piece CID through the `/piece/{cid}` route. Upon receiving such a request, the server looks up the requested piece and serves the content directly from the provider's unsealed sector. Any errors encountered during retrieval, such as the requested piece not being found, result in appropriate HTTP error responses (404, 500, etc.). ### IPFS Gateway Integration In addition to supporting piece-based retrievals, the Curio HTTP server also integrates IPFS gateway functionality. The `/ipfs/{cid}` route allows users to request content by CID. This is handled by the `frisbii` server backed by a `blockstore`, which fetches and returns the content corresponding to the requested IPFS CID. ### Caching and Headers Content served by the Curio retrieval server is immutable. For this reason, the server sets caching headers to allow clients and CDNs to cache content for extended periods. Specifically: * **ETag**: A unique identifier based on the piece CID ensures that cached content can be validated efficiently. * **Cache-Control**: Responses are marked as immutable and cacheable for up to one year, reducing the load on the retrieval server for repeat requests. By leveraging these caching mechanisms, the Curio HTTP server ensures that frequently accessed content is delivered quickly and with minimal resource consumption. # Migrating From Boost A step-by-step Guide to Migrating From Boost to Curio Migrating from Boost to Curio involves transitioning all deal data, including Boost deals, legacy deals, and Direct Data Onboarding (DDO) deals, to Curio’s storage and indexing system. Below is a detailed guide that walks through each step of the migration process. ## Build the `migrate-curio` CLI Tool * First, you need to build the `migrate-curio` tool, which is part of the Boost code base. This tool will handle the migration of deals. You must be running Boost version v2.4.7 or later. * Run the following command to build the tool. ```bash make migrate-curio ``` ## Preparation for Migration {% hint style="warning" %} Please read the [Curio market documentation](/curio-market/storage-market) carefully before proceeding. Curio market should be fully [configured](/curio-market/storage-market#enabling-storage-market) before proceeding with migration. {% endhint %} * Ensure that both Boost and Curio setups are ready for the migration. **Boost must be shutdown and no deal should be in process**. You will need the following details: * Path to the Boost repository (default is `~/.boost`) * Backup your Boost repository * `boostd-data` service must be up and running as migration needs to get some data from it if required. * Database credentials for Curio’s HarmonyDB (host, username, password, port) ## Run the Migration Tool Use the `migrate-curio` tool to begin the migration process. ```bash ./migrate-curio --boost-repo /path/to/boost-repo \ --db-host "127.0.0.1" \ --db-name "yugabyte" \ --db-user "yugabyte" \ --db-password "yugabyte" \ --db-port "5433" \ migrate ``` This command starts the migration, pulling data from the Boost repository and pushing it into Curio’s database. ## Migration Details The migration consists of 3 phases: ### Migrate Boost Deals 1. **Retrieve Active and Completed Deals:** * The tool will first query Boost’s database (`boost.db`) for active and completed deals. * It retrieves all Boost deals, including those that are still active or recently completed. 2. **Filtering Deals:** * Some deals may not be eligible for migration. The following deals are **skipped**: * Deals where the checkpoint is below the "add piece" stage. * Deals with a fatal retry error. * Deals with a sector ID of 0 or where the sector is no longer alive. * Deals that have already been migrated (tracked in `migrate-curio.db`). 3. **Processing Each Deal:** * For each eligible deal: * The deal is inserted into the `market_mk12_deals` table in Curio. * Additional information about the deal, such as proposal details, client peer ID, and the CID of the published deal, is migrated. * If the deal's sector is unsealed, it is added to the indexing and announcement pipeline (`market_mk12_deal_pipeline_migration`). ### Migrate Legacy Deals 1. **Load Legacy Deals:** * Legacy deals are stored in Boost’s LevelDB (`LID`). The tool queries the deals and processes each one. * It uses the `go-ds-versioning` library to access the old FSM (Finite State Machine) for managing legacy deals. 2. **Skipping Unnecessary Legacy Deals:** * Deals that have expired, have invalid sector numbers, or are already migrated are skipped. * Deals that do not have a chain deal ID or are past their expiration epoch are excluded from migration. 3. **Processing Legacy Deals:** * Legacy deals are inserted into Curio’s database: * Signed proposal CID, piece size, start and end epochs, and other deal-specific details are migrated. * Deals are not indexed. ### Migrate Direct Data Onboarding (DDO) Deals 1. **Retrieve DDO Deals:** * The tool queries the DDO database for all deals created using the Direct Data Onboarding method. 2. **Validating Claims:** * For each DDO deal, the tool verifies that the sector matches the claim in the deal. If the sector is no longer active, the deal is skipped. 3. **Processing DDO Deals:** * Eligible DDO deals are migrated to Curio’s `market_direct_deals` table. * If the sector is unsealed, the deal is also added to the `market_mk12_deal_pipeline_migration` table for indexing and announcement. ### **Deal Announcements** For all deals that have been migrated and are in unsealed sectors, Curio handles their indexing and IPNI (Indexing Provider Network Interface) announcements, ensuring they are publicly discoverable. ## Cleanup of Local Index Directory (LID) (optional) * After the migration, you can clean up the old LevelDB (`LID`) data that was used by Boost:\\ ```bash migrate-curio --boost-repo cleanup leveldb ``` * If the `LID` store was using YugabyteDB for storing indexes, use the following command\\ ```bash ./migrate-curio cleanup yugabyte \ --hosts "127.0.0.1" \ --username "yugabyte" \ --password "yugabyte" \ --connect-string "postgresql://postgres:postgres@localhost" \ --i-know-what-i-am-doing ``` This command will drop the `idx` keyspace and remove relevant tables in YugabyteDB. ## Monitoring and Finalizing Migration * Once the migration is complete, monitor the logs to ensure all deals have been correctly migrated and indexed. * Verify that the new Curio storage system has all the Boost, legacy, and DDO deals. * Verify the Curio is creating new indexing and IPNI jobs for migrated deals in batches. # Market 2.0 Market 2.0 is Curio’s unified, extensible deal interface between clients and storage providers. It exists to abstract away deal-type-specific integration.\ A client can request PoRep-style storage, PDP operations, or future specialized services through one consistent model and one consistent API surface. ## Deal Model Every deal is expressed as: * `id`: deal reference * `client`: requester identity * `products`: requested services * `data source` (optional): operation input data when required This model reflects real deal-making: who is asking, what they want, how it is referenced, and what inputs are needed. ## Clients Use these guides to submit and manage deals: 1. [HTTP API: create, update, and query deals](/market-2.0/http-api) 2. [Deal intake paths and upload ordering](/market-2.0/deal-processing#deal-intake-paths) 3. [Lifecycle and status tracking](/market-2.0/deal-processing) 4. [Data model and source/format requirements](/market-2.0/architecture#data-model) 5. [Error codes and troubleshooting](/market-2.0/http-api#endpoint-details) Contact your storage provider when: 1. You receive repeated `500` or `503` responses. 2. A deal stays non-terminal longer than expected. 3. A contract-related rejection needs provider policy confirmation. ## Storage Providers Use these guides to operate and support Market 2.0: 1. [Lifecycle and operational behavior](/market-2.0/deal-processing) 2. [Contract integration and provider allowlisting](/market-2.0/contracts) 3. [DDO contract review and allowlisting](/market-2.0/contracts/ddo-contract-review) 4. [Error codes for support triage](/market-2.0/http-api#endpoint-details) Operational note: 1. Product/source controls and contract allowlisting are provider policy decisions managed through the Curio GUI. 2. DDO contracts must be vetted before allowlisting. ## Developers ### SDK and Client Library Developers 1. [HTTP interaction model](/market-2.0/http-api) 2. [Lifecycle semantics](/market-2.0/deal-processing) 3. [Error handling contract](/market-2.0/http-api#endpoint-details) 4. [Data model and validation context](/market-2.0/architecture) 5. [Data source and format constraints](/market-2.0/architecture#data-model) ### Contract Integrators 1. [Contract integration guide](/market-2.0/contracts) 2. [Current DDO contract interface (CurioDealView v1)](/market-2.0/contracts/curiodealview) 3. [DDO contract review and builder guide](/market-2.0/contracts/ddo-contract-review) Current integration surface: 1. DDO uses `CurioDealViewV1` today. 2. Additional interfaces may be introduced in future product/version evolution without changing the top-level Market 2.0 deal model. ### Curio Product Developers 1. [Extending products](https://github.com/filecoin-project/curio/blob/main/documentation/en/market-2.0/extending-mk20.md) 2. [DDO product behavior](/market-2.0/products/ddo_v1) 3. [Retrieval product behavior](/market-2.0/products/retrieval_v1) 4. [PDP product behavior](/market-2.0/products/pdp_v1) Design rule: 1. Extend an existing product when behavior remains backward compatible and JSON evolution is safe. 2. Introduce a new product when behavior, lifecycle, or compatibility contracts diverge. # Architecture ## What This Page Covers This page explains what Market 2.0 is designed to do, why key model choices were made, how the system works end to end, and what problems it does not try to solve. ## Purpose Market 2.0 is the common interaction layer between clients and storage providers. It is designed so PoRep-style storage requests, PDP requests, and future specialized requests all use one stable deal envelope instead of separate APIs per request type. ## Core Deal Envelope ```go type Deal struct { Identifier ulid.ULID `json:"identifier"` Client string `json:"client"` Data *DataSource `json:"data,omitempty"` Products Products `json:"products"` } type Products struct { DDOV1 *DDOV1 `json:"ddo_v1,omitempty"` RetrievalV1 *RetrievalV1 `json:"retrieval_v1,omitempty"` PDPV1 *PDPV1 `json:"pdp_v1,omitempty"` } ``` Field meaning: 1. `identifier`: deal reference. 2. `client`: requester identity. 3. `products`: requested behavior. 4. `data`: operation input when required. Product-specific structs and fields are documented in: 1. [DDO v1](/market-2.0/products/ddo_v1) 2. [Retrieval v1](/market-2.0/products/retrieval_v1) 3. [PDP v1](/market-2.0/products/pdp_v1) ## Why `client` Is Text `client` is intentionally a text field, not a strict Filecoin address type. Why: 1. Market 2.0 must support identity schemes beyond native Filecoin address formats. 2. Current CurioAuth support is Filecoin-address based. 3. L2 or product-specific identity formats can be supported later without changing the deal envelope. Identity safety comes from authentication and identity matching, not from enforcing one address type in the schema. ## Deal Identifier (ULID) `identifier` uses ULID. 1. ULID specification: 2. Implementations (many languages): 3. Go implementation: 4. JavaScript/TypeScript implementation: ULID is used as a portable, sortable, globally unique deal reference. ## Authentication and Authorization This section documents current auth behavior. Header format: `Authorization: CurioAuth ::` Supported key types: 1. `secp256k1` 2. `bls` 3. `delegated` Signing input: 1. Build message bytes as `addressBytes || uppercaseRequestMethod || escapedRequestPath || RFC3339MinuteTimestamp`. 2. Hash with SHA-256. 3. Sign the digest. `addressBytes` means Filecoin address bytes (`address.Address.Bytes()`), not the human-readable address string. `escapedRequestPath` is the request URL path without the query string. The request body and query string are not part of the signed message. The auth header is therefore a short-lived path-scoped credential and must be protected in transit. Verification window: 1. Current minute. 2. Previous minute, to tolerate requests crossing a minute boundary in transit. Authorization checks: 1. Signature must verify for the request method, request path, and one of the accepted timestamps. 2. Client allow/deny policy is read from `market_mk20_clients`. 3. If client is not listed, fallback behavior uses `DenyUnknownClients` config. 4. For routes with `{id}`, authenticated client must own that deal. Route scope: 1. Market API routes are authenticated. 2. `/info/*` OpenAPI routes are public. ## Data Model ```go type DataSource struct { PieceCID cid.Cid `json:"piece_cid"` Format PieceDataFormat `json:"format"` SourceHTTP *DataSourceHTTP `json:"source_http,omitempty"` SourceAggregate *DataSourceAggregate `json:"source_aggregate,omitempty"` SourceOffline *DataSourceOffline `json:"source_offline,omitempty"` SourceHttpPut *DataSourceHttpPut `json:"source_http_put,omitempty"` } type PieceDataFormat struct { Car *FormatCar `json:"car,omitempty"` Aggregate *FormatAggregate `json:"aggregate,omitempty"` Raw *FormatBytes `json:"raw,omitempty"` } ``` Validation rules: 1. `piece_cid` must be valid PieceCID v2. 2. Exactly one source type is allowed. 3. Exactly one format is allowed. 4. Source type must be enabled by provider policy. ## Aggregation Current aggregation support: 1. `AggregateTypeV1` only. 2. This is datasegment/PODSI-style aggregation based on FRC-0058: Why aggregation matters in Market 2.0: 1. It allows subpiece-aware indexing and retrieval flows. 2. It enables retrieval of subpieces, including IPFS-style single-CID usage from subpiece data paths when retrieval/indexing settings are enabled. Current constraints: 1. Aggregate-of-aggregate is rejected. 2. `source_aggregate` mode and pre-aggregated `format.aggregate.sub` mode have different validation rules. 3. Subpiece order is meaningful and must match segment order. Future direction: 1. Additional aggregation types can be added as new FRCs are finalized. 2. The top-level Market 2.0 deal envelope remains unchanged. ## Product Composition Rules 1. At least one top-level product must be present. 2. `ddo_v1` and `pdp_v1` cannot be used together in one deal. 3. `ddo_v1` requires `retrieval_v1`. 4. With `ddo_v1`, `retrieval_v1.announce_piece` must be false. ## How a Deal Moves Through the System 1. Client submits deal intent. 2. Market 2.0 validates identity, payload, and product composition. 3. Market 2.0 selects product execution path. 4. Market 2.0 selects ingest path (immediate source or upload-first). 5. Processing continues through common lifecycle states. 6. Deal reaches terminal success (`complete`) or terminal failure (`failed`). Lifecycle states: 1. `accepted` 2. `uploading` 3. `processing` 4. `sealing` 5. `indexing` 6. `failed` 7. `complete` ## External Contract Boundary Market 2.0 supports product-specific contract verification without changing the outer deal model. Example: 1. DDO verifies through `CurioDealViewV1`, primarily with `version()` and `verifyDeal(...)`. 2. The interface also exposes `getDealState(...)` for deal-state queries. 3. Provider allowlist controls which contracts are accepted. 4. Interface versioning is explicit. ## Best Practices 1. Keep `identifier` immutable. 2. Keep `client` stable within the chosen identity namespace. 3. Query provider capabilities before creating deals. 4. Keep product payloads explicit. 5. Vet contracts before allowlisting. 6. Extend existing products only when backward compatibility is preserved. 7. Add a new product when behavior or lifecycle contracts diverge. ## What Market 2.0 Does Not Solve 1. Commercial negotiation and legal agreement terms. 2. Automatic trust in external contracts. 3. Universal payment/settlement semantics across all products. 4. Product-specific business policy decisions outside declared product contracts. 5. Operational monitoring and recovery by itself. # HTTP API Base path: `/market/mk20`\ `{id}` is the deal identifier (ULID). ## Authentication All Market 2.0 API routes require auth except `/info/*`. Auth details are documented in: [Architecture: Authentication and Authorization](/market-2.0/architecture#authentication-and-authorization) ## API Compatibility Market 2.0 APIs are expected to evolve conservatively, with backward compatibility as a primary consideration. ## Swagger and OpenAPI Specs Use the running storage provider spec first for integration and debugging. Running provider endpoints: 1. `GET /market/mk20/info/` 2. `GET /market/mk20/info/swagger.yaml` 3. `GET /market/mk20/info/swagger.json` Source tree specs: 1. [`market/mk20/http/swagger.yaml`](https://github.com/filecoin-project/curio/blob/main/market/mk20/http/swagger.yaml) 2. [`market/mk20/http/swagger.json`](https://github.com/filecoin-project/curio/blob/main/market/mk20/http/swagger.json) Version note: 1. A running provider may be on a different Curio version than this documentation. 2. If behavior differs, use that provider's `/info/swagger.*` as the operational reference. ## Endpoint Map 1. `POST /deal` 2. `GET /status/{id}` 3. `POST /update/{id}` 4. `GET /products` 5. `GET /sources` 6. `GET /contracts` 7. `POST /uploads/{id}` 8. `GET /uploads/{id}` 9. `PUT /uploads/{id}/{chunkNum}` 10. `POST /uploads/finalize/{id}` 11. `PUT /upload/{id}` 12. `POST /upload/{id}` ## Endpoint Details ## `POST /deal` Creates a deal. Request: 1. Body: `mk20.Deal` JSON. Success: 1. `200` `Ok` Error codes: 1. `400` `ErrBadProposal` 2. `401` `ErrUnAuthorized` 3. `404` `ErrDealNotFound` 4. `422` `ErrUnsupportedDataSource` 5. `423` `ErrUnsupportedProduct` 6. `424` `ErrProductNotEnabled` 7. `425` `ErrProductValidationFailed` 8. `426` `ErrDealRejectedByMarket` 9. `429` `ErrServiceOverloaded` 10. `430` `ErrMalformedDataSource` 11. `440` `ErrMarketNotEnabled` 12. `441` `ErrDurationTooShort` 13. `500` `ErrServerInternalError` 14. `503` `ErrServiceMaintenance` Response note: 1. Error body is plain text: `Reason: `. ## `GET /status/{id}` Returns deal status by product. Success: 1. `200` `mk20.DealProductStatusResponse` Error codes: 1. `400` invalid/missing id 2. `401` `ErrUnAuthorized` 3. `404` `ErrDealNotFound` 4. `500` `ErrServerInternalError` ## `POST /update/{id}` Updates an existing deal in supported update paths. The update path can fill missing `data` or add product fields that are not already present. Existing `data` and existing product fields are not replaced. Request: 1. Body: `mk20.Deal` JSON. Success: 1. `200` `Ok` Error codes: 1. `400` `ErrBadProposal` 2. `401` `ErrUnAuthorized` 3. `404` `ErrDealNotFound` 4. `422` `ErrUnsupportedDataSource` 5. `423` `ErrUnsupportedProduct` 6. `424` `ErrProductNotEnabled` 7. `425` `ErrProductValidationFailed` 8. `426` `ErrDealRejectedByMarket` 9. `429` `ErrServiceOverloaded` 10. `430` `ErrMalformedDataSource` 11. `440` `ErrMarketNotEnabled` 12. `441` `ErrDurationTooShort` 13. `500` `ErrServerInternalError` 14. `503` `ErrServiceMaintenance` ## `GET /products` Returns enabled products for this provider. Success: 1. `200` `{ "products": [...] }` Error codes: 1. `401` `ErrUnAuthorized` 2. `500` `ErrServerInternalError` ## `GET /sources` Returns enabled data source types for this provider. Success: 1. `200` `{ "sources": [...] }` Error codes: 1. `401` `ErrUnAuthorized` 2. `500` `ErrServerInternalError` ## `GET /contracts` Returns allowlisted DDO market contracts for this provider. Success: 1. `200` `{ "contracts": [...] }` Error codes: 1. `401` `ErrUnAuthorized` 2. `404` no supported contracts found 3. `500` `ErrServerInternalError` ## `POST /uploads/{id}` Starts chunked upload session. Request: 1. Body: `StartUpload` (`raw_size`, `chunk_size`). Success: 1. `200` `UploadStartCodeOk` Error codes: 1. `400` `UploadStartCodeBadRequest` 2. `401` `ErrUnAuthorized` 3. `404` `UploadStartCodeDealNotFound` 4. `409` `UploadStartCodeAlreadyStarted` 5. `500` `UploadStartCodeServerError` ## `GET /uploads/{id}` Returns chunked upload progress. Success: 1. `200` `UploadStatusCodeOk` Error codes: 1. `400` invalid/missing id 2. `401` `ErrUnAuthorized` 3. `404` `UploadStatusCodeDealNotFound` 4. `425` `UploadStatusCodeUploadNotStarted` 5. `500` `UploadStatusCodeServerError` ## `PUT /uploads/{id}/{chunkNum}` Uploads one chunk. Success: 1. `200` `UploadOk` Error codes: 1. `400` `UploadBadRequest` 2. `401` `ErrUnAuthorized` 3. `404` `UploadNotFound` 4. `409` `UploadChunkAlreadyUploaded` 5. `429` `UploadRateLimit` 6. `500` `UploadServerError` ## `POST /uploads/finalize/{id}` Finalizes chunked upload. Body may be empty or full `mk20.Deal`. Success: 1. `200` `Ok` Error codes: 1. `400` `ErrBadProposal` 2. `401` `ErrUnAuthorized` 3. `404` `ErrDealNotFound` 4. `422` `ErrUnsupportedDataSource` 5. `423` `ErrUnsupportedProduct` 6. `424` `ErrProductNotEnabled` 7. `425` `ErrProductValidationFailed` 8. `426` `ErrDealRejectedByMarket` 9. `429` `ErrServiceOverloaded` 10. `430` `ErrMalformedDataSource` 11. `440` `ErrMarketNotEnabled` 12. `441` `ErrDurationTooShort` 13. `500` `ErrServerInternalError` 14. `503` `ErrServiceMaintenance` ## `PUT /upload/{id}` Uploads full payload in serial flow. Success: 1. `200` `UploadOk` Error codes: 1. `400` `UploadBadRequest` 2. `401` `ErrUnAuthorized` 3. `404` `UploadStartCodeDealNotFound` 4. `500` `UploadServerError` ## `POST /upload/{id}` Finalizes serial upload. Body may be empty or full `mk20.Deal`. Success: 1. `200` `Ok` Error codes: 1. `400` `ErrBadProposal` 2. `401` `ErrUnAuthorized` 3. `404` `ErrDealNotFound` 4. `422` `ErrUnsupportedDataSource` 5. `423` `ErrUnsupportedProduct` 6. `424` `ErrProductNotEnabled` 7. `425` `ErrProductValidationFailed` 8. `426` `ErrDealRejectedByMarket` 9. `429` `ErrServiceOverloaded` 10. `430` `ErrMalformedDataSource` 11. `440` `ErrMarketNotEnabled` 12. `441` `ErrDurationTooShort` 13. `500` `ErrServerInternalError` 14. `503` `ErrServiceMaintenance` ## Operational Notes 1. Upload finalize endpoints accept an empty body when the deal already has data details. If the accepted deal omitted data details, finalize must include a full deal payload so MK20 can bind the uploaded piece to the deal. 2. Provider policies for products, sources, and contracts affect acceptance behavior. 3. When behavior differs from docs, trust the running provider `/info/swagger.*`. # Deal Processing This page defines the deal intake paths, lifecycle progression, status semantics, and pipeline behavior for Market 2.0. ## Deal Intake Paths All intake paths use the same deal model and the same deal identifier (`id`, ULID). They differ only in when and how piece bytes are provided. ### 1. Provider-Pull Deal Use this path when the storage provider should fetch data from client-supplied sources. API sequence: 1. `POST /deal` 2. `GET /status/{id}` until terminal state No upload endpoints are used. ### 2. Accept-Then-Upload Deal Use this path when a client wants acceptance first, then uploads bytes. API sequence: 1. `POST /deal` 2. Upload bytes using one mode: 3. Serial: `PUT /upload/{id}` then `POST /upload/{id}` 4. Chunked: `POST /uploads/{id}` then `PUT /uploads/{id}/{chunkNum}` (repeat) then `POST /uploads/finalize/{id}` 5. `GET /status/{id}` until terminal state ### 3. Upload-First-Then-Describe Deal Use this path when bytes are uploaded first and full deal details are supplied at finalize. This requires an initial product flow that can be accepted without data details. DDO deals require `data` during initial acceptance, so DDO upload flows should use `source_http_put` in the initial deal instead. API sequence: 1. `POST /deal` 2. Upload bytes using one mode: 3. Serial: `PUT /upload/{id}` 4. Chunked: `POST /uploads/{id}` then `PUT /uploads/{id}/{chunkNum}` (repeat) 5. Finalize with full deal payload: 6. Serial: `POST /upload/{id}` 7. Chunked: `POST /uploads/finalize/{id}` 8. `GET /status/{id}` until terminal state Current behavior: this path is supported in both serial and chunked finalize flows for products that can enter upload-waiting without initial data details. ### 4. Control-Only Operation (No Piece Ingestion) Use this path for PDP lifecycle operations that do not ingest piece data (for example dataset create, dataset delete, or piece-remove operations). API sequence: 1. `POST /deal` 2. `GET /status/{id}` until terminal state No upload endpoints are used. ## Deal Lifecycle Once accepted, deals follow a common lifecycle model: 1. `accepted`: proposal admitted. 2. `uploading` (optional): waiting for or receiving client-uploaded bytes. 3. `processing`: product execution has started. 4. `sealing` (when applicable): sealing work in progress. 5. `indexing` (when applicable): indexing and retrieval preparation in progress. 6. Terminal outcome: `complete` or `failed`. Not every deal uses every intermediate state. Upload-less operations typically skip `uploading`. ## Status Checks Use `GET /status/{id}` for lifecycle tracking. Status is returned per product present in the deal. Status values: 1. `accepted` 2. `uploading` 3. `processing` 4. `sealing` 5. `indexing` 6. `complete` (terminal success) 7. `failed` (terminal failure) Use upload endpoints for byte-transfer progress (`/upload*`, `/uploads*`). Use `/status/{id}` for lifecycle state. ## Pipelines Pipelines are the asynchronous execution layer behind lifecycle transitions. What pipelines do: 1. Run ingest work for pull and upload flows. 2. Run product-specific execution once data is ready. 3. Advance deals to terminal outcome. When pipeline execution starts: 1. Pull-based deals: after acceptance. 2. Upload-based deals: after upload finalize. 3. Control-only operations: after acceptance. Operational guidance: 1. If an upload-based deal appears stalled, verify finalize was called. 2. If status remains non-terminal unexpectedly, provider-side operational inspection is required. # Products Products are the service contract between client intent and provider execution in Market 2.0. This section documents the currently supported product families, how they fit into the common deal model, and how to evolve product behavior without breaking compatibility. In practice, products answer one question: what operation should Curio perform for this deal. ## Current Products 1. [DDO v1](/market-2.0/products/ddo_v1) 2. [Retrieval v1](/market-2.0/products/retrieval_v1) 3. [PDP v1](/market-2.0/products/pdp_v1) ## Product Model Basics 1. Product-specific payloads live under `deal.products`. 2. Product-specific validation must be explicit and deterministic. 3. Product behavior should map cleanly to Market 2.0 lifecycle/status behavior. 4. Product errors should map to existing MK20 error codes where possible. ## Product Definition Checklist A product definition should clearly specify: 1. Payload schema under `products.`. 2. Product name and compatibility scope. 3. Required vs optional fields. 4. Validation rules and rejection conditions. 5. Source/format assumptions (if data is required). 6. Lifecycle/status effects. 7. Error mapping expectations. 8. Operational controls required by storage providers. ## Extending Products ### When to Extend an Existing Product Extend an existing product when: 1. Behavior remains backward compatible. 2. Existing clients can continue to parse and use payloads safely. 3. JSON marshal/unmarshal compatibility is preserved. ### When to Add a New Product Add a new product when: 1. Behavior or lifecycle differs materially. 2. Compatibility contracts diverge from existing product semantics. 3. Overloading an existing product would create ambiguous behavior. ### Implementation Checklist 1. Add product type in MK20 deal/product types. 2. Register product name and validation rules. 3. Add intake/processing/finalize handling. 4. Add upload-path handling if product supports `source_http_put`. 5. Extend status derivation if lifecycle differs. 6. Add DB migrations for product state. 7. Add WebRPC/UI support if operator controls are required. 8. Add/update product docs in this directory. ### Compatibility Rules 1. Do not weaken auth and deal ownership guarantees. 2. Keep piece identity invariants intact. 3. Avoid breaking existing product behavior. ### Documentation Requirements For each product, document: 1. Use case 2. Payload fields 3. Validation rules 4. Lifecycle/status effects 5. Common rejection scenarios # DDO v1 ## What `ddo_v1` Is For `ddo_v1` is the storage onboarding product for MK20. Use it when a client wants a provider to ingest a piece and run normal sealing/indexing flow, optionally tied to an external market contract. ## Struct Definition ```go type DDOV1 struct { Provider address.Address `json:"provider"` StartEpoch *abi.ChainEpoch `json:"start_epoch"` Duration abi.ChainEpoch `json:"duration"` AllocationId *verifreg.AllocationId `json:"allocation_id,omitempty"` MarketAddress string `json:"market_address"` MarketDealID *uint64 `json:"market_deal_id"` NotificationAddress address.Address `json:"notification_address"` NotificationPayload []byte `json:"notification_payload,omitempty"` } ``` ## Required And Optional Fields * Required: * `provider` * `duration` * Optional: * `start_epoch` * `allocation_id` * `market_address` * `market_deal_id` (required if `market_address` is set) * `notification_address` and `notification_payload` (must be provided together) ## Field Behavior * `start_epoch` is optional. If set, Curio validates it against chain head and `ExpectedPoRepSealDuration`, and uses it as the deal schedule start during ingestion. * `duration` is always part of the DDO payload. For verified allocations, it must still satisfy allocation term bounds. * `market_address` and `market_deal_id` opt the deal into external contract verification. * `notification_address` and `notification_payload`, when set, are attached to the eventual piece activation manifest during ingestion. They are not evaluated by `verifyDeal(...)`. ## Core Validation Rules * `ddo_v1` must be enabled by the provider. * Provider must be a valid address and not in MK20 disabled-miner config. * If `start_epoch` is set, it must be greater than `0`. * If `allocation_id` is missing, minimum duration is enforced (`>= 518400`). * If `allocation_id` is set, it must not be `NoAllocationID`. * If `market_address` is set, it must be a valid `0x` hex address. * Notification address and payload must either both be set or both be unset. ## Market Contract Verification When `market_address` is provided, MK20 performs read-only contract verification before acceptance. Verification behavior: 1. Contract must exist in `ddo_contracts` and be allowed. 2. Contract must implement `CurioDealViewV1` and return `version() == 1`. 3. MK20 builds `CurioDealView` from local deal values and calls `verifyDeal(...)`. 4. `verifyDeal(...)` must return `true`. This call is a read-only intake gate. It checks whether the market contract accepts the proposed deal. It does not by itself guarantee payment, payout, or notification success. Curio passes these values into `verifyDeal(...)`: * `dealId` * `state` = `Open` * provider actor id * client identity bytes * piece CID v2 * `start_epoch` or `0` * duration * `allocation_id` or `0` * `finalizedEpoch` = `0` If `verifyDeal` reverts with `DealNotFound(uint256)`, MK20 rejects the deal as market-missing. ## Notification Callback Behavior If both `notification_address` and `notification_payload` are set, Curio carries them into the piece activation manifest during ingestion as a `DataActivationNotification`. Operationally: 1. This path runs later than `verifyDeal(...)`, during sector activation work. 2. Contracts that rely on notification callbacks for settlement or state transitions should be reviewed separately from intake-time contract verification. 3. Provider setups that depend on notification callbacks generally keep `Subsystems.RequireNotificationSuccess = true` (the default). See [Default Curio Configuration](/configuration/default-curio-configuration). ## Additional Sanitize Checks Before Pipeline Entry * `retrieval_v1` must be present. * `retrieval_v1.announce_piece` must be false. * Provider must be in Curio miner set. * Deal data must be present. * Piece size must fit provider sector size. * Raw format cannot be indexed. * Notification address must resolve on chain when set. * Client must pass provider allow/deny policy. * If `start_epoch` is set, it must be greater than or equal to current chain height plus `ExpectedPoRepSealDuration`. * If `allocation_id` is set, MK20 validates allocation ownership, provider, term, data, size, and expiration consistency. * For verified allocations, the allocation owner may be either the client or the market contract address. * If both `allocation_id` and `start_epoch` are set, allocation expiration must be greater than or equal to `start_epoch`. # Retrieval v1 ## What `retrieval_v1` Is For `retrieval_v1` controls indexing and announcement behavior for data submitted through MK20 products. ## Struct Definition ```go type RetrievalV1 struct { Indexing bool `json:"indexing"` AnnouncePayload bool `json:"announce_payload"` AnnouncePiece bool `json:"announce_piece"` } ``` ## Fields * `indexing`: enable indexing flow. * `announce_payload`: announce payload-level information. * `announce_piece`: announce piece-level information. ## Validation Rules * `retrieval_v1` must be enabled by provider. * `announce_payload` requires `indexing = true`. ## Product Compatibility Rules * `ddo_v1` requires `retrieval_v1`. * With `ddo_v1`, `announce_piece` must be false. * Additional context-dependent checks are applied during DDO/PDP sanitize stages. # PDP v1 ## What `pdp_v1` Is For `pdp_v1` drives PDP dataset and piece lifecycle operations through MK20. ## Struct Definition ```go type PDPV1 struct { CreateDataSet bool `json:"create_data_set"` DeleteDataSet bool `json:"delete_data_set"` AddPiece bool `json:"add_piece"` DeletePiece bool `json:"delete_piece"` DataSetID *uint64 `json:"data_set_id,omitempty"` RecordKeeper string `json:"record_keeper"` PieceIDs []uint64 `json:"piece_ids,omitempty"` ExtraData []byte `json:"extra_data,omitempty"` } ``` ## Action Model Exactly one of these flags must be set in each deal: * `create_data_set` * `delete_data_set` * `add_piece` * `delete_piece` ## Key Fields * `data_set_id`: required for all actions except `create_data_set`. * `record_keeper`: required when creating a dataset. * `piece_ids`: required for `delete_piece`. * `extra_data`: verifier/service payload. It is required for `create_data_set`, `add_piece`, and `delete_piece`, and must be absent for `delete_data_set`. ## Validation Rules * `pdp_v1` must be enabled by provider. * Provider must have PDP signing key configured (`eth_keys.role = 'pdp'`). * Only one action flag is allowed per deal. * `create_data_set`: * `data_set_id` must be absent * `record_keeper` must be a valid hex address * `extra_data` must be present and no larger than 4 KiB * `delete_data_set` / `add_piece`: * `data_set_id` must be present * dataset must exist and be active * `delete_data_set`: * `extra_data` must be absent * `add_piece`: * `extra_data` must be present and no larger than 8 KiB * `delete_piece`: * `data_set_id` must be present * `piece_ids` must be present * all listed pieces must exist and be active in that dataset * `extra_data` must be present and no larger than 256 bytes ## Runtime Sanitize Rules * `add_piece` requires `retrieval_v1`. * `offline` source is rejected for PDP flows. * Raw-format data cannot use `announce_payload`. * Dataset and piece ownership checks are revalidated. ## Processing Behavior * `add_piece` with HTTP/aggregate data goes directly into PDP processing pipelines. * `add_piece` with `put` source (or upload-first mode) enters upload-waiting first. * Create/delete operations enqueue PDP lifecycle actions without upload flow. # Contracts This section covers Market 2.0 contract integration for product developers and contract integrators. ## Contract Development Basics When designing a contract to integrate with Curio: 1. Prefer read-only verification interfaces for deal checks. 2. Use explicit interface versioning. 3. Keep verification input and deal-state semantics stable and deterministic. 4. Use a deterministic not-found contract error for missing deal references. 5. Treat provider allowlisting as a separate operational policy. ## DDO Current DDO integration uses: 1. [CurioDealView v1](/market-2.0/contracts/curiodealview) 2. [DDO contract review and allowlisting](/market-2.0/contracts/ddo-contract-review) Future products can add their own contract interfaces without changing the top-level Market 2.0 deal envelope. # DDO: CurioDealView v1 This page defines the DDO market contract interface expected by Market 2.0. ## Contract Control for Storage Providers Storage providers decide which DDO market contracts Curio will trust. From the Curio UI, each contract is either: 1. Allowed: new DDO deals can reference this contract. 2. Blocked: new DDO deals using this contract are rejected. 3. Removed: contract is no longer configured; new DDO deals using this contract are rejected. ### What This Means Operationally 1. Allowing a contract enables new intake for that contract. 2. Blocking or removing a contract stops new intake immediately. 3. This policy applies to new submissions and deal updates that introduce DDO market verification. ### Adding a New Contract Safely Before allowing a contract, providers should verify: 1. The contract implements `CurioDealViewV1` correctly. 2. `verifyDeal` checks the same deal semantics Curio will pass in (`provider`, `client`, `piece CID`, `start epoch`, `duration`, `allocation`, finalization fields). 3. `getDealState` uses the expected `Open` / `Active` / `Finalized` meanings. 4. The contract's state transitions and finalization behavior are clear for payout and replay safety. Only after this review should the contract be set to Allowed. Operational review guidance is documented in: 1. [DDO contract review and allowlisting](/market-2.0/contracts/ddo-contract-review) ## Interface Contracts integrated with Curio for DDO verification should implement `ICurioDealViewV1`. ```solidity interface ICurioDealViewV1 { error DealNotFound(uint256 dealId); enum DealState { Open, Active, Finalized } struct CurioDealView { uint256 dealId; DealState state; uint256 providerActorId; bytes clientId; bytes pieceCidV2; uint256 startEpoch; uint256 duration; uint256 allocationId; uint256 finalizedEpoch; } function version() external pure returns (uint256); function verifyDeal(CurioDealView calldata deal) external view returns (bool); function getDealState(uint256 dealId) external view returns (DealState); } ``` ## Versioning `version()` must return `1` for this interface. ## `CurioDealView` Field Semantics 1. `dealId`: the market deal identifier referenced by `market_deal_id`. 2. `state`: Curio sends `Open` during intake verification. 3. `startEpoch`: must be `0` when unused. 4. `allocationId`: must be `0` when unused. 5. `finalizedEpoch`: must be `0` for non-finalized intake verification. ## Curio Verification Checks When `market_address` is set for a DDO deal, Curio performs read-only verification: 1. Contract is allowlisted by the provider. 2. `version()` is supported and returns `1`. 3. Curio builds a `CurioDealView` from local deal details and calls `verifyDeal(...)`. 4. `verifyDeal(...)` must return `true`. If `verifyDeal` reverts with `DealNotFound(uint256)`, Curio treats the market deal as missing. The interface also includes `getDealState(...)` for deal-state queries. ## Read-Only Boundary 1. Curio does not call write methods for this verification path. 2. This verification path only decides whether the contract accepts the proposed deal during intake. 3. Payment, settlement, and notification-driven state transitions are outside this read-only boundary. # DDO Contract Review and Allowlisting This page is for storage providers reviewing DDO market contracts, and for builders deploying them for Market 2.0. ## What Curio Actually Checks When a DDO deal includes `market_address`, Curio performs read-only intake verification before accepting the deal. Current checks: 1. `market_address` must exist in `ddo_contracts` and be marked `allowed = true`. 2. `market_deal_id` must be present. 3. Curio calls `version()` and requires `1`. 4. Curio builds `ICurioDealViewV1.CurioDealView` from the local deal and calls `verifyDeal(...)` via `eth_call`. 5. `verifyDeal(...)` must return `true`. 6. If `verifyDeal(...)` reverts with `DealNotFound(uint256)`, Curio rejects the deal as market-missing. What this does not mean: 1. This call only answers whether the contract accepts the proposed deal at intake time. 2. It does not by itself prove payment, payout, callback success, or settlement behavior. 3. `AddMarketContract` only validates address syntax and that the address resolves to a Filecoin actor before inserting it into Curio policy state. 4. Curio does not inspect proxy admin, upgrade controls, ownership, or source verification for the contract. ## Provider Control States Provider policy is separate from contract code compatibility. Current states: 1. Added and allowed: new DDO deals can reference the contract. 2. Added but blocked (`allowed = false`): new DDO deals using that contract are rejected. 3. Removed: contract is no longer configured; new DDO deals using it are rejected. 4. Existing accepted deals continue through processing even if the contract is later blocked or removed. 5. `GET /contracts` returns only allowed contracts. These controls are managed through Curio operator tooling and UI. ## Provider Review Checklist Before allowing a DDO market contract, providers should review: 1. Verified contract source and ABI from a source they trust. 2. The deployed address, preferably confirmed through a trusted builder channel. 3. `verifyDeal(...)` behavior, including validation of `providerActorId`, `clientId`, `pieceCidV2`, `startEpoch`, `duration`, `allocationId`, and the expected `state` / `finalizedEpoch` semantics. 4. Notification-driven settlement or callback paths separately from the intake verification path. 5. Ownership, upgrade authority, and any governance or timelock around the contract if it is proxy-based or upgradeable. Allocation note: 1. For allocation-backed DDO flows, Curio can resolve the allocation against either the deal client or the market contract address. 2. If the market contract allocates on behalf of end users, providers should make sure that ownership model is intentional and documented. ## Upgradeable Contract Risk Curio does not distinguish immutable and upgradeable contracts during allowlisting. Providers should treat upgradeable contracts as an ongoing trust decision: 1. Review who controls upgrades. 2. Review whether upgrades are gated by a timelock or multisig. 3. Re-review the contract when implementation or ownership changes. 4. Block or remove the contract if governance or implementation changes become unclear. ## Builder Guidance Builders integrating a DDO market contract should: 1. Publish verified source and a reliable ABI reference. 2. Implement `CurioDealViewV1` exactly and return `version() == 1`. 3. Keep `verifyDeal(...)` deterministic and read-only. 4. Validate the fields Curio sends rather than blindly accepting every call. 5. Use `DealNotFound(uint256)` for missing market deal identifiers. 6. Keep notification-driven callbacks efficient and document the expected failure behavior for providers if the contract depends on them. 7. Communicate upgrade process and governance model clearly if the contract is upgradeable. ## Related Pages 1. [CurioDealView v1 interface](/market-2.0/contracts/curiodealview) 2. [DDO product behavior](/market-2.0/products/ddo_v1) # Snap Deals This guide explains how to enable snap-deals in Curio. ## Simplified explanation Snap-deals allow storage providers to accept deals from users and place that user’s data into a block of storage that had already been committed. That was a bit of a mouthful, so picture it like this. Imagine there is a town with a very long shelf. Anyone in this town can store anything they want on this shelf. When a townsperson wants to store something, they give that *thing* to a storage provider. The storage provider builds a wooden box, puts the townsperson’s stuff into the box, and then puts the box on the shelf.
A shelf representing the Filecoin network.

Sector as a shelf

Some of the boxes have useful stuff in them, like photographs, music, or videos. But sometimes, the storage providers don’t have any townspeople lining up to put useful stuff into the boxes. So instead, they put packing peanuts in the box and put that on the shelf. This means that there are a lot of boxes being made to just hold packing peanuts. Making boxes takes a long time and takes a lot of work from the storage provider.
Types of data in a Filecoin sector.

Data boxes

Instead of creating a new box every time someone wants to store something, it’d be better if we could just replace the packing peanuts with useful stuff! Since nobody cares about the packing peanuts, nobody is going to be unhappy with throwing them out. And the storage provider gets to put useful stuff on the shelf without having to create a new box! Things are better for the townsperson, too, since they don’t have to wait for the storage provider to create a new box!
Emptying sectors of dummy data to fill them with real data.

Replacing data

This is a simplified view of how Snap-deals work. Instead of a storage provider creating an entirely new sector to store a client’s data, they can put the client’s data into a committed capacity sector. The data becomes available faster, things are less expensive for the storage provider, and more of the network’s storage capacity gets utilised! ## How to enable snap-deals To enable the snap deals pipeline in a Curio cluster, user needs to enable the snap deal specific tasks on the machines that have GPU resources. Apart from this, the deal ingestion pipeline needs to be updated to pass the deals to the snap deal pipeline instead of the PoRep sealing pipeline. {% hint style="warning" %} Data can be ingested using either the Snap Deals pipeline or the PoRep pipeline at any given time, but not both simultaneously. {% endhint %} ## FastSnap (SnapDeals UpdateEncode acceleration) Curio’s SnapDeals `UpdateEncode` path has a **fast mode** (“fastsnap”) that uses the batch sealing CUDA toolchain (`extern/supraseal`) to accelerate TreeR generation and uses Curio-native snap encoding. * **Capability check**: run ```bash curio test supra system-info ``` Look for **“Can run fast TreeR: yes”**. * **Fallback mode**: if the host lacks AVX-512 (AMD64v4) or a usable CUDA GPU, Curio automatically falls back to a CPU path for TreeR generation. * **Troubleshooting / force fallback**: set: ```bash export DISABLE_SUPRA_TREE_R=1 ``` This forces the CPU fallback TreeR path (useful to isolate batch sealing/toolchain issues). ### Configuration {% hint style="warning" %} When switching between Snap and PoRep deal pipeline, you must ensure that no sectors are being sealed or snapped. All pipelines must be empty before making a switch. {% endhint %} #### Curio Market 1. Enable snap deals on base layer. Enabling it on base layer is very important so that no node in the cluster accidentally forwards deal to PoRep pipeline. 2. Save the layer and exit. [Enable snap tasks](#enable-snap-tasks) and restart all the nodes. ``` [Ingest] DoSnap = true ``` #### Boost Adapter (Deprecated) {% hint style="warning" %} Boost adapter is no longer supported with new Curio releases. {% endhint %} 1. Create or update the market layer ([if one is already created](https://github.com/filecoin-project/curio/blob/main/documentation/en/enabling-market.md#enable-market-adapter-in-curio)) for the minerID where you wish to use snap deals pipeline.
```shell curio config add --title mt01000 ``` \ Add an entry like:
``` [Subsystems] EnableParkPiece = true BoostAdapters = ["t10000:127.0.0.1:32100"] [Ingest] DoSnap = true ``` \ Press `ctrl + D` to save and exit.\ Or edit the existing layer.
```shell curio config edit mt01000 ``` \ Enable the snap deals for ingestion:
``` [Subsystems] EnableParkPiece = true BoostAdapters = ["t10000:127.0.0.1:32100"] [Ingest] DoSnap = true ``` \ Save the layer and exit. 2. Add the new market configuration layer to the appropriate nodes based on the [best practices](/best-practices). ### Enable snap tasks 1. Add the `upgrade` layer already shipped with Curio to the `/etc/curio.env` file on the Curio nodes where GPU resources are available.
``` CURIO_LAYERS=gui,seal,post,upgrade <----- Add the "upgrade" layer CURIO_ALL_REMAINING_FIELDS_ARE_OPTIONAL=true CURIO_DB_HOST=yugabyte1,yugabyte2,yugabyte3 CURIO_DB_USER=yugabyte CURIO_DB_PASSWORD=yugabyte CURIO_DB_PORT=5433 CURIO_DB_NAME=yugabyte CURIO_REPO_PATH=~/.curio CURIO_NODE_NAME=ChangeMe FIL_PROOFS_USE_MULTICORE_SDR=1 ```
2. Restart the Curio services on the node.
``` systemctl restart curio ``` *** ## Troubleshooting Snap Deals ingestion ### Error: `allocating sector numbers: no suitable sectors found` Plain-English meaning: * Curio tried to pick an existing **CC sector** to upgrade (snap), but none matched the constraints. Common causes: * There are no CC sectors available to upgrade for that miner. * Expiration constraints: the candidate sectors cannot satisfy the deal end-epoch requirements. * Snap pipeline not actually enabled where ingestion runs (layer mismatch). * Required snap tasks are not running on any GPU-capable node. What to check first: 1. Confirm ingestion is routed to Snap: ```toml [Ingest] DoSnap = true ``` 2. Confirm at least one node is running the `upgrade` layer (or equivalent snap task enablement) and has GPU resources. 3. Check in the Curio UI whether the miner has CC sectors expected to be eligible for upgrades. What to include when asking for help: * deal UUID + piece CID * the full error line + surrounding logs * your `CURIO_LAYERS` for the ingest node and the GPU node ### Error: `skipped scheduling ParkPiece ... out of available Storage` Plain-English meaning: * Curio refused to start ParkPiece because there isn’t enough eligible free space on any attached storage path. What to do: * Verify free space on the storage paths attached to the node running ParkPiece. * Check `AllowTypes` / `DenyTypes` in `/sectorstorage.json` (if `unsealed` is effectively disallowed everywhere, ParkPiece can’t place data). * Check the Curio config knob `ParkPieceMinFreeStoragePercent` (default is 5%) and whether your current free space is below that threshold. See: * [Storage Configuration](/storage-configuration) ### Error: `no suitable data URL found for piece_id ` Plain-English meaning: * Curio has a piece to ingest, but can’t find any usable location for the bytes (most commonly: no data URL was stored for the parked piece). What to do: * Ensure you have added a data URL (and any required headers) for the deal/piece. * Verify the market/ingest layer is enabled on the node that runs ParkPiece. See: * [Storage Market](/curio-market/storage-market) * [Curio Market troubleshooting](/curio-market/troubleshooting) # Batch Sealing This page explains how to set up Curio batch sealing (extern/supraseal) {% hint style="danger" %} **Disclaimer:** Batch sealing is currently in **BETA**. Use with caution and expect potential issues or changes in future versions. Some additional manual system configuration is required. Batch sealing only supports **CC sectors** at the moment. The recommended workflow for onboarding deal data is to batch-seal CC sectors and then upgrade them with SnapDeals. If you enable batch sealing on a node but do **not** enable SnapDeals in the cluster, deals may be routed into the CC/batch pipeline which will seal empty sectors (discarding deal data). Make sure SnapDeals are enabled if you intend to onboard real data deals. {% endhint %} ## CC Scheduler (batch sealing only) Curio includes a **CC Scheduler** UI and DB table (`sectors_cc_scheduler`) used to decide how many CC sectors to queue for batch sealing per SP. * The **CC Scheduler is only used for batch sealing**. * Do not enable or rely on it for deals. Curio’s web UI exposes this as the **CC Scheduler** page/tab. ## Key Features * Seals multiple sectors (up to 128) in a single batch * Up to 16x better core utilisation efficiency * Optimized to utilize CPU and GPU resources efficiently * Uses raw NVMe devices for layer storage instead of RAM ## Requirements * CPU with at least 4 cores per CCX (AMD) or equivalent * NVMe drives with high IOPS (10-20M total IOPS recommended) * GPU for PC2 phase (NVIDIA RTX 3090 or better recommended) * 1GB hugepages configured (minimum 36 pages) * Ubuntu or compatible Linux distribution (**GCC 12 or 13 required**, doesn't need to be system-wide) * At least 256GB RAM, ALL MEMORY CHANNELS POPULATED * Without **all** memory channels populated sealing **performance will suffer drastically** * NUMA-Per-Socket (NPS) set to 1 ## Storage Recommendations You need 2 sets of NVMe drives: 1. Drives for layers: * Total 10-20M IOPS * Capacity for 11 x 32G x batchSize x pipelines * Raw unformatted block devices (SPDK will take them over) * Each drive should be able to sustain \~2GiB/s of writes * This requirement isn't understood well yet, it's possible that lower write rates are fine. More testing is needed. 2. Drives for P2 output: * With a filesystem * Fast with sufficient capacity (\~70G x batchSize x pipelines) * Can be remote storage if fast enough (\~500MiB/s/GPU) The following table shows the number of NVMe drives required for different batch sizes. The drive count column indicates `N + M` where `N` is the number of drives for layer data (SPDK) and `M` is the number of drives for P2 output (filesystem). The iops/drive column shows the minimum iops **per drive** required for the batch size. Batch size indicated with `2x` means dual-pipeline drive setup. IOPS requirements are calculated simply by dividing total target 10M IOPS by the number of drives. In reality, depending on CPU core speed this may be too low or higher than necessary. When ordering a system with barely enough IOPS plan to have free drive slots in case you need to add more drives later. | Batch Size | 3.84TB | 7.68TB | 12.8TB | 15.36TB | 30.72TB | | ------------ | ------ | ------ | ------ | ------- | ------- | | 32 | 4 + 1 | 2 + 1 | 1 + 1 | 1 + 1 | 1 + 1 | | ^ iops/drive | 2500K | 5000K | 10000K | 10000K | 10000K | | 64 (2x 32) | 7 + 2 | 4 + 1 | 2 + 1 | 2 + 1 | 1 + 1 | | ^ iops/drive | 1429K | 2500K | 5000K | 5000K | 10000K | | 128 (2x 64) | 13 + 3 | 7 + 2 | 4 + 1 | 4 + 1 | 2 + 1 | | ^ iops/drive | 770K | 1429K | 2500K | 2500K | 5000K | | 2x 128 | 26 + 6 | 13 + 3 | 8 + 2 | 7 + 2 | 4 + 1 | | ^ iops/drive | 385K | 770K | 1250K | 1429K | 2500K | ## Hardware Recommendations Currently, the community is trying to determine the best hardware configurations for batch sealing. Some general observations are: * Single socket systems will be easier to use at full capacity * You want a lot of NVMe slots, on PCIe Gen4 platforms with large batch sizes you may use 20-24 3.84TB NVMe drives * In general you'll want to make sure all memory channels are populated * You need 4\~8 physical cores (not threads) for batch-wide compute, then on each CCX you'll lose 1 core for a "coordinator" * Each thread computes 2 sectors * On zen2 and earlier hashers compute only one sector per thread * Large (many-core) CCX-es are typically better {% hint style="info" %} Please consider contributing to the [batch sealing hardware examples](https://github.com/filecoin-project/curio/discussions/140). {% endhint %} ## Setup ### Check NUMA setup: ```bash numactl --hardware ``` You should expect to see `available: 1 nodes (0)`. If you see more than one node you need to go into your UEFI and set `NUMA Per Socket` (or a similar setting) to 1. ### Configure hugepages: This can be done by adding the following to `/etc/default/grub`. You need 36 1G hugepages for the batch sealer. ```bash GRUB_CMDLINE_LINUX_DEFAULT="hugepages=36 default_hugepagesz=1G hugepagesz=1G" ``` Then run `sudo update-grub` and reboot the machine. Or at runtime: ```bash sudo sysctl -w vm.nr_hugepages=36 ``` Then check /proc/meminfo to verify the hugepages are available: ```bash cat /proc/meminfo | grep Huge ``` Expect output like: ``` AnonHugePages: 0 kB ShmemHugePages: 0 kB FileHugePages: 0 kB HugePages_Total: 36 HugePages_Free: 36 HugePages_Rsvd: 0 HugePages_Surp: 0 Hugepagesize: 1048576 kB ``` Check that `HugePages_Free` is equal to 36, the kernel can sometimes use some of the hugepages for other purposes. ### Dependencies CUDA 12.x or later is required (11.x won't work). The build process requires **GCC 12 or 13** — system-wide or as `gcc-12`/`g++-12` (or `gcc-13`/`g++-13`) installed locally. {% hint style="warning" %} **CUDA / GCC compatibility:** Your CUDA toolkit version determines which GCC it supports as a host compiler: * **CUDA 12.0–12.5:** supports GCC up to 12.x only → use `gcc-12`/`g++-12` * **CUDA 12.6+:** supports GCC up to 13.2 → either GCC 12 or 13 works * **CUDA 13.0+:** supports GCC 13+ → use `gcc-13`/`g++-13` If you get nvcc errors about unsupported compiler versions, check your CUDA/GCC pairing. {% endhint %} * On Arch install GCC 12 or 13 via your distro/AUR as appropriate * On Ubuntu/Debian install `gcc-12` and `g++-12` (or `gcc-13`/`g++-13`) ```shell # For CUDA 12.0–12.5: sudo apt install gcc-12 g++-12 # For CUDA 12.6+ or CUDA 13+: sudo apt install gcc-13 g++-13 ``` * In addition to general build dependencies (listed on the [installation page](/installation)), you need `libgmp-dev` and `libconfig++-dev` ```shell sudo apt install libgmp-dev libconfig++-dev ``` {% hint style="info" %} For SnapDeals “fastsnap” troubleshooting (fast TreeR path), you can check CPU/CUDA prerequisites with: ```bash curio test supra system-info ``` {% endhint %} ### Building Build and install the batch-capable Curio binary: ```bash make curio make sptool ``` ```shell make install ``` For calibnet ```bash make calibnet make calibnet-sptool ``` ```shell make install ``` {% hint style="warning" %} The build should be run on the target machine. Binaries won't be portable between CPU generations due to different AVX512 support. {% endhint %} ### Setup NVMe devices for SPDK: {% hint style="success" %} SPDK setup can be done automatically using the Curio CLI command: {% endhint %} ```bash sudo curio batch setup ``` This command will: * Download SPDK if not already available * Configure 1GB hugepages (36 pages by default) * Bind NVMe devices for use with batch sealing You can customize the number of hugepages: ```bash sudo curio batch setup --hugepages 36 --min-pages 36 ``` Alternatively, if you need to manually check SPDK status or unbind devices, you can use: ```bash cd extern/supraseal/deps/spdk-v24.05/ # Check status sudo ./scripts/setup.sh status # Manually run setup (not normally needed) sudo env NRHUGE=36 ./scripts/setup.sh ``` ### Benchmark NVME IOPS Please make sure to benchmark the raw NVME IOPS before proceeding with further configuration to verify that IOPS requirements are fulfilled. ```bash cd extern/supraseal/deps/spdk-v24.05/ # repeat -b with all devices you plan to use with supraseal # NOTE: You want to test with ALL devices so that you can see if there are any bottlenecks in the system ./build/examples/perf -b 0000:85:00.0 -b 0000:86:00.0... -q 64 -o 4096 -w randread -t 10 ``` The output should look like below ``` ======================================================== Latency(us) Device Information : IOPS MiB/s Average min max PCIE (0000:04:00.0) NSID 1 from core 0: 889422.78 3474.31 71.93 10.05 1040.94 PCIE (0000:41:00.0) NSID 1 from core 0: 890028.08 3476.67 71.88 10.69 1063.32 PCIE (0000:42:00.0) NSID 1 from core 0: 890035.08 3476.70 71.88 10.66 1001.86 PCIE (0000:86:00.0) NSID 1 from core 0: 889259.28 3473.67 71.95 10.62 1003.83 PCIE (0000:87:00.0) NSID 1 from core 0: 889179.58 3473.36 71.95 10.55 993.32 PCIE (0000:88:00.0) NSID 1 from core 0: 889272.18 3473.72 71.94 10.38 974.63 PCIE (0000:c1:00.0) NSID 1 from core 0: 889815.08 3475.84 71.90 10.97 1044.70 PCIE (0000:c2:00.0) NSID 1 from core 0: 889691.08 3475.36 71.91 11.04 1036.57 PCIE (0000:c3:00.0) NSID 1 from core 0: 890082.78 3476.89 71.88 10.44 1023.32 ======================================================== Total : 8006785.90 31276.51 71.91 10.05 1063.32 ``` With ideally >10M IOPS total for all devices. ### PC2 output storage Attach scratch space storage for PC2 output (batch sealer needs \~70GB per sector in batch - 32GiB for the sealed sector, and 36GiB for the cache directory with TreeC/TreeR and aux files) ## Usage 1. Start the Curio node with the batch sealer layer ```bash curio run --layers batch-machine1 ``` 2. Add a batch of CC sectors: ```bash curio seal start --now --cc --count 32 --actor f01234 --duration-days 365 ``` 3. Monitor progress - you should see a "Batch..." task running in the [Curio GUI](/curio-gui) 4. PC1 will take 3.5-5 hours, followed by PC2 on GPU 5. After batch completion, the storage will be released for the next batch ## Configuration * Run `curio calc batch-cpu` on the target machine to determine supported batch sizes for your CPU
Example batch-cpu output ``` # EPYC 7313 16-Core Processor root@udon:~# ./curio calc batch-cpu Basic CPU Information Processor count: 1 Core count: 16 Thread count: 32 Threads per core: 2 Cores per L3 cache (CCX): 4 L3 cache count (CCX count): 4 Hasher Threads per CCX: 6 Sectors per CCX: 12 --------- Batch Size: 16 sectors Required Threads: 8 Required CCX: 2 Required Cores: 6 hasher (+4 minimum for non-hashers) Enough cores available for hashers ✔ Non-hasher cores: 10 Enough cores for coordination ✔ pc1 writer: 1 pc1 reader: 2 pc1 orchestrator: 3 pc2 reader: 4 pc2 hasher: 5 pc2 hasher_cpu: 6 pc2 writer: 7 pc2 writer_cores: 3 c1 reader: 7 Unoccupied Cores: 0 { sectors = 16; coordinators = ( { core = 10; hashers = 2; }, { core = 12; hashers = 6; } ) } --------- Batch Size: 32 sectors Required Threads: 16 Required CCX: 3 Required Cores: 11 hasher (+4 minimum for non-hashers) Enough cores available for hashers ✔ Non-hasher cores: 5 Enough cores for coordination ✔ ! P2 hasher will share a core with P1 writer, performance may be impacted ! P2 hasher_cpu will share a core with P2 reader, performance may be impacted pc1 writer: 1 pc1 reader: 2 pc1 orchestrator: 3 pc2 reader: 0 pc2 hasher: 1 pc2 hasher_cpu: 0 pc2 writer: 4 pc2 writer_cores: 1 c1 reader: 0 Unoccupied Cores: 0 { sectors = 32; coordinators = ( { core = 5; hashers = 4; }, { core = 8; hashers = 6; }, { core = 12; hashers = 6; } ) } --------- Batch Size: 64 sectors Required Threads: 32 Required CCX: 6 Required Cores: 22 hasher (+4 minimum for non-hashers) Not enough cores available for hashers ✘ Batch Size: 128 sectors Required Threads: 64 Required CCX: 11 Required Cores: 43 hasher (+4 minimum for non-hashers) Not enough cores available for hashers ✘ ```
* Create a new layer configuration for the batch sealer, e.g. batch-machine1: ```toml [Subsystems] EnableBatchSeal = true [Seal] # This field is optional. In most setups, NVMe devices can be inferred automatically if this configuration is omitted. LayerNVMEDevices = [ "0000:88:00.0", "0000:86:00.0", # Add PCIe addresses for all NVMe devices to use ] # Set to your desired batch size (what the batch-cpu command says your CPU supports AND what you have nvme space for) BatchSealBatchSize = 32 # pipelines can be either 1 or 2; 2 pipelines double storage requirements but in correctly balanced systems makes # layer hashing run 100% of the time, nearly doubling throughput BatchSealPipelines = 2 # Set to true for Zen2 or older CPUs for compatibility SingleHasherPerThread = false ``` ### Environment Variables | Variable | Description | | ---------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `DISABLE_SPDK_SETUP=1` | When set, disables automatic SPDK setup (hugepage configuration and NVMe device binding) during supraseal initialization. Useful for advanced users who want to manually manage SPDK configuration, map drives, or control hugepage-numa assignment. | ## Optimization * Balance batch size, CPU cores, and NVMe drives to keep PC1 running constantly * Ensure sufficient GPU capacity to complete PC2 before next PC1 batch finishes * Monitor CPU, GPU and NVMe utilization to identify bottlenecks * Monitor hasher core utilisation ## Troubleshooting ### Node doesn't start / isn't visible in the UI * Ensure hugepages are configured correctly * Check NVMe device IOPS and capacity * If spdk setup fails, try to `wipefs -a` the NVMe devices (this will wipe partitions from the devices, be careful!) ### Performance issues You can monitor performance by looking at "hasher" core utilisation in e.g. `htop`. To identify hasher cores, call `curio calc supraseal-config --batch-size 128` (with the correct batch size), and look for `coordinators` ```go topology: ... { pc1: { writer = 1; ... hashers_per_core = 2; sector_configs: ( { sectors = 128; coordinators = ( { core = 59; hashers = 8; }, { core = 64; hashers = 14; }, { core = 72; hashers = 14; }, { core = 80; hashers = 14; }, { core = 88; hashers = 14; } ) } ) }, pc2: { ... } ``` In this example, cores 59, 64, 72, 80, and 88 are "coordinators", with two hashers per core, meaning that * In first group core 59 is a coordinator, cores 60-63 are hashers (4 hasher cores / 8 hasher threads) * In second group core 64 is a coordinator, cores 65-71 are hashers (7 hasher cores / 14 hasher threads) * And so on Coordinator cores will usually sit at 100% utilisation, hasher threads **SHOULD** sit at 100% utilisation, anything less indicates a bottleneck in the system, like not enough NVMe IOPS, not enough Memory bandwidth, or incorrect NUMA setup. To troubleshoot: * Read the requirements at the top of this page very carefully * [Benchmark NVME IOPS](#benchmark-nvme-iops) * Validate GPU setup if PC2 is slow * Review logs for any errors during batch processing ### Slower than expected NVMe speed If the [NVME Benchmark](#benchmark-nvme-iops) shows lower than expected IOPS, you can try formatting the NVMe devices with SPDK: ```bash cd extern/supraseal/deps/spdk-v24.05/ ./build/examples/nvme_manage ``` Go through the menus like this ``` NVMe Management Options [1: list controllers] [2: create namespace] [3: delete namespace] [4: attach namespace to controller] [5: detach namespace from controller] [6: format namespace or controller] [7: firmware update] [8: opal] [9: quit] 6 0000:04:00.00 SAMSUNG MZQL23T8HCLS-00A07 S64HNG0W829861 6 0000:41:00.00 SAMSUNG MZQL23T8HCLS-00A07 S64HNG0W829862 6 0000:42:00.00 SAMSUNG MZQL23T8HCLS-00A07 S64HNG0W829860 6 0000:86:00.00 SAMSUNG MZQL23T8HCLS-00A07 S64HNG0W829794 6 0000:87:00.00 SAMSUNG MZQL23T8HCLS-00A07 S64HNG0W829798 6 0000:88:00.00 SAMSUNG MZQL23T8HCLS-00A07 S64HNG0W829837 6 0000:c1:00.00 SAMSUNG MZQL23T8HCLS-00A07 S64HNG0W829795 6 0000:c2:00.00 SAMSUNG MZQL23T8HCLS-00A07 S64HNG0W829836 6 0000:c3:00.00 SAMSUNG MZQL23T8HCLS-00A07 S64HNG0W829797 6 0000:c4:00.00 SAMSUNG MZQL23T8HCLS-00A07 S64HNG0W829850 6 Please Input PCI Address(domain:bus:dev.func): 0000:c4:00.00 Please Input Namespace ID (1 - 32): 1 ## Select 1 Please Input Secure Erase Setting: 0: No secure erase operation requested 1: User data erase 2: Cryptographic erase 0 Supported LBA formats: 0: 512 data bytes 1: 4096 data bytes Please input LBA format index (0 - 1): 1 ## Select 4096 data bytes Warning: use this utility at your own risk. This command will format your namespace and all data will be lost. This command may take several minutes to complete, so do not interrupt the utility until it completes. Press 'Y' to continue with the format operation. y ``` Then you might see a difference in performance like this: ``` Latency(us) Device Information : IOPS MiB/s Average min max PCIE (0000:c1:00.0) NSID 1 from core 0: 721383.71 2817.91 88.68 11.20 591.51 ## before PCIE (0000:86:00.0) NSID 1 from core 0: 1205271.62 4708.09 53.07 11.87 446.84 ## after ``` *** ## Troubleshooting batch commit / all-or-nothing failures Support frequently sees failures of the form: * `all-or-nothing: Batch successes 67/68, Batch failing: [code=.. at idx=..] ...` What it means: * Some batch operations are **atomic**: a single failing entry can cause the whole batch to fail. How to debug (operator workflow): 1. In the Curio UI, locate the failing **CommitBatch / PreCommitBatch** task and open its details. 2. Correlate the batch failure with the per-sector tasks immediately before it (often the underlying failure is earlier). 3. Check chain/message errors: many failures are actually wallet funding, fee, or chain state errors. 4. Collect: * `sp_id` * the task ID(s) * the full error message including the `idx=...` How to recover safely: * If you suspect one “bad” sector is poisoning the batch, temporarily reduce batching (batch size / concurrency) to isolate. * Avoid manual DB edits unless you know exactly what you are doing (take a backup first). If you’re stuck: * follow `documentation/en/troubleshooting/collect-debug-info.md` and include the task IDs. # Scaling Curio cluster This page describes how to add additional nodes or miner IDs to a Curio cluster ## Migrating additional lotus-miner to Curio To migrate your second or later `lotus-miner` to an existing Curio cluster, you need to follow the same steps as before. The only exception would be that there is no need to install a new YugabyteDB cluster. After installing the `curio` binary on the `lotus-miner` node, you can run `curio guided-setup` to start the migration. ## Initialising additional Miner IDs in existing Curio cluster The process to initiate a new minerID on the network is same as when you [initialise a new Curio cluster with a new minerID.](/setup#initiating-a-new-curio-cluster) The only exception would be that there is no need to install a new YugabyteDB cluster ## Migrating lotus-worker to Curio cluster Once you have migrated a minerID to the Curio cluster, you would need to repurpose all of your `lotus-worker` nodes attached to the migrated minerID as Curio nodes. 1. [Install](/installation) `curio` binary on the `lotus-worker` node. 2. [Configure the service ENV file](/curio-service#environment-variables-configuration) with correct details. 3. Start the new `curio` node and verify in GUI that the new node is now part of the cluster. 4. [Attach the existing storage](/storage-configuration#attach-existing-storage-to-curio) to the Curio node. 5. Repeat for the rest of the workers. ## Adding nodes to Curio cluster To add new nodes to an existing Curio cluster, please follow the below process. * [Install](/installation) `curio` binary. * [Configure the service ENV file](/curio-service#environment-variables-configuration) with correct details. * Start the new `curio` node and verify in GUI that the new node is now part of the cluster. * [Attach any new or existing storage](/storage-configuration) if required. * Repeat for any additional nodes to be attached. # Curio GUI This page describe how to access the Curio GUI and what information is available there. ## Accessing Curio GUI The default port is `4701` to access the Curio GUI. To enable GUI on a Curio node user must start a Curio node with `gui` layer. This is a pre-built layer shipped with Curio binaries. ### Changing default GUI port You can change the default GUI port by setting a different IP address and port in the "base" layer of the configuration. We highly recommend not specifying the GUI address in other layers to avoid confusion. ``` curio config edit base ``` This will open the "base" layer in your default text editor. ``` # The address that should listen for Web GUI requests. # # type: string #GuiAddress = "0.0.0.0:4701" should be changed to below # The address that should listen for Web GUI requests. # # type: string GuiAddress = "127.0.0.1:4702" ``` Save the configuration and restart the Curio service on the node running GUI layer to access the GUI on the new address and port. ## GUI menu and dashboards {% hint style="danger" %} Curio web UI is currently under development. Some UI pages might change over time and could be different from the screenshots and description below. {% endhint %} ### Home Page

Curio home page

Chain Connectivity: Chain sync status of all available Lotus daemon nodes Cluster Machines: A quick list of all Curio nodes in the cluster PoRep Pipeline: A quick summary of cluster wide sealing sectors Actor Summary: Summary of minerIDs served by this Curio cluster ### Configuration page All configuration layers can be found and edited from the `configuration` page of the GUI. It also allows adding new configuration layers via GUI.

Configuration page

Editing a configuration layer

Editing a config layer

### Sectors Curio GUI can be used to browse the list of all sectors from all the minerID served by the Curio cluster. This is the GUI replacement of `lotus-miner sectors list` from `lotus-miner`.

Curio sector list

### PoRep Pipeline This page can be used to glance as the sectors current being sealed and historically sealed by Curio cluster. It details sector moving through the various sealing stages and status of each stage.

PoRep pipeline

Users can click on "DETAILS" and get more detailed information about a sector. This page will tell you about the pieces, storage etc.

Sector details

### Node details On the home page, in the "Cluster Machines" list, a user can click on the machine name to get a more detailed view of each node. It will list attached storage and status of tasks handled by that particular machine.

Node details

# Garbage Collection Garbage collection and cleanup process in Curio ## Sealing Pipeline cleanup The **SDRPipelineGC** is a periodic task in the system that ensures the efficiency and effectiveness of the storage sealing process. It is responsible for cleaning up completed entries in the sealing pipeline. ### Process The GC operates by removing sealing pipeline entries that have finished the sealing process. These entries have their metadata already stored in the long-term sector metadata table (also known as the `sectors_meta` table). This operation aids in maintaining the pipeline fluid and uncluttered, enhancing overall system performance. ### Handling failed sector In case a sector fails the sealing process, its corresponding entry can be manually removed via the web user interface (WebUI). This function allows for active management of pipeline entries and ensures that failed entries do not stall the pipeline. ## Storage Cleanup The `StorageGCMark` component is responsible for sweeping through all sector files in the system. A sector file will be marked in the `storage_removal_marks` table, under the following conditions: * The sector is not "pinned" in the `storage_gc_pins` table. (A sector pin indicates that the sector should not be removed, even after it has expired.) * The sector is not present in the sealing table called `sectors_sdr_pipeline`. * The sector has been labeled as a "failed" sector. (Note that "failed" sectors must first be removed from the pipeline table before they can have their data garbage collected.) * The sector is not present in the miner actor precommit sector set. * The sector is not present in the `Live` or `Unproven` sector sets. ### Approval and Removal Removal marks from the `StorageGCMark` process need to be approved separately. This approval is currently available only through the WebUI. In the future, there may be extensions to allow auto-approval, backed by custom policies for selection.

Storage GC approval

Once a removal mark has been granted approval, the periodic `StorageGCSweep` task will review all approved removal marks. This task will then proceed to delete the files which have been approved for removal. This final stage ensures that only necessary data remains in the system, optimizing storage and improving the overall system's functionality. ### Removing a failed sector For the removal of a sector that has failed the sealing process, users should go to the "PoRep" page on the WebUI and select the "DETAILS" link corresponding to the sector in question. This action will redirect them to a page where a "Remove" button is available. On clicking this button, the failed sector will be extricated from the SDR pipeline table, making it available for the StorageGCMark process to mark it for garbage collection. However, the removal of the sector will only ensue after it gains the requisite approval. This approval can be provided on the "Storage GC Info" page. On receipt of the approval, the StorageGCSweep task will review the mark and proceed to delete the sector files, effectively removing the failed sector from the system

How GC a failed sector

# Best Practices Curio best practices 1. YugybteDB backing the Curio cluster should be multi node to avoid single point of failure. 2. All miner IDs should be part of the base layer. We highly recommend not creating separate layers for different MinerIDs but using different layers for control addresses if required. 3. No worker should be dedicated to specific minerIDs. All Curio nodes should be setup to allow jobs for any minerIDs. 4. Multiple workers should be started with `--post` layer to allow fast wdPost and winPost turn around time. 5. We recommend running 1 GUI layer enabled node. A cluster wide GUI can be access via this node without putting any additional strain of read operations on the DB. 6. The unsealed and sealed copies should not be stored in the same storage location. Curio will allow automatic regeneration of sealed and unsealed in future if one is lost. 7. The **`DoSnap`** configuration should be set at the **base layer** to ensure that each node has the correct deal handling. This ensures consistency across all nodes and prevents misconfiguration. 8. Market requires a domain name in Curio. A domain name should be prepared before configuring the market. 9. Users utilizing a **reverse proxy** for delegated TLS configuration should enable the **market** service on multiple nodes to ensure **high availability** for the HTTP server. Additionally, **libp2p** should be enabled on multiple nodes to maintain **high availability** for peer-to-peer communications. 10. It is recommended to create a distinct layer for each market adapter (Deprecated), corresponding to each minerID. This configuration enables precise control, allowing for the assignment of specific minerIDs to either the Snap Deals pipeline or the PoRep pipeline. 11. It is advised to run the market adapter (Deprecated) on the same node that will execute the TreeD task for the PoRep pipeline or the Encode task for the Snap Deals pipeline. # Administration Administration guides for managing your Curio cluster This section covers administrative tasks for maintaining and managing your Curio cluster. ## Guides * [Node Maintenance & Cordoning](/administration/node-maintenance) - How to safely cordon nodes for maintenance without disrupting sealing pipelines * [YugabyteDB Backup](/administration/yugabyte-backup) - How to backup and restore your database * [YugabyteDB Troubleshooting](/administration/yugabyte-troubleshooting) - Common YB migration/runtime issues and how to resolve them * [Indexing / CheckIndex Troubleshooting](/administration/indexing-checkindex-troubleshooting) - Diagnose deal indexing and CheckIndex pipeline errors # Node Maintenance & Cordoning How to safely cordon Curio nodes for maintenance without disrupting in-progress sealing pipelines. ## What cordoning does Running `curio cordon` (or setting `unschedulable = true` in the WebUI) tells the Harmony scheduler to **stop scheduling new tasks** on a node. Tasks that are already running will finish, but no new work will be picked up. This is intentionally simple: cordon → wait for running tasks to finish → do maintenance → restart → uncordon. ## How cordoning affects the sealing pipeline Cordoning blocks **all** new task scheduling on the node, including pipeline continuation tasks like TreeD, TreeRC, SyntheticProofs, and Finalize. If a sector's data is **location-bound** to the cordoned node (the sector cache lives on local storage), those follow-up tasks cannot run on another node either. **What this means in practice:** * If you cordon a node that has sectors mid-pipeline (e.g., SDR completed but TreeRC not yet started), those sectors will be **paused** until the node is uncordoned. * The sectors are not lost — they will resume once the node is uncordoned and the scheduler picks them up again. * However, if the node stays cordoned for too long, sectors may **expire** (miss their precommit deadline), wasting the SDR work. {% hint style="warning" %} **Non-batched sealing operators:** SDR takes many hours. If you cordon a node and leave it cordoned past the sector's precommit deadline, that SDR work is lost. Plan maintenance windows accordingly. {% endhint %} ## Recommended workflows ### Quick maintenance (restart, upgrade, config change) For short interruptions where downtime is minutes, not hours: 1. **Check the pipeline** — in the WebUI, verify what stages are in progress on the node. 2. `curio cordon ` — stop new work from being scheduled. 3. **Wait** for currently-running tasks to complete (watch the WebUI or logs). 4. Do your maintenance (restart, upgrade, etc.). 5. `curio uncordon ` — resume scheduling. Pipeline sectors that were paused (waiting for their next stage) will resume automatically after uncordon. ### Planned extended maintenance (hours) If the node will be down for an extended period: 1. **Stop starting new sectors** first — pause deal intake or CC sector creation so no new SDR work begins on the node. 2. **Wait for in-progress pipelines to clear** — let sectors finish through Finalize before cordoning. Monitor via the pipeline view in the WebUI. 3. `curio cordon ` once pipelines are drained. 4. Do your maintenance. 5. `curio uncordon ` when ready. ### Decommissioning a node or long outage If you know a node will be offline for a long time (longer than precommit deadlines): 1. **Attach the node's storage to another node** — use `curio attach` on a different machine to make the sector data accessible elsewhere. 2. Other nodes with access to the storage paths can then pick up the remaining pipeline tasks. 3. Cordon and shut down the original node. See [Storage Configuration](/storage-configuration) for details on attaching storage. ## Key points to remember | Aspect | Behavior | | ----------------------------- | ---------------------------------------------------------------------------- | | Running tasks | Finish normally on the cordoned node | | New task scheduling | Blocked — no new work starts | | Location-bound pipeline tasks | Paused until uncordon (data is on local storage, other nodes can't run them) | | Sector data | Safe — nothing is deleted or moved by cordoning | | Precommit deadlines | Still apply — sectors can expire if cordoned too long | ## Common mistakes * **Cordoning during active SDR without a plan to uncordon quickly.** SDR is the longest pipeline stage. If you cordon right after SDR completes, the follow-up stages (TreeD, TreeRC, etc.) are blocked. Either wait for the full pipeline to clear first, or ensure you'll uncordon in time. * **Forgetting to uncordon after maintenance.** The node will sit idle, and any paused pipelines will remain stuck. Set a reminder or check the WebUI after maintenance. * **Cordoning when you should be migrating storage.** If the node is going away permanently, cordon alone won't help — you need to move or re-attach the storage paths so other nodes can access the sector data. # YugabyteDB Backup How to backup and restore your YugabyteDB database for Curio Maintaining regular backups of your YugabyteDB database is critical for disaster recovery and before performing operations like software downgrades. This guide covers essential backup and restore procedures for your Curio cluster's database. {% hint style="danger" %} **Always create a backup before running `curio toolbox downgrade`** or performing any major cluster operations. Database schema changes during upgrades may not be reversible without a backup. {% endhint %} ## Important: Curio uses a DB *and* a schema Curio connects to YugabyteDB over **YSQL** (Postgres protocol). Two configuration values matter: * **Database name** (`CURIO_DB_NAME` / `--db-name`) – default is `yugabyte` * **Schema** inside that database – Curio uses schema **`curio`** by default So, on a default setup, you typically want to back up the **`yugabyte`** database (and it will include the `curio` schema). Before running backups, confirm what Curio is configured to use: ```bash # If you run Curio with env vars echo "$CURIO_DB_NAME" # If Curio is installed on PATH, you can inspect flags/defaults curio --help | grep -E "db-(name|host|port)" ``` If you’re running Curio via systemd or containers, check the env/flags there: ```bash # systemd (example) systemctl cat curio | sed -n '1,200p' # docker compose (example) docker compose config | sed -n '1,200p' ``` ## Prerequisites * Access to your YugabyteDB cluster * The `ysql_dump` and `ysqlsh` utilities (included with YugabyteDB installation) * Sufficient disk space for backup files ## Backup Methods ### Method 1: Using ysql\_dump (Recommended) The `ysql_dump` utility creates a logical backup of your database that can be restored to any YugabyteDB cluster. #### Full database backup ```bash ysql_dump -h -p 5433 -U -d -F c -f curio_backup_$(date +%Y%m%d_%H%M%S).dump ``` **Parameters:** * `-h`: YugabyteDB host address * `-p`: YSQL port (default: 5433) * `-U`: Database username * `-d`: Database name (often `yugabyte` for Curio defaults) * `-F c`: Custom format (compressed, supports parallel restore) * `-f`: Output filename #### Example with typical Curio defaults ```bash ysql_dump -h 127.0.0.1 -p 5433 -U yugabyte -d yugabyte -F c -f curio_backup_$(date +%Y%m%d_%H%M%S).dump ``` #### Curio schema backup If you want to back up only Curio’s schema (and its data), not other schemas in the DB: ```bash ysql_dump -h -p 5433 -U -d --schema=curio -F c -f curio_schema_$(date +%Y%m%d_%H%M%S).dump ``` If you want **schema-only (DDL only)** for Curio (no data): ```bash ysql_dump -h -p 5433 -U -d --schema=curio --schema-only -f curio_schema_$(date +%Y%m%d_%H%M%S).sql ``` ### Method 2: Using ysqlsh with COPY For smaller exports or specific tables: ```bash ysqlsh -h -p 5433 -U -d -c "\COPY TO 'table_backup.csv' WITH CSV HEADER" ``` ## Restore Procedures ### Restore from ysql\_dump backup {% hint style="warning" %} Ensure all Curio nodes are **stopped** before restoring a backup. {% endhint %} #### Full restore (entire database) ```bash # Drop and recreate the database (ONLY if you intend to restore the full DB) ysqlsh -h -p 5433 -U -c "DROP DATABASE IF EXISTS ;" ysqlsh -h -p 5433 -U -c "CREATE DATABASE ;" # Restore from backup pg_restore -h -p 5433 -U -d -F c curio_backup_YYYYMMDD_HHMMSS.dump ``` #### Restore only the Curio schema If you created a Curio-schema dump (`--schema=curio`), restore into the existing DB: ```bash pg_restore -h -p 5433 -U -d -F c curio_schema_YYYYMMDD_HHMMSS.dump ``` If you created a Curio schema-only SQL file (`--schema-only`), apply it with ysqlsh: ```bash ysqlsh -h -p 5433 -U -d -f curio_schema_YYYYMMDD_HHMMSS.sql ``` ## Automated Backup Script (example) ```bash #!/bin/bash # curio-db-backup.sh BACKUP_DIR="/path/to/backups" YB_HOST="127.0.0.1" YB_PORT="5433" YB_USER="yugabyte" YB_DB="yugabyte" # Curio default DB name RETENTION_DAYS=7 mkdir -p "$BACKUP_DIR" BACKUP_FILE="$BACKUP_DIR/curio_backup_$(date +%Y%m%d_%H%M%S).dump" ysql_dump -h "$YB_HOST" -p "$YB_PORT" -U "$YB_USER" -d "$YB_DB" -F c -f "$BACKUP_FILE" if [ $? -eq 0 ]; then echo "Backup created successfully: $BACKUP_FILE" find "$BACKUP_DIR" -name "curio_backup_*.dump" -mtime +$RETENTION_DAYS -delete else echo "Backup failed!" exit 1 fi ``` ## Best Practices 1. **Regular backups**: schedule automated daily backups for production clusters 2. **Test restores**: periodically verify backups by performing test restores 3. **Off-site storage**: store backup copies in a different location or cloud storage 4. **Pre-upgrade backups**: always create a fresh backup before upgrading or downgrading Curio 5. **Monitor backup size**: ensure adequate storage capacity ## Troubleshooting ### Connection issues ```bash # Verify YugabyteDB is running yugabyted status # Check connectivity ysqlsh -h -p 5433 -U yugabyte -c "SELECT version();" ``` ### Permission errors Ensure your database user has sufficient privileges: ```sql GRANT ALL PRIVILEGES ON DATABASE TO ; ``` You may also need privileges on the `curio` schema and its objects depending on how your cluster is secured. ## Additional Resources * [YugabyteDB Backup and Restore Documentation](https://docs.yugabyte.com/preview/manage/backup-restore/) * [ysql\_dump Reference](https://docs.yugabyte.com/preview/admin/ysql-dump/) * [YugabyteDB Best Practices](https://docs.yugabyte.com/preview/develop/best-practices-ysql/) *** ## Multi-node cluster caveats (practical ops notes) If you run a multi-node Yugabyte cluster: * Validate that your backup method matches your recovery goal. * Logical dumps (`ysql_dump`) are typically easiest for portability. * If you rely on Yugabyte-native distributed backups/snapshots, ensure you test restores. Before a restore/downgrade: * stop all Curio writers * confirm no nodes are still writing tasks/metrics See also: * `documentation/en/administration/yugabyte-troubleshooting.md` # YugabyteDB troubleshooting Operational troubleshooting for YugabyteDB when used as Curio’s control plane. Many “Curio issues” are actually **control-plane issues** (YugabyteDB health, schema, connectivity, or performance). Before doing anything risky: * take a backup (see [YugabyteDB Backup](/administration/yugabyte-backup)) * stop Curio writers if you are restoring/downgrading *** ## 1) Quick health checklist On the Yugabyte nodes: * `yugabyted status` * check `yb-master` and `yb-tserver` logs for FATALs From a Curio node: ```bash ysqlsh -h "$CURIO_DB_HOST" -p "${CURIO_DB_PORT:-5433}" -U "$CURIO_DB_USER" -d "${CURIO_DB_NAME:-yugabyte}" -c "select 1;" ``` If the above is slow/intermittent, fix DB health first. *** ## 2) Connection string / multi-host setups Curio commonly uses a comma-separated host list: ```bash CURIO_DB_HOST=yugabyte1,yugabyte2,yugabyte3 CURIO_DB_PORT=5433 CURIO_DB_NAME=yugabyte ``` Tips: * Ensure all hosts are reachable from all Curio nodes. * If you disable load balancing in Curio, it may pin to the first host; keep that host stable. *** ## 3) ulimit / file descriptor exhaustion Symptoms: * DB is unstable under load, or tserver logs mention file descriptor issues. What to do: * Ensure you applied recommended `nofile` limits (see PDP guide and Yugabyte docs). * Restart services after changing persistent limits. *** ## 4) Upgrade / downgrade safety Rules of thumb: * Back up before changing Curio or Yugabyte versions. * Stop Curio before restoring a DB dump. * If a schema migration fails, capture: * the migration filename * full error output * Yugabyte version Common migration blocker: * `Rewriting of YB table is not yet implemented (SQLSTATE 0A000)` *** ## 5) Common SQLSTATE errors in support ### `SQLSTATE 23505` (duplicate key) Often indicates: * concurrency/retries inserting the same identity row. Action: * verify whether the app is progressing or stuck. * if stuck, correlate with task errors and consider a targeted restart of the relevant Curio subsystem. ### `relation "curio." does not exist` Usually indicates: * schema migrations didn’t run, or you are connected to the wrong DB name/schema. Action: * confirm `CURIO_DB_NAME` and that schema `curio` exists. *** ## 6) Backups Follow the backup guidance here: * [YugabyteDB Backup](/administration/yugabyte-backup) # Indexing & CheckIndex troubleshooting Troubleshooting indexing/IPNI and the CheckIndex task in Curio. This page is built from recurring support incidents in `#fil-curio-help`. ## What indexing is (plain English) Curio maintains a local index so retrieval can answer: “Which piece contains this multihash, and at what offset?”. If you use IPNI, Curio may also publish advertisements so the wider retrieval ecosystem can discover your content. ## What `CheckIndex` does `CheckIndex` is a background task that periodically checks whether indexing and announcements are complete and schedules follow-up work when something is missing or needs retrying. Code reference: * `tasks/indexing/task_check_indexes.go` ## First response playbook (safe steps) When indexing looks broken, do these in order: 1. Check DB health first * Many “indexing errors” are actually Yugabyte slowness/unavailability. * See: `documentation/en/administration/yugabyte-troubleshooting.md` 2. Check whether you *have unsealed data* available * Re-indexing typically requires access to an **unsealed copy**. * If your cluster intentionally does not keep unsealed data, expect errors like “no unsealed copy found” when attempting reindex. 3. Reduce overload (temporary) * If tasks are piling up, temporarily reduce indexing concurrency on the node(s) that run market/indexing. * Knobs live in Curio configuration (see `configuration/default-curio-configuration.md`). ## Common symptoms and what they usually mean ### Many `CheckIndex` tasks at once / “storm” Typical causes: * DB health issues causing retries. * A node repeatedly restarting and re-enqueueing work. * A configuration mismatch causing work to never complete. What to collect: * counts of tasks by type * the oldest CheckIndex task ID + its error logs ### `SQLSTATE 23505 duplicate key value violates unique constraint ...` Plain-English meaning: * Two workers attempted to create the same “identity” row concurrently. What to do: * Determine if it’s transient noise (pipeline still progresses) or if the same deal/piece is stuck. * Collect: deal UUID, piece CID, task IDs, and the full error span. ### `piece missing in indexstore` / `no unsealed copy of sector found for reindexing` Plain-English meaning: * Curio is being asked to build/serve an index for content, but cannot find the bytes required to construct it. What to do: * Confirm at least one storage path allows `unsealed` and has the data. * Confirm the node doing indexing can access the path. ## Quick DB inspection (YSQL) > These queries assume default schema `curio`. Count tasks by name: ```sql select name, count(*) from curio.harmony_task group by name order by count(*) desc; ``` List recent CheckIndex tasks: ```sql select id, posted_time, update_time, owner_id, retries from curio.harmony_task where name = 'CheckIndex' order by posted_time desc limit 50; ``` ## Last resort: deleting stuck CheckIndex tasks (use with caution) Only do this if you understand the impact and have a DB backup. 1. Stop Curio on the node running indexing/market. 2. Delete tasks: ```sql delete from curio.harmony_task where name = 'CheckIndex'; ``` 3. Start Curio again and monitor task creation rate. If the storm returns immediately, the root cause is still present (DB health, configuration, or missing data). # Logging This guide describes how to update logging preferences in Curio. ## Log file configuration Each Curio node generates Go logs which are directed to `/var/log/curio/curio.log` file by default if you are running Curio as a systemd service. ### Redirect Go logs to a file By default, Curio redirect all logs to the standard output if not running as a systemd service. To change this behaviour, add the following variable to the `.bashrc` file and restart the `curio` process to start redirecting all logs to the file. ```shell export GOLOG_OUTPUT=FILE >> ~/.bashrc export GOLOG_FILE="$HOME/curio.log" >> ~/.bashrc && source ~/.bashrc ``` ### Redirect Rust logs to a standard output By default the `fil_logger` library used by `rust-fil-proof` doesn’t log anything. You can change this by setting the RUST\_LOG environment variable to another level. This will show log output on stderr which can be redirected to a file either by systemd or in the shell while launching the `curio` process manually. Using systemd service file: ```bash export RUST_LOG=info >> /etc/curio.env systemctl restart curio.service ``` Running Curio manually: ```shell export RUST_LOG=info >> ~/.bashrc && source ~/.bashrc ``` The log-level can be chosen between 5 options: * trace * debug * info * warn * error ### Change logging verbosity The verbosity of the `curio` logs can be changed without restarting the service or process. The following command can be used to list different subsystems within the `curio` process and change the verbosity of individual subsystem to get more/less detailed logs. ```shell curio cli --machine log list ``` To change the verbosity, please run: ```shell curio cli --machine log set-level --system chain debug ``` The log-level can be chosen between 4 options: * debug * info * warn * error You can specify multiple subsystems to change the log level of multiple subsystems at once. ```shell curio cli --machine log set-level --system chain --system chainxchg debug ``` # Curio CLI Curio command-line interface Curio ships with 2 binaries called `curio` and `sptool` by default. ## Storage Provider Address (`SP_ADDRESS`) Most `sptool` commands require a miner actor address and will not run without one. Set `SP_ADDRESS` in your environment so you do not have to pass `--actor` on every command: ```bash export SP_ADDRESS=f01234 ``` You can still override the address for a single command with `--actor`. Without `SP_ADDRESS` or `--actor`, `sptool` refuses to run. ## Curio Binary The command line interface (CLI) for Curio operates slightly differently from typical software. Some commands, such as those related to storage, make API calls in the backend, while others interact directly with the database to perform the required actions. Commands that make API calls require authentication since the Curio API is permissioned. This authentication involves passing a token and address with the API call. The Curio CLI handles this authentication seamlessly, eliminating the need to set environment variables for the token and address. It retrieves the authentication and actor details from the database and generates the API call accordingly. This design allows you to make changes to a remote node without having direct access to it. For example, you can detach storage in Node 2 from Node 1. All such commands are nested under the `cli` subcommand. To perform actions on a remote machine, you must provide the correct IP address and port using the `--machine` flag in the format `--machine=10.0.0.1:12300`. This documentation explains the unique aspects of the Curio CLI, including its authentication process and how to interact with remote nodes. The `curio` CLI references can be found [here](/curio-cli/curio). ## Sptool Binary Certain administrative and monitoring operations require updating or fetching information about the minerID from the chain. These operations do not need access to the database and therefore are not included in the Curio binary. Instead, these commands are hosted under the `sptool` binary. The `sptool` binary provides an interface with the Filecoin blockchain for operations required by a storage provider. `sptool` requires a miner actor on every command. Set [`SP_ADDRESS`](#storage-provider-address-sp_address) (or pass `--actor`) before using it. The `sptool` CLI references can be found [here](/curio-cli/sptool).
# Curio ``` NAME: curio - Filecoin decentralized storage network provider USAGE: curio [global options] command [command options] VERSION: 1.28.4 COMMANDS: cli Execute cli commands run Start a Curio process config Manage node config by layers. The layer 'base' will always be applied at Curio start-up. test Utility functions for testing web, gui Start Curio web interface guided-setup Run the guided setup for migrating from lotus-miner to Curio or Creating a new Curio miner seal Manage the sealing pipeline unseal Manage unsealed data market fetch-params Fetch proving parameters calc Math Utils toolbox Tool Box for Curio batch Manage batch sealing operations help, h Shows a list of commands or help for one command GLOBAL OPTIONS: --color use color in display output (default: depends on output being a TTY) --db-host value Comma-separated list of hostnames for yugabyte cluster (default: "127.0.0.1") [$CURIO_DB_HOST, $CURIO_HARMONYDB_HOSTS] --db-host-cql value Comma-separated list of hostnames for yugabyte cluster (default: <--db-host>) [$CURIO_DB_HOST_CQL] --db-name value Name of the Postgres database in Yugabyte cluster (default: "yugabyte") [$CURIO_DB_NAME, $CURIO_HARMONYDB_NAME] --db-user value Username for connecting to the Postgres database in Yugabyte cluster (default: "yugabyte") [$CURIO_DB_USER, $CURIO_HARMONYDB_USERNAME] --db-password value Password for connecting to the Postgres database in Yugabyte cluster (default: "yugabyte") [$CURIO_DB_PASSWORD, $CURIO_HARMONYDB_PASSWORD] --db-port value Port for connecting to the Postgres database in Yugabyte cluster (default: "5433") [$CURIO_DB_PORT, $CURIO_HARMONYDB_PORT] --db-cassandra-port value Port for connecting to the Cassandra database in Yugabyte cluster (default: 9042) [$CURIO_DB_CASSANDRA_PORT, $CURIO_INDEXDB_PORT] --db-load-balance Enable load balancing for connecting to the Postgres database in Yugabyte cluster (default: true) [$CURIO_DB_LOAD_BALANCE, $CURIO_HARMONYDB_LOAD_BALANCE] --db-readonly Open the database in read-only mode (skip schema upgrades and harmony_machines writes) (default: false) [$CURIO_DB_READONLY] --repo-path value (default: "~/.curio") [$CURIO_REPO_PATH] --vv enables very verbose mode, useful for debugging the CLI (default: false) --help, -h show help --version, -v print the version ``` ## curio cli ``` NAME: curio cli - Execute cli commands USAGE: curio cli [command options] COMMANDS: info Get Curio node info storage manage sector storage log Manage logging wait-api Wait for Curio api to come online stop Stop a running Curio process cordon Cordon a machine, set it to maintenance mode uncordon Uncordon a machine, resume scheduling index-sample Provides a sample of CIDs from an indexed piece help, h Shows a list of commands or help for one command OPTIONS: --machine value machine host:port (curio run --listen address) --help, -h show help ``` ### curio cli info ``` NAME: curio cli info - Get Curio node info USAGE: curio cli info [command options] OPTIONS: --help, -h show help ``` ### curio cli storage ``` NAME: curio cli storage - manage sector storage USAGE: curio cli storage [command options] DESCRIPTION: Sectors can be stored across many filesystem paths. These commands provide ways to manage the storage a Curio node will use to store sectors long term for proving (references as 'store') as well as how sectors will be stored while moving through the sealing pipeline (references as 'seal'). COMMANDS: attach attach local storage path detach detach local storage path list list local storage paths find find sector in the storage system generate-vanilla-proof generate vanilla proof for a sector redeclare redeclare sectors in a local storage path help, h Shows a list of commands or help for one command OPTIONS: --help, -h show help ``` #### curio cli storage attach ``` NAME: curio cli storage attach - attach local storage path USAGE: curio cli storage attach [command options] [path] DESCRIPTION: Storage can be attached to a Curio node using this command. The storage volume list is stored local to the Curio node in storage.json set in curio run. We do not recommend manually modifying this value without further understanding of the storage system. Each storage volume contains a configuration file which describes the capabilities of the volume. When the '--init' flag is provided, this file will be created using the additional flags. Weight A high weight value means data will be more likely to be stored in this path Seal Data for the sealing process will be stored here Store Finalized sectors that will be moved here for long term storage and be proven over time OPTIONS: --init initialize the path first (default: false) --weight value (for init) path weight (default: 10) --seal (for init) use path for sealing (default: false) --store (for init) use path for long-term storage (default: false) --max-storage value (for init) limit storage space for sectors (expensive for very large paths!) --groups value [ --groups value ] path group names --allow-to value [ --allow-to value ] path groups allowed to pull data from this path (allow all if not specified) --allow-types value [ --allow-types value ] file types to allow storing in this path --deny-types value [ --deny-types value ] file types to deny storing in this path --allow-miners value [ --allow-miners value ] miners to allow storing in this path --deny-miners value [ --deny-miners value ] miners to deny storing in this path --help, -h show help ``` #### curio cli storage detach ``` NAME: curio cli storage detach - detach local storage path USAGE: curio cli storage detach [command options] [path] OPTIONS: --really-do-it (default: false) --help, -h show help ``` #### curio cli storage list ``` NAME: curio cli storage list - list local storage paths USAGE: curio cli storage list [command options] OPTIONS: --local only list local storage paths (default: false) --help, -h show help ``` #### curio cli storage find ``` NAME: curio cli storage find - find sector in the storage system USAGE: curio cli storage find [command options] [miner address] [sector number] OPTIONS: --help, -h show help ``` #### curio cli storage generate-vanilla-proof ``` NAME: curio cli storage generate-vanilla-proof - generate vanilla proof for a sector USAGE: curio cli storage generate-vanilla-proof [command options] [miner address] [sector number] OPTIONS: --help, -h show help ``` #### curio cli storage redeclare ``` NAME: curio cli storage redeclare - redeclare sectors in a local storage path USAGE: curio cli storage redeclare [command options] [id] DESCRIPTION: --machine flag in cli command should point to the node where storage to redeclare is attached OPTIONS: --all redeclare all storage paths (default: false) --drop-missing Drop index entries with missing files (default: true) --help, -h show help ``` ### curio cli log ``` NAME: curio cli log - Manage logging USAGE: curio cli log [command options] COMMANDS: list List log systems set-level Set log level help, h Shows a list of commands or help for one command OPTIONS: --help, -h show help ``` #### curio cli log list ``` NAME: curio cli log list - List log systems USAGE: curio cli log list [command options] OPTIONS: --help, -h show help ``` #### curio cli log set-level ``` NAME: curio cli log set-level - Set log level USAGE: curio cli log set-level [command options] [level] DESCRIPTION: Set the log level for logging systems: The system flag can be specified multiple times. eg) log set-level --system chain --system chainxchg debug Available Levels: debug info warn error Environment Variables: GOLOG_LOG_LEVEL - Default log level for all log systems GOLOG_LOG_FMT - Change output log format (json, nocolor) GOLOG_FILE - Write logs to file GOLOG_OUTPUT - Specify whether to output to file, stderr, stdout or a combination, i.e. file+stderr OPTIONS: --system value [ --system value ] limit to log system --help, -h show help ``` ### curio cli wait-api ``` NAME: curio cli wait-api - Wait for Curio api to come online USAGE: curio cli wait-api [command options] OPTIONS: --timeout value duration to wait till fail (default: 30s) --help, -h show help ``` ### curio cli stop ``` NAME: curio cli stop - Stop a running Curio process USAGE: curio cli stop [command options] OPTIONS: --help, -h show help ``` ### curio cli cordon ``` NAME: curio cli cordon - Cordon a machine, set it to maintenance mode USAGE: curio cli cordon [command options] OPTIONS: --help, -h show help ``` ### curio cli uncordon ``` NAME: curio cli uncordon - Uncordon a machine, resume scheduling USAGE: curio cli uncordon [command options] OPTIONS: --help, -h show help ``` ### curio cli index-sample ``` NAME: curio cli index-sample - Provides a sample of CIDs from an indexed piece USAGE: curio cli index-sample [command options] piece-cid OPTIONS: --json output in json format (default: false) --help, -h show help ``` ## curio run ``` NAME: curio run - Start a Curio process USAGE: curio run [command options] OPTIONS: --listen value host address and port the worker api will listen on (default: "0.0.0.0:12300") [$CURIO_LISTEN] --nosync don't check full-node sync status (default: false) --manage-fdlimit manage open file limit (default: true) --layers value, -l value, --layer value [ --layers value, -l value, --layer value ] list of layers to be interpreted (atop defaults). Default: base [$CURIO_LAYERS] --name value custom node name [$CURIO_NODE_NAME] --help, -h show help ``` ## curio config ``` NAME: curio config - Manage node config by layers. The layer 'base' will always be applied at Curio start-up. USAGE: curio config [command options] COMMANDS: default, defaults Print default node config set, add, update, create Set a config layer or the base by providing a filename or stdin. get, cat, show Get a config layer by name. You may want to pipe the output to a file, or use 'less' list, ls List config layers present in the DB. interpret, view, stacked, stack Interpret stacked config layers by this version of curio, with system-generated comments. remove, rm, del, delete Remove a named config layer. edit edit a config layer new-cluster Create new configuration for a new cluster help, h Shows a list of commands or help for one command OPTIONS: --help, -h show help ``` ### curio config default ``` NAME: curio config default - Print default node config USAGE: curio config default [command options] OPTIONS: --no-comment don't comment default values (default: false) --help, -h show help ``` ### curio config set ``` NAME: curio config set - Set a config layer or the base by providing a filename or stdin. USAGE: curio config set [command options] a layer's file name OPTIONS: --title value title of the config layer (req'd for stdin) --help, -h show help ``` ### curio config get ``` NAME: curio config get - Get a config layer by name. You may want to pipe the output to a file, or use 'less' USAGE: curio config get [command options] layer name OPTIONS: --help, -h show help ``` ### curio config list ``` NAME: curio config list - List config layers present in the DB. USAGE: curio config list [command options] OPTIONS: --help, -h show help ``` ### curio config interpret ``` NAME: curio config interpret - Interpret stacked config layers by this version of curio, with system-generated comments. USAGE: curio config interpret [command options] a list of layers to be interpreted as the final config OPTIONS: --layers value [ --layers value ] comma or space separated list of layers to be interpreted (base is always applied) --help, -h show help ``` ### curio config remove ``` NAME: curio config remove - Remove a named config layer. USAGE: curio config remove [command options] OPTIONS: --help, -h show help ``` ### curio config edit ``` NAME: curio config edit - edit a config layer USAGE: curio config edit [command options] [layer name] OPTIONS: --editor value editor to use (default: "vim") [$EDITOR] --source value source config layer (default: ) --allow-overwrite allow overwrite of existing layer if source is a different layer (default: false) --no-source-diff save the whole config into the layer, not just the diff (default: false) --no-interpret-source do not interpret source layer (default: true if --source is set) --help, -h show help ``` ### curio config new-cluster ``` NAME: curio config new-cluster - Create new configuration for a new cluster USAGE: curio config new-cluster [command options] [SP actor address...] OPTIONS: --help, -h show help ``` ## curio test ``` NAME: curio test - Utility functions for testing USAGE: curio test [command options] COMMANDS: window-post, wd, windowpost, wdpost Compute a proof-of-spacetime for a sector (requires the sector to be pre-sealed). These will not send to the chain. debug Collection of debugging utilities supra Supra consensus testing utilities help, h Shows a list of commands or help for one command OPTIONS: --help, -h show help ``` ### curio test window-post ``` NAME: curio test window-post - Compute a proof-of-spacetime for a sector (requires the sector to be pre-sealed). These will not send to the chain. USAGE: curio test window-post [command options] COMMANDS: here, cli Compute WindowPoSt for performance and configuration testing. task, scheduled, schedule, async, asynchronous Test the windowpost scheduler by running it on the next available curio. If tasks fail all retries, you will need to ctrl+c to exit. vanilla Compute WindowPoSt vanilla proofs and verify them. help, h Shows a list of commands or help for one command OPTIONS: --help, -h show help ``` #### curio test window-post here ``` NAME: curio test window-post here - Compute WindowPoSt for performance and configuration testing. USAGE: curio test window-post here [command options] [deadline index] DESCRIPTION: Note: This command is intended to be used to verify PoSt compute performance. It will not send any messages to the chain. Since it can compute any deadline, output may be incorrectly timed for the chain. OPTIONS: --deadline value deadline to compute WindowPoSt for (default: 0) --layers value [ --layers value ] list of layers to be interpreted (atop defaults). Default: base --partition value partition to compute WindowPoSt for (default: 0) --addr value SP ID to compute WindowPoSt for --help, -h show help ``` #### curio test window-post task ``` NAME: curio test window-post task - Test the windowpost scheduler by running it on the next available curio. If tasks fail all retries, you will need to ctrl+c to exit. USAGE: curio test window-post task [command options] OPTIONS: --deadline value deadline to compute WindowPoSt for (default: 0) --layers value [ --layers value ] list of layers to be interpreted (atop defaults). Default: base --addr value SP ID to compute WindowPoSt for --help, -h show help ``` #### curio test window-post vanilla ``` NAME: curio test window-post vanilla - Compute WindowPoSt vanilla proofs and verify them. USAGE: curio test window-post vanilla [command options] OPTIONS: --deadline value deadline to compute WindowPoSt for (default: 0) --layers value [ --layers value ] list of layers to be interpreted (atop defaults). Default: base --partition value partition to compute WindowPoSt for (default: 0) --addr value SP ID to compute WindowPoSt for --help, -h show help ``` ### curio test debug ``` NAME: curio test debug - Collection of debugging utilities USAGE: curio test debug [command options] COMMANDS: ipni-piece-chunks generate ipni chunks from a file debug-snsvc proofsvc-client Interact with the remote proof service help, h Shows a list of commands or help for one command OPTIONS: --help, -h show help ``` #### curio test debug ipni-piece-chunks ``` NAME: curio test debug ipni-piece-chunks - generate ipni chunks from a file USAGE: curio test debug ipni-piece-chunks [command options] OPTIONS: --help, -h show help ``` #### curio test debug debug-snsvc ``` NAME: curio test debug debug-snsvc USAGE: curio test debug debug-snsvc [command options] COMMANDS: deposit Deposit FIL into the Router contract (client) client-initiate-withdrawal Initiate a withdrawal request from the client's deposit client-complete-withdrawal Complete a pending client withdrawal after the withdrawal window elapses client-cancel-withdrawal Cancel a pending client withdrawal request redeem-client Redeem a client voucher (service role) redeem-provider Redeem a provider voucher (provider role) service-initiate-withdrawal Initiate a withdrawal request from the service pool service-complete-withdrawal Complete a pending service withdrawal after the withdrawal window elapses service-cancel-withdrawal Cancel a pending service withdrawal request service-deposit Deposit funds into the service pool (service role) get-client-state Query the state of a client get-provider-state Query the state of a provider get-service-state Query the service state create-client-voucher Create a client voucher create-provider-voucher Create a provider voucher propose-service-actor Propose a new service actor accept-service-actor Accept a proposed service actor validate-client-voucher Validate a client voucher signature validate-provider-voucher Validate a provider voucher signature help, h Shows a list of commands or help for one command OPTIONS: --help, -h show help ``` **curio test debug debug-snsvc deposit** ``` NAME: curio test debug debug-snsvc deposit - Deposit FIL into the Router contract (client) USAGE: curio test debug debug-snsvc deposit [command options] OPTIONS: --from value Sender address --amount value Amount in FIL --help, -h show help ``` **curio test debug debug-snsvc client-initiate-withdrawal** ``` NAME: curio test debug debug-snsvc client-initiate-withdrawal - Initiate a withdrawal request from the client's deposit USAGE: curio test debug debug-snsvc client-initiate-withdrawal [command options] OPTIONS: --from value Client sender address --amount value Withdrawal amount (in FIL) --help, -h show help ``` **curio test debug debug-snsvc client-complete-withdrawal** ``` NAME: curio test debug debug-snsvc client-complete-withdrawal - Complete a pending client withdrawal after the withdrawal window elapses USAGE: curio test debug debug-snsvc client-complete-withdrawal [command options] OPTIONS: --from value Client sender address --help, -h show help ``` **curio test debug debug-snsvc client-cancel-withdrawal** ``` NAME: curio test debug debug-snsvc client-cancel-withdrawal - Cancel a pending client withdrawal request USAGE: curio test debug debug-snsvc client-cancel-withdrawal [command options] OPTIONS: --from value Client sender address --help, -h show help ``` **curio test debug debug-snsvc redeem-client** ``` NAME: curio test debug debug-snsvc redeem-client - Redeem a client voucher (service role) USAGE: curio test debug debug-snsvc redeem-client [command options] OPTIONS: --from value Service sender address --client value Client actor --amount value Cumulative amount (FIL) --nonce value Voucher nonce (default: 0) --sig value Voucher signature (hex) --help, -h show help ``` **curio test debug debug-snsvc redeem-provider** ``` NAME: curio test debug debug-snsvc redeem-provider - Redeem a provider voucher (provider role) USAGE: curio test debug debug-snsvc redeem-provider [command options] OPTIONS: --from value Provider sender address --provider value Provider actor --amount value Cumulative amount (FIL) --nonce value Voucher nonce (default: 0) --sig value Voucher signature (hex) --help, -h show help ``` **curio test debug debug-snsvc service-initiate-withdrawal** ``` NAME: curio test debug debug-snsvc service-initiate-withdrawal - Initiate a withdrawal request from the service pool USAGE: curio test debug debug-snsvc service-initiate-withdrawal [command options] OPTIONS: --amount value Withdrawal amount (in FIL) --from value Service sender address --help, -h show help ``` **curio test debug debug-snsvc service-complete-withdrawal** ``` NAME: curio test debug debug-snsvc service-complete-withdrawal - Complete a pending service withdrawal after the withdrawal window elapses USAGE: curio test debug debug-snsvc service-complete-withdrawal [command options] OPTIONS: --from value Service sender address --help, -h show help ``` **curio test debug debug-snsvc service-cancel-withdrawal** ``` NAME: curio test debug debug-snsvc service-cancel-withdrawal - Cancel a pending service withdrawal request USAGE: curio test debug debug-snsvc service-cancel-withdrawal [command options] OPTIONS: --from value Service sender address --help, -h show help ``` **curio test debug debug-snsvc service-deposit** ``` NAME: curio test debug debug-snsvc service-deposit - Deposit funds into the service pool (service role) USAGE: curio test debug debug-snsvc service-deposit [command options] OPTIONS: --from value Service sender address --amount value Amount to deposit (FIL) --help, -h show help ``` **curio test debug debug-snsvc get-client-state** ``` NAME: curio test debug debug-snsvc get-client-state - Query the state of a client USAGE: curio test debug debug-snsvc get-client-state [command options] OPTIONS: --client value Client actor address --help, -h show help ``` **curio test debug debug-snsvc get-provider-state** ``` NAME: curio test debug debug-snsvc get-provider-state - Query the state of a provider USAGE: curio test debug debug-snsvc get-provider-state [command options] OPTIONS: --provider value Provider actor address --help, -h show help ``` **curio test debug debug-snsvc get-service-state** ``` NAME: curio test debug debug-snsvc get-service-state - Query the service state USAGE: curio test debug debug-snsvc get-service-state [command options] OPTIONS: --help, -h show help ``` **curio test debug debug-snsvc create-client-voucher** ``` NAME: curio test debug debug-snsvc create-client-voucher - Create a client voucher USAGE: curio test debug debug-snsvc create-client-voucher [command options] OPTIONS: --client value Client actor address --amount value Amount to redeem (FIL) --help, -h show help ``` **curio test debug debug-snsvc create-provider-voucher** ``` NAME: curio test debug debug-snsvc create-provider-voucher - Create a provider voucher USAGE: curio test debug debug-snsvc create-provider-voucher [command options] OPTIONS: --provider value Provider actor address --amount value Amount to redeem (FIL) --nonce value Voucher nonce (default: 0) --service value Service sender address --help, -h show help ``` **curio test debug debug-snsvc propose-service-actor** ``` NAME: curio test debug debug-snsvc propose-service-actor - Propose a new service actor USAGE: curio test debug debug-snsvc propose-service-actor [command options] OPTIONS: --from value Service sender address --new-service-actor value New service actor address --help, -h show help ``` **curio test debug debug-snsvc accept-service-actor** ``` NAME: curio test debug debug-snsvc accept-service-actor - Accept a proposed service actor USAGE: curio test debug debug-snsvc accept-service-actor [command options] OPTIONS: --from value Service sender address --help, -h show help ``` **curio test debug debug-snsvc validate-client-voucher** ``` NAME: curio test debug debug-snsvc validate-client-voucher - Validate a client voucher signature USAGE: curio test debug debug-snsvc validate-client-voucher [command options] OPTIONS: --client value Client actor address --amount value Cumulative amount (FIL) --nonce value Voucher nonce (default: 0) --sig value Voucher signature (hex) --help, -h show help ``` **curio test debug debug-snsvc validate-provider-voucher** ``` NAME: curio test debug debug-snsvc validate-provider-voucher - Validate a provider voucher signature USAGE: curio test debug debug-snsvc validate-provider-voucher [command options] OPTIONS: --provider value Provider actor address --amount value Cumulative amount (FIL) --nonce value Voucher nonce (default: 0) --sig value Voucher signature (hex) --help, -h show help ``` #### curio test debug proofsvc-client ``` NAME: curio test debug proofsvc-client - Interact with the remote proof service USAGE: curio test debug proofsvc-client [command options] COMMANDS: create-voucher Create a client voucher submit Submit a proof request status Check proof status help, h Shows a list of commands or help for one command OPTIONS: --help, -h show help ``` **curio test debug proofsvc-client create-voucher** ``` NAME: curio test debug proofsvc-client create-voucher - Create a client voucher USAGE: curio test debug proofsvc-client create-voucher [command options] OPTIONS: --client value --amount value --help, -h show help ``` **curio test debug proofsvc-client submit** ``` NAME: curio test debug proofsvc-client submit - Submit a proof request USAGE: curio test debug proofsvc-client submit [command options] OPTIONS: --c1 value path to lotus-bench c1 json --miner value miner address --client-id value (default: 0) --nonce value (default: 0) --amount value --sig value --help, -h show help ``` **curio test debug proofsvc-client status** ``` NAME: curio test debug proofsvc-client status - Check proof status USAGE: curio test debug proofsvc-client status [command options] OPTIONS: --id value --help, -h show help ``` ### curio test supra ``` NAME: curio test supra - Supra consensus testing utilities USAGE: curio test supra [command options] COMMANDS: system-info Display CPU and CUDA information relevant for supraseal tree-r-file Test tree-r-file snap-encode Test snap-encode help, h Shows a list of commands or help for one command OPTIONS: --help, -h show help ``` #### curio test supra system-info ``` NAME: curio test supra system-info - Display CPU and CUDA information relevant for supraseal USAGE: curio test supra system-info [command options] OPTIONS: --help, -h show help ``` #### curio test supra tree-r-file ``` NAME: curio test supra tree-r-file - Test tree-r-file USAGE: curio test supra tree-r-file [command options] OPTIONS: --last-layer-filename value Last layer filename --data-filename value Data filename --output-dir value Output directory --sector-size value Sector size (default: 0) --help, -h show help ``` #### curio test supra snap-encode ``` NAME: curio test supra snap-encode - Test snap-encode USAGE: curio test supra snap-encode [command options] OPTIONS: --sealed-filename value Sealed filename --unsealed-filename value Unsealed filename --update-filename value Update filename --sector-size value Sector size (bytes). Supported: 2048, 8388608, 549755813888, 34359738368, 68719476736 (default: 0) --commd value Unsealed CommD CID (v1) --commk value SectorKey CommR (commK) CID (v1) --membuffer Use memory buffer instead of disk (load and store) (default: false) --help, -h show help ``` ## curio web ``` NAME: curio web - Start Curio web interface USAGE: curio web [command options] DESCRIPTION: Start an instance of Curio web interface. This creates the 'web' layer if it does not exist, then calls run with that layer. In --db-readonly / CURIO_DB_READONLY mode, no config layer is written; the GUI is enabled in-memory. OPTIONS: --gui-listen value Address to listen for the GUI on (default: "0.0.0.0:4701") --nosync don't check full-node sync status (default: false) --layers value [ --layers value ] list of layers to be interpreted (atop defaults). Default: base --help, -h show help ``` ## curio guided-setup ``` NAME: curio guided-setup - Run the guided setup for migrating from lotus-miner to Curio or Creating a new Curio miner USAGE: curio guided-setup [command options] OPTIONS: --help, -h show help ``` ## curio seal ``` NAME: curio seal - Manage the sealing pipeline USAGE: curio seal [command options] COMMANDS: start Start new sealing operations manually events List pipeline events help, h Shows a list of commands or help for one command OPTIONS: --help, -h show help ``` ### curio seal start ``` NAME: curio seal start - Start new sealing operations manually USAGE: curio seal start [command options] OPTIONS: --actor value Specify actor address to start sealing sectors for --now Start sealing sectors for all actors now (not on schedule) (default: false) --cc Start sealing new CC sectors (default: false) --count value Number of sectors to start (default: 1) --synthetic Use synthetic PoRep (default: false) --layers value [ --layers value ] list of layers to be interpreted (atop defaults). Default: base --duration-days value, -d value How long to commit sectors for (default: 1278 (3.5 years)) --help, -h show help ``` ### curio seal events ``` NAME: curio seal events - List pipeline events USAGE: curio seal events [command options] OPTIONS: --actor value Filter events by actor address; lists all if not specified --sector value Filter events by sector number; requires --actor to be specified (default: 0) --last value Limit output to the last N events (default: 100) --help, -h show help ``` ## curio unseal ``` NAME: curio unseal - Manage unsealed data USAGE: curio unseal [command options] COMMANDS: info Get information about unsealed data list-sectors List data from the sectors_unseal_pipeline and sectors_meta tables set-target-state Set the target unseal state for a sector set-target-by-pieces Set the target unseal state for sectors containing the given piece CIDs check Check data integrity in unsealed sector files help, h Shows a list of commands or help for one command OPTIONS: --help, -h show help ``` ### curio unseal info ``` NAME: curio unseal info - Get information about unsealed data USAGE: curio unseal info [command options] [minerAddress] [sectorNumber] OPTIONS: --help, -h show help ``` ### curio unseal list-sectors ``` NAME: curio unseal list-sectors - List data from the sectors_unseal_pipeline and sectors_meta tables USAGE: curio unseal list-sectors [command options] OPTIONS: --sp-id value, -s value Filter by storage provider ID (default: 0) --output value, -o value Output file path (default: stdout) --help, -h show help ``` ### curio unseal set-target-state ``` NAME: curio unseal set-target-state - Set the target unseal state for a sector USAGE: curio unseal set-target-state [command options] DESCRIPTION: Set the target unseal state for a specific sector. : The storage provider ID : The sector number : The target state (true, false) The unseal target state indicates to curio how an unsealed copy of the sector should be maintained. If the target state is true, curio will ensure that the sector is unsealed. If the target state is false, curio will ensure that there is no unsealed copy of the sector. If the target state is none, curio will not change the current state of the sector. Currently when the curio will only start new unseal processes when the target state changes from another state to true. When the target state is false, and an unsealed sector file exists, the GC mark step will create a removal mark for the unsealed sector file. The file will only be removed after the removal mark is accepted. OPTIONS: --help, -h show help ``` ### curio unseal set-target-by-pieces ``` NAME: curio unseal set-target-by-pieces - Set the target unseal state for sectors containing the given piece CIDs USAGE: curio unseal set-target-by-pieces [command options] [piece-cid ...] DESCRIPTION: Resolve each piece CID to sector(s) via market_piece_deal, then set target_unseal_state for those sectors. Accepts piece CID v1 or v2. Use --target-state to specify the desired state (true or false). Request all at once to minimize the sectors needed to be unsealed (if you have pieces stored in multiple sectors). Use --stdin to read piece CIDs one per line from stdin (for large lists). OPTIONS: --target-state Target state: true (ensure unsealed), false (ensure no unsealed copy) (default: true) --stdin Read piece CIDs one per line from stdin instead of from arguments (default: false) --help, -h show help ``` ### curio unseal check ``` NAME: curio unseal check - Check data integrity in unsealed sector files USAGE: curio unseal check [command options] DESCRIPTION: Create a check task for a specific sector, wait for its completion, and output the result. : The storage provider ID : The sector number OPTIONS: --help, -h show help ``` ## curio market ``` NAME: curio market USAGE: curio market [command options] COMMANDS: seal start sealing a deal sector early add-url Add URL to fetch data for offline deals move-to-escrow Moves funds from the deal collateral wallet into escrow with the storage market actor ddo Create a new offline verified DDO deal for Curio help, h Shows a list of commands or help for one command OPTIONS: --help, -h show help ``` ### curio market seal ``` NAME: curio market seal - start sealing a deal sector early USAGE: curio market seal [command options] OPTIONS: --actor value Specify actor address to start sealing sectors for --synthetic Use synthetic PoRep (default: false) --help, -h show help ``` ### curio market add-url ``` NAME: curio market add-url - Add URL to fetch data for offline deals USAGE: curio market add-url [command options] OPTIONS: --file value CSV file location to use for multiple deal input. Each line in the file should be in the format 'uuid,raw size,url,header1,header2...' --header HEADER, -H HEADER [ --header HEADER, -H HEADER ] Custom HEADER to include in the HTTP request --url URL, -u URL URL to send the request to --help, -h show help ``` ### curio market move-to-escrow ``` NAME: curio market move-to-escrow - Moves funds from the deal collateral wallet into escrow with the storage market actor USAGE: curio market move-to-escrow [command options] OPTIONS: --actor value Specify actor address to start sealing sectors for --max-fee value maximum fee in FIL user is willing to pay for this message (default: "0.5") --wallet value Specify wallet address to send the funds from --help, -h show help ``` ### curio market ddo ``` NAME: curio market ddo - Create a new offline verified DDO deal for Curio USAGE: curio market ddo [command options] OPTIONS: --actor value Specify actor address for the deal --remove-unsealed-copy Remove unsealed copies of sector containing this deal (default: false) --skip-ipni-announce indicates that deal index should not be announced to the IPNI (default: false) --start-epoch value start epoch by when the deal should be proved by provider on-chain (default: 2 days from now) (default: 0) --help, -h show help ``` ## curio fetch-params ``` NAME: curio fetch-params - Fetch proving parameters USAGE: curio fetch-params [command options] [sectorSize] OPTIONS: --help, -h show help ``` ## curio calc ``` NAME: curio calc - Math Utils USAGE: curio calc [command options] COMMANDS: batch-cpu Analyze and display the layout of batch sealer threads supraseal-config Generate a supra_seal configuration help, h Shows a list of commands or help for one command OPTIONS: --actor value --help, -h show help ``` ### curio calc batch-cpu ``` NAME: curio calc batch-cpu - Analyze and display the layout of batch sealer threads USAGE: curio calc batch-cpu [command options] DESCRIPTION: Analyze and display the layout of batch sealer threads on your CPU. It provides detailed information about CPU utilization for batch sealing operations, including core allocation, thread distribution for different batch sizes. OPTIONS: --dual-hashers (default: true) --help, -h show help ``` ### curio calc supraseal-config ``` NAME: curio calc supraseal-config - Generate a supra_seal configuration USAGE: curio calc supraseal-config [command options] DESCRIPTION: Generate a supra_seal configuration for a given batch size. This command outputs a configuration expected by SupraSeal. Main purpose of this command is for debugging and testing. The config can be used directly with SupraSeal binaries to test it without involving Curio. OPTIONS: --dual-hashers Zen3 and later supports two sectors per thread, set to false for older CPUs (default: true) --batch-size value, -b value (default: 0) --help, -h show help ``` ## curio toolbox ``` NAME: curio toolbox - Tool Box for Curio USAGE: curio toolbox [command options] COMMANDS: fix-msg Updated DB with message data missing from chain node downgrade Downgrade a cluster's database to a previous software version. fix-boost-migration Fix Boost migration help, h Shows a list of commands or help for one command OPTIONS: --help, -h show help ``` ### curio toolbox fix-msg ``` NAME: curio toolbox fix-msg - Updated DB with message data missing from chain node USAGE: curio toolbox fix-msg [command options] OPTIONS: --all Update data for messages in wait queue (default: false) --help, -h show help ``` ### curio toolbox downgrade ``` NAME: curio toolbox downgrade - Downgrade a cluster's database to a previous software version. USAGE: curio toolbox downgrade [command options] DESCRIPTION: If, however, the upgrade has a serious bug and you need to downgrade, first shutdown all nodes in your cluster and then run this command. Finally, only start downgraded nodes. OPTIONS: --last_good_date value YYYYMMDD when your cluster had the preferred schema. Ex: 20251128 (default: 0) --help, -h show help ``` ### curio toolbox fix-boost-migration ``` NAME: curio toolbox fix-boost-migration - Fix Boost migration USAGE: curio toolbox fix-boost-migration [command options] OPTIONS: --check check how many entries need to be fixed (default: false) --db-file value location of boost.db file --boostd-data-hosts value [ --boostd-data-hosts value ] yugabyte hosts to connect to over cassandra interface eg '127.0.0.1' --boostd-data-username value yugabyte username to connect to over cassandra interface eg 'cassandra' --boostd-data-password value yugabyte password to connect to over cassandra interface eg 'cassandra' --help, -h show help ``` ## curio batch ``` NAME: curio batch - Manage batch sealing operations USAGE: curio batch [command options] COMMANDS: setup Setup SPDK for batch sealing (configures hugepages and binds NVMe devices) help, h Shows a list of commands or help for one command OPTIONS: --help, -h show help ``` ### curio batch setup ``` NAME: curio batch setup - Setup SPDK for batch sealing (configures hugepages and binds NVMe devices) USAGE: curio batch setup [command options] DESCRIPTION: Setup SPDK for batch sealing operations. This command automatically: - Downloads SPDK if not already available - Configures 1GB hugepages (36 pages minimum) - Binds NVMe devices for use with SupraSeal Requires root/sudo access for SPDK setup operations. OPTIONS: --hugepages value Number of 1GB hugepages to configure (default: 36) --min-pages value Minimum number of hugepages required (default: 36) --help, -h show help ``` # Sptool ``` NAME: sptool - Manage Filecoin Miner Actor USAGE: sptool [global options] command [command options] VERSION: 1.28.4 COMMANDS: actor Manage Filecoin Miner Actor Metadata info Print miner actor info sectors interact with sector store proving View proving information toolbox some tools to fix some problems help, h Shows a list of commands or help for one command GLOBAL OPTIONS: --log-level value (default: "info") --actor value miner actor to manage [$SP_ADDRESS] --verbose, --vv enable verbose logging (default: false) --help, -h show help --version, -v print the version ``` ## sptool actor ``` NAME: sptool actor - Manage Filecoin Miner Actor Metadata USAGE: sptool actor [command options] COMMANDS: set-addresses, set-addrs set addresses that your miner can be publicly dialed on withdraw withdraw available balance to beneficiary repay-debt pay down a miner's debt set-peer-id set the peer id of your miner set-owner Set owner address (this command should be invoked twice, first with the old owner as the senderAddress, and then with the new owner) control Manage control addresses propose-change-worker Propose a worker address change confirm-change-worker Confirm a worker address change compact-allocated compact allocated sectors bitfield propose-change-beneficiary Propose a beneficiary address change confirm-change-beneficiary Confirm a beneficiary address change new-miner Initializes a new miner actor help, h Shows a list of commands or help for one command OPTIONS: --help, -h show help ``` ### sptool actor set-addresses ``` NAME: sptool actor set-addresses - set addresses that your miner can be publicly dialed on USAGE: sptool actor set-addresses [command options] OPTIONS: --from value optionally specify the account to send the message from --gas-limit value set gas limit (default: 0) --unset unset address (default: false) --help, -h show help ``` ### sptool actor withdraw ``` NAME: sptool actor withdraw - withdraw available balance to beneficiary USAGE: sptool actor withdraw [command options] [amount (FIL)] OPTIONS: --confidence value number of block confirmations to wait for (default: 5) --beneficiary send withdraw message from the beneficiary address (default: false) --help, -h show help ``` ### sptool actor repay-debt ``` NAME: sptool actor repay-debt - pay down a miner's debt USAGE: sptool actor repay-debt [command options] [amount (FIL)] OPTIONS: --from value optionally specify the account to send funds from --help, -h show help ``` ### sptool actor set-peer-id ``` NAME: sptool actor set-peer-id - set the peer id of your miner USAGE: sptool actor set-peer-id [command options] OPTIONS: --gas-limit value set gas limit (default: 0) --help, -h show help ``` ### sptool actor set-owner ``` NAME: sptool actor set-owner - Set owner address (this command should be invoked twice, first with the old owner as the senderAddress, and then with the new owner) USAGE: sptool actor set-owner [command options] [newOwnerAddress senderAddress] OPTIONS: --really-do-it Actually send transaction performing the action (default: false) --help, -h show help ``` ### sptool actor control ``` NAME: sptool actor control - Manage control addresses USAGE: sptool actor control [command options] COMMANDS: list Get currently set control addresses. Note: This excludes most roles as they are not known to the immediate chain state. set Set control address(-es) help, h Shows a list of commands or help for one command OPTIONS: --help, -h show help ``` #### sptool actor control list ``` NAME: sptool actor control list - Get currently set control addresses. Note: This excludes most roles as they are not known to the immediate chain state. USAGE: sptool actor control list [command options] OPTIONS: --verbose (default: false) --help, -h show help ``` #### sptool actor control set ``` NAME: sptool actor control set - Set control address(-es) USAGE: sptool actor control set [command options] [...address] OPTIONS: --really-do-it Actually send transaction performing the action (default: false) --help, -h show help ``` ### sptool actor propose-change-worker ``` NAME: sptool actor propose-change-worker - Propose a worker address change USAGE: sptool actor propose-change-worker [command options] [address] OPTIONS: --really-do-it Actually send transaction performing the action (default: false) --help, -h show help ``` ### sptool actor confirm-change-worker ``` NAME: sptool actor confirm-change-worker - Confirm a worker address change USAGE: sptool actor confirm-change-worker [command options] [address] OPTIONS: --really-do-it Actually send transaction performing the action (default: false) --help, -h show help ``` ### sptool actor compact-allocated ``` NAME: sptool actor compact-allocated - compact allocated sectors bitfield USAGE: sptool actor compact-allocated [command options] OPTIONS: --mask-last-offset value Mask sector IDs from 0 to 'highest_allocated - offset' (default: 0) --mask-upto-n value Mask sector IDs from 0 to 'n' (default: 0) --really-do-it Actually send transaction performing the action (default: false) --help, -h show help ``` ### sptool actor propose-change-beneficiary ``` NAME: sptool actor propose-change-beneficiary - Propose a beneficiary address change USAGE: sptool actor propose-change-beneficiary [command options] [beneficiaryAddress quota expiration] OPTIONS: --really-do-it Actually send transaction performing the action (default: false) --overwrite-pending-change Overwrite the current beneficiary change proposal (default: false) --actor value specify the address of miner actor --help, -h show help ``` ### sptool actor confirm-change-beneficiary ``` NAME: sptool actor confirm-change-beneficiary - Confirm a beneficiary address change USAGE: sptool actor confirm-change-beneficiary [command options] [minerID] OPTIONS: --really-do-it Actually send transaction performing the action (default: false) --existing-beneficiary send confirmation from the existing beneficiary address (default: false) --new-beneficiary send confirmation from the new beneficiary address (default: false) --help, -h show help ``` ### sptool actor new-miner ``` NAME: sptool actor new-miner - Initializes a new miner actor USAGE: sptool actor new-miner [command options] OPTIONS: --worker value, -w value worker key to use for new miner initialisation --owner value, -o value owner key to use for new miner initialisation --from value, -f value address to send actor(miner) creation message from --sector-size value specify sector size to use for new miner initialisation --confidence value number of block confirmations to wait for (default: 5) --help, -h show help ``` ## sptool info ``` NAME: sptool info - Print miner actor info USAGE: sptool info [command options] OPTIONS: --help, -h show help ``` ## sptool sectors ``` NAME: sptool sectors - interact with sector store USAGE: sptool sectors [command options] COMMANDS: status Get the on-chain status of a sector by its number list List sectors precommits Print on-chain precommit info check-expire Inspect expiring sectors expired Get or cleanup expired sectors extend Extend expiring sectors while not exceeding each sector's max life terminate Forcefully terminate a sector (WARNING: This means losing power and pay a one-time termination penalty(including collateral) for the terminated sector) compact-partitions removes dead sectors from partitions and reduces the number of partitions used if possible help, h Shows a list of commands or help for one command OPTIONS: --help, -h show help ``` ### sptool sectors status ``` NAME: sptool sectors status - Get the on-chain status of a sector by its number USAGE: sptool sectors status [command options] OPTIONS: --help, -h show help ``` ### sptool sectors list ``` NAME: sptool sectors list - List sectors USAGE: sptool sectors list [command options] OPTIONS: --help, -h show help ``` ### sptool sectors precommits ``` NAME: sptool sectors precommits - Print on-chain precommit info USAGE: sptool sectors precommits [command options] OPTIONS: --help, -h show help ``` ### sptool sectors check-expire ``` NAME: sptool sectors check-expire - Inspect expiring sectors USAGE: sptool sectors check-expire [command options] OPTIONS: --cutoff value skip sectors whose current expiration is more than epochs from now, defaults to 60 days (default: 172800) --help, -h show help ``` ### sptool sectors expired ``` NAME: sptool sectors expired - Get or cleanup expired sectors USAGE: sptool sectors expired [command options] OPTIONS: --expired-epoch value epoch at which to check sector expirations (default: WinningPoSt lookback epoch) --help, -h show help ``` ### sptool sectors extend ``` NAME: sptool sectors extend - Extend expiring sectors while not exceeding each sector's max life USAGE: sptool sectors extend [command options] DESCRIPTION: NOTE: --new-expiration, --from and --to flags have multiple formats: 1. Absolute epoch number: 2. Relative epoch number: +, e.g. +1000, means 1000 epochs from now 3. Relative day number: +d, e.g. +10d, means 10 days from now The --extension flag has two formats: 1. Number of epochs to extend by: 2. Number of days to extend by: d Extensions will be clamped at either the maximum sector extension of 3.5 years/1278 days or the sector's maximum lifetime which currently is 5 years. OPTIONS: --from value only consider sectors whose current expiration epoch is in the range of [from, to], defaults to: now + 120 (1 hour) (default: "+120") --to value only consider sectors whose current expiration epoch is in the range of [from, to], defaults to: now + 92160 (32 days) (default: "+92160") --sector-file value provide a file containing one sector number in each line, ignoring above selecting criteria --exclude value optionally provide a file containing excluding sectors --extension value try to extend selected sectors by this number of epochs, defaults to 540 days (default: "540d") --new-expiration value try to extend selected sectors to this epoch, ignoring extension --only-cc only extend CC sectors (useful for making sector ready for snap upgrade) (default: false) --no-cc don't extend CC sectors (exclusive with --only-cc) (default: false) --drop-claims drop claims for sectors that can be extended, but only by dropping some of their verified power claims (default: false) --tolerance value don't try to extend sectors by fewer than this number of epochs, defaults to 7 days (default: 20160) --max-fee value use up to this amount of FIL for one message. pass this flag to avoid message congestion. (default: "0") --max-sectors value the maximum number of sectors contained in each message (default: 500) --really-do-it pass this flag to really extend sectors, otherwise will only print out json representation of parameters (default: false) --help, -h show help ``` ### sptool sectors terminate ``` NAME: sptool sectors terminate - Forcefully terminate a sector (WARNING: This means losing power and pay a one-time termination penalty(including collateral) for the terminated sector) USAGE: sptool sectors terminate [command options] [sectorNum1 sectorNum2 ...] OPTIONS: --actor value specify the address of miner actor --really-do-it pass this flag if you know what you are doing (default: false) --from value specify the address to send the terminate message from --help, -h show help ``` ### sptool sectors compact-partitions ``` NAME: sptool sectors compact-partitions - removes dead sectors from partitions and reduces the number of partitions used if possible USAGE: sptool sectors compact-partitions [command options] OPTIONS: --deadline value the deadline to compact the partitions in (default: 0) --partitions value [ --partitions value ] list of partitions to compact sectors in --really-do-it Actually send transaction performing the action (default: false) --help, -h show help ``` ## sptool proving ``` NAME: sptool proving - View proving information USAGE: sptool proving [command options] COMMANDS: info View current state information deadlines View the current proving period deadlines information deadline View the current proving period deadline information by its index faults View the currently known proving faulty sectors information help, h Shows a list of commands or help for one command OPTIONS: --help, -h show help ``` ### sptool proving info ``` NAME: sptool proving info - View current state information USAGE: sptool proving info [command options] OPTIONS: --help, -h show help ``` ### sptool proving deadlines ``` NAME: sptool proving deadlines - View the current proving period deadlines information USAGE: sptool proving deadlines [command options] OPTIONS: --all, -a Count all sectors (only live sectors are counted by default) (default: false) --help, -h show help ``` ### sptool proving deadline ``` NAME: sptool proving deadline - View the current proving period deadline information by its index USAGE: sptool proving deadline [command options] OPTIONS: --sector-nums, -n Print sector/fault numbers belonging to this deadline (default: false) --bitfield, -b Print partition bitfield stats (default: false) --help, -h show help ``` ### sptool proving faults ``` NAME: sptool proving faults - View the currently known proving faulty sectors information USAGE: sptool proving faults [command options] OPTIONS: --help, -h show help ``` ## sptool toolbox ``` NAME: sptool toolbox - some tools to fix some problems USAGE: sptool toolbox [command options] COMMANDS: spark Manage Smart Contract PeerID used by Spark mk12-client mk12 client for Curio mk20-client mk20 client for Curio stats Curio Node Stats help, h Shows a list of commands or help for one command OPTIONS: --help, -h show help ``` ### sptool toolbox spark ``` NAME: sptool toolbox spark - Manage Smart Contract PeerID used by Spark USAGE: sptool toolbox spark [command options] COMMANDS: delete-peer Delete PeerID from Spark Smart Contract help, h Shows a list of commands or help for one command OPTIONS: --help, -h show help ``` #### sptool toolbox spark delete-peer ``` NAME: sptool toolbox spark delete-peer - Delete PeerID from Spark Smart Contract USAGE: sptool toolbox spark delete-peer [command options] OPTIONS: --really-do-it Send the message to the smart contract (default: false) --help, -h show help ``` ### sptool toolbox mk12-client ``` NAME: sptool toolbox mk12-client - mk12 client for Curio USAGE: sptool toolbox mk12-client [command options] COMMANDS: init Initialise curio mk12 client repo deal Make an online deal with Curio deal-status offline-deal Make an offline deal with Curio allocate Create new allocation[s] for verified deals list-allocations Lists all allocations for a client address(wallet) market-add Add funds to the Storage Market actor market-withdraw Withdraw funds from the Storage Market actor commp generate-rand-car creates a randomly generated dense car wallet Manage mk12 client wallets help, h Shows a list of commands or help for one command OPTIONS: --mk12-client-repo value repo directory for mk12 client (default: "~/.curio-client") [$CURIO_MK12_CLIENT_REPO] --help, -h show help ``` #### sptool toolbox mk12-client init ``` NAME: sptool toolbox mk12-client init - Initialise curio mk12 client repo USAGE: sptool toolbox mk12-client init [command options] OPTIONS: --help, -h show help ``` #### sptool toolbox mk12-client deal ``` NAME: sptool toolbox mk12-client deal - Make an online deal with Curio USAGE: sptool toolbox mk12-client deal [command options] OPTIONS: --http-url value http url to CAR file --http-headers value [ --http-headers value ] http headers to be passed with the request (e.g key=value) --car-size value size of the CAR file: required for online deals (default: 0) --provider value storage provider on-chain address --commp value commp of the CAR file --piece-size value size of the CAR file as a padded piece (default: 0) --payload-cid value root CID of the CAR file --start-epoch-head-offset value start epoch by when the deal should be proved by provider on-chain after current chain head (default: 0) --start-epoch value start epoch by when the deal should be proved by provider on-chain (default: 0) --duration value duration of the deal in epochs (default: 518400) --provider-collateral value deal collateral that storage miner must put in escrow; if empty, the min collateral for the given piece size will be used (default: 0) --storage-price value storage price in attoFIL per epoch per GiB (default: 1) --verified whether the deal funds should come from verified client data-cap (default: false) --remove-unsealed-copy indicates that an unsealed copy of the sector in not required for fast retrieval (default: false) --wallet value wallet address to be used to initiate the deal --skip-ipni-announce indicates that deal index should not be announced to the IPNI(Network Indexer) (default: false) --http make the deal over HTTP instead of libp2p (default: false) --help, -h show help ``` #### sptool toolbox mk12-client deal-status ``` NAME: sptool toolbox mk12-client deal-status USAGE: sptool toolbox mk12-client deal-status [command options] OPTIONS: --provider value storage provider on-chain address --deal-uuid value --wallet value the wallet address that was used to sign the deal proposal --http make the deal over HTTP instead of libp2p (default: false) --help, -h show help ``` #### sptool toolbox mk12-client offline-deal ``` NAME: sptool toolbox mk12-client offline-deal - Make an offline deal with Curio USAGE: sptool toolbox mk12-client offline-deal [command options] OPTIONS: --provider value storage provider on-chain address --commp value commp of the CAR file --piece-size value size of the CAR file as a padded piece (default: 0) --payload-cid value root CID of the CAR file --start-epoch-head-offset value start epoch by when the deal should be proved by provider on-chain after current chain head (default: 0) --start-epoch value start epoch by when the deal should be proved by provider on-chain (default: 0) --duration value duration of the deal in epochs (default: 518400) --provider-collateral value deal collateral that storage miner must put in escrow; if empty, the min collateral for the given piece size will be used (default: 0) --storage-price value storage price in attoFIL per epoch per GiB (default: 1) --verified whether the deal funds should come from verified client data-cap (default: false) --remove-unsealed-copy indicates that an unsealed copy of the sector in not required for fast retrieval (default: false) --wallet value wallet address to be used to initiate the deal --skip-ipni-announce indicates that deal index should not be announced to the IPNI(Network Indexer) (default: false) --http make the deal over HTTP instead of libp2p (default: false) --help, -h show help ``` #### sptool toolbox mk12-client allocate ``` NAME: sptool toolbox mk12-client allocate - Create new allocation[s] for verified deals USAGE: sptool toolbox mk12-client allocate [command options] DESCRIPTION: The command can accept a CSV formatted file in the format 'pieceCid,pieceSize,miner,tmin,tmax,expiration' OPTIONS: --miner value, -m value, --provider value, -p value [ --miner value, -m value, --provider value, -p value ] storage provider address[es] --piece-cid value, --piece value data piece-cid to create the allocation --piece-size value, --size value piece size to create the allocation (default: 0) --wallet value the wallet address that will used create the allocation --quiet do not print the allocation list (default: false) --term-min value, --tmin value The minimum duration which the provider must commit to storing the piece to avoid early-termination penalties (epochs). Default is 180 days. (default: 518400) --term-max value, --tmax value The maximum period for which a provider can earn quality-adjusted power for the piece (epochs). Default is 5 years. (default: 5256000) --expiration value The latest epoch by which a provider must commit data before the allocation expires (epochs). Default is 60 days. (default: 172800) --piece-file value, --pf value file containing piece information to create the allocation. Each line in the file should be in the format 'pieceCid,pieceSize,miner,tmin,tmax,expiration' --batch-size value number of extend requests per batch. If set incorrectly, this will lead to out of gas error (default: 500) --confidence value number of block confirmations to wait for (default: 5) --assume-yes, -y, --yes automatic yes to prompts; assume 'yes' as answer to all prompts and run non-interactively (default: false) --evm-client-contract value f4 address of EVM contract to spend DataCap from --json, -j print output in JSON format (default: false) --help, -h show help ``` #### sptool toolbox mk12-client list-allocations ``` NAME: sptool toolbox mk12-client list-allocations - Lists all allocations for a client address(wallet) USAGE: sptool toolbox mk12-client list-allocations [command options] OPTIONS: --miner value, -m value, --provider value, -p value Storage provider address. If provided, only allocations against this minerID will be printed --wallet value the wallet address that will used create the allocation --json, -j print output in JSON format (default: false) --help, -h show help ``` #### sptool toolbox mk12-client market-add ``` NAME: sptool toolbox mk12-client market-add - Add funds to the Storage Market actor USAGE: sptool toolbox mk12-client market-add [command options] DESCRIPTION: Send signed message to add funds for the default wallet to the Storage Market actor. Uses 2x current BaseFee and a maximum fee of 1 nFIL. This is an experimental utility, do not use in production. OPTIONS: --assume-yes, -y, --yes automatic yes to prompts; assume 'yes' as answer to all prompts and run non-interactively (default: false) --wallet value move balance from this wallet address to its market actor --help, -h show help ``` #### sptool toolbox mk12-client market-withdraw ``` NAME: sptool toolbox mk12-client market-withdraw - Withdraw funds from the Storage Market actor USAGE: sptool toolbox mk12-client market-withdraw [command options] OPTIONS: --assume-yes, -y, --yes automatic yes to prompts; assume 'yes' as answer to all prompts and run non-interactively (default: false) --wallet value move balance to this wallet address from its market actor --help, -h show help ``` #### sptool toolbox mk12-client commp ``` NAME: sptool toolbox mk12-client commp USAGE: sptool toolbox mk12-client commp [command options] OPTIONS: --help, -h show help ``` #### sptool toolbox mk12-client generate-rand-car ``` NAME: sptool toolbox mk12-client generate-rand-car - creates a randomly generated dense car USAGE: sptool toolbox mk12-client generate-rand-car [command options] OPTIONS: --size value, -s value The size of the data to turn into a car (default: 8000000) --chunksize value, -c value Size of chunking that should occur (default: 512) --maxlinks value, -l value Max number of leaves per level (default: 8) --help, -h show help ``` #### sptool toolbox mk12-client wallet ``` NAME: sptool toolbox mk12-client wallet - Manage mk12 client wallets USAGE: sptool toolbox mk12-client wallet [command options] COMMANDS: new Generate a new key of the given type list List wallet address balance Get account balance export export keys import import keys default, get-default Get default wallet address set-default Set default wallet address delete Delete an account from the wallet sign Sign a message help, h Shows a list of commands or help for one command OPTIONS: --help, -h show help ``` **sptool toolbox mk12-client wallet new** ``` NAME: sptool toolbox mk12-client wallet new - Generate a new key of the given type USAGE: sptool toolbox mk12-client wallet new [command options] [bls|secp256k1|delegated (default secp256k1)] OPTIONS: --help, -h show help ``` **sptool toolbox mk12-client wallet list** ``` NAME: sptool toolbox mk12-client wallet list - List wallet address USAGE: sptool toolbox mk12-client wallet list [command options] OPTIONS: --addr-only, -a Only print addresses (default: false) --id, -i Output ID addresses (default: false) --help, -h show help ``` **sptool toolbox mk12-client wallet balance** ``` NAME: sptool toolbox mk12-client wallet balance - Get account balance USAGE: sptool toolbox mk12-client wallet balance [command options] [address] OPTIONS: --help, -h show help ``` **sptool toolbox mk12-client wallet export** ``` NAME: sptool toolbox mk12-client wallet export - export keys USAGE: sptool toolbox mk12-client wallet export [command options] [address] OPTIONS: --help, -h show help ``` **sptool toolbox mk12-client wallet import** ``` NAME: sptool toolbox mk12-client wallet import - import keys USAGE: sptool toolbox mk12-client wallet import [command options] [ (optional, will read from stdin if omitted)] OPTIONS: --format value specify input format for key (default: "hex-lotus") --as-default import the given key as your new default key (default: false) --help, -h show help ``` **sptool toolbox mk12-client wallet default** ``` NAME: sptool toolbox mk12-client wallet default - Get default wallet address USAGE: sptool toolbox mk12-client wallet default [command options] OPTIONS: --help, -h show help ``` **sptool toolbox mk12-client wallet set-default** ``` NAME: sptool toolbox mk12-client wallet set-default - Set default wallet address USAGE: sptool toolbox mk12-client wallet set-default [command options] [address] OPTIONS: --help, -h show help ``` **sptool toolbox mk12-client wallet delete** ``` NAME: sptool toolbox mk12-client wallet delete - Delete an account from the wallet USAGE: sptool toolbox mk12-client wallet delete [command options]
OPTIONS: --help, -h show help ``` **sptool toolbox mk12-client wallet sign** ``` NAME: sptool toolbox mk12-client wallet sign - Sign a message USAGE: sptool toolbox mk12-client wallet sign [command options] OPTIONS: --help, -h show help ``` ### sptool toolbox mk20-client ``` NAME: sptool toolbox mk20-client - mk20 client for Curio USAGE: sptool toolbox mk20-client [command options] COMMANDS: init Initialise curio mk12 client repo commp deal Make a mk20 deal with Curio pdp-deal Make a mk20 PDP deal with Curio aggregate Create a new aggregate from a list of CAR files upload Upload a file to the storage provider chunk-upload Upload a file in chunks to the storage provider deal-status Get status of a Mk20 deal help, h Shows a list of commands or help for one command OPTIONS: --mk12-client-repo value repo directory for mk12 client (default: "~/.curio-client") [$CURIO_MK12_CLIENT_REPO] --help, -h show help ``` #### sptool toolbox mk20-client init ``` NAME: sptool toolbox mk20-client init - Initialise curio mk12 client repo USAGE: sptool toolbox mk20-client init [command options] OPTIONS: --help, -h show help ``` #### sptool toolbox mk20-client commp ``` NAME: sptool toolbox mk20-client commp USAGE: sptool toolbox mk20-client commp [command options] OPTIONS: --help, -h show help ``` #### sptool toolbox mk20-client deal ``` NAME: sptool toolbox mk20-client deal - Make a mk20 deal with Curio USAGE: sptool toolbox mk20-client deal [command options] OPTIONS: --http-url value http url to CAR file --http-headers value [ --http-headers value ] http headers to be passed with the request (e.g key=value) --provider value storage provider on-chain address --pcidv2 value pcidv2 of the CAR file --duration value duration of the deal in epochs (default: 518400) --market-address value market contract address of the deal --market-deal-id value market deal ID (default: 0) --allocation value allocation id of the deal (default: 0) --indexing indicates that an deal should be indexed (default: true) --wallet value wallet address to be used to initiate the deal --announce indicates that deal should be announced to the IPNI(Network Indexer) (default: true) --aggregate value aggregate file path for the deal --put used HTTP put as data source (default: false) --help, -h show help ``` #### sptool toolbox mk20-client pdp-deal ``` NAME: sptool toolbox mk20-client pdp-deal - Make a mk20 PDP deal with Curio USAGE: sptool toolbox mk20-client pdp-deal [command options] OPTIONS: --http-url value http url to CAR file --http-headers value [ --http-headers value ] http headers to be passed with the request (e.g key=value) --provider value PDP providers's URL --pcidv2 value pcidv2 of the CAR file --wallet value wallet address to be used to initiate the deal --aggregate value aggregate file path for the deal --put used HTTP put as data source (default: false) --add-piece add piece (default: false) --add-dataset add dataset (default: false) --remove-piece remove piece (default: false) --remove-dataset remove dataset (default: false) --record-keeper value record keeper address --piece-id value [ --piece-id value ] root IDs --dataset-id value dataset IDs (default: 0) --help, -h show help ``` #### sptool toolbox mk20-client aggregate ``` NAME: sptool toolbox mk20-client aggregate - Create a new aggregate from a list of CAR files USAGE: sptool toolbox mk20-client aggregate [command options] OPTIONS: --files value [ --files value ] list of CAR files to aggregate --piece-size value piece size of the aggregate (default: 0) --out output the aggregate file (default: true) --help, -h show help ``` #### sptool toolbox mk20-client upload ``` NAME: sptool toolbox mk20-client upload - Upload a file to the storage provider USAGE: sptool toolbox mk20-client upload [command options] OPTIONS: --provider value PDP providers's URL --deal value deal id to upload to --help, -h show help ``` #### sptool toolbox mk20-client chunk-upload ``` NAME: sptool toolbox mk20-client chunk-upload - Upload a file in chunks to the storage provider USAGE: sptool toolbox mk20-client chunk-upload [command options] OPTIONS: --provider value storage provider on-chain address --deal value deal id to upload to --chunk-size value chunk size to be used for the upload (default: "4 MiB") --wallet value wallet address to be used to initiate the deal --help, -h show help ``` #### sptool toolbox mk20-client deal-status ``` NAME: sptool toolbox mk20-client deal-status - Get status of a Mk20 deal USAGE: sptool toolbox mk20-client deal-status [command options] OPTIONS: --provider value PDP providers's URL --id value deal id --wallet value wallet address to be used to initiate the deal --help, -h show help ``` ### sptool toolbox stats ``` NAME: sptool toolbox stats - Curio Node Stats USAGE: sptool toolbox stats [command options] OPTIONS: --help, -h show help ``` # Developer Developer-oriented references for Curio build, API, and tooling behavior. This section contains technical references intended for Curio developers and advanced operators. * [API](/developer/api) * [Build and Make Variables](/developer/make-variables) # API Curio API references ## Groups * [Allocate](#Allocate) * [AllocatePieceToSector](#AllocatePieceToSector) * [DefaultGroup](#DefaultGroup) * [Cordon](#Cordon) * [Info](#Info) * [Shutdown](#Shutdown) * [Uncordon](#Uncordon) * [Version](#Version) * [Index](#Index) * [IndexSamples](#IndexSamples) * [Log](#Log) * [LogList](#LogList) * [LogSetLevel](#LogSetLevel) * [Storage](#Storage) * [StorageAddLocal](#StorageAddLocal) * [StorageDetachLocal](#StorageDetachLocal) * [StorageFindSector](#StorageFindSector) * [StorageGenerateVanillaProof](#StorageGenerateVanillaProof) * [StorageInfo](#StorageInfo) * [StorageInit](#StorageInit) * [StorageList](#StorageList) * [StorageLocal](#StorageLocal) * [StorageRedeclare](#StorageRedeclare) * [StorageStat](#StorageStat) ### Allocate #### AllocatePieceToSector There are not yet any comments for this method. Perms: write Inputs: ```json [ "f01234", { "PublishCid": { "/": "bafy2bzacea3wsdh6y3a36tb3skempjoxqpuyompjbmfeyf34fi3uy6uue42v4" }, "DealID": 5432, "DealProposal": { "PieceCID": { "/": "bafy2bzacea3wsdh6y3a36tb3skempjoxqpuyompjbmfeyf34fi3uy6uue42v4" }, "PieceSize": 1032, "VerifiedDeal": true, "Client": "f01234", "Provider": "f01234", "Label": "", "StartEpoch": 10101, "EndEpoch": 10101, "StoragePricePerEpoch": "0", "ProviderCollateral": "0", "ClientCollateral": "0" }, "DealSchedule": { "StartEpoch": 10101, "EndEpoch": 10101 }, "PieceActivationManifest": { "CID": { "/": "bafy2bzacea3wsdh6y3a36tb3skempjoxqpuyompjbmfeyf34fi3uy6uue42v4" }, "Size": 1032, "VerifiedAllocationKey": { "Client": 1000, "ID": 0 }, "Notify": [ { "Address": "f01234", "Payload": "Ynl0ZSBhcnJheQ==" } ] }, "KeepUnsealed": true }, 9, { "Scheme": "string value", "Opaque": "string value", "User": {}, "Host": "string value", "Path": "string value", "Fragment": "string value", "RawQuery": "string value", "RawPath": "string value", "RawFragment": "string value", "ForceQuery": true, "OmitHost": true }, { "Authorization": [ "Bearer ey.." ] } ] ``` Response: ```json { "Sector": 9, "Offset": 1032 } ``` ### DefaultGroup #### Cordon Perms: admin Inputs: `null` Response: `{}` #### Info Perms: read Inputs: `null` Response: ```json { "ID": 123, "CPU": 123, "RAM": 9, "GPU": 1, "HostPort": "string value", "LastContact": "0001-01-01T00:00:00Z", "Unschedulable": true, "Name": { "String": "string value", "Valid": true }, "StartupTime": { "Time": "0001-01-01T00:00:00Z", "Valid": true }, "Tasks": { "String": "string value", "Valid": true }, "Layers": { "String": "string value", "Valid": true }, "Miners": { "String": "string value", "Valid": true } } ``` #### Shutdown Perms: admin Inputs: `null` Response: `{}` #### Uncordon Perms: admin Inputs: `null` Response: `{}` #### Version There are not yet any comments for this method. Perms: admin Inputs: `null` Response: ```json [ 123 ] ``` ### Index #### IndexSamples Perms: admin Inputs: ```json [ { "/": "bafy2bzacea3wsdh6y3a36tb3skempjoxqpuyompjbmfeyf34fi3uy6uue42v4" } ] ``` Response: ```json [ "Bw==" ] ``` ### Log The log method group has logging methods #### LogList There are not yet any comments for this method. Perms: read Inputs: `null` Response: ```json [ "string value" ] ``` #### LogSetLevel Perms: admin Inputs: ```json [ "string value", "string value" ] ``` Response: `{}` ### Storage The storage method group contains are storage administration method #### StorageAddLocal Perms: admin Inputs: ```json [ "string value" ] ``` Response: `{}` #### StorageDetachLocal Perms: admin Inputs: ```json [ "string value" ] ``` Response: `{}` #### StorageFindSector Perms: admin Inputs: ```json [ { "Miner": 1000, "Number": 9 }, 1, 34359738368, true ] ``` Response: ```json [ { "ID": "76f1988b-ef30-4d7e-b3ec-9a627f4ba5a8", "URLs": [ "string value" ], "BaseURLs": [ "string value" ], "Weight": 42, "CanSeal": true, "CanStore": true, "Primary": true, "AllowTypes": [ "string value" ], "DenyTypes": [ "string value" ], "AllowMiners": [ "string value" ], "DenyMiners": [ "string value" ] } ] ``` #### StorageGenerateVanillaProof Perms: admin Inputs: ```json [ "f01234", 9 ] ``` Response: `"Ynl0ZSBhcnJheQ=="` #### StorageInfo Perms: admin Inputs: ```json [ "76f1988b-ef30-4d7e-b3ec-9a627f4ba5a8" ] ``` Response: ```json { "ID": "76f1988b-ef30-4d7e-b3ec-9a627f4ba5a8", "URLs": [ "string value" ], "Weight": 42, "MaxStorage": 42, "CanSeal": true, "CanStore": true, "Groups": [ "string value" ], "AllowTo": [ "string value" ], "AllowTypes": [ "string value" ], "DenyTypes": [ "string value" ], "AllowMiners": [ "string value" ], "DenyMiners": [ "string value" ] } ``` #### StorageInit There are not yet any comments for this method. Perms: admin Inputs: ```json [ "string value", { "ID": "76f1988b-ef30-4d7e-b3ec-9a627f4ba5a8", "Weight": 42, "CanSeal": true, "CanStore": true, "MaxStorage": 42, "Groups": [ "string value" ], "AllowTo": [ "string value" ], "AllowTypes": [ "string value" ], "DenyTypes": [ "string value" ], "AllowMiners": [ "string value" ], "DenyMiners": [ "string value" ] } ] ``` Response: `{}` #### StorageList Perms: admin Inputs: `null` Response: ```json { "76f1988b-ef30-4d7e-b3ec-9a627f4ba5a8": [ { "Miner": 1000, "Number": 100, "SectorFileType": 2 } ] } ``` #### StorageLocal Perms: admin Inputs: `null` Response: ```json { "76f1988b-ef30-4d7e-b3ec-9a627f4ba5a8": "/data/path" } ``` #### StorageRedeclare Perms: admin Inputs: ```json [ "string value", true ] ``` Response: `{}` #### StorageStat Perms: admin Inputs: ```json [ "76f1988b-ef30-4d7e-b3ec-9a627f4ba5a8" ] ``` Response: ```json { "Capacity": 9, "Available": 9, "FSAvailable": 9, "Reserved": 9, "Max": 9, "Used": 9 } ``` # Build and Make Variables This project uses a modular Makefile layout: * `Makefile`: entrypoint; includes all `scripts/makefiles/*.mk` fragments. * `scripts/makefiles/00-vars.mk`: shared defaults and derived variables. * `scripts/makefiles/05-help.mk`: `make help` and quick usage guidance. * `scripts/makefiles/10-deps.mk`: dependency bootstrap (`ffi`, `blst`, `supraseal`, submodules). * `scripts/makefiles/20-test.mk`: test and coverage targets. * `scripts/makefiles/30-build.mk`: binary build/install/cleanup targets. * `scripts/makefiles/40-gen.mk`: code/doc generation targets. * `scripts/makefiles/50-docker.mk`: docker/devnet targets. * `scripts/makefiles/60-abi.mk`: ABI to Go generation pattern rule. The refactor preserves behavior but makes execution paths easier to reason about. ## How overrides work You can override variables at invocation time: ```bash make build FFI_USE_OPENCL=1 make gen GOCC=gotip GOCACHE_CLEAN=1 make docker/devnet build_lotus=1 lotus_version=v1.35.0 ``` You can also export them from the environment: ```bash export GOFLAGS='-mod=mod' export FFI_USE_OPENCL=1 make build ``` Command-line values take precedence over file defaults. ## Build target matrix These targets share the same pipeline and differ only in tags: * `make build` * Builds: `curio`, `sptool` * Tag behavior: uses `CURIO_TAGS` as-is. * Default tags: `CURIO_TAGS_BASE` (`cunative`) plus conditional extras. * `make calibnet` * Builds: `curio`, `sptool` (mainnet-calibration network) * Tag behavior: appends `calibnet` to `CURIO_TAGS` then runs `build`. * `make debug` * Builds: `curio`, `sptool` * Tag behavior: appends `debug` to `CURIO_TAGS` then runs `build`. * `make 2k` * Builds: `curio`, `sptool` (dev-net / 2k) * Tag behavior: appends `2k` to `CURIO_TAGS` then runs `build`. * `make curio-pdp` (synonym: `make skiff`) * Builds: `curio` (PDP-only daemon from `./cmd/skiff`; no worker RPC or deal market) * Uses `SKIFF_TAGS` (`CURIO_TAGS` + `nosupraseal` + `skiff`) and skips filecoin-ffi `BUILD_DEPS`. * Overwrites the `curio` binary produced by `make curio`. Equivalent one-off forms are also available: * `make calibnet-curio` * `make calibnet-sptool` * `make calibnet-curio-pdp` (synonym: `make calibnet-skiff`) * `make cu2k` * `make 2k-curio-pdp` (synonym: `make 2k-skiff`) ## Important default context * On Linux, the default `make build` path does **not** add the `nosupraseal` tag. That means `lib/supraffi` Linux implementations (`//go:build linux && !nosupraseal`) are compiled in. * On CUDA-capable Linux hosts, Curio can use the SupraSeal-backed fast TreeR path for snap encode. If prerequisites are not met, Curio falls back to filecoin-ffi TreeR logic at runtime. * See `lib/ffiselect/ffidirect/ffi-direct.go` (`TreeRFile`) for the fast-path + fallback decision. * Even if you do not plan to run SupraSeal batch sealing, keeping default (non-`nosupraseal`) builds is still useful for snap encode performance paths. * On Linux CUDA builds, filecoin-ffi now defaults `FFI_USE_CUDA_SUPRASEAL=1` (FFI-internal feature selection only). ## `FFI_USE_OPENCL` and `FFI_USE_CUDA_SUPRASEAL` behavior by OS and CUDA availability `FFI_USE_OPENCL` controls Curio-side build routing: * filecoin-ffi backend selection inputs (`FFI_USE_CUDA`/OpenCL direction), * whether linux supraseal dependency build is included, * whether `nosupraseal` is auto-added to `CURIO_TAGS`. `FFI_USE_CUDA_SUPRASEAL` controls only filecoin-ffi's internal GPU feature choice: * it does not affect Curio `nosupraseal` tagging, * it does not affect whether linux `build/.supraseal-install` is included. Defaults and guardrails: * `FFI_USE_CUDA` is derived as `1` unless `FFI_USE_OPENCL=1`. * `FFI_USE_CUDA_SUPRASEAL` defaults to `1` only when `UNAME_S=Linux` and `FFI_USE_CUDA=1`; otherwise default `0`. * effective value is forced to `0` whenever `FFI_USE_CUDA=0`. ### Behavior matrix | OS | `nvcc` in `PATH` | `FFI_USE_OPENCL` value | default `FFI_USE_CUDA_SUPRASEAL` | Result | | ----- | ---------------- | ---------------------- | -------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | Linux | Yes | unset/empty | `1` | Build succeeds. `FFI_USE_CUDA=1`, FFI defaults to `cuda-supraseal`, linux supraseal dep included unless `DISABLE_SUPRASEAL=1`, tags default to `cunative` (or `cunative nosupraseal` when `DISABLE_SUPRASEAL=1`). | | Linux | No | unset/empty | `1` | Build fails early in `curio-libfilecoin` with CUDA-required error. | | Linux | Yes/No | `1` | `0` | Build succeeds. `FFI_USE_CUDA=0`, effective `FFI_USE_CUDA_SUPRASEAL=0`, direct filecoin-ffi OpenCL install path (prebuilt eligible), supraseal dep skipped, `nosupraseal` auto-added to tags, `build.IsOpencl=1`. | | Linux | Yes | `0` | `1` | Build succeeds. `FFI_USE_CUDA=1` (unambiguous CUDA), FFI defaults to `cuda-supraseal`, supraseal dep included unless `DISABLE_SUPRASEAL=1`. | | Linux | No | `0` | `1` | Build fails (same CUDA-required guard as default). | | macOS | Yes/No | unset/empty | `0` | Build succeeds. Darwin filecoin-ffi install path is used, linux supraseal dep block not applicable, tags default to `cunative` (or `cunative nosupraseal` when `DISABLE_SUPRASEAL=1`). | | macOS | Yes/No | `1` | `0` | Build succeeds. Mainly affects Curio tags/ldflags (`nosupraseal`, `build.IsOpencl=1`). Darwin path does not have Linux CUDA guard. | Notes: * On macOS, presence/absence of CUDA libraries does not drive Curio's Makefile decision flow. * In filecoin-ffi source builds, Darwin already chooses OpenCL feature path by default. * If either `FFI_USE_OPENCL=1` or `DISABLE_SUPRASEAL=1`, linux `build/.supraseal-install` is not added. * `FFI_USE_CUDA_SUPRASEAL` can be overridden (for example `FFI_USE_CUDA_SUPRASEAL=0`) without changing Curio tags/dependency gating. ## Detailed `make build` scenarios This section answers "what exactly is in the binary and what path was taken?" for common cases. ### Quick outcome table | Scenario | Example command | `CURIO_TAGS` used by `curio`/`sptool` | FFI dependency path | FFI GPU feature | `build/.supraseal-install` step | `build.IsOpencl` ldflag in `curio` | | ------------------------------------------------ | ------------------------------------------------- | ------------------------------------- | -------------------------------------------------------------- | -------------------------------------------- | ------------------------------- | ---------------------------------- | | Linux default (CUDA path) | `make build` | `cunative` | Linux `curio-libfilecoin` path, `FFI_USE_CUDA=1` | `cuda-supraseal` (default) | Yes | empty string | | macOS default | `make build` | `cunative` | Darwin direct `make -C extern/filecoin-ffi .install-filcrypto` | Darwin OpenCL-style default in filecoin-ffi | No (linux-only dep block) | empty string | | Linux OpenCL path | `make build FFI_USE_OPENCL=1` | `cunative nosupraseal` | Linux direct `make -C extern/filecoin-ffi .install-filcrypto` | `opencl` | No | `1` | | Linux CUDA but disable supraseal in Curio binary | `make build FFI_USE_OPENCL=0 DISABLE_SUPRASEAL=1` | `cunative nosupraseal` | Linux `curio-libfilecoin` path, `FFI_USE_CUDA=1` | `cuda-supraseal` (default unless overridden) | No | `0` | ### 1. Linux + `make build` (default CUDA-style path) Command: ```bash make build ``` Resulting behavior: * `build` invokes `curio` and `sptool`. * `BUILD_DEPS` includes: * `setup-cgo-env` * `build/.filecoin-install` * `ffi-version-check` * `build/.blst-install` * `build/.supraseal-install` (because linux and `FFI_USE_OPENCL != 1`) * `build/.filecoin-install` uses `curio-libfilecoin` (linux path), which: * requires `nvcc` on this CUDA path * passes `FFI_USE_CUDA=1` when `FFI_USE_OPENCL` is unset/empty * defaults `FFI_USE_CUDA_SUPRASEAL=1` on Linux CUDA path * tags default to `CURIO_TAGS="cunative"` (no `nosupraseal`) * `curio` compile specifics: * `GOAMD64=v3` is set in the build command * `CGO_LDFLAGS_ALLOW='.*'` is set on Linux for `curio` * `-X github.com/filecoin-project/curio/build.IsOpencl=` (empty) * `sptool` builds with the same tag set (`cunative`) * `sptool` uses the default `CGO_LDFLAGS_ALLOW` pattern (not Linux `curio` override) ### 2. macOS + `make build` Command: ```bash make build ``` Resulting behavior: * `BUILD_DEPS` still runs setup/version/blst checks. * `build/.filecoin-install` uses Darwin-specific path: * `make -C extern/filecoin-ffi .install-filcrypto` * it does **not** use `curio-libfilecoin` runtime `nvcc` gate. * linux-only supraseal dependency block is not added. * `build/.blst-install` still runs (`bash scripts/build-blst.sh`) because BLST is always in `BUILD_DEPS`. * tags are still `CURIO_TAGS="cunative"` by default. * `curio` ldflag `build.IsOpencl` remains empty unless you set `FFI_USE_OPENCL`. * even without `nosupraseal` tag, non-linux builds compile non-linux supraffi stubs by file-level build constraints (`!linux`). ### 3. Linux + OpenCL GPU path Command: ```bash make build FFI_USE_OPENCL=1 ``` Resulting behavior: * `FFI_USE_OPENCL=1` changes both deps and tags: * linux supraseal dependency block is skipped * `CURIO_TAGS_EXTRA` adds `nosupraseal` * final tags become `CURIO_TAGS="cunative nosupraseal"` * `build/.filecoin-install` uses direct filecoin-ffi install path on Linux OpenCL: * `make -C extern/filecoin-ffi .install-filcrypto` * prebuilt libfilcrypto is eligible (unless `FFI_BUILD_FROM_SOURCE=1` is set externally) * `curio` ldflag embeds `build.IsOpencl=1`. ### 4. Linux + CUDA, but you do not want supraseal in the Curio binary If your goal is "CUDA filecoin-ffi path, but compile binary without supraseal code", use: ```bash make build FFI_USE_OPENCL=0 DISABLE_SUPRASEAL=1 ``` What this does: * `FFI_USE_OPENCL=0` keeps CUDA mode explicit (`FFI_USE_CUDA=1`). * `DISABLE_SUPRASEAL=1` adds `nosupraseal`, so supraffi linux implementations are excluded at compile time. * linux supraseal dependency build step is skipped. * FFI still defaults to `FFI_USE_CUDA_SUPRASEAL=1` unless you override it. ## What `nosupraseal` means `nosupraseal` is a Go build tag that selects non-supraseal code paths in `lib/supraffi`. Relevant files: * supraseal-enabled linux files: * `lib/supraffi/seal.go` * `lib/supraffi/cuda_linux.go` * `lib/supraffi/seal_nvme.go` * `lib/supraffi/spdk_setup.go` * fallback/stub files selected by `!linux || nosupraseal`: * `lib/supraffi/seal_nonlinux.go` * `lib/supraffi/cuda_nonlinux.go` Practical effect: * with `nosupraseal`, linux supraseal integration code is not linked into the binary. * calls into supraffi functionality will use stub behavior (errors/panics for unavailable operations). * on non-linux, those stub files are selected even without explicitly adding `nosupraseal`. ## Common `make build` use cases * Linux host without CUDA toolkit: ```bash make build FFI_USE_OPENCL=1 ``` * Build calibnet binary set with OpenCL path: ```bash make calibnet FFI_USE_OPENCL=1 ``` * Build debug variant with explicit Go flags: ```bash make debug GOFLAGS='-mod=mod -trimpath' ``` * Linux CUDA path, but disable filecoin-ffi `cuda-supraseal` feature only: ```bash make build FFI_USE_OPENCL=0 FFI_USE_CUDA_SUPRASEAL=0 ``` * Build with fully explicit tags (overrides computed tag composition): ```bash make build CURIO_TAGS='cunative debug calibnet' ``` * Build host-optimized native curio at a fixed ISA level: ```bash make curio-native GOAMD64_NATIVE=v2 ``` ## Variables you are expected to override ### Build and toolchain * `GOCC` (default: `go`) * Purpose: Go command used for `build/run/generate`. * Override when: testing with a non-default Go binary (`gotip`, custom wrapper). * Example: `make build GOCC=/usr/local/go/bin/go` * `GOFLAGS` (default: empty) * Purpose: standard Go flags propagated into `go build`/`go run`/`go generate`. * Override when: changing module behavior, enabling race, changing trimpath, etc. * Example: `make build GOFLAGS='-mod=mod -trimpath'` * `FFI_USE_OPENCL` (default: unset) * Purpose: controls GPU backend assumptions for FFI-related dependency builds. * Expected values: * unset/empty or any value other than `1`: CUDA path (`FFI_USE_CUDA=1`). * `1`: OpenCL path (`FFI_USE_CUDA=0`). * Override when: * Linux host has no CUDA toolkit (`nvcc`) and you want OpenCL. * Running `make gen` is already forced to `FFI_USE_OPENCL=1` by target export. * Examples: * `make deps FFI_USE_OPENCL=1` * `make build FFI_USE_OPENCL=0` * `FFI_USE_CUDA_SUPRASEAL` (default: computed) * Purpose: controls whether filecoin-ffi uses `cuda-supraseal` in CUDA builds. * Default behavior: * Linux with `FFI_USE_CUDA=1`: defaults to `1`. * all other cases: defaults to `0`. * if `FFI_USE_CUDA=0`, effective value is forced to `0`. * Override when: you want CUDA FFI build path but want the non-`cuda-supraseal` FFI feature set. * Example: `make build FFI_USE_OPENCL=0 FFI_USE_CUDA_SUPRASEAL=0` * Important: this variable does not affect Curio `nosupraseal` tags or linux supraseal dependency gating. * `DISABLE_SUPRASEAL` (default: `0`) * Purpose: force `nosupraseal` tag and skip Linux supraseal dependency build. * Expected values: * `0`: default behavior. * `1`: disable supraseal in Curio build/tag flow. * Override when: Linux host has CUDA and you want CUDA-backed FFI build but a Curio binary compiled without supraseal paths. * Example: `make build FFI_USE_OPENCL=0 DISABLE_SUPRASEAL=1` * `CGO_LDFLAGS_ALLOW` (default: quoted value of `CGO_LDFLAGS_ALLOW_PATTERN`) * Purpose: linker flag allowlist for cgo. * Override when: integrating custom toolchains requiring different allow patterns. * Usually do not override directly; prefer default. * `CURIO_TAGS_BASE` (default: `cunative`) * Purpose: baseline build tags for binaries. * Override when: customizing feature profile globally. * Example: `make build CURIO_TAGS_BASE='cunative debug'` * `CURIO_TAGS` (default: computed from `CURIO_TAGS_BASE` plus conditional extras) * Purpose: final tag string passed to most builds. * Override when: you need exact control and do not want computed defaults. * Example: `make curio CURIO_TAGS='cunative calibnet debug'` * `GOAMD64_NATIVE` (default: auto-detected on linux/amd64 via `/proc/cpuinfo`) * Purpose: ISA level for `curio-native` target. * Override when: reproducibility across hosts or intentionally targeting older CPUs. * Example: `make curio-native GOAMD64_NATIVE=v2` ### Generation and caching * `GOCACHE_CLEAN` (default: unset) * Purpose: when set to `1`, `make gensimple` runs `go clean -cache` first. * Override when: generation repeatedly fails due Go build cache contention/corruption. * Example: `make gen GOCACHE_CLEAN=1` ### Testing and coverage * `COVERAGE_DIR` (default: `coverage`) * Purpose: output directory for coverage artifacts. * Override when: CI artifact path conventions differ. * Example: `make cov COVERAGE_DIR=build/coverage` ### Docker/devnet * `build_lotus` (default: `0`) * Purpose: * `0`: use prebuilt lotus image. * `1`: clone/build lotus image locally. * Override when: testing against a custom lotus branch/tag implementation. * `lotus_version` (default: `v1.35.1`) * Purpose: lotus version for prebuilt image tag or local clone branch/tag. * Override when: validating compatibility with another lotus version. * `curio_docker_user` (default: `curio`) * Purpose: image name prefix for locally built images. * Override when: publishing/testing in your own registry namespace. * `curio_base_image` (default: `$(curio_docker_user)/curio-all-in-one:latest-debug`) * Purpose: base image reference passed into docker builds. * Override when: pinning to an alternate base image/tag. * `ffi_from_source` (default: `0`) * Purpose: docker build arg controlling FFI source build behavior in image builds. * Override when: image tests require source-built FFI. * `lotus_base_image` (default when `build_lotus=0`: prebuilt GHCR image) * Purpose: lotus image consumed by curio docker build flow. * Override when: testing against a custom already-built lotus image. * `docker_args` (default: empty) * Purpose: appended raw args to docker build commands. * Override when: injecting platform/buildkit/cache args. * Example: `make docker/curio docker_args='--platform linux/amd64 --no-cache'` ## Variables that are internal/derived (usually do not override) * `FFI_PATH`, `BLST_PATH`, `SUPRA_FFI_PATH` * `BUILD_DEPS`, `MODULES`, `BINS`, `CLEAN` * `UNAME_S`, `NVCC_PATH`, `CUDA_PATH`, `CUDA_LIB_PATH` * `FFI_USE_CUDA`, `FFI_USE_CUDA_SUPRASEAL_EFFECTIVE` * `CURIO_TAGS_EXTRA`, `CURIO_TAGS_CSV` * `TEST_ENV_VARS` * `CGO_LDFLAGS_ALLOW_PATTERN` If you override internal variables, do so only when actively developing the Makefile itself. If you need these overridden as a regular pattern, consider opening a Github issue or talking with support. ## About `make test` Current state in this repository: * `make test` Runs `go test -v -tags="cgo,fvm" -timeout 30m ./itests/...`. It can be used to run integration tests locally. ## Safe override recipes * Linux without CUDA: * `make deps FFI_USE_OPENCL=1` * `make build FFI_USE_OPENCL=1` * Build calibnet binaries with explicit tags: * `make calibnet CURIO_TAGS='cunative calibnet'` * Linux CUDA path but compile Curio without supraseal: * `make build FFI_USE_OPENCL=0 DISABLE_SUPRASEAL=1` * Linux CUDA path but disable only filecoin-ffi `cuda-supraseal`: * `make build FFI_USE_OPENCL=0 FFI_USE_CUDA_SUPRASEAL=0` * Recover from unstable generation cache: * `make gen GOCACHE_CLEAN=1` * Build docker devnet with locally built lotus: * `make docker/devnet build_lotus=1 lotus_version=v1.35.1` # Docker Devnet How to run a local network with Curio using docker ## Prerequisites To ensure a stable and functional network, the Curio devnet requires running multiple binaries in parallel. To simplify this process, we have packaged the devnet using Docker. Please make sure to install the latest version of Docker on your system before proceeding. * Install Docker - ## Building Docker images Build images from the root of the Curio repository ``` make clean docker/devnet ``` * If you need to build containers using a specific version of lotus then provide the version as a parameter. The version must be a tag of [Lotus git repo](https://github.com/filecoin-project/lotus). We are shipping images for all releases from Lotus in our [Github image repo](https://github.com/filecoin-shipyard/lotus-containers/pkgs/container/lotus-containers).\\ ```bash make clean docker/devnet lotus_version=v1.29.2 ``` \\ * If the branch or tag you requested does not exist in our [Github image repository](https://github.com/filecoin-shipyard/lotus-containers/pkgs/container/lotus-containers) then you can build the lotus image manually.\\ ```bash make clean docker/devnet lotus_version=test/branch1 build_lotus=1 ``` ## Start devnet Docker stack * Run ``` make devnet/up ``` * It will spin up `lotus`, `lotus-miner`, `yugabyte`, `curio` and `piece-server` containers. All temporary data will be saved in `./docker/data` folder. * The initial setup could take up to 5 min or more as it needs to download Filecoin proof parameters. During the initial setup, it is normal to see error messages in the log. Containers are waiting for the lotus to be ready. It may timeout several times. Restart is expected to be managed by `docker`. * Try opening the Curio GUI . Devnet is ready to operate when the URL opens and indicates no errors on the startup page. * You can inspect the status using `cd docker/devnet && docker compose logs -f`. ## Make a deal in devnet 1. Login to `piece-server` container either via docker desktop UI or with below command ```shell docker exec -it piece-server /bin/bash ``` 2. Run the below command to make a deal and follow the on-screen instructions. ```shell ./sample/make-a-deal.sh ``` # Experimental Features This section covers the current experimental features available in Curio Curio is developing new features on a regular basis as part of the overall development. This section covers the experimental features released by Curio along with details on how to use them. It is **not** recommended to run experimental features in production environments. The features should be tested as per your requirements, and any issues or requests should be reported to the team via Github or Slack. Once the new features have been tested and vetted, they may be released as part of a stable Curio release and all documentation concerning those features will be moved to an appropriate section of this site. Current experimental features are listed below. {% content-ref url="/pages/cG0uFxJMC1wm4sY0PEr7" %} [GPU Over Provisioning](/experimental-features/gpu-over-provisioning) {% endcontent-ref %} {% content-ref url="/pages/prgTowu8zIKmj20tUd9t" %} [Snark Market](/experimental-features/snark-market) {% endcontent-ref %} {% content-ref url="/pages/TRjFWDJ4OmcU4UXacVNm" %} [Snark Market (Consumer)](/experimental-features/snark-market-consumer) {% endcontent-ref %} {% content-ref url="/pages/K4tcL73v6kMmfmNNijuu" %} [cuzk Proving Daemon](/experimental-features/cuzk-proving-daemon) {% endcontent-ref %} # GPU Over Provisioning This page explains how to allow Curio to run more than multiple GPU tasks on a single GPU at the same time ## Overview The `HARMONY_GPU_OVERPROVISION_FACTOR` environment variable enables GPU over-provisioning by allowing each physical GPU to present itself as multiple logical GPUs. When set to a value greater than 1, this feature allows a single GPU to handle multiple independent processes concurrently. ## Usage ### Enabling Over provisioning Set the `HARMONY_GPU_OVERPROVISION_FACTOR` environment variable to the desired over-provisioning factor. #### **Example** ```bash export HARMONY_GPU_OVERPROVISION_FACTOR=2 ``` * **Effect**: Each physical GPU is treated as two logical GPUs. * **Application**: In a snap encode worker, this setting allows each GPU to handle two independent encode processes simultaneously. #### Example with Service File **/etc/curio.env File** ```sh CURIO_LAYERS=gui,post CURIO_ALL_REMAINING_FIELDS_ARE_OPTIONAL=true CURIO_DB_HOST=yugabyte1,yugabyte2,yugabyte3 CURIO_DB_USER=yugabyte CURIO_DB_PASSWORD=yugabyte CURIO_DB_PORT=5433 CURIO_DB_NAME=yugabyte CURIO_REPO_PATH=~/.curio CURIO_NODE_NAME=ChangeMe FIL_PROOFS_USE_MULTICORE_SDR=1 HARMONY_GPU_OVERPROVISION_FACTOR=2 ``` ## Considerations * **Workload Compatibility**: Ideal for workloads that are not heavily memory-bound. * **Snap Encode Workloads**: Generally suitable for over-provisioning. * **SNARK Workloads**: May encounter memory limitations, especially on GPUs with lower memory capacity. * **GPU Specifications**: Enterprise GPUs with higher memory are better suited for over-provisioning. * **Performance Testing**: It's important to test and validate the optimal over-provisioning factor for your specific hardware and workloads. ### Benefits * **Increased Throughput**: Potentially improves processing capacity per GPU. * **Enhanced Utilization**: Makes better use of GPU resources that might otherwise be underutilized. ### Limitations * **Memory Constraints**: Over-provisioning can lead to memory bottlenecks on GPUs with limited memory. * **Potential Instability**: Running multiple processes on a single GPU may affect system stability and performance. ### Recommendations * **Start with Lower Values**: Begin with an over-provisioning factor of 2 and monitor system performance. * **Monitor Resource Usage**: Keep an eye on GPU memory usage, temperatures, and overall system load. * **Increment Gradually**: Adjust the over-provisioning factor incrementally to find the optimal balance. ### Feedback and Support As this is an experimental feature, we encourage users to provide feedback on their experience. Your insights are valuable for improving GPU over-provisioning support in future releases. # Enable PDP This guide walks you through setting up a PDP-enabled Filecoin Storage Provider using Lotus, YugabyteDB, and Curio {% hint style="info" %} For a PDP-only node without PoRep/sealing (skiff), see the [Curio-PDP runbook](/experimental-features/curio-pdp) and [Skiff binary](https://github.com/filecoin-project/curio/blob/main/documentation/en/skiff-binary.md). {% endhint %} {% hint style="danger" %} **ALPHA FEATURE - UNDER DEVELOPMENT** This documentation covers the PDP (Proof of Data Possession) feature, which is currently in alpha and under active development. This tool is intended for testing and experimental use only. For production use and submitting real deals with live PDP Storage Providers, please use the [Synapse SDK](https://github.com/FilOzone/synapse-sdk). {% endhint %} ## 🚀 Prerequisites {% hint style="warning" %} **Note:** This guide is written specifically for **Ubuntu 22.04**. If you are using a different Linux distribution, refer to the relevant documentation for package installation and compatibility. {% endhint %} Before starting, make sure you have a user with **sudo privileges**. This section prepares your system for the PDP stack. *** ### ⚙️ Hardware requirements * **RAM**: 32 GiB+ * **CPU**: 8 Core+ * **Storage**: * 1 TiB Fast storage (NVMe/SSD) * 10 TiB Long-term storage (HDD) * **GPU**: Not required * **Connectivity**: Public HTTPS endpoint (domain) *** ### 🧰 System Package Installation ```sh sudo apt update && sudo apt upgrade -y && sudo apt install -y \ mesa-opencl-icd ocl-icd-opencl-dev gcc git jq pkg-config curl clang \ build-essential hwloc libhwloc-dev libarchive-dev wget ntp python-is-python3 aria2 ``` *** ### :hammer: Install Go (match `go.mod`) Curio’s minimum Go version is set in the Curio repo at `go.mod`. Example (current repo min is **1.26.2**): ```sh sudo rm -rf /usr/local/go wget https://go.dev/dl/go1.26.2.linux-amd64.tar.gz sudo tar -C /usr/local -xzf go1.26.2.linux-amd64.tar.gz echo 'export PATH=$PATH:/usr/local/go/bin' >> ~/.bashrc source ~/.bashrc go version ``` {% hint style="success" %} You should see something like: `go version go1.26.2 linux/amd64` {% endhint %} *** ### :wrench: Install Rust ```sh curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh ``` {% hint style="info" %} When prompted, choose the option 1) Proceed with standard installation (default — just press Enter). {% endhint %} ```sh source $HOME/.cargo/env rustc --version ``` {% hint style="success" %} You should see something like: `rustc 1.86.0 (05f9846f8 2025-03-31)` {% endhint %} *** ### 🔐 Add Go and Rust to Secure Sudo Path ```sh sudo tee /etc/sudoers.d/dev-paths <
Lotus Documentationhttps://lotus.filecoin.io/lotus/get-started/what-is-lotus//files/hYl19U06Px0ZUgI4sqRGhttps://lotus.filecoin.io/lotus/get-started/what-is-lotus/
Lotus Support ChannelsFilecoin Slack - #fil-lotus-help/files/9Nk64EGiSiQSyBWdnfwmhttps://filecoinproject.slack.com/archives/CPFTWMY7N
### 🔧 Build Lotus Daemon Clone and check out Lotus: ```sh git clone https://github.com/filecoin-project/lotus.git cd lotus git checkout $(curl -s https://api.github.com/repos/filecoin-project/lotus/releases/latest | jq -r .tag_name) ``` **Build and Install for Mainnet** ```sh make clean && make lotus sudo make install-daemon lotus --version ``` **Build and Install for Calibration** ```sh make clean && make GOFLAGS="-tags=calibnet" lotus sudo make install-daemon lotus --version ``` {% hint style="success" %} You should see something like: `lotus version 1.32.2+calibnet+git.ff88d8269` {% endhint %} *** ### 📦 Import a Snapshot and Start the Daemon Download the Snapshot **Mainnet:** ```sh aria2c -x5 -o snapshot.car.zst https://forest-archive.chainsafe.dev/latest/mainnet/ ``` **Calibration:** ```sh aria2c -x5 -o snapshot.car.zst https://forest-archive.chainsafe.dev/latest/calibnet/ ``` **Import and Start the Daemon** ```sh lotus daemon --import-snapshot snapshot.car.zst --remove-existing-chain --halt-after-import nohup lotus daemon > ~/lotus.log 2>&1 & ``` {% hint style="info" %} If you encounter errors related to `EnableEthRPC` or `EnableIndexer`, run the following command and restart Lotus {% endhint %} ```sh sed -i 's/^\( *\)#*EnableEthRPC = .*/\1EnableEthRPC = true/; s/^\( *\)#*EnableIndexer = .*/\1EnableIndexer = true/' ~/.lotus/config.toml ``` **Monitor Sync Progress** ```sh lotus sync wait ``` To monitor continuously: ```sh lotus sync wait --watch ``` **Monitor Logs** ```sh tail -f ~/lotus.log ``` *** ## 🐘 Running YugabyteDB 🧠 Curio uses YugabyteDB to store metadata about deals, sealing operations, and PDP submissions.
Cover image
Yugabyte Documentationhttps://docs.yugabyte.com/preview/tutorials/quick-start/linux//files/T8ayxg9WluJEOg1CwsVThttps://docs.yugabyte.com/preview/tutorials/quick-start/linux/
Yugabyte Support ChannelsFilecoin Slack - #fil-curio-help - Yugabyte Slack/files/JSzWtmVgzubPKREGWNrBhttps://filecoinproject.slack.com/archives/C06LF5YP8S3
### 🛠 Set ulimit configuration {% hint style="warning" %} Before starting Yugabyte, you must increase the default `ulimit` values to ensure system limits do not interfere with the database. {% endhint %} To do this: #### 🔁 **Persist new limits across reboots** Add these lines to `/etc/security/limits.conf`: ```sh echo "$(whoami) soft nofile 1048576" | sudo tee -a /etc/security/limits.conf echo "$(whoami) hard nofile 1048576" | sudo tee -a /etc/security/limits.conf ``` This ensures the increased limits are automatically applied to future sessions. #### ⚡ **Apply limit immediately (for current shell only)** ```sh ulimit -n 1048576 ``` Verify: ```sh ulimit -n ``` {% hint style="success" %} This should output `1048576`. {% endhint %} ### ⚙️ Install Yugabyte ```sh wget https://software.yugabyte.com/releases/2.25.1.0/yugabyte-2.25.1.0-b381-linux-x86_64.tar.gz tar xvfz yugabyte-2.25.1.0-b381-linux-x86_64.tar.gz cd yugabyte-2.25.1.0 ./bin/post_install.sh ``` ### 🚀 Start the DB ```sh ./bin/yugabyted start \ --advertise_address 127.0.0.1 \ --master_flags rpc_bind_addresses=127.0.0.1 \ --tserver_flags rpc_bind_addresses=127.0.0.1 ``` {% hint style="warning" %} If you encounter locale-related errors when starting Yugabyte for the first time, run: {% endhint %} ```sh sudo locale-gen en_US.UTF-8 ``` {% hint style="success" %} Visit `http://127.0.0.1:15433` to confirm successful installation. This is the YugabyteDB web UI — it should display the dashboard if the service is running correctly and all nodes are healthy. {% endhint %} {% hint style="info" %} You can also check your Yugabyte cluster details directly in the CLI with: {% endhint %} ```sh ./bin/yugabyted status ``` *** ## 🧱 Installing and Configuring Curio 🧠 Curio is the core PDP client that coordinates sealing, interacts with Lotus and submits PDP proofs.
Curio Documentationhttps://docs.curiostorage.org//files/JSzWtmVgzubPKREGWNrBhttps://docs.curiostorage.org/
Curio Support ChannelsFilecoin Slack - #fil-curio-help/files/9Nk64EGiSiQSyBWdnfwmhttps://filecoinproject.slack.com/archives/C06LF5YP8S3
### ⚙️ System Configuration Before you proceed with the installation, you should increase the UDP buffer size: ```sh sudo sysctl -w net.core.rmem_max=2097152 sudo sysctl -w net.core.rmem_default=2097152 ``` To make this change persistent across reboots: ```sh echo 'net.core.rmem_max=2097152' | sudo tee -a /etc/sysctl.conf echo 'net.core.rmem_default=2097152' | sudo tee -a /etc/sysctl.conf ``` ### 🔬 Build Curio Clone the repository and switch to the PDP branch: ```sh git clone https://github.com/filecoin-project/curio.git cd curio git checkout pdpM3d ``` {% hint style="info" %} Curio is compiled for a specific Filecoin network at build time. Choose the appropriate build command below. {% endhint %} Mainnet ```sh make clean build ``` Calibration ```sh make clean calibnet ``` {% hint style="info" %} This step will take a few minutes to complete. {% endhint %} ### ✅ Install and Verify Curio Run the following to install the compiled binary: ```sh sudo make install ``` This will place curio in `/usr/local/bin` Verify the installation: ```sh curio --version ``` Expected example output: ```sh curio version 1.24.4+calibnet+git_f954c0a_2025-04-06T15:46:32-04:00 ``` *** ### 🔧 Guided Setup Curio provides a utility to help you set up a new miner interactively. Run the following command: ```sh curio guided-setup ``` #### 1️⃣ Select Curio Installation Type Use the arrow keys to navigate the guided setup menu and select "**Setup non-Storage Provider cluster**". #### 2️⃣ Enter Your YugabyteDB Connection Details If you used the default installation steps from this guide, the following values should work: * Host: `127.0.0.1` * Port: `5433` * Username: `yugabyte` * Password: `yugabyte` * Database: `yugabyte` You can verify these settings by running the following command from the Yugabyte directory: ```sh ./bin/yugabyted status ``` After selecting "**Continue to connect and update schema**", Curio will automatically create the required tables and schema in the database. #### 3️⃣ Telemetry (Optional) You'll be asked whether to share anonymised or signed telemetry with the Curio team to help improve the software. Select your preference and continue. #### 4️⃣ Save Database Configuration At the final step of the guided setup, you'll be prompted to choose where to save your database configuration file. Use the arrow keys to select a location. A common default is: ```sh /home/your-username/curio.env ``` Once selected, setup will complete, and the miner configuration will be stored. #### 5️⃣ Launch the Curio Web GUI To explore the Curio interface visually, start the GUI layer: ```sh curio run --layers=gui ``` Then, open your browser and go to: ```sh http://127.0.0.1:4701 ``` This will launch the Curio web GUI locally. *** ## 🧪 Enabling FWSS PDP 🧠 This section enables **FWSS Proof of Data Possession (PDP)** on your SP node using Curio. These steps guide you through running a standalone PDP service using Curio and pdptool.
Cover image
PDP Support Channels/files/9Nk64EGiSiQSyBWdnfwmhttps://filecoinproject.slack.com/archives/C0717TGU7V2
### 📦 Attach Storage Locations With Curio running with the GUI layer: ```sh curio run --layers=gui ``` Run the following commands in your Curio CLI to attach storage paths: ```sh curio cli storage attach --init --seal /fast-storage/path curio cli storage attach --init --store /long-term-storage/path ``` {% hint style="info" %} Your fast-storage path should point to high-performance storage media such as NVMe or SSD {% endhint %} *** ### 🔧 Add a PDP Configuration Layer Browse to the **Configurations** page of the Curio GUI. Create a new layer named **pdp** and enable the following under Subsystems: {% hint style="info" %} You may find it helpful to search for the setting names in your browser. {% endhint %} * ✅ `EnableParkPiece` * ✅ `EnablePDP` * ✅ `EnableCommP` * ✅ `EnableMoveStorage` In the **HTTP** section: * ✅ Enable: `true` * 🌐 DomainName: `your domain (e.g., pdp.mydomain.com)` * 📡 ListenAddress: `0.0.0.0:443` {% hint style="info" %} **Tip:** You must point your domain's A record to your server's public IP address for Let's Encrypt to issue a certificate. {% endhint %} *** ### 💰 Import your Filecoin Wallet Private Key: {% hint style="warning" %} There are several ways to obtain private keys for Ethereum addresses. In this guide, we will use a new delegated FIL wallet address. {% endhint %} Create a new delegated wallet: ```sh lotus wallet new delegated ``` ```sh # Example output: t410fuo4dghaeiqzokiqnxruzdr6e3cjktnxprrc56bi ``` {% hint style="info" %} You can display your Lotus wallets at any time by running: {% endhint %} ```sh lotus wallet list ``` Export & convert your new delegated wallet address private key: ```sh lotus wallet export | xxd -r -p | jq -r '.PrivateKey' | base64 -d | xxd -p -c 32 ``` ```sh # Example output: d4c2e3f9a716bb0e47fa91b2cf4a29870be3c5982fd6eafed71e8ac3f9c0b127 ``` Browse to the **PDP** page of the Curio GUI and in the **Owner Address** section: * Select **Import Key** * Copy the previously generated private wallet key into the **Private Key (Hex)** field. * Select **Import Key** {% hint style="success" %} Your 0x wallet address - the delegated Ethereum address derived from your Filecoin delegated wallet private key - will be added to the **Owner Address** section of the Curio PDP page. {% endhint %} Make sure to send a small amount of FIL or tFIL (testnet FIL) to your 0x wallet - we recommend 8 FIL for Mainnet & 5 tFIL for Calibration to ensure uninterrupted PDP operation during initial setup and testing. [Calibration test FIL faucet information](https://docs.filecoin.io/smart-contracts/developing-contracts/get-test-tokens). {% hint style="warning" %} **Important:** Secure your private key material. Don't expose or store it in plain text without protection. {% endhint %} *** ### 🚀 Restart and Verify Restart Curio with both layers: ```sh curio run --layers=gui,pdp ``` {% hint style="info" %} If you encounter errors related to `EnableEthRPC` or `EnableIndexer`, run the following command and restart Lotus {% endhint %} ```sh sed -i 's/^\( *\)#*EnableEthRPC = .*/\1EnableEthRPC = true/; s/^\( *\)#*EnableIndexer = .*/\1EnableIndexer = true/' ~/.lotus/config.toml ``` {% hint style="info" %} If you encounter errors binding to port 443 when starting Curio with the pdp configuration layer, run: {% endhint %} ```sh sudo setcap 'cap_net_bind_service=+ep' /usr/local/bin/curio ``` Test the PDP service: {% hint style="info" %} If `pdptool` is not installed, clone and build Curio: {% endhint %} ```sh git clone https://github.com/filecoin-project/curio.git cd curio/cmd/pdptool go build . ``` ```sh ./pdptool ping --service-url https://your-domain.com --service-name public ``` {% hint style="info" %} Always use `public` for the `--service-name` flag {% endhint %} {% hint style="success" %} Expected output: {% endhint %} ```sh Ping successful: Service is reachable and JWT token is valid. ``` *** ## 🎉 You're Done! You've successfully launched a **PDP-enabled Filecoin Storage Provider** stack. Your system is now: * ✅ Syncing with the Filecoin network via Lotus * ✅ Recording deal and sector metadata in YugabyteDB * ✅ Operating Curio to manage sealing and coordination * ✅ Enabled Proof of Data Possession (PDP) * ✅ Connected to your PDP-enabled storage provider *** ## 🔜 Next Steps * :heavy\_check\_mark: Register your FWSS node * :link: Explore FWSS & PDP tools & resources at [https://www.filecoin.services](https://www.filecoin.services/) * 💬 Join the community - Filecoin Slack - [#fil-pdp](https://filecoinproject.slack.com/archives/C0717TGU7V2) # Curio-PDP runbook Curio-PDP is the lightweight PDP storage provider build (`make curio-pdp`, Go tag `skiff`). It runs PDP proving and the FWSS registration flow without PoRep/sealing, MK20 market code, or `filecoin-ffi`. For the skiff binary overview and build flags, see [Skiff binary](https://github.com/filecoin-project/curio/blob/main/documentation/en/skiff-binary.md). For full-stack Curio with Yugabyte and optional PDP alongside sealing, see [Enable PDP](/experimental-features/enable-pdp). ## Architecture and data stores | Deployment | HarmonyDB (tasks, config, PDP state) | Piece index (multihash → offset) | | --------------------- | ------------------------------------ | ------------------------------------ | | **Full Curio** | Yugabyte (YSQL) | Yugabyte YCQL / Cassandra-compatible | | **Curio-PDP (skiff)** | **Yugabyte (YSQL)** | **Yugabyte YCQL** | | **Tests / CI** | Postgres | Scylla (CQL) | Curio-PDP is intentionally lighter on compute and dependencies: no PoRep/sealing, MK20 market code, or `filecoin-ffi`. Operators still run **Dockerized Yugabyte** for HarmonyDB and piece indexing — the same YSQL + YCQL stack as full Curio, bundled via `docker/skiff`. Piece payload files live on disk under writable paths discovered under `/data` (see [Storage](#storage)). Index data lives in Yugabyte YCQL and must be backed up with the database (see [Yugabyte backup](/administration/yugabyte-backup)). ## Prerequisites * **Docker** and **Docker Compose** (recommended deployment path) * External **Lotus-compatible chain node** (Lotus, Forest, etc.) — set `FULLNODE_API_INFO` or `[APIs].ChainApiInfo` * Writable storage under `/data` (see [Storage](#storage)) * Optional public **HTTPS domain** when exposing the PDP HTTP API (`HTTP.DomainName` in config) * FIL/tFIL to fund the PDP signing wallet before FWSS registration ## First-time setup ### 1. Configure `docker/skiff/.env` Copy or edit `docker/skiff/.env` before starting the stack. **Chain node (required).** Skiff does not embed a chain node. Set `FULLNODE_API_INFO` to a Lotus-compatible RPC endpoint that matches your skiff build network (`skiff` = mainnet, `calibnet-skiff` = calibration, etc.). If Lotus (or another chain node) runs on the **same machine** as Docker, use an address the skiff **container** can reach — not `127.0.0.1` inside the container: ```bash # Lotus on the Docker host (macOS / Windows / Linux with host-gateway) FULLNODE_API_INFO=/ip4/host.docker.internal/tcp/1234/http # Lotus on another host on your LAN FULLNODE_API_INFO=/ip4/192.168.1.50/tcp/1234/http ``` See [Skiff binary — Chain API](https://github.com/filecoin-project/curio/blob/main/documentation/en/skiff-binary.md#chain-api) for config-layer alternatives (`[APIs].ChainApiInfo`). **Storage paths** default to `./data/` under `docker/skiff/`. Adjust `YUGABYTE_DATA`, `SKIFF_REPO_DATA`, and `SKIFF_STORAGE` if needed. ### 2. Start Yugabyte + Skiff (Docker) From the repo root: ```bash cd docker/skiff docker compose up -d ``` This starts: * **Yugabyte** — YSQL on port `5433`, YCQL on port `9042`, web UI on `15433` * **Skiff** — local admin GUI on `127.0.0.1:4701`; public PDP API on `80`/`443` only Persistent data defaults to `docker/skiff/data/` (Yugabyte, repo state, and piece storage). {% hint style="warning" %} **Public firewall: open only TCP 80 and 443.** The admin GUI on port `4701` is unauthenticated and for local operator access only — do not publish it to the internet. The Compose file maps host `127.0.0.1:4701` for the same reason. Use SSH port forwarding if you need remote GUI access (see [PDP signing wallet](#3-pdp-signing-wallet-admin-gui)). {% endhint %} HarmonyDB migrations run on connect and create the same `curio` schema as full Curio. The piece `IndexStore` connects to Yugabyte YCQL on the same host (port `9042` by default via `--db-cassandra-port` / `CURIO_DB_CASSANDRA_PORT`). For a native skiff binary against the same Yugabyte stack (without the skiff container), export: ```bash export CURIO_DB_HOST=127.0.0.1 export CURIO_DB_PORT=5433 export CURIO_DB_USER=yugabyte export CURIO_DB_PASSWORD=yugabyte export CURIO_DB_NAME=yugabyte export CURIO_REPO_PATH=~/.curio export SKIFF_MACHINE_HOST=127.0.0.1:skiff export FULLNODE_API_INFO=/ip4/127.0.0.1/tcp/1234/http ``` ### 3. PDP signing wallet (admin GUI) Skiff needs a **PDP signing key** stored in HarmonyDB (`eth_keys` with `role=pdp`) before FWSS registration. Configure it through the admin GUI — the key is **not** set in `.env`. **Open the GUI** * On the Docker host: **** * From a remote machine (SSH tunnel): ```bash ssh -L 4701:127.0.0.1:4701 user@your-server ``` Then browse to **** on your laptop. Go to **PDP** → wallet section. | Action | When to use | | ---------- | ---------------------------------------------------------------------------------------------------------------- | | **Create** | Generate a new secp256k1 key on this node; the private key is shown **once** — save it before closing the dialog | | **Import** | Paste a hex private key for an existing 0x address you already control | Only one PDP key is allowed per cluster. After create or import, fund the displayed **0x address** with enough FIL/tFIL for registration and ongoing on-chain messages (see [Enable PDP — Import your Filecoin Wallet Private Key](/experimental-features/enable-pdp#import-your-filecoin-wallet-private-key) for recommended amounts and a Lotus delegated-wallet import workflow). {% hint style="danger" %} The GUI has no login. Anyone who can reach port `4701` can manage keys and config. Keep it on localhost or behind an SSH tunnel only. {% endhint %} The wallet private key is stored in Yugabyte and survives container restarts as long as `YUGABYTE_DATA` is preserved. Back up Yugabyte before redeploying (see [Yugabyte backup](/administration/yugabyte-backup)). ### 4. Register with FWSS In the GUI **Register** tab, complete provider registration, then verify with: ```bash pdptool ping --service-url https://your-domain.com --service-name public ``` ### Native binary (optional) If running skiff outside Docker, start Yugabyte first (see `docker/skiff/docker-compose.yaml` for the reference single-node command), then: ```bash ./curio # curio-pdp build ``` On first start, skiff **auto-seeds the `base` config layer** with PDP defaults (`EnablePDP`, `EnableWebGui`, `GuiAddress`, `StorageRPCSecret`). If a separate `pdp` layer already exists from a prior full-Curio setup, it is merged into `base` once at startup. Configure the PDP wallet via the same GUI steps above. ## Configuration model Skiff reads **only the `base` layer** at runtime. Do not rely on separate `pdp` or `gui` layers — put operational settings in `base` (or let auto-seed populate defaults and edit via the GUI). Typical `base` values: * `Subsystems.EnablePDP = true` (forced on) * `Subsystems.EnableWebGui = true` * `Subsystems.GuiAddress = "127.0.0.1:4701"` (never bind the GUI to `0.0.0.0` on a host reachable from the internet) * `HTTP.Enable = false` until a domain is configured for the public API Only **TCP 80 and 443** should be exposed on your public firewall for FWSS registration and client traffic. See [Curio HTTP server](/curio-market/curio-http-server) for TLS and reverse-proxy options. Skiff requires an external Lotus-compatible chain node. Set `FULLNODE_API_INFO` or `[APIs].ChainApiInfo` in the `base` layer (see [Skiff binary — Chain API](https://github.com/filecoin-project/curio/blob/main/documentation/en/skiff-binary.md#chain-api)). ## Storage Curio-PDP stores piece payloads on local disk. Mount drives at **`/data`** (or bind-mount volumes beneath it). On startup the node scans `/data` and **every subdirectory**, probes each for write access, and uses every writable location as storage. Unwritable paths are skipped. Missing `sectorstore.json` files are created automatically in each writable location. Additionally, you can use a path other than `/data`: ```bash DATA_STORAGE=/var/lib/curio-data ./curio ``` You can also set `[Subsystems].DataPath` in the `base` config layer, or pass `--data=/var/lib/curio-data`. ## Moving between deployment profiles Full Curio and Curio-PDP both use **Yugabyte (YSQL + YCQL)**. CI uses Postgres + Scylla and is not an operator deployment profile. To move **relational PDP state and piece indexes**: 1. Back up Yugabyte YSQL and YCQL (see [Yugabyte backup](/administration/yugabyte-backup)). 2. Restore into the target Yugabyte instance. 3. Copy **piece files** separately; payloads are not in the database dump. 4. If the imported DB has a separate `pdp` config layer, skiff merges it into `base` on next startup. There is no dedicated migration tool — Yugabyte backup/restore plus file copy is sufficient. ## Troubleshooting | Symptom | Check | | ------------------------------------------------ | ------------------------------------------------------------------------------------------------------------------------------------------- | | `docker compose up` fails on `FULLNODE_API_INFO` | Set chain RPC in `docker/skiff/.env`; see [Configure .env](#1-configure-dockerskiffenv) | | Skiff cannot reach chain node | From inside the container, `127.0.0.1` is not the Docker host — use `host.docker.internal` or the node's LAN IP | | Alert: PDP wallet not configured | [PDP signing wallet](#3-pdp-signing-wallet-admin-gui) → Create or Import; verify `eth_keys` has `role=pdp` | | Yugabyte connection errors | `docker compose ps`, `CURIO_DB_*`, YSQL on `5433`, YCQL on `9042`; see [Yugabyte troubleshooting](/administration/yugabyte-troubleshooting) | | No storage paths | Drives mounted under `/data` (or `DATA_STORAGE` / `--data`); write permissions on discovered paths | | Registration fails | Wallet funded; `HTTP.DomainName` / TLS; chain node synced and reachable | | Startup warning about missing key | Expected until wallet is configured; clears after key insert | # Snark Market This page explains how to set up a provider for the Curio Snark Market (experimental). > ⚠️ **Experimental Feature in Testing**\ > This feature is currently experimental and under active testing. Interfaces, behaviors, and requirements **may change without notice**. *** ## 🔧 What is the Snark Market? The Snark Market allows any Curio node — **including Storage Providers with spare sealing/GPU capacity** — to sell or buy proof computation in exchange for FIL. It is designed for GPU nodes that want to participate in Filecoin proof offloading, enabling a decentralized proof marketplace. Storage Providers can also **consume** proof compute — buying proofs from the market to save GPU capacity. See [Snark Market (Consumer)](/experimental-features/snark-market-consumer) for that setup. This guide will walk you through how to: * Enable proof selling on GPU nodes * Set up the pricing and wallet * See activity and settlement stats in the UI *** ## ⚙️ Prerequisites Before enabling the Snark Market on your node: * You must be running a Curio node with GPU capabilities. * You must have a working web UI (Your browser can access ports exposed by Curio). * You need a Lotus node installed and the Filecoin mainnet chain synced.\ Refer to lotus documentation here:\ * You need **YugabyteDB** installed.\ 👉 Follow the official setup instructions here:\ *** ## ⚙️ System Requirements * Modern **NVIDIA GPU** (recommended 12GB+ VRAM) * 70GB base system RAM + \~220 GB Per GPU * Lower amount is acceptable, but you won't be able to use the much faster CUDA C2 feature (batch sealing toolchain)\ Meaning \~10mins/proof instead of 2. * Curio **v1.27.0 or later** (Snark Market is included in official releases) * FIL balance on Mainnet *** ## 🚀 Setup Instructions ### 0. (Optional/Recommended) Benchmark your system You can skip most of the complex setup and learn how your hardware performs with lotus-bench * Follow build instructions from * For faster CUDA C2 (batch sealing toolchain) lotus-bench do:\ `RUSTFLAGS="-C target-cpu=native -g" FFI_BUILD_FROM_SOURCE=1 FFI_USE_CUDA_SUPRASEAL=1 make clean deps lotus-bench` * Download example snark inputs from (\~51MB) * Run `./lotus-bench simple commit2 c1.json`, wait a few minutes for results\ On the first run lotus-bench may need to download the SNARK proving parameters ### 1. Install Curio First, install dependencies (as per the [main install guide](https://docs.curiostorage.org/setup)): ```bash sudo apt update sudo apt install -y build-essential pkg-config libssl-dev curl git clang cmake golang ``` Then, build Curio (for CUDA C2 support, build from source): ```bash git checkout v1.27.2 # or latest release tag git submodule update make clean RUSTFLAGS="-C target-cpu=native -g" FFI_BUILD_FROM_SOURCE=1 FFI_USE_CUDA_SUPRASEAL=1 make clean build all sudo make install ``` Run basic setup. * If needed set database variables (defaults work with local YugabyteDB install)\ `CURIO_DB_NAME`, `CURIO_DB_USER`, `CURIO_DB_HOST` (can be a comma separated list of tserver IPs, one is also ok), `CURIO_DB_PASSWORD` * Ensure your lotus-node is running * Run `curio guided-setup` -> select `Setup non-Storage Provider cluster` * Create empty snark-provider configuration layer -> `echo | curio config create --title snark-provider` After all is done, [setup, enable and start the curio service](https://docs.curiostorage.org/curio-service#service-file-for-curio) * For `CURIO_LAYERS` use `gui,snark-provider` *** ## 🛠️ Step-by-step Setup ### 2. Enable the Market in Layer Configuration Ensure you're **not running on a WindowPoSt node**. This is only supported on GPU-based PoRep or Snap nodes. In the Web UI: 1. Go to `Overview` → `Configuration`, select the `snark-provider` if created previously 2. Find the **Subsystems** section 3. Set `EnableProofShare` to true 4. Save and restart the node
Enable PROOFSHARE toggle in configuration

Enable PROOFSHARE from the configuration layer

*** ### 3. Configure Provider Settings Navigate to `Snark Market` in the sidebar. Under **Provider Settings**: * Enable the settings checkbox * **Create a new `f1` wallet** (do *not* reuse existing wallet)\ Use `lotus wallet new secp256k1` in the CLI\ ⚠️ *Please note: This wallet can be changed later, but it is tricky* * Set **Price (FIL/p)** to `0.005` (recommended for testing) * Single proof (`p`) should take roughly two minutes to compute, your price should be calculated based on how many proofs per hour per GPU you expect to compute and your cost to run the GPU. The snark marketplace automatically adjusts the market price to match supply to demand. * Click **Update Settings**
Snark Wallet Setup

Snark Market provider settings with wallet and price configured

*** ### 4. Verify Your Node Once you've configured the provider settings: * Your node will automatically begin queueing proof work * The dashboard will update with: * Active Asks * SNARK Queue * Payment Summaries * Recent Settlements * You can also view the [global public dashboard](https://mainnet.snass.fsp.sh/ui/)
Snark Market Dashboard

Overview showing queue, asks, settlements, and active proofs

*** ## 🪙 Wallet Setup Notes * Create a **new `f1` address** and fund it (e.g. `0.1 FIL`) * This wallet receives SNARK proof rewards * Ensure the wallet remains **unlocked** *** ## 📈 Pricing & Payments * Price is set per \~130M proof constraints (default granularity) * Suggested starting point price: `0.005 FIL` * Settlements occur automatically when network gas fee to settle is less than 0.2% of the balance to settle *** ## 🧪 Notes * Your provider must complete **50 challenge proofs** per "work-slot" * Each "work-slot" is allows for either one "work ask" in the market - listing of readiness to take on work on one proof for some minimum price, or one "assigned proof" which is being actively worked on. * Number of work slots is calculated by dividing "completed challenge proofs" by **50** * Failure to complete assigned work within the deadline reduces "completed challenge proofs" balance by **50** proofs * Withdrawing an ask from the market (i.e. in order to adjust ask price) reduces "completed challenge proofs" balance by **1** proof * "challenge proofs" are gained only by completing unpaid challenge proofs, which are assigned when attempting to create a work ask while the challenge balance is too low. * The service *may* assign real proofs as challenges, but only if they were failed by other providers and no other provider can be found who can take the payment * Proofs must complete within **45 minutes**, or your node will lose its active slot and need to **re-earn trust** * The system is **fault-tolerant** and retries failed work automatically * You may be assigned retry proofs which come with non-current, potentially lower than current, still higher than your minimum price per proof. * You can **scale horizontally** by running more GPU workers with the same setup *** Let us know on Slack if you’re testing `#fil-curio-help` — we’ll be actively monitoring for feedback and performance 🚀 # Snark Market (Consumer) This page explains how to set up a consumer (GPU saver) for the Curio Snark Market (experimental). > ⚠️ **Experimental Feature in Testing**\ > This feature is currently experimental and under active testing. Interfaces, behaviors, and requirements **may change without notice**. *** ## 🔧 What is the Snark Market (Consumer)? The Snark Market allows Storage Providers to **buy** proof computation from the market instead of running GPUs locally. You offload PoRep and Snap proof work to providers in exchange for FIL — a "GPU saver" approach that lets you complete the C2 (proof) phase of sealing without local GPU capacity. To sell proof compute (be a provider), see [Snark Market](/experimental-features/snark-market). *** ## ⚙️ Prerequisites Before enabling the Snark Market consumer on your node: * You must be running a Curio node with a **sealing pipeline** (PoRep or Snap tasks). * You must have a working web UI (your browser can access ports exposed by Curio). * You need a Lotus node installed and the Filecoin mainnet chain synced.\ Refer to lotus documentation here:\ * You need **YugabyteDB** installed.\ 👉 Follow the official setup instructions here:\ *** ## ⚙️ System Requirements * **No GPU required** for consumers — you buy proofs from the market. * Standard sealing node RAM and storage. * Curio **v1.27.0 or later** (Snark Market is included in official releases). * FIL balance on Mainnet (to pay for proof compute). *** ## 🛠️ Step-by-step Setup ### 1. Enable Remote Proofs in Configuration 1. Go to `Overview` → `Configuration`, and select your miner or sealing layer. 2. Find the **Subsystems** section. 3. Set `EnableRemoteProofs` to true. 4. Save and restart the node. *** ### 2. Add and Fund Client Wallets Navigate to **Snark Market** in the sidebar. Under **Client Wallets**: * **Add Wallet**: Click **Add Wallet** and enter an f1 address you control. You can use an existing wallet (e.g. worker, collateral) or create a dedicated one — SPs typically already have funded wallets. * **Deposit**: Click **Deposit** to move FIL from that wallet's chain balance into the payment router. You need available balance in the router to pay for proofs. *** ### 3. Configure Client Settings Under **Client Settings** (right side of the page): 1. Accept the **Client Terms of Service** when prompted. 2. Click **Add SP**. Enter your SP address and the client wallet address (use a wallet you added in Client Wallets). 3. For each SP row: * Check **Enabled**. * Set **Wallet** to the f1 address you will use for payments. * Set **buy\_delay\_secs** — delay before outsourcing work (allows local GPU to take it first if you have one). * Enable **do\_porep** and/or **do\_snap** for the proof types you want to buy. * Set **FIL/P** (max price per proof; market price must be ≤ this). 4. Click **Save**. *** ## 📈 Pricing **FIL/P** is the maximum you are willing to pay per **P**. One **P** (one "proof unit") equals the cost of a single **32 GiB C2 (PoRep) proof** — the SNARK proof generated after C1 during sector sealing. | Proof type | Multiplier | Cost formula | Example at 0.005 FIL/P | | ------------------------------ | ---------- | ------------- | ---------------------- | | **32 GiB C2** (PoRep) | 1× | 1 × (FIL/P) | **0.005 FIL** | | **32 GiB Snap** (UpdateEncode) | 1.6× | 1.6 × (FIL/P) | **0.008 FIL** | Example: If you set **FIL/P = 0.005** and the market price is at or below that, a 32 GiB C2 will cost \~0.005 FIL and a 32 GiB Snap proof will cost \~0.008 FIL. The market price fluctuates; your configured FIL/P is the maximum you will pay — proofs are only purchased when the current market price is at or below your limit. Check the [public dashboard](https://mainnet.snass.fsp.sh/ui/) for the current **Min price**; use it to set your FIL/P accordingly. *** ## 🔄 Balance Manager (Optional) To automatically top up client wallet balances: 1. Go to **Wallet** → **Balance Manager**. 2. Click **Add SnarkMarket Client Rule**. 3. Set **Subject** to your client wallet address. 4. Set **Low** and **High** watermarks (in FIL). 5. Save. The Balance Manager will deposit FIL from the wallet's chain balance into the router when the available balance falls below the low watermark. *** ## ✅ Verify * **View Requests**: For each SP, click **View Requests** to see in-flight and completed proof requests. * **Client Messages**: Shows deposit and withdrawal status for client wallets. *** ## 🧪 Notes * **buy\_delay\_secs** gives your local GPU (if any) time to take work before outsourcing. Set to 0 to outsource immediately when local capacity is idle. * Work is only sent to the market when the current price is ≤ your configured max FIL/P. * Ensure client wallets stay funded; otherwise proofs cannot be purchased. *** Let us know on Slack if you're testing `#fil-curio-help` — we'll be actively monitoring for feedback and performance 🚀 # Market 2.0 API OpenAPI spec of Market 2.0 API. {% hint style="warning" %} These APIs can change without any notice as they are still being developed. This page is meant to be a reference for client libraries and SDK to allow parallel development. {% endhint %} ## List of supported DDO contracts > List of supported DDO contracts ```json {"openapi":"3.1.1","info":{"title":"Curio Market 2.0 API","version":"0.0.1"},"paths":{"/contracts":{"get":{"description":"List of supported DDO contracts","summary":"List of supported DDO contracts","responses":{"200":{"description":"Array of contract addresses supported by a system or application.","content":{"application/json":{"schema":{"$ref":"#/components/schemas/mk20.SupportedContracts"}}}},"500":{"description":"Internal Server Error","content":{"application/json":{"schema":{"type":"string"}}}}}}}},"components":{"schemas":{"mk20.SupportedContracts":{"type":"object","properties":{"contracts":{"description":"Contracts represents a list of supported contract addresses in string format.","type":"array","items":{"type":"string"}}}}}}} ``` ## List of supported products > List of supported products ```json {"openapi":"3.1.1","info":{"title":"Curio Market 2.0 API","version":"0.0.1"},"paths":{"/products":{"get":{"description":"List of supported products","summary":"List of supported products","responses":{"200":{"description":"Array of products supported by the SP","content":{"application/json":{"schema":{"$ref":"#/components/schemas/mk20.SupportedProducts"}}}},"500":{"description":"Internal Server Error","content":{"application/json":{"schema":{"type":"string"}}}}}}}},"components":{"schemas":{"mk20.SupportedProducts":{"type":"object","properties":{"products":{"description":"Contracts represents a list of supported contract addresses in string format.","type":"array","items":{"type":"string"}}}}}}} ``` ## List of supported dats sources > List of supported data sources ```json {"openapi":"3.1.1","info":{"title":"Curio Market 2.0 API","version":"0.0.1"},"paths":{"/sources":{"get":{"description":"List of supported data sources","summary":"List of supported dats sources","responses":{"200":{"description":"Array of dats sources supported by the SP","content":{"application/json":{"schema":{"$ref":"#/components/schemas/mk20.SupportedDataSources"}}}},"500":{"description":"Internal Server Error","content":{"application/json":{"schema":{"type":"string"}}}}}}}},"components":{"schemas":{"mk20.SupportedDataSources":{"type":"object","properties":{"sources":{"description":"Contracts represents a list of supported contract addresses in string format.","type":"array","items":{"type":"string"}}}}}}} ``` ## List of supported DDO contracts > List of supported DDO contracts ```json {"openapi":"3.1.1","info":{"title":"Curio Market 2.0 API","version":"0.0.1"},"paths":{"/status/{id}":{"get":{"description":"List of supported DDO contracts","summary":"List of supported DDO contracts","parameters":[{"schema":{"type":"string"},"description":"id","name":"id","in":"path","required":true}],"responses":{"200":{"description":"the status response for deal products with their respective deal statuses","content":{"application/json":{"schema":{"$ref":"#/components/schemas/mk20.DealProductStatusResponse"}}}},"400":{"description":"Bad Request - Invalid input or validation error","content":{"application/json":{"schema":{"type":"string"}}}},"500":{"description":"Internal Server Error","content":{"application/json":{"schema":{"type":"string"}}}}}}}},"components":{"schemas":{"mk20.DealProductStatusResponse":{"type":"object","properties":{"ddo_v1":{"description":"DDOV1 holds the DealStatusResponse for product \"ddo_v1\".","allOf":[{"$ref":"#/components/schemas/mk20.DealStatusResponse"}]},"pdp_v1":{"description":"PDPV1 represents the DealStatusResponse for the product pdp_v1.","allOf":[{"$ref":"#/components/schemas/mk20.DealStatusResponse"}]}}},"mk20.DealStatusResponse":{"type":"object","properties":{"error_msg":{"description":"ErrorMsg is an optional field containing error details associated with the deal's current state if an error occurred.","type":"string"},"status":{"description":"State indicates the current processing state of the deal as a DealState value.","allOf":[{"$ref":"#/components/schemas/mk20.DealState"}]}}},"mk20.DealState":{"type":"string","enum":["accepted","uploading","processing","sealing","indexing","failed","complete"]}}}} ``` ## Update the deal details of existing deals. > Useful for adding adding additional products and updating PoRep duration ```json {"openapi":"3.1.1","info":{"title":"Curio Market 2.0 API","version":"0.0.1"},"paths":{"/update/{id}":{"get":{"description":"Useful for adding adding additional products and updating PoRep duration","summary":"Update the deal details of existing deals.","parameters":[{"schema":{"type":"string"},"description":"id","name":"id","in":"path","required":true}],"responses":{"200":{"description":"Ok represents a successful operation with an HTTP status code of 200","content":{"application/json":{"schema":{"$ref":"#/components/schemas/mk20.DealCode"}}}},"400":{"description":"Bad Request - Invalid input or validation error","content":{"application/json":{"schema":{"type":"string"}}}},"404":{"description":"ErrDealNotFound indicates that the specified deal could not be found, corresponding to the HTTP status code 404","content":{"application/json":{"schema":{"$ref":"#/components/schemas/mk20.DealCode"}}}},"422":{"description":"ErrUnsupportedDataSource indicates the specified data source is not supported or disabled for use in the current context","content":{"application/json":{"schema":{"$ref":"#/components/schemas/mk20.DealCode"}}}},"423":{"description":"ErrUnsupportedProduct indicates that the requested product is not supported by the provider","content":{"application/json":{"schema":{"$ref":"#/components/schemas/mk20.DealCode"}}}},"424":{"description":"ErrProductNotEnabled indicates that the requested product is not enabled on the provider","content":{"application/json":{"schema":{"$ref":"#/components/schemas/mk20.DealCode"}}}},"425":{"description":"ErrProductValidationFailed indicates a failure during product-specific validation due to invalid or missing data","content":{"application/json":{"schema":{"$ref":"#/components/schemas/mk20.DealCode"}}}},"426":{"description":"ErrDealRejectedByMarket indicates that a proposed deal was rejected by the market for not meeting its acceptance criteria or rules","content":{"application/json":{"schema":{"$ref":"#/components/schemas/mk20.DealCode"}}}},"429":{"description":"ErrServiceOverloaded indicates that the service is overloaded and cannot process the request at the moment","content":{"application/json":{"schema":{"$ref":"#/components/schemas/mk20.DealCode"}}}},"430":{"description":"ErrMalformedDataSource indicates that the provided data source is incorrectly formatted or contains invalid data","content":{"application/json":{"schema":{"$ref":"#/components/schemas/mk20.DealCode"}}}},"440":{"description":"ErrMarketNotEnabled indicates that the market is not enabled for the requested operation","content":{"application/json":{"schema":{"$ref":"#/components/schemas/mk20.DealCode"}}}},"441":{"description":"ErrDurationTooShort indicates that the provided duration value does not meet the minimum required threshold","content":{"application/json":{"schema":{"$ref":"#/components/schemas/mk20.DealCode"}}}},"500":{"description":"ErrServerInternalError indicates an internal server error with a corresponding error code of 500","content":{"application/json":{"schema":{"$ref":"#/components/schemas/mk20.DealCode"}}}},"503":{"description":"ErrServiceMaintenance indicates that the service is temporarily unavailable due to maintenance, corresponding to HTTP status code 503","content":{"application/json":{"schema":{"$ref":"#/components/schemas/mk20.DealCode"}}}}},"requestBody":{"content":{"application/json":{"schema":{"$ref":"#/components/schemas/mk20.Deal"}}},"description":"mk20.Deal in json format","required":true}}}},"components":{"schemas":{"mk20.DealCode":{"type":"integer","enum":[200,401,400,404,430,422,423,424,425,426,500,503,429,440,441]},"mk20.Deal":{"type":"object","properties":{"client":{"description":"Client wallet for the deal","allOf":[{"$ref":"#/components/schemas/address.Address"}]},"data":{"description":"Data represents the source of piece data and associated metadata.","allOf":[{"$ref":"#/components/schemas/mk20.DataSource"}]},"identifier":{"description":"Identifier represents a unique identifier for the deal in UUID format.","type":"array","items":{"type":"integer"}},"products":{"description":"Products represents a collection of product-specific information associated with a deal","allOf":[{"$ref":"#/components/schemas/mk20.Products"}]}}},"address.Address":{"type":"object"},"mk20.DataSource":{"type":"object","properties":{"format":{"description":"Format defines the format of the piece data, which can include CAR, Aggregate, or Raw formats.","allOf":[{"$ref":"#/components/schemas/mk20.PieceDataFormat"}]},"piece_cid":{"description":"PieceCID represents the unique identifier (pieceCID V2) for a piece of data, stored as a CID object.","allOf":[{"$ref":"#/components/schemas/cid.Cid"}]},"source_aggregate":{"description":"SourceAggregate represents an aggregated source, comprising multiple data sources as pieces.","allOf":[{"$ref":"#/components/schemas/mk20.DataSourceAggregate"}]},"source_http":{"description":"SourceHTTP represents the HTTP-based source of piece data within a deal, including raw size and URLs for retrieval.","allOf":[{"$ref":"#/components/schemas/mk20.DataSourceHTTP"}]},"source_httpput":{"description":"SourceHTTPPut // allow clients to push piece data after deal accepted, sort of like offline import","allOf":[{"$ref":"#/components/schemas/mk20.DataSourceHttpPut"}]},"source_offline":{"description":"SourceOffline defines the data source for offline pieces, including raw size information.","allOf":[{"$ref":"#/components/schemas/mk20.DataSourceOffline"}]}}},"mk20.PieceDataFormat":{"type":"object","properties":{"aggregate":{"description":"Aggregate holds a reference to the aggregated format of piece data.","allOf":[{"$ref":"#/components/schemas/mk20.FormatAggregate"}]},"car":{"description":"Car represents the optional CAR file format, including its metadata and versioning details.","allOf":[{"$ref":"#/components/schemas/mk20.FormatCar"}]},"raw":{"description":"Raw represents the raw format of the piece data, encapsulated as bytes.","allOf":[{"$ref":"#/components/schemas/mk20.FormatBytes"}]}}},"mk20.FormatAggregate":{"type":"object","properties":{"sub":{"description":"Sub holds a slice of DataSource, representing details of sub pieces aggregated under this format.\nThe order must be same as segment index to avoid incorrect indexing of sub pieces in an aggregate","type":"array","items":{"$ref":"#/components/schemas/mk20.DataSource"}},"type":{"description":"Type specifies the type of aggregation for data pieces, represented by an AggregateType value.","allOf":[{"$ref":"#/components/schemas/mk20.AggregateType"}]}}},"mk20.AggregateType":{"type":"integer","enum":[0,1]},"mk20.FormatCar":{"type":"object"},"mk20.FormatBytes":{"type":"object"},"cid.Cid":{"type":"object"},"mk20.DataSourceAggregate":{"type":"object","properties":{"pieces":{"type":"array","items":{"$ref":"#/components/schemas/mk20.DataSource"}}}},"mk20.DataSourceHTTP":{"type":"object","properties":{"urls":{"description":"URLs lists the HTTP endpoints where the piece data can be fetched.","type":"array","items":{"$ref":"#/components/schemas/mk20.HttpUrl"}}}},"mk20.HttpUrl":{"type":"object","properties":{"fallback":{"description":"Fallback indicates whether this URL serves as a fallback option when other URLs fail.","type":"boolean"},"headers":{"description":"HTTPHeaders represents the HTTP headers associated with the URL.","allOf":[{"$ref":"#/components/schemas/http.Header"}]},"priority":{"description":"Priority indicates the order preference for using the URL in requests, with lower values having higher priority.","type":"integer"},"url":{"description":"URL specifies the HTTP endpoint where the piece data can be fetched.","type":"string"}}},"http.Header":{"type":"object","additionalProperties":{"type":"array","items":{"type":"string"}}},"mk20.DataSourceHttpPut":{"type":"object"},"mk20.DataSourceOffline":{"type":"object"},"mk20.Products":{"type":"object","properties":{"ddo_v1":{"description":"DDOV1 represents a product v1 configuration for Direct Data Onboarding (DDO)","allOf":[{"$ref":"#/components/schemas/mk20.DDOV1"}]},"pdp_v1":{"description":"PDPV1 represents product-specific configuration for PDP version 1 deals.","allOf":[{"$ref":"#/components/schemas/mk20.PDPV1"}]},"retrieval_v1":{"description":"RetrievalV1 represents configuration for retrieval settings in the system, including indexing and announcement flags.","allOf":[{"$ref":"#/components/schemas/mk20.RetrievalV1"}]}}},"mk20.DDOV1":{"type":"object","properties":{"allocation_id":{"description":"AllocationId represents an aggregated allocation identifier for the deal.","allOf":[{"$ref":"#/components/schemas/github_com_filecoin-project_go-state-types_builtin_v16_verifreg.AllocationId"}]},"contract_address":{"description":"ContractAddress specifies the address of the contract governing the deal","type":"string"},"contract_verify_method":{"description":"ContractDealIDMethod specifies the method name to verify the deal and retrieve the deal ID for a contract","type":"string"},"contract_verify_method_params":{"description":"ContractDealIDMethodParams represents encoded parameters for the contract verify method if required by the contract","type":"array","items":{"type":"integer"}},"duration":{"description":"Duration represents the deal duration in epochs. This value is ignored for the deal with allocationID.\nIt must be at least 518400","type":"integer"},"notification_address":{"description":"NotificationAddress specifies the address to which notifications will be relayed to when sector is activated","type":"string"},"notification_payload":{"description":"NotificationPayload holds the notification data typically in a serialized byte array format.","type":"array","items":{"type":"integer"}},"piece_manager":{"description":"Actor providing AuthorizeMessage (like f1/f3 wallet) able to authorize actions such as managing ACLs","allOf":[{"$ref":"#/components/schemas/address.Address"}]},"provider":{"description":"Provider specifies the address of the provider","allOf":[{"$ref":"#/components/schemas/address.Address"}]}}},"github_com_filecoin-project_go-state-types_builtin_v16_verifreg.AllocationId":{"type":"integer","enum":[0]},"mk20.PDPV1":{"type":"object","properties":{"add_root":{"description":"AddRoot indicated that this deal is meant to add root to a given ProofSet. ProofSetID must be defined.","type":"boolean"},"create_proof_set":{"description":"CreateProofSet indicated that this deal is meant to create a new ProofSet for the client by storage provider.","type":"boolean"},"delete_proof_set":{"description":"DeleteProofSet indicated that this deal is meant to delete an existing ProofSet created by SP for the client.\nProofSetID must be defined.","type":"boolean"},"delete_root":{"description":"DeleteRoot indicates whether the root of the data should be deleted. ProofSetID must be defined.","type":"boolean"},"extra_data":{"description":"ExtraData can be used to send additional information to service contract when Verifier action like AddRoot, DeleteRoot etc. are performed.","type":"array","items":{"type":"integer"}},"proof_set_id":{"description":"ProofSetID is PDP verified contract proofset ID. It must be defined for all deals except when CreateProofSet is true.","type":"integer"},"record_keeper":{"description":"RecordKeeper specifies the record keeper contract address for the new PDP proofset.","type":"string"},"root_ids":{"description":"RootIDs is a list of root ids in a proof set.","type":"array","items":{"type":"integer"}}}},"mk20.RetrievalV1":{"type":"object","properties":{"announce_payload":{"description":"AnnouncePayload indicates whether the payload should be announced to IPNI.","type":"boolean"},"announce_piece":{"description":"AnnouncePiece indicates whether the piece information should be announced to IPNI.","type":"boolean"},"indexing":{"description":"Indexing indicates if the deal is to be indexed in the provider's system to support CIDs based retrieval","type":"boolean"}}}}}} ``` ## Finalizes the serial upload process > Finalizes the serial upload process once data has been uploaded ```json {"openapi":"3.1.1","info":{"title":"Curio Market 2.0 API","version":"0.0.1"},"paths":{"/upload/{id}":{"post":{"description":"Finalizes the serial upload process once data has been uploaded","summary":"Finalizes the serial upload process","parameters":[{"schema":{"type":"string"},"description":"id","name":"id","in":"path","required":true}],"responses":{"200":{"description":"Ok represents a successful operation with an HTTP status code of 200","content":{"application/json":{"schema":{"$ref":"#/components/schemas/mk20.DealCode"}}}},"400":{"description":"Bad Request - Invalid input or validation error","content":{"application/json":{"schema":{"type":"string"}}}},"404":{"description":"ErrDealNotFound indicates that the specified deal could not be found, corresponding to the HTTP status code 404","content":{"application/json":{"schema":{"$ref":"#/components/schemas/mk20.DealCode"}}}},"422":{"description":"ErrUnsupportedDataSource indicates the specified data source is not supported or disabled for use in the current context","content":{"application/json":{"schema":{"$ref":"#/components/schemas/mk20.DealCode"}}}},"423":{"description":"ErrUnsupportedProduct indicates that the requested product is not supported by the provider","content":{"application/json":{"schema":{"$ref":"#/components/schemas/mk20.DealCode"}}}},"424":{"description":"ErrProductNotEnabled indicates that the requested product is not enabled on the provider","content":{"application/json":{"schema":{"$ref":"#/components/schemas/mk20.DealCode"}}}},"425":{"description":"ErrProductValidationFailed indicates a failure during product-specific validation due to invalid or missing data","content":{"application/json":{"schema":{"$ref":"#/components/schemas/mk20.DealCode"}}}},"426":{"description":"ErrDealRejectedByMarket indicates that a proposed deal was rejected by the market for not meeting its acceptance criteria or rules","content":{"application/json":{"schema":{"$ref":"#/components/schemas/mk20.DealCode"}}}},"429":{"description":"ErrServiceOverloaded indicates that the service is overloaded and cannot process the request at the moment","content":{"application/json":{"schema":{"$ref":"#/components/schemas/mk20.DealCode"}}}},"430":{"description":"ErrMalformedDataSource indicates that the provided data source is incorrectly formatted or contains invalid data","content":{"application/json":{"schema":{"$ref":"#/components/schemas/mk20.DealCode"}}}},"440":{"description":"ErrMarketNotEnabled indicates that the market is not enabled for the requested operation","content":{"application/json":{"schema":{"$ref":"#/components/schemas/mk20.DealCode"}}}},"441":{"description":"ErrDurationTooShort indicates that the provided duration value does not meet the minimum required threshold","content":{"application/json":{"schema":{"$ref":"#/components/schemas/mk20.DealCode"}}}},"500":{"description":"ErrServerInternalError indicates an internal server error with a corresponding error code of 500","content":{"application/json":{"schema":{"$ref":"#/components/schemas/mk20.DealCode"}}}},"503":{"description":"ErrServiceMaintenance indicates that the service is temporarily unavailable due to maintenance, corresponding to HTTP status code 503","content":{"application/json":{"schema":{"$ref":"#/components/schemas/mk20.DealCode"}}}}},"requestBody":{"content":{"application/json":{"schema":{"$ref":"#/components/schemas/mk20.Deal"}}},"description":"mk20.deal in json format"}}}},"components":{"schemas":{"mk20.DealCode":{"type":"integer","enum":[200,401,400,404,430,422,423,424,425,426,500,503,429,440,441]},"mk20.Deal":{"type":"object","properties":{"client":{"description":"Client wallet for the deal","allOf":[{"$ref":"#/components/schemas/address.Address"}]},"data":{"description":"Data represents the source of piece data and associated metadata.","allOf":[{"$ref":"#/components/schemas/mk20.DataSource"}]},"identifier":{"description":"Identifier represents a unique identifier for the deal in UUID format.","type":"array","items":{"type":"integer"}},"products":{"description":"Products represents a collection of product-specific information associated with a deal","allOf":[{"$ref":"#/components/schemas/mk20.Products"}]}}},"address.Address":{"type":"object"},"mk20.DataSource":{"type":"object","properties":{"format":{"description":"Format defines the format of the piece data, which can include CAR, Aggregate, or Raw formats.","allOf":[{"$ref":"#/components/schemas/mk20.PieceDataFormat"}]},"piece_cid":{"description":"PieceCID represents the unique identifier (pieceCID V2) for a piece of data, stored as a CID object.","allOf":[{"$ref":"#/components/schemas/cid.Cid"}]},"source_aggregate":{"description":"SourceAggregate represents an aggregated source, comprising multiple data sources as pieces.","allOf":[{"$ref":"#/components/schemas/mk20.DataSourceAggregate"}]},"source_http":{"description":"SourceHTTP represents the HTTP-based source of piece data within a deal, including raw size and URLs for retrieval.","allOf":[{"$ref":"#/components/schemas/mk20.DataSourceHTTP"}]},"source_httpput":{"description":"SourceHTTPPut // allow clients to push piece data after deal accepted, sort of like offline import","allOf":[{"$ref":"#/components/schemas/mk20.DataSourceHttpPut"}]},"source_offline":{"description":"SourceOffline defines the data source for offline pieces, including raw size information.","allOf":[{"$ref":"#/components/schemas/mk20.DataSourceOffline"}]}}},"mk20.PieceDataFormat":{"type":"object","properties":{"aggregate":{"description":"Aggregate holds a reference to the aggregated format of piece data.","allOf":[{"$ref":"#/components/schemas/mk20.FormatAggregate"}]},"car":{"description":"Car represents the optional CAR file format, including its metadata and versioning details.","allOf":[{"$ref":"#/components/schemas/mk20.FormatCar"}]},"raw":{"description":"Raw represents the raw format of the piece data, encapsulated as bytes.","allOf":[{"$ref":"#/components/schemas/mk20.FormatBytes"}]}}},"mk20.FormatAggregate":{"type":"object","properties":{"sub":{"description":"Sub holds a slice of DataSource, representing details of sub pieces aggregated under this format.\nThe order must be same as segment index to avoid incorrect indexing of sub pieces in an aggregate","type":"array","items":{"$ref":"#/components/schemas/mk20.DataSource"}},"type":{"description":"Type specifies the type of aggregation for data pieces, represented by an AggregateType value.","allOf":[{"$ref":"#/components/schemas/mk20.AggregateType"}]}}},"mk20.AggregateType":{"type":"integer","enum":[0,1]},"mk20.FormatCar":{"type":"object"},"mk20.FormatBytes":{"type":"object"},"cid.Cid":{"type":"object"},"mk20.DataSourceAggregate":{"type":"object","properties":{"pieces":{"type":"array","items":{"$ref":"#/components/schemas/mk20.DataSource"}}}},"mk20.DataSourceHTTP":{"type":"object","properties":{"urls":{"description":"URLs lists the HTTP endpoints where the piece data can be fetched.","type":"array","items":{"$ref":"#/components/schemas/mk20.HttpUrl"}}}},"mk20.HttpUrl":{"type":"object","properties":{"fallback":{"description":"Fallback indicates whether this URL serves as a fallback option when other URLs fail.","type":"boolean"},"headers":{"description":"HTTPHeaders represents the HTTP headers associated with the URL.","allOf":[{"$ref":"#/components/schemas/http.Header"}]},"priority":{"description":"Priority indicates the order preference for using the URL in requests, with lower values having higher priority.","type":"integer"},"url":{"description":"URL specifies the HTTP endpoint where the piece data can be fetched.","type":"string"}}},"http.Header":{"type":"object","additionalProperties":{"type":"array","items":{"type":"string"}}},"mk20.DataSourceHttpPut":{"type":"object"},"mk20.DataSourceOffline":{"type":"object"},"mk20.Products":{"type":"object","properties":{"ddo_v1":{"description":"DDOV1 represents a product v1 configuration for Direct Data Onboarding (DDO)","allOf":[{"$ref":"#/components/schemas/mk20.DDOV1"}]},"pdp_v1":{"description":"PDPV1 represents product-specific configuration for PDP version 1 deals.","allOf":[{"$ref":"#/components/schemas/mk20.PDPV1"}]},"retrieval_v1":{"description":"RetrievalV1 represents configuration for retrieval settings in the system, including indexing and announcement flags.","allOf":[{"$ref":"#/components/schemas/mk20.RetrievalV1"}]}}},"mk20.DDOV1":{"type":"object","properties":{"allocation_id":{"description":"AllocationId represents an aggregated allocation identifier for the deal.","allOf":[{"$ref":"#/components/schemas/github_com_filecoin-project_go-state-types_builtin_v16_verifreg.AllocationId"}]},"contract_address":{"description":"ContractAddress specifies the address of the contract governing the deal","type":"string"},"contract_verify_method":{"description":"ContractDealIDMethod specifies the method name to verify the deal and retrieve the deal ID for a contract","type":"string"},"contract_verify_method_params":{"description":"ContractDealIDMethodParams represents encoded parameters for the contract verify method if required by the contract","type":"array","items":{"type":"integer"}},"duration":{"description":"Duration represents the deal duration in epochs. This value is ignored for the deal with allocationID.\nIt must be at least 518400","type":"integer"},"notification_address":{"description":"NotificationAddress specifies the address to which notifications will be relayed to when sector is activated","type":"string"},"notification_payload":{"description":"NotificationPayload holds the notification data typically in a serialized byte array format.","type":"array","items":{"type":"integer"}},"piece_manager":{"description":"Actor providing AuthorizeMessage (like f1/f3 wallet) able to authorize actions such as managing ACLs","allOf":[{"$ref":"#/components/schemas/address.Address"}]},"provider":{"description":"Provider specifies the address of the provider","allOf":[{"$ref":"#/components/schemas/address.Address"}]}}},"github_com_filecoin-project_go-state-types_builtin_v16_verifreg.AllocationId":{"type":"integer","enum":[0]},"mk20.PDPV1":{"type":"object","properties":{"add_root":{"description":"AddRoot indicated that this deal is meant to add root to a given ProofSet. ProofSetID must be defined.","type":"boolean"},"create_proof_set":{"description":"CreateProofSet indicated that this deal is meant to create a new ProofSet for the client by storage provider.","type":"boolean"},"delete_proof_set":{"description":"DeleteProofSet indicated that this deal is meant to delete an existing ProofSet created by SP for the client.\nProofSetID must be defined.","type":"boolean"},"delete_root":{"description":"DeleteRoot indicates whether the root of the data should be deleted. ProofSetID must be defined.","type":"boolean"},"extra_data":{"description":"ExtraData can be used to send additional information to service contract when Verifier action like AddRoot, DeleteRoot etc. are performed.","type":"array","items":{"type":"integer"}},"proof_set_id":{"description":"ProofSetID is PDP verified contract proofset ID. It must be defined for all deals except when CreateProofSet is true.","type":"integer"},"record_keeper":{"description":"RecordKeeper specifies the record keeper contract address for the new PDP proofset.","type":"string"},"root_ids":{"description":"RootIDs is a list of root ids in a proof set.","type":"array","items":{"type":"integer"}}}},"mk20.RetrievalV1":{"type":"object","properties":{"announce_payload":{"description":"AnnouncePayload indicates whether the payload should be announced to IPNI.","type":"boolean"},"announce_piece":{"description":"AnnouncePiece indicates whether the piece information should be announced to IPNI.","type":"boolean"},"indexing":{"description":"Indexing indicates if the deal is to be indexed in the provider's system to support CIDs based retrieval","type":"boolean"}}}}}} ``` ## Upload the deal data > Allows uploading data for deals in a single stream. Suitable for small deals. ```json {"openapi":"3.1.1","info":{"title":"Curio Market 2.0 API","version":"0.0.1"},"paths":{"/upload/{id}":{"put":{"description":"Allows uploading data for deals in a single stream. Suitable for small deals.","summary":"Upload the deal data","parameters":[{"schema":{"type":"string"},"description":"id","name":"id","in":"path","required":true}],"responses":{"200":{"description":"UploadOk indicates a successful upload operation, represented by the HTTP status code 200","content":{"application/json":{"schema":{"$ref":"#/components/schemas/mk20.UploadCode"}}}},"400":{"description":"Bad Request - Invalid input or validation error","content":{"application/json":{"schema":{"type":"string"}}}},"404":{"description":"UploadStartCodeDealNotFound represents a 404 status indicating the deal was not found during the upload start process","content":{"application/json":{"schema":{"$ref":"#/components/schemas/mk20.UploadStartCode"}}}},"500":{"description":"UploadServerError indicates a server-side error occurred during the upload process, represented by the HTTP status code 500","content":{"application/json":{"schema":{"$ref":"#/components/schemas/mk20.UploadCode"}}}}},"requestBody":{"content":{"application/json":{"schema":{"type":"array","items":{"type":"integer"}}}},"description":"raw binary","required":true}}}},"components":{"schemas":{"mk20.UploadCode":{"type":"integer","enum":[200,400,404,409,500]},"mk20.UploadStartCode":{"type":"integer","enum":[200,400,404,409,500]}}}} ``` ## Finalizes the upload process > Finalizes the upload process once all the chunks are uploaded. ```json {"openapi":"3.1.1","info":{"title":"Curio Market 2.0 API","version":"0.0.1"},"paths":{"/uploads/finalize/{id}":{"post":{"description":"Finalizes the upload process once all the chunks are uploaded.","summary":"Finalizes the upload process","parameters":[{"schema":{"type":"string"},"description":"id","name":"id","in":"path","required":true}],"responses":{"200":{"description":"Ok represents a successful operation with an HTTP status code of 200","content":{"application/json":{"schema":{"$ref":"#/components/schemas/mk20.DealCode"}}}},"400":{"description":"Bad Request - Invalid input or validation error","content":{"application/json":{"schema":{"type":"string"}}}},"404":{"description":"ErrDealNotFound indicates that the specified deal could not be found, corresponding to the HTTP status code 404","content":{"application/json":{"schema":{"$ref":"#/components/schemas/mk20.DealCode"}}}},"422":{"description":"ErrUnsupportedDataSource indicates the specified data source is not supported or disabled for use in the current context","content":{"application/json":{"schema":{"$ref":"#/components/schemas/mk20.DealCode"}}}},"423":{"description":"ErrUnsupportedProduct indicates that the requested product is not supported by the provider","content":{"application/json":{"schema":{"$ref":"#/components/schemas/mk20.DealCode"}}}},"424":{"description":"ErrProductNotEnabled indicates that the requested product is not enabled on the provider","content":{"application/json":{"schema":{"$ref":"#/components/schemas/mk20.DealCode"}}}},"425":{"description":"ErrProductValidationFailed indicates a failure during product-specific validation due to invalid or missing data","content":{"application/json":{"schema":{"$ref":"#/components/schemas/mk20.DealCode"}}}},"426":{"description":"ErrDealRejectedByMarket indicates that a proposed deal was rejected by the market for not meeting its acceptance criteria or rules","content":{"application/json":{"schema":{"$ref":"#/components/schemas/mk20.DealCode"}}}},"429":{"description":"ErrServiceOverloaded indicates that the service is overloaded and cannot process the request at the moment","content":{"application/json":{"schema":{"$ref":"#/components/schemas/mk20.DealCode"}}}},"430":{"description":"ErrMalformedDataSource indicates that the provided data source is incorrectly formatted or contains invalid data","content":{"application/json":{"schema":{"$ref":"#/components/schemas/mk20.DealCode"}}}},"440":{"description":"ErrMarketNotEnabled indicates that the market is not enabled for the requested operation","content":{"application/json":{"schema":{"$ref":"#/components/schemas/mk20.DealCode"}}}},"441":{"description":"ErrDurationTooShort indicates that the provided duration value does not meet the minimum required threshold","content":{"application/json":{"schema":{"$ref":"#/components/schemas/mk20.DealCode"}}}},"500":{"description":"ErrServerInternalError indicates an internal server error with a corresponding error code of 500","content":{"application/json":{"schema":{"$ref":"#/components/schemas/mk20.DealCode"}}}},"503":{"description":"ErrServiceMaintenance indicates that the service is temporarily unavailable due to maintenance, corresponding to HTTP status code 503","content":{"application/json":{"schema":{"$ref":"#/components/schemas/mk20.DealCode"}}}}},"requestBody":{"content":{"application/json":{"schema":{"$ref":"#/components/schemas/mk20.Deal"}}},"description":"mk20.deal in json format"}}}},"components":{"schemas":{"mk20.DealCode":{"type":"integer","enum":[200,401,400,404,430,422,423,424,425,426,500,503,429,440,441]},"mk20.Deal":{"type":"object","properties":{"client":{"description":"Client wallet for the deal","allOf":[{"$ref":"#/components/schemas/address.Address"}]},"data":{"description":"Data represents the source of piece data and associated metadata.","allOf":[{"$ref":"#/components/schemas/mk20.DataSource"}]},"identifier":{"description":"Identifier represents a unique identifier for the deal in UUID format.","type":"array","items":{"type":"integer"}},"products":{"description":"Products represents a collection of product-specific information associated with a deal","allOf":[{"$ref":"#/components/schemas/mk20.Products"}]}}},"address.Address":{"type":"object"},"mk20.DataSource":{"type":"object","properties":{"format":{"description":"Format defines the format of the piece data, which can include CAR, Aggregate, or Raw formats.","allOf":[{"$ref":"#/components/schemas/mk20.PieceDataFormat"}]},"piece_cid":{"description":"PieceCID represents the unique identifier (pieceCID V2) for a piece of data, stored as a CID object.","allOf":[{"$ref":"#/components/schemas/cid.Cid"}]},"source_aggregate":{"description":"SourceAggregate represents an aggregated source, comprising multiple data sources as pieces.","allOf":[{"$ref":"#/components/schemas/mk20.DataSourceAggregate"}]},"source_http":{"description":"SourceHTTP represents the HTTP-based source of piece data within a deal, including raw size and URLs for retrieval.","allOf":[{"$ref":"#/components/schemas/mk20.DataSourceHTTP"}]},"source_httpput":{"description":"SourceHTTPPut // allow clients to push piece data after deal accepted, sort of like offline import","allOf":[{"$ref":"#/components/schemas/mk20.DataSourceHttpPut"}]},"source_offline":{"description":"SourceOffline defines the data source for offline pieces, including raw size information.","allOf":[{"$ref":"#/components/schemas/mk20.DataSourceOffline"}]}}},"mk20.PieceDataFormat":{"type":"object","properties":{"aggregate":{"description":"Aggregate holds a reference to the aggregated format of piece data.","allOf":[{"$ref":"#/components/schemas/mk20.FormatAggregate"}]},"car":{"description":"Car represents the optional CAR file format, including its metadata and versioning details.","allOf":[{"$ref":"#/components/schemas/mk20.FormatCar"}]},"raw":{"description":"Raw represents the raw format of the piece data, encapsulated as bytes.","allOf":[{"$ref":"#/components/schemas/mk20.FormatBytes"}]}}},"mk20.FormatAggregate":{"type":"object","properties":{"sub":{"description":"Sub holds a slice of DataSource, representing details of sub pieces aggregated under this format.\nThe order must be same as segment index to avoid incorrect indexing of sub pieces in an aggregate","type":"array","items":{"$ref":"#/components/schemas/mk20.DataSource"}},"type":{"description":"Type specifies the type of aggregation for data pieces, represented by an AggregateType value.","allOf":[{"$ref":"#/components/schemas/mk20.AggregateType"}]}}},"mk20.AggregateType":{"type":"integer","enum":[0,1]},"mk20.FormatCar":{"type":"object"},"mk20.FormatBytes":{"type":"object"},"cid.Cid":{"type":"object"},"mk20.DataSourceAggregate":{"type":"object","properties":{"pieces":{"type":"array","items":{"$ref":"#/components/schemas/mk20.DataSource"}}}},"mk20.DataSourceHTTP":{"type":"object","properties":{"urls":{"description":"URLs lists the HTTP endpoints where the piece data can be fetched.","type":"array","items":{"$ref":"#/components/schemas/mk20.HttpUrl"}}}},"mk20.HttpUrl":{"type":"object","properties":{"fallback":{"description":"Fallback indicates whether this URL serves as a fallback option when other URLs fail.","type":"boolean"},"headers":{"description":"HTTPHeaders represents the HTTP headers associated with the URL.","allOf":[{"$ref":"#/components/schemas/http.Header"}]},"priority":{"description":"Priority indicates the order preference for using the URL in requests, with lower values having higher priority.","type":"integer"},"url":{"description":"URL specifies the HTTP endpoint where the piece data can be fetched.","type":"string"}}},"http.Header":{"type":"object","additionalProperties":{"type":"array","items":{"type":"string"}}},"mk20.DataSourceHttpPut":{"type":"object"},"mk20.DataSourceOffline":{"type":"object"},"mk20.Products":{"type":"object","properties":{"ddo_v1":{"description":"DDOV1 represents a product v1 configuration for Direct Data Onboarding (DDO)","allOf":[{"$ref":"#/components/schemas/mk20.DDOV1"}]},"pdp_v1":{"description":"PDPV1 represents product-specific configuration for PDP version 1 deals.","allOf":[{"$ref":"#/components/schemas/mk20.PDPV1"}]},"retrieval_v1":{"description":"RetrievalV1 represents configuration for retrieval settings in the system, including indexing and announcement flags.","allOf":[{"$ref":"#/components/schemas/mk20.RetrievalV1"}]}}},"mk20.DDOV1":{"type":"object","properties":{"allocation_id":{"description":"AllocationId represents an aggregated allocation identifier for the deal.","allOf":[{"$ref":"#/components/schemas/github_com_filecoin-project_go-state-types_builtin_v16_verifreg.AllocationId"}]},"contract_address":{"description":"ContractAddress specifies the address of the contract governing the deal","type":"string"},"contract_verify_method":{"description":"ContractDealIDMethod specifies the method name to verify the deal and retrieve the deal ID for a contract","type":"string"},"contract_verify_method_params":{"description":"ContractDealIDMethodParams represents encoded parameters for the contract verify method if required by the contract","type":"array","items":{"type":"integer"}},"duration":{"description":"Duration represents the deal duration in epochs. This value is ignored for the deal with allocationID.\nIt must be at least 518400","type":"integer"},"notification_address":{"description":"NotificationAddress specifies the address to which notifications will be relayed to when sector is activated","type":"string"},"notification_payload":{"description":"NotificationPayload holds the notification data typically in a serialized byte array format.","type":"array","items":{"type":"integer"}},"piece_manager":{"description":"Actor providing AuthorizeMessage (like f1/f3 wallet) able to authorize actions such as managing ACLs","allOf":[{"$ref":"#/components/schemas/address.Address"}]},"provider":{"description":"Provider specifies the address of the provider","allOf":[{"$ref":"#/components/schemas/address.Address"}]}}},"github_com_filecoin-project_go-state-types_builtin_v16_verifreg.AllocationId":{"type":"integer","enum":[0]},"mk20.PDPV1":{"type":"object","properties":{"add_root":{"description":"AddRoot indicated that this deal is meant to add root to a given ProofSet. ProofSetID must be defined.","type":"boolean"},"create_proof_set":{"description":"CreateProofSet indicated that this deal is meant to create a new ProofSet for the client by storage provider.","type":"boolean"},"delete_proof_set":{"description":"DeleteProofSet indicated that this deal is meant to delete an existing ProofSet created by SP for the client.\nProofSetID must be defined.","type":"boolean"},"delete_root":{"description":"DeleteRoot indicates whether the root of the data should be deleted. ProofSetID must be defined.","type":"boolean"},"extra_data":{"description":"ExtraData can be used to send additional information to service contract when Verifier action like AddRoot, DeleteRoot etc. are performed.","type":"array","items":{"type":"integer"}},"proof_set_id":{"description":"ProofSetID is PDP verified contract proofset ID. It must be defined for all deals except when CreateProofSet is true.","type":"integer"},"record_keeper":{"description":"RecordKeeper specifies the record keeper contract address for the new PDP proofset.","type":"string"},"root_ids":{"description":"RootIDs is a list of root ids in a proof set.","type":"array","items":{"type":"integer"}}}},"mk20.RetrievalV1":{"type":"object","properties":{"announce_payload":{"description":"AnnouncePayload indicates whether the payload should be announced to IPNI.","type":"boolean"},"announce_piece":{"description":"AnnouncePiece indicates whether the piece information should be announced to IPNI.","type":"boolean"},"indexing":{"description":"Indexing indicates if the deal is to be indexed in the provider's system to support CIDs based retrieval","type":"boolean"}}}}}} ``` ## Status of deal upload > Return a json struct detailing the current status of a deal upload. ```json {"openapi":"3.1.1","info":{"title":"Curio Market 2.0 API","version":"0.0.1"},"paths":{"/uploads/{id}":{"get":{"description":"Return a json struct detailing the current status of a deal upload.","summary":"Status of deal upload","parameters":[{"schema":{"type":"string"},"description":"id","name":"id","in":"path","required":true}],"responses":{"200":{"description":"The status of a file upload process, including progress and missing chunks","content":{"application/json":{"schema":{"$ref":"#/components/schemas/mk20.UploadStatus"}}}},"400":{"description":"Bad Request - Invalid input or validation error","content":{"application/json":{"schema":{"type":"string"}}}},"404":{"description":"UploadStatusCodeDealNotFound indicates that the requested deal was not found, corresponding to status code 404","content":{"application/json":{"schema":{"$ref":"#/components/schemas/mk20.UploadStatusCode"}}}},"425":{"description":"UploadStatusCodeUploadNotStarted indicates that the upload process has not started yet","content":{"application/json":{"schema":{"$ref":"#/components/schemas/mk20.UploadStatusCode"}}}},"500":{"description":"UploadStatusCodeServerError indicates an internal server error occurred during the upload process, corresponding to status code 500","content":{"application/json":{"schema":{"$ref":"#/components/schemas/mk20.UploadStatusCode"}}}}}}}},"components":{"schemas":{"mk20.UploadStatus":{"type":"object","properties":{"missing":{"description":"Missing represents the number of chunks that are not yet uploaded.","type":"integer"},"missing_chunks":{"description":"MissingChunks is a slice containing the indices of missing chunks.","type":"array","items":{"type":"integer"}},"total_chunks":{"description":"TotalChunks represents the total number of chunks required for the upload.","type":"integer"},"uploaded":{"description":"Uploaded represents the number of chunks successfully uploaded.","type":"integer"},"uploaded_chunks":{"description":"UploadedChunks is a slice containing the indices of successfully uploaded chunks.","type":"array","items":{"type":"integer"}}}},"mk20.UploadStatusCode":{"type":"integer","enum":[200,404,425,500]}}}} ``` ## Starts the upload process > Initializes the upload for a deal. Each upload must be initialized before chunks can be uploaded for a deal. ```json {"openapi":"3.1.1","info":{"title":"Curio Market 2.0 API","version":"0.0.1"},"paths":{"/uploads/{id}":{"post":{"description":"Initializes the upload for a deal. Each upload must be initialized before chunks can be uploaded for a deal.","summary":"Starts the upload process","parameters":[{"schema":{"type":"string"},"description":"id","name":"id","in":"path","required":true}],"responses":{"200":{"description":"UploadStartCodeOk indicates a successful upload start request with status code 200","content":{"application/json":{"schema":{"$ref":"#/components/schemas/mk20.UploadStartCode"}}}},"400":{"description":"Bad Request - Invalid input or validation error","content":{"application/json":{"schema":{"type":"string"}}}},"404":{"description":"UploadStartCodeDealNotFound represents a 404 status indicating the deal was not found during the upload start process","content":{"application/json":{"schema":{"$ref":"#/components/schemas/mk20.UploadStartCode"}}}},"409":{"description":"UploadStartCodeAlreadyStarted indicates that the upload process has already been initiated and cannot be started again","content":{"application/json":{"schema":{"$ref":"#/components/schemas/mk20.UploadStartCode"}}}},"500":{"description":"UploadStartCodeServerError indicates an error occurred on the server while processing an upload start request","content":{"application/json":{"schema":{"$ref":"#/components/schemas/mk20.UploadStartCode"}}}}},"requestBody":{"content":{"application/json":{"schema":{"$ref":"#/components/schemas/mk20.StartUpload"}}},"description":"Metadata for initiating an upload operation","required":true}}}},"components":{"schemas":{"mk20.UploadStartCode":{"type":"integer","enum":[200,400,404,409,500]},"mk20.StartUpload":{"type":"object","properties":{"chunk_size":{"description":"ChunkSize defines the size of each data chunk to be used during the upload process.","type":"integer"},"raw_size":{"description":"RawSize indicates the total size of the data to be uploaded in bytes.","type":"integer"}}}}}} ``` ## Upload a file chunk > Allows uploading chunks for a deal file. Method can be called in parallel to speed up uploads. ```json {"openapi":"3.1.1","info":{"title":"Curio Market 2.0 API","version":"0.0.1"},"paths":{"/uploads/{id}/{chunkNum}":{"put":{"description":"Allows uploading chunks for a deal file. Method can be called in parallel to speed up uploads.","summary":"Upload a file chunk","parameters":[{"schema":{"type":"string"},"description":"id","name":"id","in":"path","required":true},{"schema":{"type":"string"},"description":"chunkNum","name":"chunkNum","in":"path","required":true}],"responses":{"200":{"description":"UploadOk indicates a successful upload operation, represented by the HTTP status code 200","content":{"application/json":{"schema":{"$ref":"#/components/schemas/mk20.UploadCode"}}}},"400":{"description":"Bad Request - Invalid input or validation error","content":{"application/json":{"schema":{"type":"string"}}}},"404":{"description":"UploadNotFound represents an error where the requested upload chunk could not be found, typically corresponding to HTTP status 404","content":{"application/json":{"schema":{"$ref":"#/components/schemas/mk20.UploadCode"}}}},"409":{"description":"UploadChunkAlreadyUploaded indicates that the chunk has already been uploaded and cannot be re-uploaded","content":{"application/json":{"schema":{"$ref":"#/components/schemas/mk20.UploadCode"}}}},"500":{"description":"UploadServerError indicates a server-side error occurred during the upload process, represented by the HTTP status code 500","content":{"application/json":{"schema":{"$ref":"#/components/schemas/mk20.UploadCode"}}}}},"requestBody":{"content":{"application/json":{"schema":{"type":"array","items":{"type":"integer"}}}},"description":"raw binary","required":true}}}},"components":{"schemas":{"mk20.UploadCode":{"type":"integer","enum":[200,400,404,409,500]}}}} ``` ## The address.Address object ```json {"openapi":"3.1.1","info":{"title":"Curio Market 2.0 API","version":"0.0.1"},"components":{"schemas":{"address.Address":{"type":"object"}}}} ``` ## The cid.Cid object ```json {"openapi":"3.1.1","info":{"title":"Curio Market 2.0 API","version":"0.0.1"},"components":{"schemas":{"cid.Cid":{"type":"object"}}}} ``` ## The github\_com\_filecoin-project\_go-state-types\_builtin\_v16\_verifreg.AllocationId object ```json {"openapi":"3.1.1","info":{"title":"Curio Market 2.0 API","version":"0.0.1"},"components":{"schemas":{"github_com_filecoin-project_go-state-types_builtin_v16_verifreg.AllocationId":{"type":"integer","enum":[0]}}}} ``` ## The http.Header object ```json {"openapi":"3.1.1","info":{"title":"Curio Market 2.0 API","version":"0.0.1"},"components":{"schemas":{"http.Header":{"type":"object","additionalProperties":{"type":"array","items":{"type":"string"}}}}}} ``` ## The mk20.AggregateType object ```json {"openapi":"3.1.1","info":{"title":"Curio Market 2.0 API","version":"0.0.1"},"components":{"schemas":{"mk20.AggregateType":{"type":"integer","enum":[0,1]}}}} ``` ## The mk20.DDOV1 object ```json {"openapi":"3.1.1","info":{"title":"Curio Market 2.0 API","version":"0.0.1"},"components":{"schemas":{"mk20.DDOV1":{"type":"object","properties":{"allocation_id":{"description":"AllocationId represents an aggregated allocation identifier for the deal.","allOf":[{"$ref":"#/components/schemas/github_com_filecoin-project_go-state-types_builtin_v16_verifreg.AllocationId"}]},"contract_address":{"description":"ContractAddress specifies the address of the contract governing the deal","type":"string"},"contract_verify_method":{"description":"ContractDealIDMethod specifies the method name to verify the deal and retrieve the deal ID for a contract","type":"string"},"contract_verify_method_params":{"description":"ContractDealIDMethodParams represents encoded parameters for the contract verify method if required by the contract","type":"array","items":{"type":"integer"}},"duration":{"description":"Duration represents the deal duration in epochs. This value is ignored for the deal with allocationID.\nIt must be at least 518400","type":"integer"},"notification_address":{"description":"NotificationAddress specifies the address to which notifications will be relayed to when sector is activated","type":"string"},"notification_payload":{"description":"NotificationPayload holds the notification data typically in a serialized byte array format.","type":"array","items":{"type":"integer"}},"piece_manager":{"description":"Actor providing AuthorizeMessage (like f1/f3 wallet) able to authorize actions such as managing ACLs","allOf":[{"$ref":"#/components/schemas/address.Address"}]},"provider":{"description":"Provider specifies the address of the provider","allOf":[{"$ref":"#/components/schemas/address.Address"}]}}},"github_com_filecoin-project_go-state-types_builtin_v16_verifreg.AllocationId":{"type":"integer","enum":[0]},"address.Address":{"type":"object"}}}} ``` ## The mk20.DataSource object ```json {"openapi":"3.1.1","info":{"title":"Curio Market 2.0 API","version":"0.0.1"},"components":{"schemas":{"mk20.DataSource":{"type":"object","properties":{"format":{"description":"Format defines the format of the piece data, which can include CAR, Aggregate, or Raw formats.","allOf":[{"$ref":"#/components/schemas/mk20.PieceDataFormat"}]},"piece_cid":{"description":"PieceCID represents the unique identifier (pieceCID V2) for a piece of data, stored as a CID object.","allOf":[{"$ref":"#/components/schemas/cid.Cid"}]},"source_aggregate":{"description":"SourceAggregate represents an aggregated source, comprising multiple data sources as pieces.","allOf":[{"$ref":"#/components/schemas/mk20.DataSourceAggregate"}]},"source_http":{"description":"SourceHTTP represents the HTTP-based source of piece data within a deal, including raw size and URLs for retrieval.","allOf":[{"$ref":"#/components/schemas/mk20.DataSourceHTTP"}]},"source_httpput":{"description":"SourceHTTPPut // allow clients to push piece data after deal accepted, sort of like offline import","allOf":[{"$ref":"#/components/schemas/mk20.DataSourceHttpPut"}]},"source_offline":{"description":"SourceOffline defines the data source for offline pieces, including raw size information.","allOf":[{"$ref":"#/components/schemas/mk20.DataSourceOffline"}]}}},"mk20.PieceDataFormat":{"type":"object","properties":{"aggregate":{"description":"Aggregate holds a reference to the aggregated format of piece data.","allOf":[{"$ref":"#/components/schemas/mk20.FormatAggregate"}]},"car":{"description":"Car represents the optional CAR file format, including its metadata and versioning details.","allOf":[{"$ref":"#/components/schemas/mk20.FormatCar"}]},"raw":{"description":"Raw represents the raw format of the piece data, encapsulated as bytes.","allOf":[{"$ref":"#/components/schemas/mk20.FormatBytes"}]}}},"mk20.FormatAggregate":{"type":"object","properties":{"sub":{"description":"Sub holds a slice of DataSource, representing details of sub pieces aggregated under this format.\nThe order must be same as segment index to avoid incorrect indexing of sub pieces in an aggregate","type":"array","items":{"$ref":"#/components/schemas/mk20.DataSource"}},"type":{"description":"Type specifies the type of aggregation for data pieces, represented by an AggregateType value.","allOf":[{"$ref":"#/components/schemas/mk20.AggregateType"}]}}},"cid.Cid":{"type":"object"},"mk20.DataSourceAggregate":{"type":"object","properties":{"pieces":{"type":"array","items":{"$ref":"#/components/schemas/mk20.DataSource"}}}},"mk20.DataSourceHTTP":{"type":"object","properties":{"urls":{"description":"URLs lists the HTTP endpoints where the piece data can be fetched.","type":"array","items":{"$ref":"#/components/schemas/mk20.HttpUrl"}}}},"mk20.HttpUrl":{"type":"object","properties":{"fallback":{"description":"Fallback indicates whether this URL serves as a fallback option when other URLs fail.","type":"boolean"},"headers":{"description":"HTTPHeaders represents the HTTP headers associated with the URL.","allOf":[{"$ref":"#/components/schemas/http.Header"}]},"priority":{"description":"Priority indicates the order preference for using the URL in requests, with lower values having higher priority.","type":"integer"},"url":{"description":"URL specifies the HTTP endpoint where the piece data can be fetched.","type":"string"}}},"http.Header":{"type":"object","additionalProperties":{"type":"array","items":{"type":"string"}}},"mk20.DataSourceHttpPut":{"type":"object"},"mk20.DataSourceOffline":{"type":"object"},"mk20.AggregateType":{"type":"integer","enum":[0,1]},"mk20.FormatCar":{"type":"object"},"mk20.FormatBytes":{"type":"object"}}}} ``` ## The mk20.DataSourceAggregate object ```json {"openapi":"3.1.1","info":{"title":"Curio Market 2.0 API","version":"0.0.1"},"components":{"schemas":{"mk20.DataSourceAggregate":{"type":"object","properties":{"pieces":{"type":"array","items":{"$ref":"#/components/schemas/mk20.DataSource"}}}},"mk20.DataSource":{"type":"object","properties":{"format":{"description":"Format defines the format of the piece data, which can include CAR, Aggregate, or Raw formats.","allOf":[{"$ref":"#/components/schemas/mk20.PieceDataFormat"}]},"piece_cid":{"description":"PieceCID represents the unique identifier (pieceCID V2) for a piece of data, stored as a CID object.","allOf":[{"$ref":"#/components/schemas/cid.Cid"}]},"source_aggregate":{"description":"SourceAggregate represents an aggregated source, comprising multiple data sources as pieces.","allOf":[{"$ref":"#/components/schemas/mk20.DataSourceAggregate"}]},"source_http":{"description":"SourceHTTP represents the HTTP-based source of piece data within a deal, including raw size and URLs for retrieval.","allOf":[{"$ref":"#/components/schemas/mk20.DataSourceHTTP"}]},"source_httpput":{"description":"SourceHTTPPut // allow clients to push piece data after deal accepted, sort of like offline import","allOf":[{"$ref":"#/components/schemas/mk20.DataSourceHttpPut"}]},"source_offline":{"description":"SourceOffline defines the data source for offline pieces, including raw size information.","allOf":[{"$ref":"#/components/schemas/mk20.DataSourceOffline"}]}}},"mk20.PieceDataFormat":{"type":"object","properties":{"aggregate":{"description":"Aggregate holds a reference to the aggregated format of piece data.","allOf":[{"$ref":"#/components/schemas/mk20.FormatAggregate"}]},"car":{"description":"Car represents the optional CAR file format, including its metadata and versioning details.","allOf":[{"$ref":"#/components/schemas/mk20.FormatCar"}]},"raw":{"description":"Raw represents the raw format of the piece data, encapsulated as bytes.","allOf":[{"$ref":"#/components/schemas/mk20.FormatBytes"}]}}},"mk20.FormatAggregate":{"type":"object","properties":{"sub":{"description":"Sub holds a slice of DataSource, representing details of sub pieces aggregated under this format.\nThe order must be same as segment index to avoid incorrect indexing of sub pieces in an aggregate","type":"array","items":{"$ref":"#/components/schemas/mk20.DataSource"}},"type":{"description":"Type specifies the type of aggregation for data pieces, represented by an AggregateType value.","allOf":[{"$ref":"#/components/schemas/mk20.AggregateType"}]}}},"mk20.AggregateType":{"type":"integer","enum":[0,1]},"mk20.FormatCar":{"type":"object"},"mk20.FormatBytes":{"type":"object"},"cid.Cid":{"type":"object"},"mk20.DataSourceHTTP":{"type":"object","properties":{"urls":{"description":"URLs lists the HTTP endpoints where the piece data can be fetched.","type":"array","items":{"$ref":"#/components/schemas/mk20.HttpUrl"}}}},"mk20.HttpUrl":{"type":"object","properties":{"fallback":{"description":"Fallback indicates whether this URL serves as a fallback option when other URLs fail.","type":"boolean"},"headers":{"description":"HTTPHeaders represents the HTTP headers associated with the URL.","allOf":[{"$ref":"#/components/schemas/http.Header"}]},"priority":{"description":"Priority indicates the order preference for using the URL in requests, with lower values having higher priority.","type":"integer"},"url":{"description":"URL specifies the HTTP endpoint where the piece data can be fetched.","type":"string"}}},"http.Header":{"type":"object","additionalProperties":{"type":"array","items":{"type":"string"}}},"mk20.DataSourceHttpPut":{"type":"object"},"mk20.DataSourceOffline":{"type":"object"}}}} ``` ## The mk20.DataSourceHTTP object ```json {"openapi":"3.1.1","info":{"title":"Curio Market 2.0 API","version":"0.0.1"},"components":{"schemas":{"mk20.DataSourceHTTP":{"type":"object","properties":{"urls":{"description":"URLs lists the HTTP endpoints where the piece data can be fetched.","type":"array","items":{"$ref":"#/components/schemas/mk20.HttpUrl"}}}},"mk20.HttpUrl":{"type":"object","properties":{"fallback":{"description":"Fallback indicates whether this URL serves as a fallback option when other URLs fail.","type":"boolean"},"headers":{"description":"HTTPHeaders represents the HTTP headers associated with the URL.","allOf":[{"$ref":"#/components/schemas/http.Header"}]},"priority":{"description":"Priority indicates the order preference for using the URL in requests, with lower values having higher priority.","type":"integer"},"url":{"description":"URL specifies the HTTP endpoint where the piece data can be fetched.","type":"string"}}},"http.Header":{"type":"object","additionalProperties":{"type":"array","items":{"type":"string"}}}}}} ``` ## The mk20.DataSourceHttpPut object ```json {"openapi":"3.1.1","info":{"title":"Curio Market 2.0 API","version":"0.0.1"},"components":{"schemas":{"mk20.DataSourceHttpPut":{"type":"object"}}}} ``` ## The mk20.DataSourceOffline object ```json {"openapi":"3.1.1","info":{"title":"Curio Market 2.0 API","version":"0.0.1"},"components":{"schemas":{"mk20.DataSourceOffline":{"type":"object"}}}} ``` ## The mk20.Deal object ```json {"openapi":"3.1.1","info":{"title":"Curio Market 2.0 API","version":"0.0.1"},"components":{"schemas":{"mk20.Deal":{"type":"object","properties":{"client":{"description":"Client wallet for the deal","allOf":[{"$ref":"#/components/schemas/address.Address"}]},"data":{"description":"Data represents the source of piece data and associated metadata.","allOf":[{"$ref":"#/components/schemas/mk20.DataSource"}]},"identifier":{"description":"Identifier represents a unique identifier for the deal in UUID format.","type":"array","items":{"type":"integer"}},"products":{"description":"Products represents a collection of product-specific information associated with a deal","allOf":[{"$ref":"#/components/schemas/mk20.Products"}]}}},"address.Address":{"type":"object"},"mk20.DataSource":{"type":"object","properties":{"format":{"description":"Format defines the format of the piece data, which can include CAR, Aggregate, or Raw formats.","allOf":[{"$ref":"#/components/schemas/mk20.PieceDataFormat"}]},"piece_cid":{"description":"PieceCID represents the unique identifier (pieceCID V2) for a piece of data, stored as a CID object.","allOf":[{"$ref":"#/components/schemas/cid.Cid"}]},"source_aggregate":{"description":"SourceAggregate represents an aggregated source, comprising multiple data sources as pieces.","allOf":[{"$ref":"#/components/schemas/mk20.DataSourceAggregate"}]},"source_http":{"description":"SourceHTTP represents the HTTP-based source of piece data within a deal, including raw size and URLs for retrieval.","allOf":[{"$ref":"#/components/schemas/mk20.DataSourceHTTP"}]},"source_httpput":{"description":"SourceHTTPPut // allow clients to push piece data after deal accepted, sort of like offline import","allOf":[{"$ref":"#/components/schemas/mk20.DataSourceHttpPut"}]},"source_offline":{"description":"SourceOffline defines the data source for offline pieces, including raw size information.","allOf":[{"$ref":"#/components/schemas/mk20.DataSourceOffline"}]}}},"mk20.PieceDataFormat":{"type":"object","properties":{"aggregate":{"description":"Aggregate holds a reference to the aggregated format of piece data.","allOf":[{"$ref":"#/components/schemas/mk20.FormatAggregate"}]},"car":{"description":"Car represents the optional CAR file format, including its metadata and versioning details.","allOf":[{"$ref":"#/components/schemas/mk20.FormatCar"}]},"raw":{"description":"Raw represents the raw format of the piece data, encapsulated as bytes.","allOf":[{"$ref":"#/components/schemas/mk20.FormatBytes"}]}}},"mk20.FormatAggregate":{"type":"object","properties":{"sub":{"description":"Sub holds a slice of DataSource, representing details of sub pieces aggregated under this format.\nThe order must be same as segment index to avoid incorrect indexing of sub pieces in an aggregate","type":"array","items":{"$ref":"#/components/schemas/mk20.DataSource"}},"type":{"description":"Type specifies the type of aggregation for data pieces, represented by an AggregateType value.","allOf":[{"$ref":"#/components/schemas/mk20.AggregateType"}]}}},"mk20.AggregateType":{"type":"integer","enum":[0,1]},"mk20.FormatCar":{"type":"object"},"mk20.FormatBytes":{"type":"object"},"cid.Cid":{"type":"object"},"mk20.DataSourceAggregate":{"type":"object","properties":{"pieces":{"type":"array","items":{"$ref":"#/components/schemas/mk20.DataSource"}}}},"mk20.DataSourceHTTP":{"type":"object","properties":{"urls":{"description":"URLs lists the HTTP endpoints where the piece data can be fetched.","type":"array","items":{"$ref":"#/components/schemas/mk20.HttpUrl"}}}},"mk20.HttpUrl":{"type":"object","properties":{"fallback":{"description":"Fallback indicates whether this URL serves as a fallback option when other URLs fail.","type":"boolean"},"headers":{"description":"HTTPHeaders represents the HTTP headers associated with the URL.","allOf":[{"$ref":"#/components/schemas/http.Header"}]},"priority":{"description":"Priority indicates the order preference for using the URL in requests, with lower values having higher priority.","type":"integer"},"url":{"description":"URL specifies the HTTP endpoint where the piece data can be fetched.","type":"string"}}},"http.Header":{"type":"object","additionalProperties":{"type":"array","items":{"type":"string"}}},"mk20.DataSourceHttpPut":{"type":"object"},"mk20.DataSourceOffline":{"type":"object"},"mk20.Products":{"type":"object","properties":{"ddo_v1":{"description":"DDOV1 represents a product v1 configuration for Direct Data Onboarding (DDO)","allOf":[{"$ref":"#/components/schemas/mk20.DDOV1"}]},"pdp_v1":{"description":"PDPV1 represents product-specific configuration for PDP version 1 deals.","allOf":[{"$ref":"#/components/schemas/mk20.PDPV1"}]},"retrieval_v1":{"description":"RetrievalV1 represents configuration for retrieval settings in the system, including indexing and announcement flags.","allOf":[{"$ref":"#/components/schemas/mk20.RetrievalV1"}]}}},"mk20.DDOV1":{"type":"object","properties":{"allocation_id":{"description":"AllocationId represents an aggregated allocation identifier for the deal.","allOf":[{"$ref":"#/components/schemas/github_com_filecoin-project_go-state-types_builtin_v16_verifreg.AllocationId"}]},"contract_address":{"description":"ContractAddress specifies the address of the contract governing the deal","type":"string"},"contract_verify_method":{"description":"ContractDealIDMethod specifies the method name to verify the deal and retrieve the deal ID for a contract","type":"string"},"contract_verify_method_params":{"description":"ContractDealIDMethodParams represents encoded parameters for the contract verify method if required by the contract","type":"array","items":{"type":"integer"}},"duration":{"description":"Duration represents the deal duration in epochs. This value is ignored for the deal with allocationID.\nIt must be at least 518400","type":"integer"},"notification_address":{"description":"NotificationAddress specifies the address to which notifications will be relayed to when sector is activated","type":"string"},"notification_payload":{"description":"NotificationPayload holds the notification data typically in a serialized byte array format.","type":"array","items":{"type":"integer"}},"piece_manager":{"description":"Actor providing AuthorizeMessage (like f1/f3 wallet) able to authorize actions such as managing ACLs","allOf":[{"$ref":"#/components/schemas/address.Address"}]},"provider":{"description":"Provider specifies the address of the provider","allOf":[{"$ref":"#/components/schemas/address.Address"}]}}},"github_com_filecoin-project_go-state-types_builtin_v16_verifreg.AllocationId":{"type":"integer","enum":[0]},"mk20.PDPV1":{"type":"object","properties":{"add_root":{"description":"AddRoot indicated that this deal is meant to add root to a given ProofSet. ProofSetID must be defined.","type":"boolean"},"create_proof_set":{"description":"CreateProofSet indicated that this deal is meant to create a new ProofSet for the client by storage provider.","type":"boolean"},"delete_proof_set":{"description":"DeleteProofSet indicated that this deal is meant to delete an existing ProofSet created by SP for the client.\nProofSetID must be defined.","type":"boolean"},"delete_root":{"description":"DeleteRoot indicates whether the root of the data should be deleted. ProofSetID must be defined.","type":"boolean"},"extra_data":{"description":"ExtraData can be used to send additional information to service contract when Verifier action like AddRoot, DeleteRoot etc. are performed.","type":"array","items":{"type":"integer"}},"proof_set_id":{"description":"ProofSetID is PDP verified contract proofset ID. It must be defined for all deals except when CreateProofSet is true.","type":"integer"},"record_keeper":{"description":"RecordKeeper specifies the record keeper contract address for the new PDP proofset.","type":"string"},"root_ids":{"description":"RootIDs is a list of root ids in a proof set.","type":"array","items":{"type":"integer"}}}},"mk20.RetrievalV1":{"type":"object","properties":{"announce_payload":{"description":"AnnouncePayload indicates whether the payload should be announced to IPNI.","type":"boolean"},"announce_piece":{"description":"AnnouncePiece indicates whether the piece information should be announced to IPNI.","type":"boolean"},"indexing":{"description":"Indexing indicates if the deal is to be indexed in the provider's system to support CIDs based retrieval","type":"boolean"}}}}}} ``` ## The mk20.DealCode object ```json {"openapi":"3.1.1","info":{"title":"Curio Market 2.0 API","version":"0.0.1"},"components":{"schemas":{"mk20.DealCode":{"type":"integer","enum":[200,401,400,404,430,422,423,424,425,426,500,503,429,440,441]}}}} ``` ## The mk20.DealProductStatusResponse object ```json {"openapi":"3.1.1","info":{"title":"Curio Market 2.0 API","version":"0.0.1"},"components":{"schemas":{"mk20.DealProductStatusResponse":{"type":"object","properties":{"ddo_v1":{"description":"DDOV1 holds the DealStatusResponse for product \"ddo_v1\".","allOf":[{"$ref":"#/components/schemas/mk20.DealStatusResponse"}]},"pdp_v1":{"description":"PDPV1 represents the DealStatusResponse for the product pdp_v1.","allOf":[{"$ref":"#/components/schemas/mk20.DealStatusResponse"}]}}},"mk20.DealStatusResponse":{"type":"object","properties":{"error_msg":{"description":"ErrorMsg is an optional field containing error details associated with the deal's current state if an error occurred.","type":"string"},"status":{"description":"State indicates the current processing state of the deal as a DealState value.","allOf":[{"$ref":"#/components/schemas/mk20.DealState"}]}}},"mk20.DealState":{"type":"string","enum":["accepted","uploading","processing","sealing","indexing","failed","complete"]}}}} ``` ## The mk20.DealState object ```json {"openapi":"3.1.1","info":{"title":"Curio Market 2.0 API","version":"0.0.1"},"components":{"schemas":{"mk20.DealState":{"type":"string","enum":["accepted","uploading","processing","sealing","indexing","failed","complete"]}}}} ``` ## The mk20.DealStatusResponse object ```json {"openapi":"3.1.1","info":{"title":"Curio Market 2.0 API","version":"0.0.1"},"components":{"schemas":{"mk20.DealStatusResponse":{"type":"object","properties":{"error_msg":{"description":"ErrorMsg is an optional field containing error details associated with the deal's current state if an error occurred.","type":"string"},"status":{"description":"State indicates the current processing state of the deal as a DealState value.","allOf":[{"$ref":"#/components/schemas/mk20.DealState"}]}}},"mk20.DealState":{"type":"string","enum":["accepted","uploading","processing","sealing","indexing","failed","complete"]}}}} ``` ## The mk20.FormatAggregate object ```json {"openapi":"3.1.1","info":{"title":"Curio Market 2.0 API","version":"0.0.1"},"components":{"schemas":{"mk20.FormatAggregate":{"type":"object","properties":{"sub":{"description":"Sub holds a slice of DataSource, representing details of sub pieces aggregated under this format.\nThe order must be same as segment index to avoid incorrect indexing of sub pieces in an aggregate","type":"array","items":{"$ref":"#/components/schemas/mk20.DataSource"}},"type":{"description":"Type specifies the type of aggregation for data pieces, represented by an AggregateType value.","allOf":[{"$ref":"#/components/schemas/mk20.AggregateType"}]}}},"mk20.DataSource":{"type":"object","properties":{"format":{"description":"Format defines the format of the piece data, which can include CAR, Aggregate, or Raw formats.","allOf":[{"$ref":"#/components/schemas/mk20.PieceDataFormat"}]},"piece_cid":{"description":"PieceCID represents the unique identifier (pieceCID V2) for a piece of data, stored as a CID object.","allOf":[{"$ref":"#/components/schemas/cid.Cid"}]},"source_aggregate":{"description":"SourceAggregate represents an aggregated source, comprising multiple data sources as pieces.","allOf":[{"$ref":"#/components/schemas/mk20.DataSourceAggregate"}]},"source_http":{"description":"SourceHTTP represents the HTTP-based source of piece data within a deal, including raw size and URLs for retrieval.","allOf":[{"$ref":"#/components/schemas/mk20.DataSourceHTTP"}]},"source_httpput":{"description":"SourceHTTPPut // allow clients to push piece data after deal accepted, sort of like offline import","allOf":[{"$ref":"#/components/schemas/mk20.DataSourceHttpPut"}]},"source_offline":{"description":"SourceOffline defines the data source for offline pieces, including raw size information.","allOf":[{"$ref":"#/components/schemas/mk20.DataSourceOffline"}]}}},"mk20.PieceDataFormat":{"type":"object","properties":{"aggregate":{"description":"Aggregate holds a reference to the aggregated format of piece data.","allOf":[{"$ref":"#/components/schemas/mk20.FormatAggregate"}]},"car":{"description":"Car represents the optional CAR file format, including its metadata and versioning details.","allOf":[{"$ref":"#/components/schemas/mk20.FormatCar"}]},"raw":{"description":"Raw represents the raw format of the piece data, encapsulated as bytes.","allOf":[{"$ref":"#/components/schemas/mk20.FormatBytes"}]}}},"mk20.AggregateType":{"type":"integer","enum":[0,1]},"mk20.FormatCar":{"type":"object"},"mk20.FormatBytes":{"type":"object"},"cid.Cid":{"type":"object"},"mk20.DataSourceAggregate":{"type":"object","properties":{"pieces":{"type":"array","items":{"$ref":"#/components/schemas/mk20.DataSource"}}}},"mk20.DataSourceHTTP":{"type":"object","properties":{"urls":{"description":"URLs lists the HTTP endpoints where the piece data can be fetched.","type":"array","items":{"$ref":"#/components/schemas/mk20.HttpUrl"}}}},"mk20.HttpUrl":{"type":"object","properties":{"fallback":{"description":"Fallback indicates whether this URL serves as a fallback option when other URLs fail.","type":"boolean"},"headers":{"description":"HTTPHeaders represents the HTTP headers associated with the URL.","allOf":[{"$ref":"#/components/schemas/http.Header"}]},"priority":{"description":"Priority indicates the order preference for using the URL in requests, with lower values having higher priority.","type":"integer"},"url":{"description":"URL specifies the HTTP endpoint where the piece data can be fetched.","type":"string"}}},"http.Header":{"type":"object","additionalProperties":{"type":"array","items":{"type":"string"}}},"mk20.DataSourceHttpPut":{"type":"object"},"mk20.DataSourceOffline":{"type":"object"}}}} ``` ## The mk20.FormatBytes object ```json {"openapi":"3.1.1","info":{"title":"Curio Market 2.0 API","version":"0.0.1"},"components":{"schemas":{"mk20.FormatBytes":{"type":"object"}}}} ``` ## The mk20.FormatCar object ```json {"openapi":"3.1.1","info":{"title":"Curio Market 2.0 API","version":"0.0.1"},"components":{"schemas":{"mk20.FormatCar":{"type":"object"}}}} ``` ## The mk20.HttpUrl object ```json {"openapi":"3.1.1","info":{"title":"Curio Market 2.0 API","version":"0.0.1"},"components":{"schemas":{"mk20.HttpUrl":{"type":"object","properties":{"fallback":{"description":"Fallback indicates whether this URL serves as a fallback option when other URLs fail.","type":"boolean"},"headers":{"description":"HTTPHeaders represents the HTTP headers associated with the URL.","allOf":[{"$ref":"#/components/schemas/http.Header"}]},"priority":{"description":"Priority indicates the order preference for using the URL in requests, with lower values having higher priority.","type":"integer"},"url":{"description":"URL specifies the HTTP endpoint where the piece data can be fetched.","type":"string"}}},"http.Header":{"type":"object","additionalProperties":{"type":"array","items":{"type":"string"}}}}}} ``` ## The mk20.PDPV1 object ```json {"openapi":"3.1.1","info":{"title":"Curio Market 2.0 API","version":"0.0.1"},"components":{"schemas":{"mk20.PDPV1":{"type":"object","properties":{"add_root":{"description":"AddRoot indicated that this deal is meant to add root to a given ProofSet. ProofSetID must be defined.","type":"boolean"},"create_proof_set":{"description":"CreateProofSet indicated that this deal is meant to create a new ProofSet for the client by storage provider.","type":"boolean"},"delete_proof_set":{"description":"DeleteProofSet indicated that this deal is meant to delete an existing ProofSet created by SP for the client.\nProofSetID must be defined.","type":"boolean"},"delete_root":{"description":"DeleteRoot indicates whether the root of the data should be deleted. ProofSetID must be defined.","type":"boolean"},"extra_data":{"description":"ExtraData can be used to send additional information to service contract when Verifier action like AddRoot, DeleteRoot etc. are performed.","type":"array","items":{"type":"integer"}},"proof_set_id":{"description":"ProofSetID is PDP verified contract proofset ID. It must be defined for all deals except when CreateProofSet is true.","type":"integer"},"record_keeper":{"description":"RecordKeeper specifies the record keeper contract address for the new PDP proofset.","type":"string"},"root_ids":{"description":"RootIDs is a list of root ids in a proof set.","type":"array","items":{"type":"integer"}}}}}}} ``` ## The mk20.PieceDataFormat object ```json {"openapi":"3.1.1","info":{"title":"Curio Market 2.0 API","version":"0.0.1"},"components":{"schemas":{"mk20.PieceDataFormat":{"type":"object","properties":{"aggregate":{"description":"Aggregate holds a reference to the aggregated format of piece data.","allOf":[{"$ref":"#/components/schemas/mk20.FormatAggregate"}]},"car":{"description":"Car represents the optional CAR file format, including its metadata and versioning details.","allOf":[{"$ref":"#/components/schemas/mk20.FormatCar"}]},"raw":{"description":"Raw represents the raw format of the piece data, encapsulated as bytes.","allOf":[{"$ref":"#/components/schemas/mk20.FormatBytes"}]}}},"mk20.FormatAggregate":{"type":"object","properties":{"sub":{"description":"Sub holds a slice of DataSource, representing details of sub pieces aggregated under this format.\nThe order must be same as segment index to avoid incorrect indexing of sub pieces in an aggregate","type":"array","items":{"$ref":"#/components/schemas/mk20.DataSource"}},"type":{"description":"Type specifies the type of aggregation for data pieces, represented by an AggregateType value.","allOf":[{"$ref":"#/components/schemas/mk20.AggregateType"}]}}},"mk20.DataSource":{"type":"object","properties":{"format":{"description":"Format defines the format of the piece data, which can include CAR, Aggregate, or Raw formats.","allOf":[{"$ref":"#/components/schemas/mk20.PieceDataFormat"}]},"piece_cid":{"description":"PieceCID represents the unique identifier (pieceCID V2) for a piece of data, stored as a CID object.","allOf":[{"$ref":"#/components/schemas/cid.Cid"}]},"source_aggregate":{"description":"SourceAggregate represents an aggregated source, comprising multiple data sources as pieces.","allOf":[{"$ref":"#/components/schemas/mk20.DataSourceAggregate"}]},"source_http":{"description":"SourceHTTP represents the HTTP-based source of piece data within a deal, including raw size and URLs for retrieval.","allOf":[{"$ref":"#/components/schemas/mk20.DataSourceHTTP"}]},"source_httpput":{"description":"SourceHTTPPut // allow clients to push piece data after deal accepted, sort of like offline import","allOf":[{"$ref":"#/components/schemas/mk20.DataSourceHttpPut"}]},"source_offline":{"description":"SourceOffline defines the data source for offline pieces, including raw size information.","allOf":[{"$ref":"#/components/schemas/mk20.DataSourceOffline"}]}}},"mk20.FormatCar":{"type":"object"},"mk20.FormatBytes":{"type":"object"},"cid.Cid":{"type":"object"},"mk20.DataSourceAggregate":{"type":"object","properties":{"pieces":{"type":"array","items":{"$ref":"#/components/schemas/mk20.DataSource"}}}},"mk20.DataSourceHTTP":{"type":"object","properties":{"urls":{"description":"URLs lists the HTTP endpoints where the piece data can be fetched.","type":"array","items":{"$ref":"#/components/schemas/mk20.HttpUrl"}}}},"mk20.HttpUrl":{"type":"object","properties":{"fallback":{"description":"Fallback indicates whether this URL serves as a fallback option when other URLs fail.","type":"boolean"},"headers":{"description":"HTTPHeaders represents the HTTP headers associated with the URL.","allOf":[{"$ref":"#/components/schemas/http.Header"}]},"priority":{"description":"Priority indicates the order preference for using the URL in requests, with lower values having higher priority.","type":"integer"},"url":{"description":"URL specifies the HTTP endpoint where the piece data can be fetched.","type":"string"}}},"http.Header":{"type":"object","additionalProperties":{"type":"array","items":{"type":"string"}}},"mk20.DataSourceHttpPut":{"type":"object"},"mk20.DataSourceOffline":{"type":"object"},"mk20.AggregateType":{"type":"integer","enum":[0,1]}}}} ``` ## The mk20.Products object ```json {"openapi":"3.1.1","info":{"title":"Curio Market 2.0 API","version":"0.0.1"},"components":{"schemas":{"mk20.Products":{"type":"object","properties":{"ddo_v1":{"description":"DDOV1 represents a product v1 configuration for Direct Data Onboarding (DDO)","allOf":[{"$ref":"#/components/schemas/mk20.DDOV1"}]},"pdp_v1":{"description":"PDPV1 represents product-specific configuration for PDP version 1 deals.","allOf":[{"$ref":"#/components/schemas/mk20.PDPV1"}]},"retrieval_v1":{"description":"RetrievalV1 represents configuration for retrieval settings in the system, including indexing and announcement flags.","allOf":[{"$ref":"#/components/schemas/mk20.RetrievalV1"}]}}},"mk20.DDOV1":{"type":"object","properties":{"allocation_id":{"description":"AllocationId represents an aggregated allocation identifier for the deal.","allOf":[{"$ref":"#/components/schemas/github_com_filecoin-project_go-state-types_builtin_v16_verifreg.AllocationId"}]},"contract_address":{"description":"ContractAddress specifies the address of the contract governing the deal","type":"string"},"contract_verify_method":{"description":"ContractDealIDMethod specifies the method name to verify the deal and retrieve the deal ID for a contract","type":"string"},"contract_verify_method_params":{"description":"ContractDealIDMethodParams represents encoded parameters for the contract verify method if required by the contract","type":"array","items":{"type":"integer"}},"duration":{"description":"Duration represents the deal duration in epochs. This value is ignored for the deal with allocationID.\nIt must be at least 518400","type":"integer"},"notification_address":{"description":"NotificationAddress specifies the address to which notifications will be relayed to when sector is activated","type":"string"},"notification_payload":{"description":"NotificationPayload holds the notification data typically in a serialized byte array format.","type":"array","items":{"type":"integer"}},"piece_manager":{"description":"Actor providing AuthorizeMessage (like f1/f3 wallet) able to authorize actions such as managing ACLs","allOf":[{"$ref":"#/components/schemas/address.Address"}]},"provider":{"description":"Provider specifies the address of the provider","allOf":[{"$ref":"#/components/schemas/address.Address"}]}}},"github_com_filecoin-project_go-state-types_builtin_v16_verifreg.AllocationId":{"type":"integer","enum":[0]},"address.Address":{"type":"object"},"mk20.PDPV1":{"type":"object","properties":{"add_root":{"description":"AddRoot indicated that this deal is meant to add root to a given ProofSet. ProofSetID must be defined.","type":"boolean"},"create_proof_set":{"description":"CreateProofSet indicated that this deal is meant to create a new ProofSet for the client by storage provider.","type":"boolean"},"delete_proof_set":{"description":"DeleteProofSet indicated that this deal is meant to delete an existing ProofSet created by SP for the client.\nProofSetID must be defined.","type":"boolean"},"delete_root":{"description":"DeleteRoot indicates whether the root of the data should be deleted. ProofSetID must be defined.","type":"boolean"},"extra_data":{"description":"ExtraData can be used to send additional information to service contract when Verifier action like AddRoot, DeleteRoot etc. are performed.","type":"array","items":{"type":"integer"}},"proof_set_id":{"description":"ProofSetID is PDP verified contract proofset ID. It must be defined for all deals except when CreateProofSet is true.","type":"integer"},"record_keeper":{"description":"RecordKeeper specifies the record keeper contract address for the new PDP proofset.","type":"string"},"root_ids":{"description":"RootIDs is a list of root ids in a proof set.","type":"array","items":{"type":"integer"}}}},"mk20.RetrievalV1":{"type":"object","properties":{"announce_payload":{"description":"AnnouncePayload indicates whether the payload should be announced to IPNI.","type":"boolean"},"announce_piece":{"description":"AnnouncePiece indicates whether the piece information should be announced to IPNI.","type":"boolean"},"indexing":{"description":"Indexing indicates if the deal is to be indexed in the provider's system to support CIDs based retrieval","type":"boolean"}}}}}} ``` ## The mk20.RetrievalV1 object ```json {"openapi":"3.1.1","info":{"title":"Curio Market 2.0 API","version":"0.0.1"},"components":{"schemas":{"mk20.RetrievalV1":{"type":"object","properties":{"announce_payload":{"description":"AnnouncePayload indicates whether the payload should be announced to IPNI.","type":"boolean"},"announce_piece":{"description":"AnnouncePiece indicates whether the piece information should be announced to IPNI.","type":"boolean"},"indexing":{"description":"Indexing indicates if the deal is to be indexed in the provider's system to support CIDs based retrieval","type":"boolean"}}}}}} ``` ## The mk20.StartUpload object ```json {"openapi":"3.1.1","info":{"title":"Curio Market 2.0 API","version":"0.0.1"},"components":{"schemas":{"mk20.StartUpload":{"type":"object","properties":{"chunk_size":{"description":"ChunkSize defines the size of each data chunk to be used during the upload process.","type":"integer"},"raw_size":{"description":"RawSize indicates the total size of the data to be uploaded in bytes.","type":"integer"}}}}}} ``` ## The mk20.SupportedContracts object ```json {"openapi":"3.1.1","info":{"title":"Curio Market 2.0 API","version":"0.0.1"},"components":{"schemas":{"mk20.SupportedContracts":{"type":"object","properties":{"contracts":{"description":"Contracts represents a list of supported contract addresses in string format.","type":"array","items":{"type":"string"}}}}}}} ``` ## The mk20.SupportedDataSources object ```json {"openapi":"3.1.1","info":{"title":"Curio Market 2.0 API","version":"0.0.1"},"components":{"schemas":{"mk20.SupportedDataSources":{"type":"object","properties":{"sources":{"description":"Contracts represents a list of supported contract addresses in string format.","type":"array","items":{"type":"string"}}}}}}} ``` ## The mk20.SupportedProducts object ```json {"openapi":"3.1.1","info":{"title":"Curio Market 2.0 API","version":"0.0.1"},"components":{"schemas":{"mk20.SupportedProducts":{"type":"object","properties":{"products":{"description":"Contracts represents a list of supported contract addresses in string format.","type":"array","items":{"type":"string"}}}}}}} ``` ## The mk20.UploadCode object ```json {"openapi":"3.1.1","info":{"title":"Curio Market 2.0 API","version":"0.0.1"},"components":{"schemas":{"mk20.UploadCode":{"type":"integer","enum":[200,400,404,409,500]}}}} ``` ## The mk20.UploadStartCode object ```json {"openapi":"3.1.1","info":{"title":"Curio Market 2.0 API","version":"0.0.1"},"components":{"schemas":{"mk20.UploadStartCode":{"type":"integer","enum":[200,400,404,409,500]}}}} ``` ## The mk20.UploadStatus object ```json {"openapi":"3.1.1","info":{"title":"Curio Market 2.0 API","version":"0.0.1"},"components":{"schemas":{"mk20.UploadStatus":{"type":"object","properties":{"missing":{"description":"Missing represents the number of chunks that are not yet uploaded.","type":"integer"},"missing_chunks":{"description":"MissingChunks is a slice containing the indices of missing chunks.","type":"array","items":{"type":"integer"}},"total_chunks":{"description":"TotalChunks represents the total number of chunks required for the upload.","type":"integer"},"uploaded":{"description":"Uploaded represents the number of chunks successfully uploaded.","type":"integer"},"uploaded_chunks":{"description":"UploadedChunks is a slice containing the indices of successfully uploaded chunks.","type":"array","items":{"type":"integer"}}}}}}} ``` ## The mk20.UploadStatusCode object ```json {"openapi":"3.1.1","info":{"title":"Curio Market 2.0 API","version":"0.0.1"},"components":{"schemas":{"mk20.UploadStatusCode":{"type":"integer","enum":[200,404,425,500]}}}} ``` # Wallet Exporter The **Wallet Exporter** is an optional telemetry component that periodically exposes comprehensive statistics about Curio wallets, miners and the messages they send. The metrics are exported via the built-in Prometheus endpoint and can be scraped by any Prometheus compatible monitoring stack. > ⚠️ IMPORTANT: **Enable the exporter on exactly *****one***** Curio node in the cluster.** Enabling it on multiple nodes would cause duplicated metric series which leads to incorrect dashboards and aggregated values. *** ## Why would I enable it? * Track wallet balances for every on-chain address that appears in the `wallet_names` table. Those are wallets with a name on the Wallets page. * Observe storage-provider balances and power in real-time. * Monitor message throughput, gas usage and fees broken down by sender, receiver and reason. * Build Grafana dashboards that correlate wallet activity with other Curio subsystems. ## How does it work? 1. Every 30 seconds (`WalletExporterInterval`) Curio executes one exporter cycle. 2. During a cycle Curio gathers the following information: * Wallet balances * Miner available balance and power (for the SP IDs configured on this node) * Newly sent messages that are waiting to land * Messages that have already landed (executed) and their execution result 3. For every data-point the exporter records an *OpenCensus measure* which is immediately made available as a Prometheus metric. ## Enabling the exporter Add the following to one layer which is active on *exactly one* node: ```toml [CurioSubsystems] EnableWalletExporter = true ``` Restart the node. The new metrics will appear at the standard metrics endpoint within \~30 seconds. ## Exported metrics | Metric name | Type | Description | | -------------------------------------- | --------- | -------------------------------------------------------------------------- | | `wallet_balance_nfil` | gauge | Wallet or miner balance in **NanoFIL** | | `wallet_power` | gauge | Storage-provider power in **bytes** (label `type` = `raw` or `qap`) | | `wallet_message_sent` | counter | Number of messages that have been **submitted** | | `wallet_message_landed` | counter | Number of messages that have **executed** on-chain | | `wallet_gas_units_requested` | counter | Gas units requested by submitted messages | | `wallet_gas_units_used` | counter | Gas units used by executed messages | | `wallet_sent_nfil` | counter | NanoFIL **value** transferred by executed messages | | `wallet_gas_paid_nfil` | counter | NanoFIL **gas fee** paid by executed messages | | `wallet_message_land_duration_seconds` | histogram | Distribution of time (in seconds) between message submission and execution | # cuzk Proving Daemon This page explains how to set up the cuzk persistent GPU SNARK proving daemon to accelerate proof computation in Curio. > **Experimental Feature**\ > This feature is currently experimental and under active development. Configuration, behavior, and interfaces may change without notice. *** ## What is cuzk? cuzk is a persistent GPU-resident SNARK proving daemon. It acts as a "proving server" that Curio delegates proof computations to over gRPC. The key difference from the default proving path (`ffiselect`) is that cuzk loads Groth16 SRS parameters **once at startup** and keeps them resident in CUDA-pinned host memory across all proofs. The default Curio code path spawns a fresh child process per proof, each of which loads the SRS from disk (30-90 seconds for 32 GiB PoRep), runs one proof, and exits. cuzk eliminates this repeated loading overhead entirely. ### Supported proof types | Proof type | Curio task | Description | | --------------- | ------------- | -------------------------------- | | PoRep C2 | `PoRep` | Seal commit phase 2 SNARK | | SnapDeals Prove | `UpdateProve` | CC sector update proof | | PSProve | `PSProve` | Snark Market proof share compute | ### How integration works When cuzk is enabled in Curio's configuration: 1. **Resource accounting bypassed**: `TypeDetails()` reports zero GPU and minimal RAM for proving tasks. Curio's harmony scheduler no longer gates these tasks on local GPU availability. 2. **Backpressure via polling**: `CanAccept()` queries the cuzk daemon's queue via `GetStatus` and rejects tasks when the queue is full (controlled by `MaxPending`). 3. **Vanilla proofs stay local**: The `Do()` method generates vanilla proofs locally (requires sector data on disk), sends them to cuzk for SNARK computation, then verifies the returned proof locally. When cuzk is **not** configured (default), all three tasks behave exactly as before. There is no behavioral change for existing deployments. *** ## Requirements * **NVIDIA GPU** with CUDA support (the cuzk daemon itself runs on the GPU machine) * **CUDA toolkit** (`nvcc` must be in PATH) * **Rust toolchain** (1.86.0 or later; managed automatically via `rust-toolchain.toml`) * **Filecoin proof parameters** downloaded (same parameters as standard Curio proving) * Sufficient system RAM for SRS residency (minimum 128 GiB, recommended 256+ GiB) *** ## Building From the Curio repository root: ```bash # Build both Curio and the cuzk daemon make curio cuzk ``` The `make cuzk` target: * Checks for `cargo` (Rust) and `nvcc` (CUDA) in PATH * Runs `cargo build --release` in `extern/cuzk/` * Copies the resulting binary to `./cuzk` To install: ```bash sudo make install # installs curio and sptool sudo make install-cuzk # installs cuzk to /usr/local/bin/cuzk ``` Note: `make cuzk` is intentionally **not** part of `make build` or `make buildall` since it requires CUDA and Rust, which are not available in all build environments (e.g., CI). *** ## Daemon Configuration The cuzk daemon reads its configuration from a TOML file (default: `/data/zk/cuzk.toml`). An example configuration is provided at `extern/cuzk/cuzk.example.toml`. ### Minimal configuration ```toml [daemon] listen = "0.0.0.0:9820" [srs] param_cache = "/var/tmp/filecoin-proof-parameters" preload = ["porep-32g"] [synthesis] partition_workers = 7 # Adjust based on RAM (see table below) ``` ### RAM-based tuning The primary tuning knob is `partition_workers`, which controls how many PoRep partitions are synthesized concurrently on the CPU. More workers keep the GPU fed but use more RAM. | System RAM | partition\_workers | gpu\_workers\_per\_device | Peak RSS | Throughput | | ---------- | ------------------ | ------------------------- | --------- | ------------- | | 128 GiB | 2 | 1 | \~110 GiB | \~152 s/proof | | 256 GiB | 7 | 1 | \~208 GiB | \~53 s/proof | | 384 GiB | 10 | 2 | \~271 GiB | \~43 s/proof | | 512+ GiB | 12 | 2 | \~400 GiB | \~38 s/proof | Memory formula: `Peak RSS = 69 + (partition_workers x 20) GiB` ### Running the daemon ```bash # With default config path cuzk # With custom config cuzk --config /path/to/cuzk.toml # Override listen address cuzk --listen unix:///run/curio/cuzk.sock # Override log level cuzk --log-level debug ``` For production, run cuzk as a systemd service: ```ini [Unit] Description=cuzk SNARK proving daemon After=network.target [Service] Type=simple ExecStart=/usr/local/bin/cuzk --config /data/zk/cuzk.toml Restart=on-failure RestartSec=10 LimitNOFILE=1048576 # Ensure CUDA libraries are available Environment=LD_LIBRARY_PATH=/usr/local/cuda/lib64 [Install] WantedBy=multi-user.target ``` *** ## Curio Configuration Add the following to your Curio configuration layer to connect to the cuzk daemon: ```toml [Cuzk] # gRPC endpoint of the cuzk daemon. # TCP: "127.0.0.1:9820" # Unix: "unix:///run/curio/cuzk.sock" Address = "127.0.0.1:9820" # Maximum total pending proofs in the cuzk queue before Curio stops # sending new tasks (backpressure). When the daemon's queue reaches # this level, CanAccept rejects new tasks until capacity frees up. MaxPending = 10 # Maximum time to wait for a proof result from the daemon. # If exceeded, the task is retried. ProveTimeout = "30m" ``` When `Address` is empty (the default), cuzk integration is disabled and all proving tasks use the standard local GPU path. ### Which Curio subsystems are affected The cuzk client is used by tasks on nodes that have these subsystems enabled: * `EnablePoRepProof = true` -- PoRep C2 proving * `EnableUpdateProve = true` -- SnapDeals update proving * `EnableProofShare = true` -- Snark Market proof computation These subsystems must still be enabled as usual. The `[Cuzk]` configuration only changes *how* the SNARK computation is performed (local GPU vs. remote daemon). *** ## Deployment Patterns ### Co-located (single machine) Run both Curio and cuzk on the same GPU machine. Use TCP localhost or a Unix socket: ``` Curio (Go) --gRPC--> cuzk (Rust) --CUDA--> GPU ``` ```toml # Curio config [Cuzk] Address = "127.0.0.1:9820" ``` This is the simplest deployment and avoids any network overhead. ### Dedicated prover (separate machines) Run Curio on CPU-only machines for sealing tasks (SDR, TreeD, etc.) and cuzk on a dedicated GPU machine. Curio connects over the network: ``` Curio (CPU node) --gRPC/TCP--> cuzk (GPU node) ``` ```toml # Curio config (on CPU node) [Cuzk] Address = "gpu-prover.internal:9820" ``` Note: vanilla proof data (up to \~200 MB for PoRep C2) is sent over gRPC, so ensure sufficient network bandwidth between the nodes. *** ## Monitoring The cuzk daemon exposes its status via the gRPC `GetStatus` RPC. Curio queries this automatically for backpressure. You can also query it manually: ```bash grpcurl -plaintext 127.0.0.1:9820 cuzk.v1.ProvingEngine/GetStatus ``` This returns the current queue state for each proof type (pending count, in-progress count). *** ## Troubleshooting ### cuzk build fails * Verify `nvcc` is in PATH: `nvcc --version` * Verify Rust toolchain: `rustup show` (should show 1.86.0 or later) * The Cargo workspace in `extern/cuzk/` depends on vendored forks of `bellperson`, `bellpepper-core`, and `supraseal-c2` under `extern/`. If you see missing crate errors, ensure `git submodule update --init --recursive` was run. ### Curio cannot connect to cuzk * Check that the daemon is running: `systemctl status cuzk` * Verify the address matches between Curio's `[Cuzk].Address` and the daemon's `[daemon].listen` * Check firewall rules if using TCP across machines * Look at Curio logs for `cuzk` entries: `journalctl -u curio | grep cuzk` ### Proofs are slow or timing out * Increase `ProveTimeout` in Curio's config if proofs legitimately take longer * Check daemon logs for queue depth. If proofs pile up, reduce `MaxPending` or add more GPU capacity * Tune `partition_workers` based on the RAM table above. Too many workers can cause memory pressure; too few starve the GPU ### Curio tasks rejected (backpressure) If Curio logs show "cuzk pipeline full, backpressuring", the daemon's queue is at capacity. Either: * Increase `MaxPending` (allows more queued proofs, uses more memory) * Add GPU capacity (second GPU, second daemon instance) * Reduce the rate of incoming sealing/snap work