Storage Market
A comprehensive guide to configuring and managing storage deals in Curio
Overview
The Curio Storage Market provides a comprehensive framework for managing deals, data retrieval, and storage through decentralized protocols. This page details the different configurations, commands, and workflows available for managing storage deals, both online and offline.
The storage market in Curio is built around several key concepts:
Deal Protocols: Protocols and workflows for deal-making, sealing, and data transfers.
Tasks: Various tasks managed by the storage market, such as commP, PSD, indexing deals, IPNI advertisement etc.
Deal Flows: The workflows for processing online and offline deals, each of which has specific tasks and checks to ensure the deal is successfully completed.
Configuration
The Curio storage market is configurable through the StorageMarketConfig
structure. This section outlines the main configuration parameters and the implications of setting them.
MK12 Configuration
The MK12
configuration encompasses all deal-related settings for the MK1.2.0 and MK1.2.1 deal protocols (commonly referred to as Boost deals). This configuration controls key parameters like batching, sealing time, and the number of deals that can be published at once.
Key Parameters
PublishMsgPeriod: Specifies the time to wait before publishing deals as a batch. Increasing this period allows more deals to be included in a single message but delays the publishing of deals. Lowering this period will result in faster publishing but fewer deals being batched together, increasing chain overhead.
MaxDealsPerPublishMsg: Controls the maximum number of deals to include in one batch. If set too high, the publish message may become too large and expensive. Setting it too low might reduce efficiency, as the node sends more messages.
MaxPublishDealFee: This defines the maximum fee you’re willing to pay per deal when sending the
PublishStorageDeals
message. The consequence of setting a low fee is that deal publishing may be delayed or fail if network congestion raises gas costs.ExpectedPoRepSealDuration: This value controls how long you expect the Proof of Replication (PoRep) sealing process to take. Deals that cannot be sealed within this time will fail.
ExpectedSnapSealDuration: Similar to PoRep, this defines the expected time for snap sealing. The duration should account for hardware speed and network delays.
SkipCommP: If set to
true
, the CommP (Commitment Proof) check is skipped before thePublishDealMessage
is sent on-chain. Skipping this step is risky because if there’s a mismatch, all deals in the sector may need to be resent.DisabledMiners: A list of miner addresses excluded from participating in deal-making. Use this option to prevent specific miners from handling deals if needed.
MaxConcurrentDealSizeGiB: MaxConcurrentDealSizeGiB is a sum of all size of all deals which are waiting to be added to a sector when the cumulative size of all deals in process reaches this number, new deals will be rejected. (Default: 0 = unlimited)
DenyUnknownClients: DenyUnknownClients determines the default behaviour for the deal of clients which are not in allow/deny list. If True then all deals coming from unknown clients will be rejected.
PieceLocator Configuration
This configuration allows you to set up remote HTTP servers that provide piece data for offline deals. A PieceLocator
config is a combination of a URL and headers for fetching pieces when requested by the miner. This is crucial for handling offline deals where data is not available immediately and must be retrieved during the commP and sealing phase.
URL: The endpoint where the piece data can be located.
Headers: Any custom headers needed for the HTTP request, such as authorization tokens.
Consequences: If the piece data is not available at the specified URL, the offline deal will fail. Make sure that the remote server is properly configured and available.
Enabling Storage Market
To enable the Curio market on a Curio node, the following configuration changes are required:
Enable the Deal Market:
Set
EnableDealMarket
totrue
in theCurioSubsystemsConfig
for at least one node. This enables deal-making capabilities on the node.
Enable CommP:
On one of the nodes where
EnableDealMarket
is set totrue
, ensure thatEnableCommP
is also set totrue
. This allows the node to compute piece commitments (CommP) before publishing storage deal messages.
Enable HTTP:
At least one node must have HTTP enabled to support:
Retrievals.
IPNI sync.
Handling storage deals.
To enable HTTP, set the
Enable
flag in theHTTPConfig
totrue
and specify theListenAddress
for the HTTP server.
Set a Domain Name:
Ensure that a valid
DomainName
is specified in theHTTPConfig
. This is mandatory for proper HTTP server functionality and essential for enabling TLS. The domain name cannot be an IP address.
HTTP Configuration Details:
If TLS is managed by a reverse proxy, enable
DelegateTLS
in theHTTPConfig
to allow the HTTP server to run without handling TLS directly.Configure additional parameters such as
ReadTimeout
,IdleTimeout
, andCompressionLevels
to ensure the server operates efficiently.
Libp2p Activation:
The
libp2p
service will automatically start on one of the servers running the HTTP server whereEnableDealMarket
is set totrue
. If more than 1 node satsifies the condition and the node running libp2p goes down then it will switch over to another node after 5 minutes.
Other Considerations:
Ensure the
MK12Config
settings underStorageMarketConfig
are properly configured for deal publishing. Key parameters include:PublishMsgPeriod
for deal batching frequency.MaxDealsPerPublishMsg
for the maximum number of deals per message.MaxPublishDealFee
to set the fee limit for publishing deals.
If handling offline deals, configure
PieceLocator
to specify the endpoints for piece retrieval.
By applying these changes, the Curio market subsystem will be activated on the specified node(s), enabling storage deals, IPNI synchronization, and retrieval functionality.
MK12 Deals (Boost Deals)
The MK12 protocol governs the entire deal process, from proposing and publishing deals to sealing and validating them. It's designed to work efficiently for both online and offline deals.
Key tasks within MK12 include:
Piece Commitment: Ensuring that the piece has been added to a sector.
Publish Storage Deals (PSD): Sending the on-chain message to register the deal.
Finding Deals: Identifying the deal ID on-chain and adding it to the sector.
MK12 Tasks
Tasks refer to operations that the system performs on deals to ensure their success. Tasks include:
CommP tasks: Commitment proof tasks, ensuring the piece information is correct by calculating the commitment locally.
PSD tasks: Tasks for sending and validating the PublishStorageDeals message.
Find Deal tasks: These tasks poll the blockchain to identify the deal’s status and retrieve its ID after the deal has been published successfully with PSD task.
Online Deal Flow
Deal Proposal: A client proposes a deal, specifying the amount of data and terms.
Data Transfer: Data is transferred immediately. The system checks that the entire piece has been received.
CommP Task: Once data is received, a commitment proof is generated.
PSD Task: The deal is published on-chain.
Sector Assignment: The deal is assigned to a sector and sealed.
Offline Deal Flow
Deal Proposal: The client proposes a deal for data that is not yet available on the miner’s node.
PieceLocator: The miner is provided with a URL where the data can be fetched later.
Data Fetch: The miner fetches the piece from the provided URL using the
PieceLocator
configuration or local database.CommP Task: Once the piece is retrieved, a commitment proof is generated.
PSD Task: The deal is published on-chain.
Sector Assignment: The deal is added to a sector and sealed.
Add data URL for offline deals
The add-url
command allows you to specify a URL from which the miner can fetch piece data for offline deals. This is essential for deals where the client does not transfer the data immediately upon deal acceptance.
Example Usage:
Options:
--url: The URL where the piece data can be fetched.
--header: Custom headers to include in the HTTP request.
Consequences: If the URL is not accessible or the headers are incorrect, the deal will fail to retrieve the data and will not be able to complete.
Move funds to escrow
The move-to-escrow
command moves funds from the deal collateral wallet to the escrow account with the storage market actor. This is necessary to lock in collateral for a deal.
Example Usage:
Options:
--actor: Specifies the actor address that should start sealing sectors for the deal.
--max-fee: Maximum fee in FIL you’re willing to pay for this message.
Consequences: If insufficient funds are moved to escrow, the deal may not be processed, and the collateral may not be secured.
Start Sealing Early
The seal
command allows you to start sealing a deal's sector early, before all the deals have been batched.
Example Usage:
Options:
--actor: Specifies the actor address.
Consequences: Sealing early can speed up the process, but it may result in inefficiencies if all deals are not batched correctly.
Last updated