Preparers API¶

配置驱动的组件构建。该模式与用法见 Preparers。

Logger_Preparer()

Bases: Preparer

Tokenizer_Preparer()

Bases: Preparer

Dataset_Preparer()

Bases: Preparer

Preparer for creating Dataset objects.

This preparer focuses on creating Dataset objects from configuration, separating the concern of "what the data is" from "how to load it".

prepare(tokenizer: PreTrainedTokenizerFast) -> AssetBERTMLMDataset

Prepare and return a Dataset object.

Parameters:

Name	Type	Description	Default
`tokenizer`	`PreTrainedTokenizerFast`	PreTrainedTokenizerFast for tokenizing the data.	required

Returns:

Name	Type	Description
`AssetBERTMLMDataset`	`AssetBERTMLMDataset`	The prepared dataset.

DataLoader_Preparer()

Bases: Preparer

Preparer for creating DataLoader objects from Dataset.

This preparer focuses on wrapping Dataset objects into DataLoader, separating the concern of "how to load data in batches" from dataset creation.

prepare(dataset: Dataset, shuffle_override: Optional[bool] = None) -> DataLoader

Prepare and return a DataLoader object.

Parameters:

Name	Type	Description	Default
`dataset`	`Dataset`	PyTorch Dataset object to wrap.	required
`shuffle_override`	`Optional[bool]`	Override the shuffle setting from config. Useful for forcing shuffle=False on validation loaders.	`None`

Returns:

Name	Type	Description
`DataLoader`	`DataLoader`	The prepared dataloader.

AssetBERT_Preparer()

Bases: Preparer

Accelerator_Preparer()

Bases: Preparer

Creates a configured HuggingFace Accelerator instance.

Handles compile cache environment variables, TF32 flags, TorchDynamoPlugin creation, and Accelerator instantiation.

Does NOT call accelerator.prepare(model, optimizer, ...) — that stays in Trainer.

prepare() -> Accelerator

Prepare and return an Accelerator instance.

Returns:

Name	Type	Description
`Accelerator`	`Accelerator`	The configured Accelerator.

Optimizer_Preparer()

Bases: Preparer