pdmlabs.RunExperiment

pdmlabs.RunExperiment#

Experiment orchestration and execution for predictive maintenance anomaly detection.

This module provides the main entry point for running predictive maintenance experiments across different learning paradigms (supervised, unsupervised, semi-supervised) with automatic hyperparameter optimization.

Key Features:

Multi-method, multi-experiment orchestration
Hyperparameter optimization using MANGO (Bayesian optimization)
Pipeline construction (preprocessing → method → postprocessing → thresholding)
MLflow integration for experiment tracking
Support for 7 experiment types with different learning strategies
Constraint-based parameter validation for different experiment types
Parallel execution for hyperparameter search

Core Functions:

run_experiment: Main entry point to execute predictive maintenance experiments get_method_type: Map experiment type to appropriate method interface is_port_in_use: Check port availability for MLflow tracking server run_mlflow_server: Start or verify MLflow tracking server

Example

>>> from pdmlabs.RunExperiment import run_experiment
>>> from pdmlabs.utils.dataset import Dataset
>>> dataset = Dataset(data, datetime_column='timestamp')
>>> methods = [IsolationForest()]
>>> param_spaces = [{'n_estimators': [100, 200]}]
>>> results = run_experiment(
...     dataset=dataset.get_rul_dataset()[0],
...     methods=methods,
...     param_space_dict_per_method=param_spaces,
...     method_names=['IF'],
...     experiments=[UnsupervisedPdMExperiment],
...     experiment_names=['Baseline'],
...     MAX_RUNS=20
... )

Functions

`get_method_type`(experiment)	Map experiment class to corresponding method interface.
`is_port_in_use`(host, port)	Check if a given port is in use on the specified host.
`run_experiment`(dataset, methods, ...[, ...])	Execute predictive maintenance anomaly detection experiments with hyperparameter optimization.
`run_mlflow_server`(mlflow_port)	Start or verify MLflow tracking server for experiment logging.

pdmlabs.RunExperiment.get_method_type(experiment)#

Map experiment class to corresponding method interface.

Determines which method interface (supervised, unsupervised, or semi-supervised) is required for a given experiment type. This ensures the correct method base class is used during method instantiation.

Parameters:: experiment (type) – Experiment class (not instance). One of: - AutoProfileSemiSupervisedPdMExperiment - IncrementalSemiSupervisedPdMExperiment - SemiSupervisedPdMExperiment - UnsupervisedPdMExperiment - SupervisedPdMExperiment - SupervisedRULPdMExperiment - Supervised_SA_PdMExperiment
Returns:: Method interface class: - SemiSupervisedMethodInterface for semi-supervised experiments - UnsupervisedMethodInterface for unsupervised experiments - SupervisedMethodInterface for supervised/RUL/SA experiments
Return type:: type
Raises:: ValueError – If experiment type is not recognized.

Examples

>>> from pdmlabs.RunExperiment import get_method_type
>>> from pdmlabs.experiment.batch.unsupervised_experiment import UnsupervisedPdMExperiment
>>> interface = get_method_type(UnsupervisedPdMExperiment)
>>> print(interface.__name__)
'UnsupervisedMethodInterface'

pdmlabs.RunExperiment.is_port_in_use(host, port)#

Check if a given port is in use on the specified host.

Attempts a socket connection to verify if a port is listening. Useful for checking if a server (e.g., MLflow UI) is already running.

Parameters:

host (str) – Host IP address or hostname (e.g., ‘127.0.0.1’, ‘localhost’, ‘0.0.0.0’).
port (int) – Port number to check (0-65535).

Returns:

True if port is in use (connection succeeds), False if available.

Return type:

bool

Examples

>>> is_port_in_use('127.0.0.1', 5000)
False  # Port 5000 is free

>>> is_port_in_use('127.0.0.1', 8080)
True   # Port 8080 is in use

Notes

Quick check: returns result once connection is attempted
Safe: uses context manager to ensure socket is properly closed
Non-blocking: does not hang on connection refused

<<<<<<< Updated upstream pdmlabs.RunExperiment.run_experiment(dataset, methods, param_space_dict_per_method, method_names, experiments, experiment_names, additional_parameters={}, MAX_RUNS=1, MAX_JOBS=1, INITIAL_RANDOM=1, profile_size=2, fit_size=None, postprocessor=<class 'pdmlabs.postprocessing.default.DefaultPostProcessor'>, preprocessor=<class 'pdmlabs.preprocessing.record_level.default.DefaultPreProcessor'>, thresholder=<class 'pdmlabs.thresholding.constant.ConstantThresholder'>, mlflow_port=None, debug=True, optimization_param='AD1_AUC', maximize=True, custom_evaluators=None)#

pdmlabs.RunExperiment.run_experiment(datasetmethodsparam_space_dict_per_methodmethod_namesexperimentsexperiment_namesadditional_parameters={}MAX_RUNS=1MAX_JOBS=1INITIAL_RANDOM=1profile_size=Noneinitial_profile_size=Nonepostprocessor=<class 'pdmlabs.postprocessing.default.DefaultPostProcessor'>preprocessor=<class 'pdmlabs.preprocessing.record_level.default.DefaultPreProcessor'>thresholder=<class 'pdmlabs.thresholding.constant.ConstantThresholder'>mlflow_port=Nonedebug=Trueoptimization_param='AD1_AUC'maximize=Truecustom_evaluators=Noneoptimizer: str = 'mango'use_cache: bool = False)#

Execute predictive maintenance anomaly detection experiments with hyperparameter optimization.

Orchestrates complete experiments: constructs pipelines, performs hyperparameter search, logs results to MLflow, and returns optimal parameters. Supports multiple methods and experiments with cross-product evaluation (each method × each experiment combination).

Parameters:

dataset (dict) – Dataset configuration dictionary (typically from Dataset.get_*_dataset() methods). Must contain: - ‘target_data’: List of test feature dataframes - ‘target_sources’: List of test source identifiers - Additional dataset metadata (see loadAnomalyDetectionDataset.py)
methods (list) – List of instantiated method classes. Each method should inherit from MethodInterface. Order must correspond to param_space_dict_per_method and method_names.
param_space_dict_per_method (list[dict]) – Hyperparameter search spaces for each method. Each dict maps parameter names to lists of candidate values. Example: [{‘n_estimators’: [50, 100], ‘max_samples’: [256, 512]}]
method_names (list[str]) – Human-readable names for each method (for logging). Used in MLflow experiment naming and artifact paths.
experiments (list[type]) –
List of experiment class types (not instances) to execute. Supported: AutoProfileSemiSupervisedPdMExperiment, IncrementalSemiSupervisedPdMExperiment,

UnsupervisedPdMExperiment, SemiSupervisedPdMExperiment, SupervisedPdMExperiment, SupervisedRULPdMExperiment, Supervised_SA_PdMExperiment

Each method will be evaluated on all experiments (cross-product).
experiment_names (list[str]) – Human-readable names for each experiment (for logging/identification).
additional_parameters (dict, default={}) – Extra hyperparameters for pipeline components (preprocessing, postprocessing, thresholding). Key format: ‘{component}_{param_name}’ (e.g., ‘postprocessor_window_length’, ‘preprocessor_features’). Values should be lists of candidate values (for grid search).
MAX_RUNS (int, default=1) – Maximum number of hyperparameter configurations to evaluate per method-experiment pair. Higher values allow more thorough exploration but increase computation.
MAX_JOBS (int, default=1) – Number of parallel processes for hyperparameter search (via MANGO). Typically 1-8 depending on system CPU cores.
INITIAL_RANDOM (int, default=1) – Number of initial random hyperparameter samples before Bayesian optimization. Provides diversity in the exploration phase.
profile_size (int or list[int], default=2) – Historical buffer size (number of samples) used by online methods. If list: allows multiple buffer sizes to be tested. For online/streaming evaluation: how much historical data to keep.
fit_size (int or list[int], optional) – Initial profile size (“warm-up” buffer) before evaluation begins. If None: defaults to profile_size. Used in AutoProfile and Incremental experiments for initialization.
postprocessor (type, default=DefaultPostProcessor) – Post-processing class for score smoothing/normalization. =======
profile_size (list[int], optional) – Candidate sizes (in records) of every profile collected after the first one in a target source. AutoProfile re-profiles at each reset date; this is the size used for those subsequent profiles. A single int is accepted and wrapped into a one-element list. If None: inferred from the data (see Notes). Only used by AutoProfileSemiSupervisedPdMExperiment.
initial_profile_size (list[int], optional) – Candidate sizes (in records) of the first profile collected in each target source, i.e. the slice from the start of the scenario to its first reset date. A single int is accepted and wrapped into a one-element list. If None: inherits the values of profile_size. Only used by AutoProfileSemiSupervisedPdMExperiment.
use_cache (bool, default=False) – Reuse metrics from a previous FINISHED MLflow run whose parameters match, instead of re-evaluating that configuration. Matching spans every run in the experiment, including earlier invocations under the same experiment name, so results then depend on leftover MLflow history. Off by default.
postprocessor (type, default=DefaultPostProcessor) – Post-processing class for score smoothing/normalization. >>>>>>> Stashed changes Will be instantiated with appropriate parameters during pipeline construction. Options: DefaultPostProcessor, MovingAveragePostProcessor, MinMaxPostProcessor, etc.
preprocessor (type, default=DefaultPreProcessor) – Pre-processing class for data preparation/transformation. Will be instantiated with appropriate parameters. Options: DefaultPreProcessor, FeatureSelector, MinMaxScaler, etc.
thresholder (type, default=ConstantThresholder) – Thresholding class to convert anomaly scores to binary labels. Will be instantiated with appropriate parameters. Options: ConstantThresholder, SurvToRUL, DynamicThresholder, etc.
mlflow_port (int, optional) – Port number for MLflow tracking UI. If provided, starts/verifies MLflow server. If None: skips MLflow setup (no experiment logging).
debug (bool, default=True) – If True: enables verbose logging and debug messages during experiment execution. If False: minimal logging, only results and warnings.
optimization_param (str, default="AD1_AUC") – Metric to optimize during hyperparameter search. Options: “AD1_AUC”, “AD2_AUC”, “avg_time_to_alarm”, “false_alarm_rate”, etc. See evaluation module for complete list of available metrics.
maximize (bool, default=True) – If True: MANGO maximizes optimization_param. If False: MANGO minimizes optimization_param. Typically True for AUC, F1-score; False for error rate, false alarms.
optimizer (str, default='mango') –
Hyperparameter optimization backend to use. Supported values:
- 'mango': Mango Bayesian optimizer (Gaussian Process surrogate, parallel via joblib).
- 'mango_random': Mango in pure random-search mode.
- 'smac': SMAC3 HyperparameterOptimizationFacade (parallel via native workers).
- 'gpyopt': GPyOpt Bayesian optimization with batch acquisition (parallel via joblib). Requires pip install pdmlabs[gpyopt].
- 'hyperopt': Hyperopt TPE (sequential only; emits a warning when MAX_JOBS > 1). Requires pip install pdmlabs[hyperopt].
- 'optuna': Optuna 5 TPE with multivariate mode and constant_liar strategy (parallel via joblib + JournalStorage). Requires pip install pdmlabs[optuna].
The value is also logged as an MLflow parameter per run for traceability.

Returns:

list[dict] – Best hyperparameters for each method-experiment combination (in execution order). Each dict maps parameter names to optimal values discovered by MANGO. Length = len(methods) × len(experiments)
Pipeline Construction – For each method-experiment pair: 1. Create PdMPipeline with: preprocessor → method → postprocessor → thresholder 2. Configure AUC resolution=30 (granularity of ROC curve) 3. Assign experiment type (Supervised/Unsupervised/SemiSupervised) 4. Build search space: pipeline params + method params + additional params
Constraint Functions – Different experiments apply parameter constraints: - AutoProfileSemiSupervisedPdMExperiment: max_wait_time constraints - IncrementalSemiSupervisedPdMExperiment: incremental + max_wait constraints - UnsupervisedPdMExperiment: SAND or distance-based constraints (KNN) or max_wait constraints

Examples

>>> from pdmlabs.RunExperiment import run_experiment
>>> from pdmlabs.utils.automatic_parameter_generation import online_technique
>>> from pdmlabs.method.unsupervised_method import IF

>>> # Load dataset
>>> dataset_obj = Dataset(data, 'timestamp', failure_column='failure')
>>> train_data, val_data = dataset_obj.get_rul_dataset()

>>> # Configure experiment
>>> methods = [IF]
>>> param_spaces = [online_technique('IF', maximum_profile=500)]
>>> best_params = run_experiment(
...     dataset=train_data,
...     methods=methods,
...     param_space_dict_per_method=param_spaces,
...     method_names=['IF'],
...     experiments=[UnsupervisedPdMExperiment],
...     experiment_names=['Online'],
...     MAX_RUNS=20,
...     MAX_JOBS=4,
...     INITIAL_RANDOM=2,
...     mlflow_port=5000,
...     optimization_param='AD1_AUC'
... )
>>> print(best_params[0])  # Best parameters for IF in Online experiment

Notes

Important: initial_profile_size defaults to profile_size if not specified. When both are None, profile_size is inferred from the target data as max_wait = ceil(min_target_scenario_len / 3), then spread over 16 integer candidates in [max_wait // 10, max_wait] (see pdmlabs.utils.automatic_parameter_generation.profile_values()). initial_profile_size then inherits those values.
MANGO parameters (num, jobs, initial_random) calculated from space size and MAX_RUNS
Constraint functions prevent invalid hyperparameter combinations
MLflow artifacts saved to: ./artifacts/{experiment_name} artifacts/
All experiments execute in sequence (not parallel), but internal MANGO uses parallelization
Method-experiment cross-product: if 2 methods × 3 experiments = 6 total runs

pdmlabs.RunExperiment.run_mlflow_server(mlflow_port)#

Start or verify MLflow tracking server for experiment logging.

Checks if MLflow UI server is already running on the specified port. If not, starts a new MLflow UI process listening on localhost.

Parameters:: mlflow_port (int) – Port number for MLflow UI server (e.g., 5000, 8080).
Returns:: Prints status messages but returns nothing.
Return type:: None

Notes

Host is hardcoded to 127.0.0.1 (localhost)
Starts MLflow in non-blocking mode (subprocess)
Assumes ‘mlflow’ command is available in system PATH
Multiple calls with same port: only first actually starts server
Server can be accessed at http://127.0.0.1:{mlflow_port}

Examples

>>> run_mlflow_server(5000)
MLflow server started at http://127.0.0.1:5000.

>>> run_mlflow_server(5000)  # Second call
MLflow server is already running at http://127.0.0.1:5000.

pdmlabs.RunExperiment

Contents

pdmlabs.RunExperiment#