General Configuration Properties
The following general configuration properties are available as part of the
cloudprovider.properties file at ./datasynapse/manager-data/hooks/Director/cloudprovider.properties. These properties are used for all cloud service providers.
Adapter - General Properties
| Property | Description |
|---|---|
ServiceProvider |
Specify the service providers to be used with DataSynapse GridServer® Director.
Valid values: AWS, AZURE, or GCP.
|
AZURE_VM_CORE, GCP_VM_CORE |
The number of CPU cores of the VM size selected for the Engine. Ensure that the VM size mentioned in the VM scale set or instance group has vCPU equal to this property.
|
AWS_ENGINE_WEIGHT, AZURE_ENGINE_WEIGHT, GCP_ENGINE_WEIGHT |
Weightage of DataSynapse GridServer® Engine for a particular cloud provider. |
BROKER_PROTOCOL |
Protocol of DataSynapse GridServer® Broker.
Example: HTTP or HTTPS.
|
BROKER_HOSTNAME |
List of DataSynapse GridServer® Broker hostname on which adapter engines will configure. It only contains standalone DataSynapse GridServer® Broker, not failover Brokers. |
AGGRESSIVE_SCALEDOWN |
Set this property to true to let the DataSynapse HPCCA scale down the DataSynapse GridServer® Engine as soon as the task is completed.
If this property is set to true, then DataSynapse HPCCA skips the ENGINEIDLETIMEOUT check and quickly scales down the DataSynapse GridServer® Engine when it finds that no task is running. If this property is set to false, then DataSynapse HPCCA starts calculating the idle timeout for the DataSynapse GridServer® Engine, and after the idle timeout exceeds the DataSynapse HPCCA spins down the DataSynapse GridServer® Engine.
|
| MINIMUM_BUSY_ENGINE | This property is used to check count of busy engine on engine daemon, If HPCCA finds the number of busy engines on engine daemon is less than or equal to MINIMUM_BUSY_ENGINE, the engines on engine daemon will be parked and becomes candidate for aggressive scaledown. |
AGGRESSIVE_SCALEDOWN_RECHECKINTERVAL |
The time interval to monitor the working state of DataSynapse GridServer® Engine for aggressive scale down. |
AWS_ENGINE_OS, AZURE_ENGINE_OS, GCP_ENGINE_OS |
It is the Engine OS for a particular cloud provider. |
WEBSERVICEUSER and WEBSERVICEPW |
The user ID and password in DataSynapse GridServer® that the adapter uses to interact with the DataSynapse GridServer® web service API.
The adapter calls the web service API to determine if DataSynapse GridServer® Engine has connected to the director.
|
GRIDSTATUSRECHECKINTERVAL |
Internal algorithm interval to check the need for VMs and either open new VMs or stop the idle ones based on the calculated need.
|
ENGINEIDLETIMEOUT |
The period for which the DataSynapse GridServer® Engine can be idle before it is flagged for termination, assuming the number of DataSynapse GridServer® Engine exceeds the MINENGINES value. |
MINENGINES and MAXENGINES |
The minimum and maximum number of DataSynapse GridServer® Engine VMs that the adapter is allowed to create. If MINENGINES is greater than 0, the adapter attempts to start DataSynapse GridServer® Engine when the server is started (provided that it does not find any VMs that appear to be DataSynapse GridServer® Engine VMs in the available region for the cloud service provider.) |
ALGORITHM |
The algorithm currently in use to determine the number of DataSynapse GridServer® Engine VMs needed at any time. Only the
BASIC algorithm is defined.
In the basic algorithm, we take the number of Engines needed as the following:
Engines needed = (number of tasks that can run on the cloud) x (the allocation percentage as a decimal)
Example
If we have 10 Engines at the moment, our allocation percentage is 0.1 and the number of tasks we are running increases to 200, the basic algorithm starts 10 new Engines (200 x 0.1 = 20 Engines, 20 Engines needed - 10 active Engines = 10 new Engines needed)
|
TASKCOUNTFLOOR |
This specifies the minimum number of pending tasks before DataSynapse GridServer® Engine VMs are started. This value is subtracted from the pending task total (minimum zero) when the BASIC algorithm determines the Engine need. |
ALLOCATIONPERCENTAGE |
This is specific to the BASIC algorithm and determines the number of DataSynapse GridServer® Engine VMs needed by multiplying the number of pending tasks by the allocation percentage to arrive at a target number of DataSynapse GridServer® Engine VMs. It then compares this value with the current number of DataSynapse GridServer® Engine to determine how many DataSynapse GridServer® Engine VMs must be created. |
CONSIDER_NON_HPCCA_ENGINES |
Set this property to true if DataSynapse HPCCA must consider non- DataSynapse HPCCA engines while calculating the engine need.
true:
For example, if the DataSynapse GridServer® Broker has 20 engines (DataSynapse HPCCA + non- DataSynapse HPCCA) and MAXENGINES property is set to 24, then DataSynapse HPCCA can open only 4 new engines until the value of MAXENGINES property is increased.
|
BATCH_VM_TERMINATION |
Set this property to true to terminate VMs in batch size.
Note: For now batch size VM termination is only supported in GCP (for AWS and Azure there are some challenges from cloud provider side).
|
BATCH_VM_COUNT |
Count of VMs to terminate in batch size.
|
USE_HPPCA_AUTO_RESTART |
Set this property to true if HPCCA needs to restart itself (after making any changes to cloudprovider.properties file) without restarting DataSynapse GridServer® Manager |
DETECT_CHANGES_INTERVAL |
Time interval for HPCCA to check for changes in cloudprovider.properties file.
|
Related information