5.5 Default 'config.properties' File
This section describes the 'archetypal' config.properties.
In other words, this documents all the configuration properties built into CloudTran,
and defines defaults as you would for Java properties.
See the documentation
of 'config.properties' files for background information.
There are four types of values you can specify:
- boolean - "true" or "false"
- classes - the fully-qualified class name of an available class. e.g. com.cloudtran.persist.JdbcLoaderFramework
- int - the simple format is a number.
You can also specify "p/d", where 'p' is the numeric value to be used in production and 'd' when DEBUG is set.
e.g. 800/200 means the default value is 800 in production, 200 when in DEBUG mode (i.e. when trace.debug is set).
- a list of int's, separated by commas. As for 'int' above, each value can be a single number or 'p/d' for production/debug.
e.g. "30/300, 60/600, 100/1000, 250/2500, 500/5000"
5.5.1 'config.overrides' Property
|
This property allows you to define a file for override properties.
The value of the property must be a valid file name (not a 'jar' resource).
It is a fatal error if this property is specified and the named file does not exist.
This is intended to make it easy to run variants of tests without a complete rebuild and redeployment.
Properties defined in the override properties file override values from config.properties;
they in turn can be overridden by system properties (-D... on the command line).
This property's value can be specified either as a system property (-D on the Java command line) or from an entry in config.properties.
One way to use this in testing would be to specify the name of a shared file that is accessible from all deployments.
The specification could be put the filename in the Application model object's 'configProperties' -
the override would be available in all deployments.
This is the second property read, immediately after the 'config.propertiesReaderClass' property.
5.5.2 'config.propertiesReaderClass' Property
|
config.propertiesReaderClass = com.cloudtran.util.ConfigPropertiesReader
This is the name of the class that reads the configuration properties.
You may want to override the default value to pull in values from somewhere other than 'config.properties'.
For example, this class could have operators type in passwords manually.
This property is the very first property used.
If you want to override it, this **must** be entered as a system property (java -Dconfig.propertiesReaderClass=x.y.z).
Once this property is read, CloudTran
- resolves the class name
- attempts to load it and instantiate an instance of it
- calls the getConfigProperties() to get the overriding properties.
If you want to just override a few parameters, you can do this:
- extend the built-in reader
- this means all the normal config.properties processing occurs
- but then override the result for special properties you want to change or protect
class MyPropertiesReader implements IConfigPropertiesReader
extends com.cloudtran.util.ConfigPropertiesReader
{
public Properties getConfigProperties()
{
Properties rv = super.getConfigProperties();
//
// now tweak 'rv' as you need
//
}
}
5.5.3 'ct.cohort.distCommitThreadPoolSize' Property
|
ct.cohort.distCommitThreadPoolSize = 32
This is the thread pool at the cohort that handles the commit/abort of sub-transactions,
on instruction from the Coordinator.
Each message sent by the Coordinator can contain many commit/abort instructions -
150 transactions being sent from the Transaction Manager to the cohort per message is not uncommon.
Each instruction is then delegated to an executor and thread in this pool.
The size of the thread pool is determined by the maximum amount of parallelism that makes sense.
Parallelism is useful because there may be a bottleneck getting to the local space, but
then any changes are sent to the backup too - this can be overlapped with the local activity.
Writing an object to a backed-up space takes about 16X more than to a non-backed-up space.
The default allocation is For now, we allocate 32.
Note that we can't aggregate on the other side of this - the commit to the space.
The SubTxInfo must be committed along with the user's info in the transaction,
so each transaction must be committed separately.
5.5.4 'ct.cohort.enrolBackoffMS' Property
|
#ct.cohort.enrolBackoffMS = 30/300, 60/600, 100/1000, 250/2500, 500/5000
This is the length of backoff time, in milliseconds, that the Cohort.enrol() will back off if the ct.cohort.maxSubtransactions is exceeded at a given cohort.
There is no default - in otherwise, by default, CloudTran failes immediately,.
If you do want to provide automatic backoff in the Cohort, then here is a possible set of values:
5.5.5 'ct.cohort.maxSubtransactions' Property
|
ct.cohort.maxSubtransactions = 10000/500
This is the maximum number of active subtransactions that can be started into a *cohort*.
A subtransaction object for these purposes is allocated when the first app program enrols into a given distributed transaction.
Further enrol() calls with the same DistributedTxAttributes do not create another cohort subtransaction.
The subtransaction is freed up when it is committed or aborted by a client call to the Transaction Manager.
(The commit call in the cohort simply flushes cached information.)
If a thread tries to enrol in a distributed transaction for the first time and this limit is breached,
the Cohort holds onto the thread and tries to allow it later, by backing off using the 'ct.cohort.enrolBackoffMS' list.
Once the limit is hit the cohort returns the TransactionExceptionCohortTooBusy.
It is a good idea to have a reasonable maximum here.
It applies back-pressure to the clients and therefore prevents a meltdown caused by a rogue app.
Tests have shown that higher values here slow the system down - so that's the downard pressure.
The upward pressure is that services in the cohort may unnecessarily time out.
By default, the maximum number of active subtransactions in production is 10,000 and 500 in debug.
5.5.6 'ct.cohort.subTxStatusLingerTimeMS' Property
|
ct.cohort.subTxStatusLingerTimeMS=60000
ct.cohort.subTxStatusLingerTimeMS is the number of milliseconds that a SubTransactionStatusEntry record should stay
in the Cohort's memory after it has completed (i.e. it has been aborted, or committed+logged+persisted).
The same consideration as for ct.txb.txStatusLingerTimeMinutes applies here, namely:
the amount of memory required to keep 'lingering' SubTransactionStatusEntry objects can be very high
so take that into account if you increase this number.
5.5.7 'ct.logger.directory' Property
|
ct.logger.directory=transactionLogs
The log directory is where the transaction logs go, for transactions sent through CloudTran.
(It has nothing to do with logging and tracing of program events.)
The path is relative to the coordinator PU project directory - e.g. myWorkSpace/myAppCoordinatorPU (which admittedly is not very clever).
If there is a crash of a complete processing unit, do not change this value - this is where
any un-committed transactions are stored.
If you reconfigure after a crash and before log replay has completed,
CloudTran will not be able to find this information.
The log directory holds the a number of subdirectory which is where the information files are stored.
The number of subdirectories is defined by the 'logSubDirectoryCount' property, described below.
The Transaction Log directories are put in the directory "<MainProject>_CoordinatorPU/.transactionLogs".
The information store in these logs is for crash recovery. It is removed when a transaction is committed.
Note that the default setting of 'ct.logger.directory' should not be used in Amazon EC2
or similar public cloud, where the local disk is no longer accessible after a crash.
In EC2 for example, you should map the log directory to an EBS (Elastic Block Storage) volume.
5.5.8 'ct.logger.diskBlockSize' Property
|
ct.logger.diskBlockSize = 4096
The size of the block size we should round up to.
Every write to the transaction log files starts on this boundary -
extra bytes at the end of a transaction's write are ignored.
This is to try to minimise a power out while writing one transaction affecting the previous transaction.
The standard block size on modern drives is 4096 - change this at your peril.
5.5.9 'ct.logger.diskSpaceLowBusyCount' Property
|
ct.logger.diskSpaceLowBusyCount = 3
This is the number of full files-worth of free space that must remain on log disk partition for processing to continue.
If there is less that this amount, then the TxBufferManager.start() method throws TransactionExceptionTxbTooBusy.
This does not affect transactions that have already started.
The default is 3, which, for the default ct.logger.fileSizeMB value of 100MB, gives 300MB of disk space.
For typical transaction sizes, this allows o(1 million) in-flight transactions to complete before disk space runs out.
5.5.10 'ct.logger.diskSpaceLowWarningCount' Property
|
ct.logger.diskSpaceLowWarningCount = 10
This is the number of full files-worth of free space that must remain on log disk partition.
If there is less than that, a "low log space" warning is issued
If not specified, the default value is 10x the size of a log file - giving space for 10 log files from this point on.
5.5.11 'ct.logger.disposer' Property
|
ct.logger.disposer = com.cloudtran.log.LogDefaultDisposer
This names the class of the log disposer, which responsible for logfile disposing.
This class must implement ILogDisposer.
The default implementation deletes the file once all transactions listed in the log file as 'persisting' have been successfully persisted.
5.5.12 'ct.logger.fileBaseName' Property
|
ct.logger.fileBaseName = Logfile_
Logfile base name. By default this names log files like "Logfile_yyyy.MM.dd.HH.mm.ss.SS"
5.5.13 'ct.logger.fileDatePattern' Property
|
ct.logger.fileDatePattern = yyyyMMdd_HHmmss.SSS
The date pattern what should be included into the logfile name.
Ends up looking like 20100211_173000.500 (11 Feb 2010, 5:30 pm, plus 500 milliseconds).
5.5.14 'ct.logger.fileExtension' Property
|
ct.logger.fileExtension = log
Extension for the log file.
5.5.15 'ct.logger.fileSizeMB' Property
|
ct.logger.fileSizeMB = 20/5
The file size of each log file in megabytes.
The considerations on this are
- increase to reduce disk allocation/switching time
- reduce to (1) improve convenience for operators (2) reduce risk of files being lost.
The default log file size is 20MB in production, 5MB in DEBUG mode
5.5.16 'ct.logger.logAfterCommit' Property
|
ct.logger.logAfterCommit=false
ct.logger.logAfterCommit enables logging of the transaction to a private disk (non-database or other datasource).
This is intended to be another improvement to resilience.
We can still reliably continue operations, without depending purely on the backed-up space for operations.
This flag is not relevant if 'ct.logger.logBeforeCommit' is set.
5.5.17 'ct.logger.logBeforeCommit' Property
|
ct.logger.logBeforeCommit=true
'ct.logger.logBeforeCommit' requires the Transaction Manager to log a transaction before returning from (i.e. positively acknowledging) a 'commit()' method.
If this flag is set, a transaction may not proceed if there is a problem writing the log to disk
5.5.18 'ct.logger.maxWriteSize' Property
|
ct.logger.maxWriteSize = 5242880
The amount of data, in bytes, below which the log thread will keep processing outstanding committed transactions.
Once the amount of data to be logged exceeds this number, the buffered data is written out.
Default is 5242880, which is 5 MB.
5.5.19 'ct.logger.threadCount' Property
|
ct.logger.threadCount=3
if ct.logger.logAfterCommit is true and ct.logger.logBeforeCommit is false, then a number of threads will be created to do the logging and deleting of logs after persistence.
ct.logger.threadCount defines the number of threads. It must be between 1 and 5 (if it is relevant).
Don't change this unless you know what you are doing. >3 is probably a waste of a thread.
5.5.20 'ct.logger.zapLogFiles' Property
|
ct.logger.zapLogFiles=false
This of course is very dangerous!!!
It will zap the log files at start of day, without attempting recovery of transactions to disk.
This will be useful after a fast recovery or during unit testing.
In production or user test deployments, it is safer to run the "Log File Replay" mechanism.
5.5.21 'ct.persist.driverClassLoaderClass' Property
|
ct.persist.driverClassLoaderClass=com.cloudtran.persist.DriverClass
Before starting the data aload process, CloudTran first tries to load all the necessary drivers.
These are the classes as named in the datasource.*.driverClass config parameters.
Should you wish to change the way this operates, you can call your own class,
which must implement the ICloudTranDriverClassLoader interface
and the 'void loadDrivers()' method.
Normally you don't need to change this property - the default driver loader can load JDBC and non-JDBC drivers.
5.5.22 'ct.persist.loadAtStartOfDay' Property
|
ct.persist.loadAtStartOfDay=true
Normally CloudTran loads all data at start of day.
This means, it reads the data for all entities and sends them to the appropriate all the PU space.
(For partitioned, spaces, CloudTran sends the data records to the correct space.)
For an entity named 'E', this data goes into instance of the class named 'XData'.
The default is that this is done.
If you want to start a space without doing this (for example, during testing), specify 'false' for this value.
Note that when loadAtStartOfDay is false, the database is not cleaned out by the load mechanism.
You should probably do this (otherwise, created objects will have a duplicate primary key).
To help you do this, master entities have a 'clearEntityGroupSpaceTables()' method,
which will delete all database rows for the complete entity group.
5.5.23 'ct.persist.maxRowsForDataSourceTransaction' Property
|
ct.persist.maxRowsForDataSourceTransaction=100/4
When collecting objects to send to the databases, CloudTran aggregates a number of transactions together.
It is possible to do this because CloudTran knows the object identity and detects when two transactions share an object
(because it is of the same class and primary key).
During the aggregate process, CloudTran checks the number of rows already collected for the next database transaction.
If this number is >= to 'ct.persist.maxRowsForDataSourceTransaction', then no more rows are added.
Tests show that it is more effective to use smaller chunks (i.e. not 512) and more threads (e.g. 15) when the database is MySQL.
For other databases, it's worth playing with this parameter and the threadsPerDatasource above.
5.5.24 'ct.persist.overallLoaderClass' Property
|
ct.persist.overallLoaderClass=com.cloudtran.persist.JdbcLoaderFramework
In production systems, the overallLoaderClass comes from the cloudsaveMappingFile, <cloudsave> element.
For testing purposes this can be overridden, in particular set to empty, to avoid loading from the database
5.5.25 'ct.persist.threadsPerDatasource' Property
|
ct.persist.threadsPerDatasource=15/2
ct.persist.threadsPerDatasource specifies the number of threads that will forward transactions to each datasource.
It must be between 1 and 20. The default is 15 (or 2 in debug mode), which is appropriate for remote databases.
For a local database (e.g. in testing - this deployment doesn't make a lot of sense in production), start with 2.
5.5.26 'ct.replayer.continueAfterError' Property
|
ct.replayer.continueAfterError = false
This property determines if the replayer continues the processing in case of unrecoverable error.
This applies to the first error seen; the replayer will always stop at the second unrecoverable error.
5.5.27 'ct.replayer.retryCount' Property
|
ct.replayer.retryCount = 5
Retry count in case of SQLRecoverableException or SQlTransientException.
5.5.28 'ct.txb.commitToCohortDelay' Property
|
ct.txb.commitToCohortDelay=2
ct.txb.commitToCohortDelay is the number of milliseconds that the coordinator will wait
to gather a group of commit or abort instructions to a particular cohort.
The higher this value, the more instructions are contained in each message sent to the cohort,
and the better the efficiency, because network bandwidth is saved.
The lower this value, the faster the message gets to the cohort: a value of 0 means
there is no wait for additional instructions to arrive before transmitting to the cohort
(although any additional instructions pending will be grouped into one message -
in high-through situations, there will be multiple instructions per message).
The average additional latency per message is roughly half of this value during high-throughput.
5.5.29 'ct.txb.commitToCohortsBeforeTxManagerCommitReturns' Property
|
ct.txb.commitToCohortsBeforeTxManagerCommitReturns=true
ct.txb.commitToCohortsBeforeTxManagerCommitReturns instructs the Transaction Manager to commit
the subtransactions at all cohorts before returning from commit call.
By default, this is true - which is the safe but slower approach.
Setting it to false improves response time - the transaction manager returns quicker.
However, there is then a small possibility of reading stale information after the transaction commits -
this can be important in test scenarios.
This will not be a problem if the application carries on to use the data in an update transaction,
because the mismatch will be caught by optimistic locking.
To present important information to the user that it is very important to make consistent, the information should be read under a transaction.
5.5.30 'ct.txb.defaultTimeoutSeconds' Property
|
ct.txb.defaultTimeoutSeconds = 10/300
The default timeout, in seconds. 10 seconds normally, 300 in debug mode.
If this is too small, transactions will time out. If it is too large, deadlocked threads will hang for that amount of time.
- seconds in production should be generous; 5 seconds would probably work well too.
5.5.31 'ct.txb.leaveDistributedTxAttributes_bizTxId_Null' Property
|
ct.txb.leaveDistributedTxAttributes_bizTxId_Null=false
ct.txb.leaveDistributedTxAttributes_bizTxId_Null, if true, tells CloudTran *not* to allocate a business transaction ID
if one is not specified in the bizTxId field of the DistributedTxAttributes parameter passed to the TxBufferManager.start() method.
The default is that this is false, which means that a business transaction id will be allocated for you, if you do not set it.
BizTxIds allocated by CloudTran look like this: "Biz81286096419765509B" - they start with "Biz8" and end with B.
They are guaranteed to be unique across the grid.
5.5.32 'ct.txb.logwriteBufferTime' Property
|
ct.txb.logwriteBufferTime = 500
This is the number of microseconds to leave for the write to process.
The default is half a millisecond (500 miscroseconds).
The actual tx write buffering time therefore is microsPerLogWrite minus logwriteBufferTime.
In practice, you don't need to tune this value:
just tune the microsPerLogWrite value.
5.5.33 'ct.txb.maxTransactions' Property
|
ct.txb.maxTransactions = 12000/600
ct.txb.maxTransactions is the maximum number of *distributed* transactions that can be started at the Transaction Manager.
These are the ones that are in play (i.e. started by a client, and not completed at the Transaction Manager).
A transaction is completed when it is committed by the client, and then logged, persisted and committed to the cohort(s).
When a transaction is completed, its data can be purged from the space, reducing the data requirements at the server.
Note that Cohort.commit() only flushes subtransactions at the cohort, it does not commit the transaction at the Transaction Manager.
Also, it does *not* include ones that are waiting to be completed (to log, cohort or persist)
The default is 12000 in production and 600 in debug.
Things will get nasty if this is too high. The 12000 number is appropriate for a single- or double-socket quad core server.
If there is a lot of memory in the Transaction Manager and you want to cater for the backends going down, then you can set this higher.
Once this limit is breach, the Transaction Manager holds onto the calling transaction and waits
for a number of milliseconds, as specified by the 'ct.txb.startBackoffMS' property.
5.5.34 'ct.txb.microsPerLogWrite' Property
|
ct.txb.microsPerLogWrite = 4000
ct.txb.microsPerLogWrite determines for how long do we wait between each write.
This number is in microseconds.
For physical disks, the idea is that this is the same as the disk's minimum access time - i.e. the time to do one rotation of the disk.
For a 7200 rpm, which does 120 revolutions per second, i.e. one revolution in 8.3 ms,
then this should be set to 8300 (8.3ms expressed in microseconds).
If this is a disk service in a Cloud (i.e. limited by disk access time)
then the time should be set so as to minimise the wasted I/O's (which seem to be the limiting factor for example on EC2/EBS).
Aim for an average write at a 4k boundary. For example, 5 rows/transaction, 100bytes each (serialised) = 500 bytes.
At maximum 2000 transactions/second, this is 1MB/sec for logging, or 1000 bytes/msec.
In that case, 4ms between writes (which is the default) gives 4KB/write every 4 ms.
5.5.35 'ct.txb.numberTxsToTransmitToCohort' Property
|
ct.txb.numberTxsToTransmitToCohort = 50/25
ct.txb.numberTxsToTransmitToCohort is the numbers of transactions that will be sent in one batch to the cohort instance.
The TxBufferCommitOrAbortThread (one per routing entry) will wait until this number of transactions have been collected before being sent to the cohort instance
5.5.36 'ct.txb.primaryKeyAllocationChunkSize' Property
|
ct.txb.primaryKeyAllocationChunkSize=1000
ct.txb.primaryKeyAllocationChunkSize is the number of keys to allocate per request to the KeyGenerationService.
The value here must balance the number of unused primary keys a client may leave unused after each interaction or run,
versus the cost of the calls to allocoate each one.
5.5.37 'ct.txb.startBackoffMS' Property
|
#ct.txb.startBackoffMS = 100/500, 300/800, 500/1250, 500/1250, 1000/2000, 2000/5000
This is the length of backoff time, in milliseconds, that the Transaction Manager start() method will back off if the ct.txb.maxTransactions is exceeded at a given cohort.
There is no default - in otherwise, by default, CloudTran failes immediately,.
If you do want to provide automatic backoff in the Transaction Manager, then here is a possible set of values:
5.5.38 'ct.txb.txStatusLingerTimeMinutes' Property
|
ct.txb.txStatusLingerTimeMinutes=2
ct.txb.txStatusLingerTimeMinutes is the number of minutes that a distributed transaction record should stay
in the Transaction Manager's memory after it has completed (i.e. it has been aborted, or committed+logged+persisted).
Leaving the distributed transaction record to linger has two purposes:
- if a client crashes and wants to check the status of a transaction,
it can call the Transaction Manager's whatHappenedTo() method.
The linger time determines how long after the transaction completes this call can give an answer:
when the transaction record is deleted, this method returns null
- if a client tries to start a transaction with the same business transaction ID as one that
has already committed or aborted, the Transaction Manager detects this situation and fails the start().
This also depends on the distributed transaction record being
in the Transaction Manager's memory.##
- the amount of memory required to keep 'lingering' distributed transaction records can be very high
so take that into account if you increase this number.
For example, at 3,000 transactions per second, a linger time of 10 minutes
will start to impact the overall memory used significantly, by order of 1-2 GB.
5.5.39 'ct.txb.writeAggregationThreads' Property
|
ct.txb.writeAggregationThreads=16
The number of threads to use to send writes to the txb space.
4* the number of cores on the Transaction Manager primary is a good starting point.
A high number will optimise timings; a low number will optimise network I/O's, which may be important in virtualised environments.
5.5.40 'datasource.[dataSource_name].property' Property
|
Datasource property definitions.
You can define the default properties for each datasource in the application model (via Eclipse modeler).
You can override these on the command line/config.properties.
Let's take an example - we want to connect to a MySql database as follows:
DB url = "jdbc:mysql://localhost:3306/warehouseexample"
username = "cloudtran"
password = "cloudtran"
Then
- the default name is "warehouseexample" - note that this is case sensitive
- the config.properties entries to specify properties are:
datasource.warehouseexample.driverClass = com.mysql.jdbc.Driver
datasource.warehouseexample.url = jdbc:mysql://localhost:3306/warehouseexample
datasource.warehouseexample.username = cloudtran
datasource.warehouseexample.password = cloudtran
If you have two datasources, they are discriminated by the second part of the name, e.g.
datasource.warehouseDB2.url = jdbc:mysql://localhost:3307/warehouseDB2
5.5.41 'trace.*' Property
|
The trace.warn/info/debug/trace properties define whether or not to trace at different priority levels.
trace.warn is the most precedent; trace.trace is least precedent, i.e. most detailed
By default, only info and warning traces are written: trace.warn=true and trace.info=true, and trace.debug=false and trace.trace=false.
The level of trace is normally overridden by a '-D...' property on the command line.
If trace.trace specified = true, .debug and .info are automatically set on.
If trace.debug specified = true, .info is automatically set on.
Default values are
- trace.warn = true
- trace.info = true
- trace.debug = false
- trace.trace = false
5.5.42 'trace.adjustNanoTimer' Property
|
trace.adjustNanoTimer = true
Cohorts and others should adjust the nanotimer to allow for round-trip delay.
This is useful in debugging to cross-reference log and trace files on different machines.
5.5.43 'trace.assert' Property
|
trace.assert defines whether assertions should be enabled.
By default (in production) assert handling is off.
You can set this true by 'trace.assert=true' in the config.properties file or on the command line (-Dtrace.assert=true).
If this is not specified, then it defaults to the value of trace.debug - so it will default to true when trace.debug is true.
5.5.44 'trace.file' Property
|
This is the trace file name for this processing unit.
If you do not specify a file (path) here, it will default to the console.
In GigaSpaces, the console log will additionally go to a file like [[GIGASPACES_HOME]]/logs/YYYY-MM-DD~hh.mm-gigaspaces-gsc_#-[[hostname]]-nnnn.log
You can do multiple runs (even simultaneously) with the same target file name.
Say you start with trace.file=MyApp.log. The second run will use the file name MyApp.log.A, the third MyApp.log.B... then MyApp.log.A2 etc.
5.5.45 'trace.fileSizeInMegaBytes' Property
|
trace.fileSizeInMegaBytes=20
The default file size for the trace file is 20 Megabytes.
After that it will roll over to the specified trace file name suffixed by "_n", where n=2, 3, etc.
5.5.46 'trace.formatForDiff' Property
|
trace.formatForDiff=false
If trace.formatForDiff is set true, timers and line numbers are omitted from the trace, to make it easier to use comparison tools, like WinDiff, on the log
5.5.47 'trace.operation.timer' Property
|
trace.operation.timer = false
records the elapsed time of individual operations, like space calls, service calls or locks.
This is for internal performance testing of the CloudTran system. You should not normally need to set this.
The usage of operation timers is to create an operation timer if this value is set.
It gets mapped into the Java variable Trace.OPERATION_TIMER.
The OperationTimer is instantiated with a message, and then the time is logged by the mark() call.
OperationTimer ot = null;
if( Trace.OPERATION_TIMER ) ot = new OperationTimer( "enrol() " + internalTxId );
...
if( Trace.OPERATION_TIMER ) ot.mark();
|
