dxFeed/QDS 3.153 has been released to public.
Available at: /dxfeed-apis/

Changes since last public release were introduced in three releases: 3.151 which is a major release focused on infrastructure, performance and onDemand historical data services integration and 3.152, 3.153 which are focused on dxFeed API extensions.

Notable changes are:
– migration to Java 7
– revised dxlib library with chunged I/O infrastructure and utilities.
– revised tape file R/W architecture giving substantial increases in performance and reliability
– direct support of onDemand historical tick data services in dxFeed API
– dxFeed API: TradeETH event which mirrors Trade behavior but contains only ETH trade information
– dxFeed API: convenience methods for bulk snapshot operations
– dxFeed API: live instrument profile service with intraday push notifications

Full list of changes:

QDS 3.153:

Change Request
* [QD-529] – dxFeed API: add to DXFeed convenient method to retrieve bulk snapshots
* [QD-565] – QD Core: Introduce ability to attach arbitrary objects to subscriptions
* [QD-590] – Tools: Instruments shall print QDS version on start

Defect Report
* [QD-587] – QD: TradeETH record shall be added to built-in QD filters feed, compfeed and regfeed.
* [QD-588] – dxFeed API: Live Instrument Profile connections report EOF with various exceptions on server side
* [QD-589] – dxFeed API: DXFeedSubscription notification on Order events happens from multiple threads

Improvement
* [QD-559] – dxFeed API: Improve performance of multiple getLastEventPromise invocations

QDS 3.152:

Change Request
* [QD-551] – dxFeed API: Introduce Underlying and Series events for calculated option-related values
* [QD-578] – dxFeed API: Hide implementation-specific properties from event classes
* [QD-579] – dxFeed API: Introduce TradeETH event
* [QD-582] – Modularity: Move SSL support to qds.jar from qds-tools.jar
* [QD-583] – Tools: Multiplexor shall support “multiplexor.qd.subscribe.*” JVM properties for permanent subscription
Defect Report
* [QD-584] – dxlib: TimeFormat formatting timezone breaks (switches zone) after parsing string with a non-default timezone
* [QD-585] – dxFeed API: InstrumentProfileCollector does not process “REMOVED” instrument type properly w.r.t. to update time
* [QD-586] – RMICommonTest hangs from time to time in testRMIException

QDS 3.151 – Major release:

• [QD-498] Migrate to Java 7
  – Check that only Java 7 APIs are used during compilation (from rt.jar + jce.jar)
  (allow com.sun.net.httpserver, com.sun.management, and sun.misc.Unsafe)
  – String.isEmpty() usage (mass search and replace)
  – StandardCharsets.UTF_8 is used instead of “UTF-8” string (mass search and replace)
  – ArrayUtil uses Arrays.copyOf
• [QD-497] dxlib: Include com.devexperts.management utilities
• [QD-501] MARS: JMX JVMSelfMonitoring.makeThreadDumps operation
  – Use com.devexperts.mars:type=JVMSelfMonitoring bean to make period thread dumps to stdout of file.
  – Supports a specified number of dumps, with a specified period, and (optionally) scheduled to start at a
  specified time.
  – Can be configured to start at JVM start with the following system options:
  -Dcom.devexperts.mars.jvm.JVMSelfMonitoring.threadDumpsCount=
  -Dcom.devexperts.mars.jvm.JVMSelfMonitoring.threadDumpsFile= (leave empty for stdout)
  -Dcom.devexperts.mars.jvm.JVMSelfMonitoring.threadDumpsPeriod= (leave empty for 1s period)
  -Dcom.devexperts.mars.jvm.JVMSelfMonitoring.threadDumpsScheduledAt= (leave empty to start immediately)
• [QD-159] dxlib: Include com.devexperts.io.Chunked*
  – com.devexperts.io.Chunked* moved to “dxlib” module.
  – BufferedOutput.totalPosition() method is introduced to track the total number of written bytes, which
  is important for StreamOutput and ChunkedOutput.
  – Get rid of Compare-And-Set owner updates and well as from other broken attempts to provide concurrency.
  Chunks and ChunkLists are NOT THREAD-SAFE. Owner tracking is “fail-fast” way to detect errors. It works correctly
  in single-thread, and only does best-effort when somebody tries to work with chunks concurrently.
  – writeFromInputStream, writeFromDataInput, and writeFromByteBuffer utility methods are promoted into BufferedOutput.
  – writeFromChunk and writeAllFromChunkList are introduced in BufferedOutput.
  – There is a special implementation of writeFromChunk in ChunkedOutput that balances the need to have fewer chunks
  by copying small chunks (less that 256 by default) into a single chunk, with a need to avoid copying large chunks
  of data by appending big chunks from the same pool to the list of chunk without actually copying any bytes.
  – readToOutputStream, readToDataOutput, and readToByteBuffer utility methods are promoted into BufferedInput.
  – Default constructor for ChunkedOutput is introduced.
  – getPool() methods are introduced in Chunk and ChunkList.
  – Debug toString() are introduced in Chunk, ChunkList, ChunkPool, ChunkedInput and ChunkedOutput.
• [QD-486] dxlib: Improve mark/reset support in io.BufferedInput
  – mark(int/long) is be deprecated, because the actual value is irrelevant for all implementations of BufferedInput.
  – mark() and unmark() methods without parameters are be introduced.
• [QD-470] Get rid of “plugins as modules” in project structure, so that “mvn clean” correctly works
  – jspc-compiler-plugin is no longer a child module. It is a separately deployed project with a fixed version 1.0
  – dxfeed-codegen is no longer a plugin
  – Switch to maven-enforcer-plugin v1.2 to get rid of extra logging (see MENFORCER-135)
• [QD-461] More speed improvements for reading/writing
  – FileWriter defers file opening until first block is to be flushed and does it in parallel thread, too
  – All QTP-related buffer sizes are moved into a separate QTPConstants class
  – AbstractByteArrayComposer/Parser classes are renamed to AbstractQTPComposer/Parser with the
  corresponding name updates for their descendant classes. ByteArrayComposer/Parser classes are
  deprecated.
  – QTP parsers are now reading bytes from an arbitrary BufferedInput (which would be typically StreamInput,
  ChunkedOutput, or ByteArrayInput) that is set with setInput method. There are no more getProcessed, getBytes,
  removeBytes methods.
  – ByteArrayParser is deprecated. It still parser from ByteArrayInput for backwards compatibility and has all
  the legacy methods (getProcessed, getBytes, removeBytes, etc).
  – InputStreamParser class is introduced as a counterpart to OutputStreamComposer that provides a facade to
  parser a single file with binary QTP data (like a snapshot) that is created with OutputStreamComposer.
  – QTP composers are now writing bytes to an arbitrary BufferedOutput (which would be typically StreamOutput or
  ChunkedOutput) that is set with setOutput method. There are no more getProcessed, getBytes, removeBytes methods.
  – ByteArrayComposer is deprecated. It still writes to ByteArrayOutput for backwards compatibility and has all
  the legacy methods (getProcessed, getBytes, removeBytes).
  – Default QTP composer implementations for Binary/Tesxt/Blob formats limit only the message size in their
  “hasCapacity” method to avoid big data and subscription messages. However, ByteArrayComposer and
  ConnectionQTPComposer override hasCapacity to limit the overall number of bytes buffered while composing message
  to avoid reallocation of byte array (in case of ByteArrayComposer) and to limit the latency
  (in case of ConnectionQTPComposer).
  – MessageConsumer.processOtherMessage(…) method with byte[] data is deprecated,
  MessageConsumerAdapter.processOtherMessage(…) method with BufferedInput data is used instead
  – BufferedInput.hasAvailable() is introduced as a faster way to check if available() > 0
  – BufferedInput.hasAvailable(bytes) is introduced as a faster way to check if available() >= bytes
  – Error dumps now include offset from the beginning of file/stream.
  – BufferedInputPart.totalPosition() is changed to return position from the start of the parent input and
  the contracts of BufferedInputPart class are documented.
  – Error dumps now include bytes following parsed position (when available) for quick visual identification
  on what kind of invalid bytes are there in the file
  – Protected ByteArrayInput.readData(int) is reintroduced as a deprecated method to restore compatibility
  with a legacy code.
  – Default chunk size is 8k (for socket operation, etc)
  – FileConnector uses its own chunk pool with chunks of 64k. The size of the chunk for a file reading/writing
  can be changed with a system property:
  -Dcom.devexperts.qd.qtp.file.chunkSize=65536
• [QD-464] FileAnalysis Tool: Support “.gz” files
  – FileAnalysis now reads files via FileReader, which gives it support for container (gzip), file format detection,
  and split files. However, it works only for binary formats.
• [QD-465] FileAnalysis Tool: Improve reports
  – FileAnalysis tool print number of times each item was encountered and its average size
  – Symbols are categorized with respect to built-in filters (fx, fut, ind, spread, fut_op, bs_opt, other)
  – Percentages in reports are always displayed with respect to the complete file size
  – Distribution of data message body bytes by record category is also printed separately for each symbol category
• [QD-462] FileConnector option to parse messages without record descriptions using a specified scheme
  – Use [schemeKnown] option to file reader to enable this behavior
  – FileAnalysis also parses additional file options.
• [QD-463] BUG FIX: FileConnector shall understand full file paths on windows
  – “file:C:Projectsmy-data-file.data” now works
  – It is also fixed in -Dscheme=… (full jar file path can be specified) and in logging configuration
• [QD-485] RMI shall log request details (who and what) for server-side request-processing errors
• [QD-487] dxFeed API: Disable WildcardSymbol subscription by default and give stronger language in JavaDoc
  – If you need to enable wildcards, you have to contract data provider for configuration and include the
  following option in “dxfeed.properties”: dxfeed.wildcard.enable=true
• [QD-508] BUG FIX: dxFeed JS API: dxfeed-webservice leaks memory when undeployed
• [QD-509] Do not track per-record stats for anonymous agents and distributors (reduces memory consumption)
• [QD-514] Support reading of partial/broken collectors debug dumps and symbol+record analysis
• [QD-516] Misc improvements in debug dump reader and collector debug infrastructure
  – Support full data dumps (ticker and history storage is dumped)
  – Support time in subscription dumps (both total and agent subscription is dumped)
  – Support queue analysis (including known time, sub time, queue status)
  – Filter any data, subscription, queue or symbol reference analysis by symbol and/or record
  – Support multiple, “+” separated, commands on a single line
  – Write everything to stdout in DebugDumpReader
  – Verification checks consistency of total subscription in addition to symbol counters
• [QD-517] FileConnector support for tcpdump-captured streams with “resyncOn” option
  – Binary TCP streams captured with tcpdump can be decoded. There is a built-in login to resync onto the
  potential beginning of the message in the stream with “resyncOn” option that takes an expected message type.
  – For example, to parse a file that is expected to contain history data, use the following command:
  qds dump file:[resyncOn=history_data,schemeKnown,speed=max]
  – Note, that scheme library/class can be provided with -Dscheme= option
  – By default, the byte stream is scanned for a message with a specified type and at most 8100 bytes long.
  – To change this threshold use -Dcom.devexperts.qd.qtp.file.BinaryFileQTPParser.maxResyncMessageLength=xxx
• [QD-538] Add method readUTFBody to BufferedInput and IOUtil.
• [QD-537] Improve subscription filter performance for en-cipher-ed symbols.
• [QD-543] Tools: ConnectionHandler (connect and compare tools) stops working (looses notification) on unexpected
  runtime exceptions
  – QD Collector implementations (Ticker, Stream, and History) are changed to make sure that if the RecordSink to
  which data is retrieved throws exception, then the record that caused the exception is already marked as
  processed and the next invocation of retrieve method skips it.
• [QD-544] dxFeed JS API Chart Demo does not work for symbols with attributes (like forex)
• [QD-546] dxFeed API: Explain limitations of getTimeSeriesPromise in JavaDoc
• [QD-545] Tools: Old tapes (without protocol descriptor) converted with “dump” tool give
  “NOT COMPATIBLE” warning on read
• [QD-547] Tools: Connect shows non user-friendly “Closing Thread[…]” message before exit
  – Internal class ConnectionHandler is renamed to ConnectionProcessor to better explain what it does
• [QD-550] FileConnector: Shall report 1 connection when reading file
• [QD-151] dxlib: Support custom Marshallers for Marshalled objects
  – Marshaller class is introduced
  – Marshalled class instance has a reference to Marshaller
  – Marshaller.Typed is a replacement for IOUtil.bytesToObjects/objectsToBytes, the later are removed
  – Marshalled.ensureBytes() method is introduced to convert object to bytes internally without necessarily
  requiring a byte array representation (that might change in the future).
  – ByteArrayOutput’s min buffer size is reduced to 64 bytes
  – ByteArrayOutput.write method is optimized for Marshaller’s use-case of converting object to a byte array
  without allocation of extra temporary arrays
  – ByteArrayOutput.needSpace method is final
• [QD-502] dxlib: Support reading/writing Marshalled objects to/from BufferedOutput/Input
  – BufferedOutput.writeMarshalled and BufferedInput.readMarshalled methods are introduced
• [QD-520] dxlib: Marshalled object shall be thread-safe and perform conversion under synchronization
• [QD-553] dxFeed API: Sample for application with permanent subscription retrieving last events
• [QD-557] dxlib: TimeFormatTest failed sometimes
• [QD-570] dxlib: Introduce com.devexperts.io.StreamCompression for transparent gzip/zip support
  – gzip supports sync flush and can be for compression/decompression of live data streams
• [QD-554] FileConnector/Tape: Rename “container” to “compression” and support zip
• [QD-575] dxlib: Static helper methods in URLInputStream to openConnection and checkConnectionResponseCode
• [QD-560] dxFeed API: Introduce Promise.awaitWithoutException convenience method
• [QD-561] dxlib: Introduce ThreadLocalPool and convenience constructors that enable override of pool sizes via
  JVM system properties
  – Also introduced LockFreePool convenience constructor that enables override of pool size via JVM system property
  – Default ChunkPool now uses ThreadLocalPool and property names to change pool sizes have changed:
  + com.devexperts.io.Chunk.threadLocalCapacity – the capacity of thread-local pool of chunks (3 by default)
  + com.devexperts.io.Chunk.poolCapacity – the capacity of global pool of chunks (4096 by default)
  + com.devexperts.io.ChunkList.threadLocalCapacity – the capacity of thread-local pool of chunk lists (3 by default)
  + com.devexperts.io.ChunkList.poolCapacity – the capacity of global pool of chunk lists (1024 by default)
  Other properties for this pool (have not changed):
  + com.devexperts.io.chunkSize – the size of the chunk (8192 by default)
  + com.devexperts.io.recyclableChunkListCapacity – the maximum size of pooled chunk list (1024 by default)
  – Compression/decompression and object serialization/deserialization pools can be sized with:
  + com.devexperts.io.ObjectSerializer.poolCapacity – the capacity of object serializer pool (16 by default)
  + com.devexperts.io.ObjectDeserializer.poolCapacity – the capacity of object deserializer pool (16 by default)
  + com.devexperts.io.Deflater.poolCapacity – the capacity of deflater pool (16 by default)
  + com.devexperts.io.Inflater.poolCapacity – the capacity of inflater pool (16 by default)
  – RecordBuffer now uses ThreadLocalPool and property names to change pool sizes have changed:
  + com.devexperts.qd.ng.RecordBuffer.threadLocalCapacity – the capacity of thread-local pool of bufs (3 by default)
  + com.devexperts.qd.ng.RecordBuffer.poolCapacity – the capacity of global pool of bufs (1024 by default)
  – QD implementation system properties to change pool sizes have changed:
  + com.devexperts.qd.impl.matrix.Distribution.poolCapacity – the capacity of distribution objects pool (2*CPUs by default)
  + com.devexperts.qd.impl.matrix.Notification.poolCapacity – the capacity of notification objects pool (2*CPUs by default)
  – QD RMI implementation system properties to change pool sizes have changed:
  + com.devexperts.rmi.Message.poolCapacity – the capacity of RMI ComposedMessage objects pool (1024 by default)
  + com.devexperts.rmi.Chunk.threadLocalCapacity – the capacity of thread-local pool of chunks (3 by default)
  + com.devexperts.rmi.Chunk.poolCapacity – the capacity of global pool of chunks (4096 by default)
  + com.devexperts.rmi.ChunkList.threadLocalCapacity – the capacity of thread-local pool of chunk lists (3 by default)
  + com.devexperts.rmi.ChunkList.poolCapacity – the capacity of global pool of chunk lists (1024 by default)
  – QDS file ChunkPool now uses ThreadLocalPool and property names to change pool sizes have changed:
  + com.devexperts.qd.qtp.file.Chunk.threadLocalCapacity – the capacity of thread-local pool of chunks (3 by default)
  + com.devexperts.qd.qtp.file.Chunk.poolCapacity – the capacity of global pool of chunks (512 by default)
  + com.devexperts.qd.qtp.file.threadLocalCapacity – the capacity of thread-local pool of chunk lists (3 by default)
  + com.devexperts.qd.qtp.file.poolCapacity – the capacity of global pool of chunk lists (128 by default)
  Other properties for this pool (have not changed):
  + com.devexperts.qd.qtp.file.chunkSize – the size of the chunk (65536 by default)
  + com.devexperts.qd.qtp.file.recyclableChunkListCapacity – the maximum size of pooled chunk list (1024 by default)
• [QD-563] QD Core: API infrastructure to define an extensible list of parameters when creating agents and distributors
  – QDAgent.Build and QDDistributor.Builder interface are introduced.
  – The recommended way of creating an agent in collector is now collector.agentBuilder().build()
  – The recommended way of creating a distributor in collector is now collector.distributorBuilder().build()
• [QD-566] dxlib: Update behavior of Promises.allOf to actually wait for completion of all promises
  – It is be better suited to retrieving multiple promise and should not cancel other promises when one of the
  promises completes exceptionally.
• [QD-567] dxFeed API: Improve getLastEventPromise JavaDoc to explain retrieval of multiple events
• [QD-154] dxFeed API: Authorization
  – DXEndpoint.user(…) and password(…) methods are introduced.
  – “dxfeed.user” and “dxfeed.password” properties are supported.
  – demo/demo becomes default user/password for all samples and all free demo services.
• [QD-124] dxFeed API: Access to historical onDemand data
  – com.dxfeed.ondemand.OnDemandService is introduced in “dxfeed-api” module.
  – Methods to control on-demand historical tick data replay are introduced.
  – Available control methods: replay, pause, setSpeed, stopAndResume, stopAndClear.
  – Available state query methods: isReplaySupported, isReplay, isClear, getTime, getSpeed.
  – add/removePropertyChangeListener to track state changes.
  – “dxfeed-ondemand” module provides implementation on OnDemandService.
  – “dxfeed-samples” modules provides “ondemand” console sample and
  “ui_quotetable_ondemand” Swing UI sample with simple replay controls.
  – Old “replay.jar” from MDS is integrated into “dxfeed-ondemand” module:
  + com.devexperts.mds.replay package becomes com.dxfeed.ondemand.impl
  + com.devexperts.mds.replay.impl package also moves to com.dxfeed.ondemand.impl
  + com.devexperts.mds.replay.impl.event package becomes com.dxfeed.ondemand.impl.event with
  all classes from this package acquiring “MDR” prefix (Event becomes MDREvent,
  MarketMaker becomes MDRMarketMaker, etc).
  – BUG FIX: MarketDataReplay becomes compatible with fresh version of dxlib. Loading
  of stored “mdrcache” file now works properly.
  – MarketDataReplay becomes compatible with modern dxFeed data scheme. It now support both
  legacy Fundamental and TradeHistory records as well as modern Summary and TimeAndSale records.
  – MarketDataReplay uses unique temp file names when writing cache.
  – OnDemandConnector is implemented. It supports “ondemand:
  ” scheme. This connector is
  used by OnDemandService implementation and must be present in the address string to make
  replay service available. This connector can be used with all qds tools like connect, dump,
  etc. Use “Help” tool to get help on “ondemand” topic to see what properties this connector supports.
  – DXEndpoint.Role.ON_DEMAND_FEED is introduced for “on-demand only” use. It does not connect to its
  until OnDemandService.replay is called and cannot be connected to an ordinary data feed at all
  (non-on-demand connectors are ignored in its connection string)
• [QD-352] BUG FIX: dxlib: TimeFormat fails to parse ISO 8601 data with “Z” for GMT
• [QD-356] BUG FIX: dxFeed API: DXFeed.detachSubscription does not work.
• [QD-359] dxFeed API: Introduce DXFeed.detachSubscriptionAndClear method
• [QD-340] dxFeed JS API: onDemand integration
  – Multiple MarketDataReplay instances with the same configuration share a single cache instance.
  – dx.feed JS object includes onDemand control methods: replay, pause, setSpeed, stopAndResume, stopAndClear.
  – dx.feed.state object provides state properties: connected, replaySupported, replay, clear, time, speed.
  – dx.feed.state.onChange can be assigned to track dx.feed.state changes.
  – JS API “Debug console” sample includes onDemand controls similar to DXFeedQuoteTableOnDemand sample.
• [QD-568] dxFeed API: Support Instrument Profile connections with live streaming updates
  – com.dxfeed.ipf.live package is introduced for clients.
  – InstrumentProfileCollector class to keep a live set of instrument profiles.
  – InstrumentProfileConnector class to connect to a live stream of instrument profile updates. It identifies
  itself with “X-Live: yes” header in the requests it makes. It can also periodically check regular files
  and regular http servers for updates with HEAD requests.
  – InstrumentProfileReader works as before, but identifies itself with “X-Live: no” header in the requests it makes.
  – com.dxfeed.ipf.services package is updated for servers.
  – InstrumentProfileServer is introduced to support live instrument profiles protocol. It correctly responds
  to HTTP HEAD requests and reports last modification time. It primarily detects the required protocol (live
  vs snapshot) via “X-Live” header, but it also support “?live” request parameter for testing purposes mainly.
  – InstrumentProfileService is deprecated and is reimplemented on top of InstrumentProfileServer, so that
  a drop-in replacement provides additional features for instrument profile providers that are currently implemented
  on top of InstrumentProfileService. Live stream is not supported, but last modification time is tracked
  and supported, which gives much better overall performance both with InstrumentProfileConnector and with
  dynamic ipf filters (that also rely on HEAD requests and last modification times).
  – Instruments tool can work in a server mode when –write : option is specified.
• [QD-576] dxFeed API: Improve javadocs style
  – Added stylesheet.css
  – Fixed javadoc problems in dxlib and in dxfeed-api modules.
  – Consistent “Provides …” wording in package.htmls
• [QD-573] FileConnector should continue to the next file on detection of corrupted file
  – Also fixed a bug with handling of broken binary files that was introduced by QD-457 parsing aggregation.
  Now, if a broken message in binary QTP format follows a set of correct messages of the same type,
  the parse processes messages from the correct message and ignore everything in the broken one.
  There is a unit-test to expose this case.
• [QD-574] Tools: connect and dump tape shall flush file even when data stops coming
  – Use the following system property to adjust the maximal time data can be kept in memory before being flushed
  + com.devexperts.qd.qtp.file.maxBufferTime (default value is “1s”)