Hi,

I am not sure this is the correct issue to comment on, a new issue might be better.

I noticed recently support for binary data transfers was added, which allows for receiving/sending data without making a copy, by providing pointers to binary data. This allows for low memory overhead in case of large data blocks. The limitation here is if data requires modifications before being sent or after being received, then a copy of the whole data block might still be needed somewhere.

For example if the data is stored in binary/raw form on the server, but the SCPI client is requesting voltages in floating point format. Then it would be an advantage to be able to construct a response in small segments of the requested block. This would reduce the memory overhead to what is required for data conversion of a small segment instead of the whole block.

A similar problem would be streaming data using SCPI (although the standard is not designed for it). For example sending an audio file to a signal generator. Where the generator would not have enough memory to handle the whole file, but would be able to process it in chunks. Another example would be an oscilloscope or logic analyzer sending a stream of samples to the client, where the size of the stream is larger then the servers memory. In the case of streaming proper backpressure is also needed, this is provided by TCP/IP or UART flow control on the lower layers and by blocking pipes in application layer.

This could be achieved by allowing the application using the SCPI parser library to parse and send data without any predefined formatting. Headers or delimiters would be handled by the application, not the parser.

Regards,
Iztok Jeras

There are more then one task resulting from your request. I just want to sumarize it if I understand it correctly - please comment, if there are some problems in following text.

In the following text ARB represents "ARBITRARY BLOCK PROGRAM DATA".

ARB is implemented only in experimental branch. Reading ARB parameter in master branch is not possible (it is not possible to do it right) because of bad design of the parser.

1. add output streaming function for definite or indefinite ARB. This could be implemented by 3 new functions. This is easy to implement. It is covered by IEEE 488.2-2004:8.7.9 and 8.7.10.
   1. function to output just ARB header with 0 for indefinite format or with some nonzero number for definite format
   2. function to output binary data until the count or until there are some data. This function can be called repeatedly.
   3. function to finish indefinite ARB (write \n)
2. add input streaming function - this is hard to implement for me now
   1. To handle definite ARB correctly, I must have full command line and input buffer must be as long as longest possible input message. There is no way for me to support input streaming now.
   2. To handle indefinite ARB, there must be some hack to stop calling SCPI_Parse and pass all incomming data to current indefinite ARB handler.
   3. I don't know, what is <^END> event. For me, it is just calling SCPI_Input with zero length data.

I can implement 1.i, 1.ii and 1.iii.

I would like to find some solution for 2.ii.

Solution for 2.i is not currently possible for me. In the future, after stabilising "experimental" branch, I will look in the way, how to solve this.

I need some advice for 2.iii.

Hi,

I was already looking at the experimental branch, I noticed some ARB code was there, but is still required a pointer to a buffer for an entire transfer. And there are other features in experimental I am interested in.

It took me some time to get hold of the IEEE standards. For now I will only read the strictly related sections.

Comments:

1.i. - A function which outputs an ARB header is what I expected. I do not mind if this function is also able to create other header types, so could be used to create non ARB responses (if there are any non ARB responses with a header).

1.ii. - I might be missing something but I do not think there are other options to detect end of data then count. Otherwise yes, this is what we need.

1.iii. - This one is not clear to me. '\n' is only part of the the second one is '^END', which is handled by a control channel not the main data channel. So only interfaces with a separate control channel can handle it. For example this is defined for GPIB, and it should be possible to implement over USB, but probably not using UART/RS232 or TCP/IP sockets over Ethernet. So availability of indefinite ARB depends on the underlying transfer interface.

2.i. - So as you describe it, currently definite ARB can not be processed in chunks, instead data is provided as a pointer with all data already loaded. What if instead of providing a memory pointer a file pointer for the pipe (UART tty*, TCP/IP, socket, ...) would be provided. The parser would still handle the header, but not the data. I do not know the architecture of the parser enough to know if this can be done. On the other hand, in most cases it makes sense for the parser to handle the pipe and return a memory pointer, so a method for choosing between memory and pipe handlers should be available.

2.ii,iii. - Again only some protocols would be able to provide <^END>, this would probably be handled as some kind of interrupt to the OS. I do not know enough here to comment.

Regarding what is <^END> and how to handle it, It is part of the IEEE 488.1 (GPIB http://en.wikipedia.org/wiki/IEEE-488 maybe pin 5 'EOI') standard. I made a quick search for GPIB Linux drivers but did not find anything obvious, probably the proper search phrase changes between standards and implementations. I would tentatively suggest not implementing indefinite transfers for now.

Regards,
Iztok Jeras

Another idea how to find out about <^END>, you could post a question to http://sigrok.org/ developers on their mailing list https://lists.sourceforge.net/lists/listinfo/sigrok-devel

Their focus is on the client side, but they also have experience with USB and firmware on the instrument side.

Or maybe developers of other GPIB, VISA, SCPI, ... client libraries could help.

Thank you very much for important informations.
For now, I can implement output streaming of definite ARB.

I forget, that it is possible to generate <^END> on controll channel on TCP. I have already an example with control channel, so maybe, this could be also a way.

My technique of parsing is to read message until the terminal character. After that, I call command callback and allow parameters parsing.

I can create new type of parser, that can read just command header and after that call callback. In this type of parser, streaming is possible, but it is not easy to implement it for me. Reason is, that direct reading of input data is not allways available.

I hope I'm not hijacking this issue, but I think my problem is covered in your discussion. I am currently working on a system with limited memory where I would like to be able to fill up most of the RAM with binary data send with the ARB command. As far as I understand the library by now it always needs to have the whole data block in memory for parsing, which requires as much parser buffer as data storage buffer.

I see two ways to solve that. Either a function for the arbitrary data which doesn't require the whole data to be available from the beginning and can be called repeatedly to get the remaining data. Or a function which just parses the header and returns the length and the current parser positions. Then one could hack something around which would copy the incoming ARB data and then calls SCPI_Parse again.

Could you implement something like that or give me a hint on how to implement that myself?

You are right. You need full buffer in the memory for parsing.

If you are using some processor like AVR with dedicated RAM and program space, you can save some RAM by moving constant definitions to program memory. Take inspiration in avr_progmem branch from Martin Vladic. 16bit+ processors are not affected by this issue.

Your proposed solutions are right but they are not so easy to implement as they sounds.

Straightforward solution is possible in multitasking environment, where we can wait for parameter in the command callback function - so just parse the command header and directly call the callback which will wait for parameters and which will handle them part by part.

Implementing this approach as state automaton or protothread will lead in very complicated solution and completely new way, how to define command callbacks.

So the hack. You can somehow tell the parser, that some commands will be processed differently (eg. by tag value). There is point after calling findCommandHeader, where the current command header is known. You must also exchange scpiParser_detectProgramMessageUnit in SCPI_Parse and SCPI_Input by some other function detecting only program header using scpiLex_ProgramHeader. You can hijack the program flow, process command and parameters by yourself and return to parser.
This is something that I don't want to do.

I don't know other solution so this is the reason why it is not part of current version.

If you come with better solution, I will be happy to incorporate it.

I've now managed to build a version with just minimal changes to the library. I disabled the range check in scpiLex_ArbitraryBlockProgramData which gives me the parsed length and a pointer to the beginning of the data even if the data is not existent yet. With that information I read the remaining data outside of the library and call SCPI_Parse again on the next chunk of data.

The only way I can think of a proper solution would be to change the parser in a way that it doesn't get a complete buffer, but instead a function which it can use to request more data. Then one could split the SCPI_ParamArbitraryBlock into one header function and one body function. The header function returns the length and the body function expects a buffer of that size as a parameter in which it memcpys the incoming data.

Nice hack and this is the reason why the parser is open-source.

Your hack will work only if the ARB parameter is the last. If you can somehow detect <^END> of command, you can easily handle also ARB indefinite format. Anyway, this is not the type of hack I would like to incorporate to current sources. :-(