How can I have multiple input sources feed into the same scanner at the same time?

If …

• your scanner is free of backtracking (verified using flex’s ‘-b’ flag),
• AND you run your scanner interactively (‘-I’ option; default unless using special table compression options),
• AND you feed it one character at a time by redefining YY_INPUT to do so,

then every time it matches a token, it will have exhausted its input buffer (because the scanner is free of backtracking). This means you can safely use select() at the point and only call yylex() for another token if select() indicates there’s data available.

That is, move the select() out from the input function to a point where it determines whether yylex() gets called for the next token.

With this approach, you will still have problems if your input can arrive piecemeal; select() could inform you that the beginning of a token is available, you call yylex() to get it, but it winds up blocking waiting for the later characters in the token.

Here’s another way: Move your input multiplexing inside of YY_INPUT. That is, whenever YY_INPUT is called, it select()’s to see where input is available. If input is available for the scanner, it reads and returns the next byte. If input is available from another source, it calls whatever function is responsible for reading from that source. (If no input is available, it blocks until some input is available.) I’ve used this technique in an interpreter I wrote that both reads keyboard input using a flex scanner and IPC traffic from sockets, and it works fine.

This document was generated on November 4, 2011 using texi2html 5.0.